14:00:23 <EmilienM> #startmeeting tripleo
14:00:25 <openstack> Meeting started Tue Aug  1 14:00:23 2017 UTC and is due to finish in 60 minutes.  The chair is EmilienM. Information about MeetBot at http://wiki.debian.org/MeetBot.
14:00:26 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
14:00:28 <openstack> The meeting name has been set to 'tripleo'
14:00:36 <EmilienM> #topic agenda
14:00:43 <EmilienM> * review past action items
14:00:45 <EmilienM> * one off agenda items
14:00:47 <EmilienM> * bugs
14:00:49 <EmilienM> * Projects releases or stable backports
14:00:51 <EmilienM> * CI
14:00:53 <EmilienM> * Specs
14:00:55 <EmilienM> * open discussion
14:00:57 <EmilienM> Anyone can use the #link, #action and #info commands, not just the moderatorǃ
14:00:58 <jrist> o/
14:00:59 <EmilienM> hello everyone
14:01:04 <rasca> o/
14:01:04 <mwhahaha> hi2u
14:01:07 <ccamacho> hi folks o/
14:01:10 <dprince> hi
14:01:15 <jkilpatr> morning
14:01:17 <marios> o/ hello
14:01:21 <openstackgerrit> Jiri Stransky proposed openstack/tripleo-common master: Allow tripleo-admin creation both for Nova-managed and split-stack  https://review.openstack.org/489613
14:01:27 <chem> o/
14:01:45 <numans> hi
14:01:49 <EmilienM> already, thanks for being here an August 1st :-) we're all survivors
14:01:54 <EmilienM> #topic review past action items
14:02:10 <EmilienM> abishop to force legacy key manager before end of pike even if barbican is not deployed
14:02:13 <EmilienM> I think it was done
14:02:18 <EmilienM> or in progress at least
14:02:22 <EmilienM> abishop to propose PTG topic about barbican key manager changes in tripleo/queens
14:02:26 <EmilienM> not sure about this one ^
14:02:31 <EmilienM> jaosorior / abishop to kick-off some discussion about a tool to migrate barbican keys from a manager to another (maybe within barbican)
14:02:36 <EmilienM> same, not sure about this one
14:02:48 <EmilienM> team involved in containerization to review https://bugs.launchpad.net/bugs/1700912 and https://bugs.launchpad.net/tripleo/+bug/1706354
14:02:48 <openstack> Launchpad bug 1700912 in tripleo "Logrotate configs must be adjusted for containerized services logs" [High,Triaged] - Assigned to Bogdan Dobrelya (bogdando)
14:02:49 <openstack> Launchpad bug 1706354 in tripleo "Containerize Logrotate with cron" [High,Triaged] - Assigned to Bogdan Dobrelya (bogdando)
14:02:53 <EmilienM> bogdando: did you get the visibility you wanted? ^
14:03:02 <EmilienM> team to review https://etherpad.openstack.org/p/tripleo-ceph-ansible (4 patches needed for m3)
14:03:03 <bogdando> EmilienM: not yet
14:03:22 <EmilienM> fultonj and gfidente are offline afik but I think everything or almost is merged
14:03:52 <jaosorior> EmilienM: me and abishop didn't really talk. Still waiting for that.
14:03:56 <EmilienM> the feature isn't working in CI yet: https://review.openstack.org/#/c/479288/
14:04:03 <EmilienM> jaosorior: ok, please talk :)
14:04:20 <jaosorior> EmilienM: will do when I see him online. Let me write a post-it note.
14:04:35 <EmilienM> jaosorior: email on ML is better then
14:04:50 <jaosorior> ok
14:04:58 <jaosorior> (I have no idea who abishop is)
14:05:04 <EmilienM> tripleoclient folks to review https://review.openstack.org/#/c/471767/ + dependency asap: DONE
14:05:21 <EmilienM> jaosorior: Alan Bishop, he's working on Cinder and storage stuffs AFIK
14:05:35 <EmilienM> jaosorior: and he works at Red Hat :-)
14:05:47 <EmilienM> ci folks to review https://review.openstack.org/#/q/topic:3nodes-oooq: still in progress
14:05:54 <EmilienM> container team to review healthcheck spec asap https://review.openstack.org/#/c/465633/: done and merged
14:05:57 <jaosorior> EmilienM: thanks!
14:06:22 <cdearborn_> o/
14:06:40 <EmilienM> #action (postponed) jaosorior and abishop to talk together about plans for queens relating to barbican backends (prepare ptg session if needed + discuss about migration tool)
14:06:56 <openstackgerrit> Dan Prince proposed openstack/tripleo-heat-templates master: Fix keystone, cinder, heat-api cron containers  https://review.openstack.org/489419
14:06:59 <EmilienM> #topic one off agenda items
14:07:05 <EmilienM> #link https://etherpad.openstack.org/p/tripleo-meeting-items
14:07:17 <EmilienM> someone wrote about performances but no name
14:07:29 <EmilienM> rook maybe? ^
14:07:36 <jkilpatr> oh that's my cue
14:07:43 <EmilienM> jkilpatr: ok please go ahead
14:07:50 <jrist> hehe
14:07:52 <jkilpatr> https://docs.google.com/presentation/d/1eQDH0UgI3UkbI7W1NNUVFnUA1v30FqJ86vHdjtOXh7s/edit#slide=id.p4
14:08:14 <jkilpatr> So for the past several months I've been working on a thirdparty job to run performance tests on promotion builds
14:08:51 <jkilpatr> after gathering several months worth of data we where able to build a machine learning based classifier that can determine if a build is performing much worse than it should and fail the build.
14:08:52 <EmilienM> jkilpatr: where the job is running exactly?
14:09:11 <jkilpatr> rdo passed phase2, mostly because that's the easiest place to get my dedicated hardware into the line.
14:09:19 <EmilienM> ok, gotcha
14:09:29 <openstackgerrit> Dan Prince proposed openstack/tripleo-heat-templates master: Drop step_config as top level docker requirement  https://review.openstack.org/442716
14:09:29 <openstackgerrit> Dan Prince proposed openstack/tripleo-heat-templates master: docker: Add unit tests on service_name  https://review.openstack.org/442755
14:09:30 <openstackgerrit> Dan Prince proposed openstack/tripleo-heat-templates master: Add docker templates to configure Ironic inspector  https://review.openstack.org/457822
14:09:32 <jkilpatr> it doesn't really work unless you can have exactly the same hardware every time, otherwise you have a hardware variable and a software variable and you can't draw any conclusions.
14:09:57 <jkilpatr> also this deploys a HA env with 2 comptues on baremetal blades. So it's good testing of 'real' deployment as well.
14:10:17 <ooolpbot> URGENT TRIPLEO TASKS NEED ATTENTION
14:10:18 <ooolpbot> https://bugs.launchpad.net/tripleo/+bug/1674681
14:10:18 <openstack> Launchpad bug 1674681 in tripleo "buildlogs.centos.org CDN issues" [Critical,Triaged]
14:11:19 <EmilienM> jkilpatr: I don't think we have the resources to run this job in the gate but I think we should run it in a periodic manner, for ex in the promotion pipeline like you proposed
14:11:40 <jkilpatr> EmilienM, I don't think you'd want this in the gate exactly, you just want it to be passing regularly before you release.
14:11:55 <EmilienM> or even more, before you promote a new repo
14:12:16 <EmilienM> so we don't promote and can investigate which commit slowed us
14:12:27 <jtomasek> o/
14:12:45 <jkilpatr> That's the idea, I hear there's some sort of thirdparty CI integration but I haven't found docs on how to get it working
14:12:54 <jkilpatr> jtomasek, is that hello or a question?
14:12:57 <EmilienM> jkilpatr: I think once we've moved the tripleo promotion workflow to RDO completely, we could add your job into this pipeline
14:13:07 <jtomasek> jkilpatr: just saying hi, sorry
14:13:15 <jkilpatr> don't worry about it.
14:13:26 <jkilpatr> EmilienM, that sounds great!
14:13:32 <EmilienM> jkilpatr: let's keep in touch
14:13:40 <EmilienM> jkilpatr: folks working on it are AFIK on PTO this week
14:13:49 <jkilpatr> The interesting thing is that it's easier to apply ML to logs + perf than just build logs, because a pass fail for most builds means different things, perf of a test acts as a 'source of truth'
14:14:06 <EmilienM> cool
14:14:10 <EmilienM> any question for jkilpatr before we move on?
14:14:41 <EmilienM> #action jkilpatr brings up perf testing discussion again when tripleo promotion workflow moved to RDO
14:15:03 <EmilienM> zshi_laptop: go ahead
14:15:22 <EmilienM> sounds like you want some review on a spec
14:15:23 <zshi_laptop> I'm requesting a review on spec :  https://review.openstack.org/#/c/483227/
14:15:34 <EmilienM> #action team to review PTP support spec https://review.openstack.org/#/c/483227/
14:15:47 <zshi_laptop> it's ptp support in tripleo, ptp is precision time protocol, can be alternative of ntp
14:15:59 <zshi_laptop> this spec proposing add ptp configuration path in tripleo
14:16:12 <zshi_laptop> but leave ntp as default time synchronization method
14:16:38 <zshi_laptop> users who want to use ptp need explicitly configuration
14:16:44 <EmilienM> zshi_laptop: I think you can go ahead and start doing the code - I see this as an optional composable service in tripleo
14:17:06 <zshi_laptop> EmilienM, cool, I have started coding
14:17:06 <EmilienM> though we'll review the spec, as you took time to write it
14:17:09 <EmilienM> ok
14:17:15 <zshi_laptop> will submit the code next week
14:17:39 <zshi_laptop> sure, thanks!
14:17:57 <EmilienM> cool
14:18:02 <EmilienM> numans: o/
14:18:05 <numans> hi
14:18:07 <numans> kolla build generates neutron-server-ovn docker image.
14:18:15 <numans> I wanted to check if it is possible to have this image uploaded into tripleoupstream docker hub ?
14:18:19 <EmilienM> dprince: ^
14:18:23 <numans> This would be required for OVN container CI job
14:18:46 <numans> and also neutron-server-odl incase odl folks need it
14:19:14 <EmilienM> numans: please check with dprince - I think he manages the docker hub thing
14:19:23 <numans> ack. will check with him
14:19:25 <numans> thanks
14:19:46 <EmilienM> and I think this manual workflow is going to change
14:19:54 <EmilienM> once we have the promotion pipeline running in RDO
14:19:59 <EmilienM> dprince can confirm when he's back
14:20:07 <openstackgerrit> Honza Pokorny proposed openstack/tripleo-ui master: Implement Zaqar logger adapter  https://review.openstack.org/459294
14:20:07 <numans> that's nice
14:20:08 <numans> thanks
14:20:21 <EmilienM> chem: o/
14:20:24 <chem> he
14:20:49 <chem> so I wanted to know if we plan a switch from "latest" tag to some version for the docker images
14:21:05 <EmilienM> another question for dprince
14:21:07 <EmilienM> :)
14:21:08 <chem> it's important for update
14:21:14 <dprince> numans: if the neutron-server-ovn is listed in our tripleo-common 'container-images' directory it should get built and pushed to dockerhub
14:21:23 <chem> EmilienM: ... I was afraid you'd say this :)
14:21:46 <numans> dprince, i will check on it. thanks
14:21:54 <EmilienM> I think containers were updated last night
14:21:58 <dprince> chem: for a default, 'latest' makes the most sense
14:21:59 <chem> anyway we have the update review in the pipeline, any help welcome https://review.openstack.org/#/q/topic:add_update_tasks
14:22:01 <chem> :)
14:22:19 <dprince> chem: when pushing images however we probably could tag them with something more meaningful
14:22:36 <chem> dprince: hi, well, we're not sure how to detect that we need to upgrade a container
14:22:41 <numans> dprince, i submitted a patch to add it in overcloud-containers.yaml - https://review.openstack.org/#/c/484394/1/container-images/overcloud_containers.yaml,
14:22:41 <dprince> chem: so long as the pull 'latest' default still translates to pull that I'd be fine w/ it I think
14:22:45 <chem> dprince: we can have this discussion outside of the mtg
14:22:54 <dprince> chem: sure, sounds good
14:23:12 <numans> but CI failed since it couldn't find it in tripleoupstream docker hub.
14:23:16 <chem> EmilienM: that's all from me
14:23:21 <numans> chem, sorry.
14:23:28 <chem> numans: np
14:23:33 <openstackgerrit> Alex Schultz proposed openstack/tripleo-puppet-elements master: Set remove-machine-id to executable  https://review.openstack.org/489618
14:23:50 <EmilienM> chem: ok thx
14:23:56 <EmilienM> anything else before we start regular agenda?
14:24:04 <EmilienM> #topic bugs
14:24:09 <EmilienM> #link https://launchpad.net/tripleo/+milestone/pike-rc1
14:24:41 <EmilienM> we have 107 Triaged and 56 In Progress
14:24:45 <EmilienM> all High or Critical
14:24:55 <EmilienM> (I moved Medium / low / wishlist to queens-1)
14:25:22 <EmilienM> right now the priority is to reduce the number of critical / high bugs targeted for pike-rc1
14:25:33 <EmilienM> at this stage - all upgrade bugs are Critical
14:25:39 <EmilienM> chem: ^ if you can help in triaging those
14:25:56 <EmilienM> is there any bug we need to discuss this week?
14:26:02 <chem> EmilienM: right, will check on them
14:26:13 <dprince> numans: I think I pushed your ovn images last night BTW https://hub.docker.com/search/?isAutomated=0&isOfficial=0&page=1&pullCount=0&q=tripleoupstream+ovn
14:26:24 <openstackgerrit> Marios Andreou proposed openstack/tripleo-heat-templates master: Adds stop and disable for libvirtd on upgrade to containers  https://review.openstack.org/489619
14:26:31 <dprince> numans: is that what you were after?
14:26:47 <numans> dprince, its neutron-server-ovn... let me check if it has now
14:27:04 <EmilienM> (moving on the agenda)
14:27:09 <EmilienM> #topic projects releases or stable backports
14:27:21 <EmilienM> I sent an email yesterday but we released pike-3 last week
14:27:37 <EmilienM> tripleoclient final was released last week - stable/pike was created for this one
14:27:45 <EmilienM> right now the focus is on rc1
14:28:05 <EmilienM> #link http://lists.openstack.org/pipermail/openstack-dev/2017-July/120357.html
14:28:20 <EmilienM> we're in Feature Freezze right now
14:28:38 <EmilienM> please do not merge code that is a feature and that doesn't have Feature Freeze Exception ( FFE )
14:29:01 <EmilienM> any question about release management.
14:29:03 <EmilienM> ?
14:29:12 <atoth> 1
14:29:22 <atoth> still not exactly clear on FFE
14:29:46 <atoth> is there some tag that gets put onto things for it?
14:30:22 <EmilienM> atoth: so let me rephrase how it works
14:30:34 <EmilienM> during the cycle, some people work on blueprints, most of them are features
14:31:01 <EmilienM> at the end of milestone 3, we stop approving code that is a feature, because we want to stabilize and produce stable release candidates
14:31:07 <EmilienM> that's call Feature Freeze
14:31:19 <EmilienM> but some features are really needed and have limited risk to break something
14:31:37 <EmilienM> so features authors ask to the PTL and the team if we can grant FFE
14:31:51 <EmilienM> if yes, the code can be approved during RC
14:31:58 <EmilienM> if no, -2 and it will wait the next cycle
14:32:04 <EmilienM> it's all documented here: https://docs.openstack.org/project-team-guide/release-management.html#common-cycle-with-development-milestones
14:32:13 <atoth> perfect, thanks
14:32:20 <EmilienM> atoth: any question in specific?
14:32:49 <atoth> Just wondering about it for the lbaas stuff :-)
14:33:12 <atoth> Have I requested it clearly enough?
14:33:12 <EmilienM> atoth: what are the patches that you need for LBaaS support in tripleo?
14:33:30 <EmilienM> (patches not merged)
14:33:45 <atoth> most are already merged into master just have to figure out why it's crashing the multinode scenario
14:34:02 <atoth> so will be new patches
14:34:12 <atoth> once figured out
14:34:22 <EmilienM> that's why I wish we could develop features with tests in the same time
14:34:31 <atoth> agreed
14:34:32 <EmilienM> in your case, the code is merged but tests are now failing
14:34:43 <EmilienM> I don't think we need an FFE for that
14:34:48 <EmilienM> the feature is already there
14:34:51 <atoth> damn newbees ;-)
14:34:52 <EmilienM> but isn't working
14:35:17 <atoth> but FFE if fix is determined?
14:35:21 <EmilienM> so from now, we're probably considering LBaaS patches as bug fixes
14:35:27 <EmilienM> as the major feature has been implemented
14:35:39 <atoth> ok, bug fixes then, ack
14:35:47 <EmilienM> alright, moving on
14:35:49 <EmilienM> #topic CI
14:36:00 <EmilienM> a bunch (if not all) CI folks are on PTO this week
14:36:15 <EmilienM> our major issue now is https://bugs.launchpad.net/tripleo/+bug/1674681
14:36:15 <openstack> Launchpad bug 1674681 in tripleo "buildlogs.centos.org CDN issues" [Critical,Triaged]
14:36:28 <EmilienM> I'm investigating today, still a lot of red in http://tripleo.org/cistatus.html
14:36:28 <openstackgerrit> Rajini Karthik proposed openstack/tripleo-heat-templates master: Add support for Dell EMC VMAX ISCSI Backend  https://review.openstack.org/489393
14:36:46 <EmilienM> please do not recheck blindly if CDN timeouts, it's still timeouting, no need to recheck
14:37:13 <EmilienM> before the meeting I was investigating why some jobs were failing, I found the CDN issue was still there but there might be something else
14:37:19 <EmilienM> any help on this is welcome
14:37:25 <EmilienM> (I'll spend my day on it probably)
14:37:38 <EmilienM> anything about CI this week?
14:38:07 <EmilienM> chem, jistr: at this stage of the cycle, we should have upgrade jobs working and if not we need to file critical bugs in launchpad
14:38:33 <chem> EmilienM: got it
14:38:42 <EmilienM> jistr: I think we should retire old upgrade jobs on master
14:38:47 <EmilienM> and keep the one you did
14:38:53 <EmilienM> from baremetal to container only
14:38:53 <chem> EmilienM: I'm relatively more concerned with update actually
14:39:12 <chem> EmilienM: and non-controller upgrade
14:39:17 <jistr> yea
14:39:18 <EmilienM> chem: I'm sad that gate-tripleo-ci-centos-7-ovb-1ctlr_1comp_1ceph-featureset024  is not testing updates AFIK
14:39:33 <EmilienM> chem: gate-tripleo-ci-centos-7-ovb-1ctlr_1comp_1ceph-featureset024  is the new ovb-updates job
14:39:36 <openstackgerrit> Rajini Karthik proposed openstack/puppet-tripleo master: Support for Dell EMC VMAX Cinder Driver  https://review.openstack.org/489624
14:39:48 <EmilienM> and AFIK it's not testing stack-update (a regression in quickstart move)
14:39:58 <EmilienM> so we don't cover overcloud updates anymore, just FYI
14:40:09 <chem> EmilienM: ack, good to know
14:40:09 <EmilienM> we might want to take care of this problem sometimes ^
14:40:27 <EmilienM> jistr: yea = let's go for it? :)
14:40:37 <chem> EmilienM: and I don't think non-controller upgrade is covered somewhere neither
14:40:40 <EmilienM> jistr: if yes, I'll work on project-config patch
14:40:47 <jistr> EmilienM: yea :) but my previous yea was mainly to +1 chem
14:40:53 <EmilienM> chem: no, we deploying 1-node overclouds
14:40:54 <jistr> main issue is that we don't have updates
14:40:58 <jistr> and non-controller upgrades
14:41:01 <EmilienM> jistr: ok so yea and yea :D
14:41:02 <jistr> the job itself runs fine
14:41:03 <chem> EmilienM: ack, noted
14:41:06 <jistr> EmilienM: yea :D
14:41:09 <chem> yeah
14:41:15 <EmilienM> stop yea'ing us
14:41:17 <EmilienM> :)
14:41:19 <adarazs> EmilienM: just want to point out that most of the folks from the CI squad are on PTO or otherwise engaged, it's mostly me who's left, so if there's some issue with quickstart/etc, please let me know.
14:41:29 <marios> chem: jistr +1 wrt upgrade-non-controller, just filed this one (from a linked BZ https://bugs.launchpad.net/tripleo/+bug/1707926 thanks to mcornea
14:41:29 <openstack> Launchpad bug 1707926 in tripleo "Ocata to Pike upgrade: libvirtd service is running on host after upgrade and nova_libvirt container keeps restarting" [Critical,In progress] - Assigned to Marios Andreou (marios-b)
14:41:36 <EmilienM> adarazs: good to know, I thought I was alone
14:42:07 <EmilienM> #action EmilienM to switch master to run new upgrade jobs and not old ones anymore
14:42:08 <chem> mcornea: jistr let's tackle those 22 upstream bugs :)
14:42:41 <EmilienM> anything else about CI?
14:42:50 <jistr> chem: tackling https://review.openstack.org/#/c/489613/ presently fwiw :/
14:43:29 <EmilienM> #topic specs
14:43:33 <EmilienM> #link https://review.openstack.org/#/q/project:openstack/tripleo-specs+status:open
14:43:38 <chem> jistr: oh boy
14:43:42 <EmilienM> do we have anything about specs to discuss this week?
14:43:51 <jistr> chem: that's blocker for ceph-ansible CI
14:43:58 <jistr> and k8s
14:44:00 <EmilienM> zshi_laptop had a spec to present before, so it's already covered
14:44:30 <EmilienM> #topic open discussion
14:44:37 <openstackgerrit> Rajini Karthik proposed openstack/puppet-tripleo master: Support for Dell EMC VMAX ISCSI Cinder Driver  https://review.openstack.org/489624
14:44:48 <EmilienM> 1. Please propose topics for PTG: https://etherpad.openstack.org/p/tripleo-ptg-queens
14:45:25 <EmilienM> 2. I won't run PTL next cycle - it's nomination time by the way, this week. Don't miss the schedule and nominate yourself if you want to be PTL
14:45:57 <EmilienM> any feedback / comment / question before we close the meeting?
14:46:04 <panda> :'(
14:46:24 <EmilienM> I hate when pandas cry :-(
14:46:37 * ccamacho cries too :P
14:46:39 <EmilienM> already, we're done
14:46:44 <EmilienM> alright$
14:46:45 <ccamacho> *** all the time
14:46:50 <EmilienM> thanks everyone
14:46:53 <EmilienM> #endmeeting