14:00:27 <EmilienM> #startmeeting tripleo
14:00:28 <openstack> Meeting started Tue Nov 29 14:00:27 2016 UTC and is due to finish in 60 minutes.  The chair is EmilienM. Information about MeetBot at http://wiki.debian.org/MeetBot.
14:00:29 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
14:00:33 <openstack> The meeting name has been set to 'tripleo'
14:00:41 <EmilienM> #topic agenda
14:00:47 <EmilienM> * review past actions items
14:00:53 <EmilienM> * one off agenda items
14:00:56 <EmilienM> * bugs
14:00:58 <EmilienM> * Projects releases or stable backports
14:00:59 <adarazs> o/
14:01:00 <EmilienM> * CI
14:01:01 <arxcruz> o/
14:01:02 <EmilienM> * Specs
14:01:04 <shardy> o/
14:01:04 <EmilienM> * open discussion
14:01:04 <michapma_alt> o/
14:01:06 <EmilienM> hello folks!
14:01:07 <akrivoka> hello \o
14:01:07 <ccamacho> Hey guys!
14:01:09 <mwhahaha> o/
14:01:09 <shardy> hey all
14:01:12 <marios> o/
14:01:12 <sshnaidm> o/
14:01:13 <thrash> o/
14:01:14 <trown> o/
14:01:14 <florianf> o/
14:01:22 <trown> happy tuesday
14:02:05 <EmilienM> #topic review past actions items
14:02:09 <gfidente> hello
14:02:12 <beagles> o/
14:02:19 <fultonj> o/
14:02:23 <EmilienM> EmilienM & jaosorior to report httpd issue on launchpad: I think the problem was solved
14:02:25 <yolanda> o/
14:02:32 <pradk> o/
14:02:33 <d0ugal> Hey
14:02:34 <EmilienM> EmilienM to communicate on ocata release schedule: done, I sent an email on openstack-dev
14:02:50 <hrybacki> o/
14:03:03 <EmilienM> EmilienM to proceed to last Liberty release and sync with tonyb about tripleo liberty EOL: done, I talked with tonyb last week
14:03:10 <EmilienM> matbu lets use oooq in tripleo-ci for periodics M to N jobs
14:03:18 <EmilienM> matbu: any update on this one? ^
14:03:25 <EmilienM> same for "matbu build upgrade jobs for N to O"
14:03:49 <matbu> oops sorry hello all
14:03:53 <EmilienM> team to review slagle's work on 3nodes job https://review.openstack.org/#/c/397441/ (and dependencies): not done, folks please review his patches
14:04:17 <EmilienM> team to reviews specs targetted for ocata
14:04:37 <EmilienM> and EmilienM to communicate on ocata specs freeze (end of o-2): done in the same email about ocata release schedule
14:04:41 <matbu> EmilienM: no from my side, i think it was depending on a sshnaidm review
14:04:59 <EmilienM> matbu: which one?
14:05:08 <matbu> i was mostly working on composable upgrade and debuging current upgrade failure in centos infra
14:05:13 <sshnaidm> matbu, the job is merged now, I think you can try with it
14:05:14 <weshay> o/
14:05:26 <cdearborn> o/
14:05:33 <matbu> sshnaidm: ack thanks, i'll start to work on this week end
14:05:42 <EmilienM> #action matbu to build upgrade jobs for N to O
14:05:49 <sshnaidm> matbu, ok, feel free to ask for anything there
14:06:08 <EmilienM> #action matbu lets use oooq in tripleo-ci for periodics M to N jobs
14:06:09 <matbu> sshnaidm: yep thx
14:06:18 <EmilienM> #topic one off agenda items
14:06:22 <EmilienM> #link https://etherpad.openstack.org/p/tripleo-meeting-items
14:06:26 <EmilienM> sshnaidm: go ahead
14:06:51 <sshnaidm> I'll copy my topic:
14:06:55 <sshnaidm> in process of transtion to quickstart we need more people in TripleO CI to review quickstart related patches. Right now the experimental oooq job is very untolerant to changes in both repos (quickstart and tripleo-ci) and requires a quick fixes to be merged urgently to work. With current time constraints we need more people involved in review and merge its patches in tripleo-ci repo. Volunteers?
14:07:24 <shardy> The untolerant to changes in both repos makes me nervous ;)
14:07:50 <sshnaidm> shardy, it was related to new repo establishment - extras, so...
14:08:27 <shardy> sshnaidm: sure, I appreciate landing the patches will help stablize things, I just wanted to clarify that's the only problem vs underlying fragility
14:08:32 <sshnaidm> thanks a lot to jaosorior trown and EmilienM, but we need a bit more people involved in review/merges
14:08:35 <trown> untolerant I think is poor wording... it is not gated
14:08:56 <shadower> sorry I'm late, hey
14:09:00 <trown> anything that is not gated is fragile
14:09:06 <EmilienM> trown: +1
14:09:10 <sshnaidm> trown, sure, sorry for my english :)
14:09:26 <trown> sshnaidm: ya no worries, was not trying to call you out, just trying to clarify :)
14:09:31 <shardy> Ok coo, provided that's the only issue, thanks for clarifying
14:10:13 <EmilienM> sshnaidm: thanks for reporting this issue
14:10:28 <EmilienM> I also have a quick off item, about our squads
14:10:29 <sshnaidm> maybe it worth to give rights for tripleo-ci for oooq jobs to current quickstart cores..?
14:10:48 <EmilienM> I noticed some people already put their names, which is awesome!
14:10:50 <EmilienM> #link https://etherpad.openstack.org/p/tripleo-squads
14:10:53 <trown> sshnaidm: that would really only add 2 extra +2 :P
14:11:12 <trown> sshnaidm: I think it will just be a bit bumpy while we get everything going
14:11:14 <sshnaidm> trown, 5 is better than 3
14:11:36 <EmilienM> we need to continue with mission statements, feel free to contribute and give your feedback!
14:12:05 <trown> I put up a rough draft for CI, but anyone feel free to modify
14:12:26 <EmilienM> trown: thanks a lot
14:12:32 <trown> also, for CI in particular I think it would be good to have a subteam meeting
14:12:38 <shardy> Do we really need mission statements?
14:12:41 <sshnaidm> trown, +1
14:12:45 <shardy> e.g "Group of people focusing on TripleO deployed in containers" seems pretty clear to me
14:12:53 <d0ugal> shardy: I guess it is useful for new people to know which squad to join.
14:12:57 <EmilienM> shardy: I thought it would be great to document the scope of every team
14:13:04 <d0ugal> shardy: otherwise one word is a bit vague :)
14:13:07 <shardy> mission - "really, really focus on deploying TripleO, in containers, yeah!"
14:13:19 <EmilienM> trown: for meetings, it will be up to the squads. If they need it or not. Nothing required here.
14:13:23 <jaosorior> EmilienM: I'm unsure where the work that I do ends up in those squads
14:13:28 <shardy> Ok, if folks find it useful then fair enough
14:13:32 <trown> containers may be a special case in that regard
14:13:53 <trown> it is pretty clear what falls under containers
14:13:55 <EmilienM> shardy: we can rename mission statement to scope of work done by the squad
14:14:21 <shardy> trown: I guess I see all of them as pretty clear, but sure if folks want to add more detail beyond the description to enumerate the scope, that's fine by me
14:14:22 <d0ugal> I am trying to understand how the Liaison works. How are they picked? When do they speak with the PTL?
14:14:47 <trown> CI though, I think it is important to distinguish that the scope is CI/dev tooling
14:15:45 <matbu> trown: i think it could be good to define CI/dev tooling then :)
14:15:49 <EmilienM> d0ugal: they are picked when someone volunteer to do it in the squad. "When do they speak with PTL"? I don't know yet
14:16:09 <shardy> d0ugal: I see it as someone willing to be the point of contact
14:16:13 <marios> d0ugal: i guess each team will work slightly differently. it could be rotated for example, or whatever works for the team
14:16:21 <EmilienM> matbu: I thought we agreed on it during the Summit?
14:16:23 <d0ugal> cool, thanks
14:16:33 <d0ugal> I was trying to determine how much time it would take before I put my name down :)
14:16:36 <EmilienM> marios: ++ for rotations
14:16:46 <trown> Also, if the squad has a seperate meeting, the Liaison would be responsible for reporting to this meeting?
14:17:21 <matbu> EmilienM: i mean not the tool itself, but does that mean that the CI squad will create/debug/improve all periodic and gating jobs for all the scopes ?
14:17:35 <EmilienM> trown: it's an option, yes. Reporting the blockers to PTL so we can take actions with other teams?
14:18:05 <EmilienM> matbu: it is something we need to discuss, in this etherpad or mailing-list. Feel free to start it!
14:18:57 <matbu> EmilienM: ack, i will add comments in the etherpad itself
14:19:06 <EmilienM> d0ugal: it shouldn't take much time as there is no specific action for now, it's just a communication thing for now I guess
14:19:26 <d0ugal> EmilienM: cool, sounds good. I copied trown and added a note next to my name :)
14:19:44 <EmilienM> also, I think it's possible for someone to be part of multiple squads in the same time
14:20:16 <EmilienM> ie; if you work on both upgrades and CI tooling/system, put your name in both squads would make sense
14:20:28 <weshay> ya
14:20:35 <d0ugal> yeah, I am in workflows and CLI/UI
14:21:07 <EmilienM> right now, we're really in alpha version so things will evolve with time :-)
14:21:23 <EmilienM> I'm moving to next topic, if any question or feedback, let us know
14:21:42 <EmilienM> #topic bugs
14:22:06 <EmilienM> #link https://launchpad.net/tripleo/+milestone/ocata-2
14:22:19 <EmilienM> 4 New, 4 Incomplete, 1 Invalid, 13 Confirmed, 48 Triaged, 34 In Progress
14:22:53 <EmilienM> it seems we have some critical bugs still open and not fixed, I'll have a look at them
14:23:08 <EmilienM> I'm also doing triage but we have so much new bugs every day in launchpad, it's hard to keep up
14:23:43 <EmilienM> I would quickly remind folks to make auto triage (status, priority, assignment and release target). I would help people who make launchpad triage
14:24:26 <EmilienM> do we have outstanding bugs this week that we need to discuss?
14:25:20 <EmilienM> seems like no
14:25:53 <EmilienM> #topic Projects releases or stable backports
14:26:27 <EmilienM> reminder: ocata-2 release is in 2 weeks
14:27:00 <EmilienM> we don't have any pending release at this time
14:27:12 <beagles> "stable backport" reminded me of https://bugs.launchpad.net/tripleo/+bug/1640812
14:27:12 <openstack> Launchpad bug 1640812 in tripleo "Network connectivity lost on node reboot" [Critical,Fix committed] - Assigned to Brent Eagles (beagles)
14:27:12 <EmilienM> we can go to next topic
14:27:24 <beagles> the fix we have for this is to os-net-config and there isn't a stable branch for mitaka
14:27:39 <beagles> yet it is flagged for mitaka backport potential
14:27:40 <EmilienM> beagles: ah, you need to submit a release in RDO itself afik
14:28:21 <EmilienM> beagles: probably in rdo.yaml layout, if you're not familiar with it, let me know. We just need to change the tag for mitaka
14:28:35 <beagles> EmilienM, ahhh... okay. Yeah, not familiar at all :)
14:29:10 <EmilienM> beagles: https://github.com/redhat-openstack/rdoinfo/blob/master/rdo.yml#L936
14:29:15 <EmilienM> #topic CI
14:29:18 <beagles> EmilienM, cool. thanks
14:29:42 <EmilienM> on CI side, I'm working on scenarios, and we moved the templates/environments to THT
14:29:56 <EmilienM> this week i'm working on helping people to add more coverage in the scenarios
14:30:20 <EmilienM> if you have any question or need help on this thing, please let me know
14:30:48 <EmilienM> also, sshnaidm is working on adding a oooq job for containers
14:30:52 <shardy> I wanted to mention that getting upgrade CI coverage is going to be come critically imporant soon
14:30:54 <EmilienM> flaper87, dprince ^
14:31:07 <weshay> +1
14:31:09 <EmilienM> shardy: yes, +1. My hope is that matbu can make progress asap...
14:31:17 <shardy> we've been working towards it for some time, but we need to prove the reimplemented upgrades for N->)
14:31:25 <shardy> N->O I mean :)
14:31:26 <EmilienM> but we haven't seen much progress lately on this topic
14:31:29 <dprince> EmilienM: ack
14:31:47 <EmilienM> matbu: do you think we can have the CI job before ocata-2 ?
14:31:52 <shardy> matbu: can you help enumerate the remaining work, perhaps start a ML thread or etherpad?
14:32:09 <matbu> shardy: for N to O ?
14:32:15 <shardy> If we need to find more help we can do it, but atm I'm not clear exactly what we're blocked on
14:32:30 <shardy> matbu: we need a way to test upgrades for all supported branches
14:32:48 <matbu> shardy: well it depend on the branches
14:32:53 <marios> matbu: i think we mean https://review.openstack.org/#/c/323750/ for starters but we'll need to plan working on a n->O too
14:32:57 <matbu> shardy: for M to N we are blocked by the timeout
14:33:03 <marios> matbu: s/we/emilien was referring to i think
14:33:14 <shardy> matbu: right, so all branches are blocked by the timeout
14:33:23 <matbu> shardy: for Mitaka branch, i guess we don't care about it (L to M i mean)
14:33:30 <shardy> we need a solution (multinode?) to that before we can move forwards, right?
14:33:46 <shardy> matbu: yup, we could start with a M->N upgrade, then add N->O
14:33:52 <matbu> shardy: and for N to O, i 'm planning to start a WIP review depending on your prototype
14:33:53 <shardy> but we need the basic proven process in place
14:34:30 <EmilienM> yeah so M to N would be the first step, as we know it works
14:34:36 <shardy> matbu: Yeah I guess we could go directly to the new model
14:34:40 <matbu> shardy: afair, the mutlinode won't work with mitaka. so last week we decide to use oooq (at least for periodic jobs)
14:34:50 <shardy> I'd have been happier if we had a green job with the old implementation, then switch over to the new approach
14:35:07 <shardy> but that's probably more work e.g writing the special-case support for moving services under httpd
14:35:47 <matbu> sshnaidm: we can make gating jobs with tripleo-ci and oooq ?
14:35:52 <shardy> matbu: Ok, sounds like this will have to be done after we get the initial N->O upgrades working then
14:36:11 <sshnaidm> matbu, I think so
14:36:13 <shardy> matbu: perhaps we can start with a small subset of services, then add coverage as we implement full upgrade support
14:36:23 <EmilienM> shardy: yes, baby steps.
14:36:25 <matbu> shardy: yes
14:36:31 <shardy> e.g a multinode job which upgrades just keystone using the new upgrade patch (which can be broken up)
14:36:37 <shardy> then we add things service by service
14:36:48 <matbu> shardy: just depending on your review, then adding more and more services and HA
14:36:54 <matbu> +1
14:36:57 <shardy> I just posted that big patch as an example, we can easily break it up
14:37:05 <shardy> Ok, that sounds like a good way forward, thanks!
14:37:28 <matbu> shardy: EmilienM just a question about the upgrade proto
14:37:30 <shardy> matbu: let me know if I can help getting that initial test in place - should I break up the patch to e.g do just keystone as a first step?
14:37:30 <ccamacho> im finishing to locally wire up the major upgrade pacemaker scripts to a HA N -> O deployment, the thing is that im getting some unbound variables out and to fix/redeploy them takes some time
14:37:38 <matbu> shardy: do we plan to landed it to ocata-2 ?
14:37:55 <shardy> matbu: possibly - let me break up the patch and see if reviews happen
14:38:08 <EmilienM> matbu: yes, it would be great
14:38:11 <shardy> a passing CI job would probably help give folks confidence in the approach
14:38:23 <matbu> shardy: k
14:38:51 <matbu> ccamacho: that's why i created pcs ansible module, it's more robust than the bash scripts i think
14:38:51 <EmilienM> like shardy, we really need to make progress on this topic, even with a very few services
14:39:19 <weshay> shardy +1
14:39:19 <matbu> shardy: btw i got a almost fully upgraded deployment with pacemaker , marios
14:39:21 <marios> matbu: ++ had a quick first pass at those they look like a good idea to me
14:39:24 <ccamacho> matbu yeahp, But I wanted to start with some working and then push it forward to your patch
14:39:41 <marios> matbu: i mean the https://review.openstack.org/#/c/403397/ ansible module
14:39:50 <matbu> marios: yep
14:39:52 <shardy> matbu: great!  Did you exclude the services that moved under httpd?
14:40:01 <shardy> I've still not had time to figure out a generic fix to those
14:40:16 <matbu> shardy: yep, i exclude some service
14:40:27 <shardy> matbu: Ok, sounds good, thanks for the update
14:40:29 <matbu> shardy: i wanted to see how handle pacemaker
14:40:41 <matbu> and if it was working consistensly
14:41:08 <matbu> shardy: marios btw i feel the approach really better for debug and for rerunning failling step
14:41:50 <matbu> shardy: but i was wondering to maybe split each playbook per steps ?
14:41:53 <marios> matbu: are we getting better error output from ansible compared to bash you mean? plus yeah by definition the steps are more discrete
14:42:01 <matbu> (maybe not the right place to discusss about that)
14:42:02 <EmilienM> marios: no
14:42:06 <EmilienM> err, matbu ^
14:42:18 <EmilienM> matbu: we thought we could use tags
14:42:38 <EmilienM> matbu: rather, having a playbook per role
14:42:56 <matbu> EmilienM: yes but we will have a big huge playbook , harder to read than some small pieces of step
14:43:11 <shardy> matbu: I'd prefer to have one playbook and run it via --tags stepN
14:43:22 <matbu> but it's just a personnal feeling :)
14:43:28 <shardy> matbu: as that's exactly how we run it via heat, but open to discussion
14:43:45 <EmilienM> maybe we can discuss about it later
14:43:48 <EmilienM> outside the meeting I mean
14:43:49 <shardy> yup
14:43:49 <matbu> shardy: yep k
14:43:52 <EmilienM> thx
14:43:53 <matbu> EmilienM: yes
14:43:56 <EmilienM> anything else about CI ?
14:44:11 <EmilienM> #topic specs
14:44:13 <EmilienM> #link https://review.openstack.org/#/q/project:openstack/tripleo-specs+status:open
14:44:42 <EmilienM> nothing to be added on my side, except, keep reviewing it when you have cycles :)
14:45:11 <EmilienM> do we have anything to talk about specs this week?
14:45:26 <EmilienM> quick reminder: ocata specs need to be merged before ocata-2 release
14:46:01 <owalsh> could I get a +W on https://review.openstack.org/#/c/388162/
14:46:45 <florianf> Something that came up on #tripleo before regaring spec reviews: Are sub project core reviewers supposed to 2+ specs related to their projects, even if the spec spans multiple projects?
14:46:54 <EmilienM> owalsh: I'll approve it shortly if no more feedback
14:47:04 <florianf> *regarding
14:47:34 <owalsh> EmilienM: thanks
14:47:37 <EmilienM> florianf: I think it's up to reviewers to feel confortable or not to +2 or +1
14:48:23 <EmilienM> we have 10 min left, I'll open the discussion
14:48:24 <florianf> OK, so the normal rule for sub org cors doesn't apply here (only +2 your own project's repos )
14:48:27 <EmilienM> #topic open discussion
14:49:20 <EmilienM> if you have any question or feedback, please go ahead now
14:49:20 <sshnaidm> I'd like you all to see the CI status page and to give your feedbacks/RFE, I'll write more detailed ML about it later: http://status-tripleoci.rhcloud.com/
14:49:26 <trown> ya, I think +2 spec related to a specialty is fine... we probably would not want to merge any spec that spanned multiple specialties without a representative from each group +2 though
14:49:46 <florianf> trown: excellent, sounds good.
14:50:12 <EmilienM> sshnaidm: nice. I would try to merge tripleo.org/cistatus.html and your page
14:50:16 <EmilienM> so we make things easier to find
14:50:18 * jpich wonders if that's worth an update to https://wiki.openstack.org/wiki/TripleO/SpecReviews to be explicit about it
14:50:47 <EmilienM> sshnaidm: your dashboard is really awesome
14:50:49 <florianf> jpich: It probably is, just to clarify...
14:51:13 <EmilienM> jpich: it sounds like a good idea
14:51:21 <jpich> florianf: I'll copy-paste some of the statements from the meeting there after
14:51:35 <jpich> Ok will do
14:51:36 <EmilienM> jpich: maybe move https://wiki.openstack.org/wiki/TripleO/SpecReviews to tripleo-specs as a policy?
14:51:49 <jpich> Yeah I was thinking about it, it would be easier to find
14:51:54 <EmilienM> so we can have a group discussion and leave Wiki (FYI OpenStack wiki will die one day)
14:52:10 <jpich> Everything does one day
14:52:20 <jpich> I'll take the policy patch as an action item :)
14:52:42 <EmilienM> #action jpich to propose a policy for TripleO Spec Reviews
14:52:45 <EmilienM> jpich: thanks a lot
14:52:52 <EmilienM> do we have anything else for this week?
14:52:54 <jpich> No problems
14:53:39 <EmilienM> thanks everyone, keep rocking! see you on #tripleo
14:53:47 <EmilienM> #endmeeting