14:00:27 #startmeeting tripleo 14:00:28 Meeting started Tue Nov 29 14:00:27 2016 UTC and is due to finish in 60 minutes. The chair is EmilienM. Information about MeetBot at http://wiki.debian.org/MeetBot. 14:00:29 Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 14:00:33 The meeting name has been set to 'tripleo' 14:00:41 #topic agenda 14:00:47 * review past actions items 14:00:53 * one off agenda items 14:00:56 * bugs 14:00:58 * Projects releases or stable backports 14:00:59 o/ 14:01:00 * CI 14:01:01 o/ 14:01:02 * Specs 14:01:04 o/ 14:01:04 * open discussion 14:01:04 o/ 14:01:06 hello folks! 14:01:07 hello \o 14:01:07 Hey guys! 14:01:09 o/ 14:01:09 hey all 14:01:12 o/ 14:01:12 o/ 14:01:13 o/ 14:01:14 o/ 14:01:14 o/ 14:01:22 happy tuesday 14:02:05 #topic review past actions items 14:02:09 hello 14:02:12 o/ 14:02:19 o/ 14:02:23 EmilienM & jaosorior to report httpd issue on launchpad: I think the problem was solved 14:02:25 o/ 14:02:32 o/ 14:02:33 Hey 14:02:34 EmilienM to communicate on ocata release schedule: done, I sent an email on openstack-dev 14:02:50 o/ 14:03:03 EmilienM to proceed to last Liberty release and sync with tonyb about tripleo liberty EOL: done, I talked with tonyb last week 14:03:10 matbu lets use oooq in tripleo-ci for periodics M to N jobs 14:03:18 matbu: any update on this one? ^ 14:03:25 same for "matbu build upgrade jobs for N to O" 14:03:49 oops sorry hello all 14:03:53 team to review slagle's work on 3nodes job https://review.openstack.org/#/c/397441/ (and dependencies): not done, folks please review his patches 14:04:17 team to reviews specs targetted for ocata 14:04:37 and EmilienM to communicate on ocata specs freeze (end of o-2): done in the same email about ocata release schedule 14:04:41 EmilienM: no from my side, i think it was depending on a sshnaidm review 14:04:59 matbu: which one? 14:05:08 i was mostly working on composable upgrade and debuging current upgrade failure in centos infra 14:05:13 matbu, the job is merged now, I think you can try with it 14:05:14 o/ 14:05:26 o/ 14:05:33 sshnaidm: ack thanks, i'll start to work on this week end 14:05:42 #action matbu to build upgrade jobs for N to O 14:05:49 matbu, ok, feel free to ask for anything there 14:06:08 #action matbu lets use oooq in tripleo-ci for periodics M to N jobs 14:06:09 sshnaidm: yep thx 14:06:18 #topic one off agenda items 14:06:22 #link https://etherpad.openstack.org/p/tripleo-meeting-items 14:06:26 sshnaidm: go ahead 14:06:51 I'll copy my topic: 14:06:55 in process of transtion to quickstart we need more people in TripleO CI to review quickstart related patches. Right now the experimental oooq job is very untolerant to changes in both repos (quickstart and tripleo-ci) and requires a quick fixes to be merged urgently to work. With current time constraints we need more people involved in review and merge its patches in tripleo-ci repo. Volunteers? 14:07:24 The untolerant to changes in both repos makes me nervous ;) 14:07:50 shardy, it was related to new repo establishment - extras, so... 14:08:27 sshnaidm: sure, I appreciate landing the patches will help stablize things, I just wanted to clarify that's the only problem vs underlying fragility 14:08:32 thanks a lot to jaosorior trown and EmilienM, but we need a bit more people involved in review/merges 14:08:35 untolerant I think is poor wording... it is not gated 14:08:56 sorry I'm late, hey 14:09:00 anything that is not gated is fragile 14:09:06 trown: +1 14:09:10 trown, sure, sorry for my english :) 14:09:26 sshnaidm: ya no worries, was not trying to call you out, just trying to clarify :) 14:09:31 Ok coo, provided that's the only issue, thanks for clarifying 14:10:13 sshnaidm: thanks for reporting this issue 14:10:28 I also have a quick off item, about our squads 14:10:29 maybe it worth to give rights for tripleo-ci for oooq jobs to current quickstart cores..? 14:10:48 I noticed some people already put their names, which is awesome! 14:10:50 #link https://etherpad.openstack.org/p/tripleo-squads 14:10:53 sshnaidm: that would really only add 2 extra +2 :P 14:11:12 sshnaidm: I think it will just be a bit bumpy while we get everything going 14:11:14 trown, 5 is better than 3 14:11:36 we need to continue with mission statements, feel free to contribute and give your feedback! 14:12:05 I put up a rough draft for CI, but anyone feel free to modify 14:12:26 trown: thanks a lot 14:12:32 also, for CI in particular I think it would be good to have a subteam meeting 14:12:38 Do we really need mission statements? 14:12:41 trown, +1 14:12:45 e.g "Group of people focusing on TripleO deployed in containers" seems pretty clear to me 14:12:53 shardy: I guess it is useful for new people to know which squad to join. 14:12:57 shardy: I thought it would be great to document the scope of every team 14:13:04 shardy: otherwise one word is a bit vague :) 14:13:07 mission - "really, really focus on deploying TripleO, in containers, yeah!" 14:13:19 trown: for meetings, it will be up to the squads. If they need it or not. Nothing required here. 14:13:23 EmilienM: I'm unsure where the work that I do ends up in those squads 14:13:28 Ok, if folks find it useful then fair enough 14:13:32 containers may be a special case in that regard 14:13:53 it is pretty clear what falls under containers 14:13:55 shardy: we can rename mission statement to scope of work done by the squad 14:14:21 trown: I guess I see all of them as pretty clear, but sure if folks want to add more detail beyond the description to enumerate the scope, that's fine by me 14:14:22 I am trying to understand how the Liaison works. How are they picked? When do they speak with the PTL? 14:14:47 CI though, I think it is important to distinguish that the scope is CI/dev tooling 14:15:45 trown: i think it could be good to define CI/dev tooling then :) 14:15:49 d0ugal: they are picked when someone volunteer to do it in the squad. "When do they speak with PTL"? I don't know yet 14:16:09 d0ugal: I see it as someone willing to be the point of contact 14:16:13 d0ugal: i guess each team will work slightly differently. it could be rotated for example, or whatever works for the team 14:16:21 matbu: I thought we agreed on it during the Summit? 14:16:23 cool, thanks 14:16:33 I was trying to determine how much time it would take before I put my name down :) 14:16:36 marios: ++ for rotations 14:16:46 Also, if the squad has a seperate meeting, the Liaison would be responsible for reporting to this meeting? 14:17:21 EmilienM: i mean not the tool itself, but does that mean that the CI squad will create/debug/improve all periodic and gating jobs for all the scopes ? 14:17:35 trown: it's an option, yes. Reporting the blockers to PTL so we can take actions with other teams? 14:18:05 matbu: it is something we need to discuss, in this etherpad or mailing-list. Feel free to start it! 14:18:57 EmilienM: ack, i will add comments in the etherpad itself 14:19:06 d0ugal: it shouldn't take much time as there is no specific action for now, it's just a communication thing for now I guess 14:19:26 EmilienM: cool, sounds good. I copied trown and added a note next to my name :) 14:19:44 also, I think it's possible for someone to be part of multiple squads in the same time 14:20:16 ie; if you work on both upgrades and CI tooling/system, put your name in both squads would make sense 14:20:28 ya 14:20:35 yeah, I am in workflows and CLI/UI 14:21:07 right now, we're really in alpha version so things will evolve with time :-) 14:21:23 I'm moving to next topic, if any question or feedback, let us know 14:21:42 #topic bugs 14:22:06 #link https://launchpad.net/tripleo/+milestone/ocata-2 14:22:19 4 New, 4 Incomplete, 1 Invalid, 13 Confirmed, 48 Triaged, 34 In Progress 14:22:53 it seems we have some critical bugs still open and not fixed, I'll have a look at them 14:23:08 I'm also doing triage but we have so much new bugs every day in launchpad, it's hard to keep up 14:23:43 I would quickly remind folks to make auto triage (status, priority, assignment and release target). I would help people who make launchpad triage 14:24:26 do we have outstanding bugs this week that we need to discuss? 14:25:20 seems like no 14:25:53 #topic Projects releases or stable backports 14:26:27 reminder: ocata-2 release is in 2 weeks 14:27:00 we don't have any pending release at this time 14:27:12 "stable backport" reminded me of https://bugs.launchpad.net/tripleo/+bug/1640812 14:27:12 Launchpad bug 1640812 in tripleo "Network connectivity lost on node reboot" [Critical,Fix committed] - Assigned to Brent Eagles (beagles) 14:27:12 we can go to next topic 14:27:24 the fix we have for this is to os-net-config and there isn't a stable branch for mitaka 14:27:39 yet it is flagged for mitaka backport potential 14:27:40 beagles: ah, you need to submit a release in RDO itself afik 14:28:21 beagles: probably in rdo.yaml layout, if you're not familiar with it, let me know. We just need to change the tag for mitaka 14:28:35 EmilienM, ahhh... okay. Yeah, not familiar at all :) 14:29:10 beagles: https://github.com/redhat-openstack/rdoinfo/blob/master/rdo.yml#L936 14:29:15 #topic CI 14:29:18 EmilienM, cool. thanks 14:29:42 on CI side, I'm working on scenarios, and we moved the templates/environments to THT 14:29:56 this week i'm working on helping people to add more coverage in the scenarios 14:30:20 if you have any question or need help on this thing, please let me know 14:30:48 also, sshnaidm is working on adding a oooq job for containers 14:30:52 I wanted to mention that getting upgrade CI coverage is going to be come critically imporant soon 14:30:54 flaper87, dprince ^ 14:31:07 +1 14:31:09 shardy: yes, +1. My hope is that matbu can make progress asap... 14:31:17 we've been working towards it for some time, but we need to prove the reimplemented upgrades for N->) 14:31:25 N->O I mean :) 14:31:26 but we haven't seen much progress lately on this topic 14:31:29 EmilienM: ack 14:31:47 matbu: do you think we can have the CI job before ocata-2 ? 14:31:52 matbu: can you help enumerate the remaining work, perhaps start a ML thread or etherpad? 14:32:09 shardy: for N to O ? 14:32:15 If we need to find more help we can do it, but atm I'm not clear exactly what we're blocked on 14:32:30 matbu: we need a way to test upgrades for all supported branches 14:32:48 shardy: well it depend on the branches 14:32:53 matbu: i think we mean https://review.openstack.org/#/c/323750/ for starters but we'll need to plan working on a n->O too 14:32:57 shardy: for M to N we are blocked by the timeout 14:33:03 matbu: s/we/emilien was referring to i think 14:33:14 matbu: right, so all branches are blocked by the timeout 14:33:23 shardy: for Mitaka branch, i guess we don't care about it (L to M i mean) 14:33:30 we need a solution (multinode?) to that before we can move forwards, right? 14:33:46 matbu: yup, we could start with a M->N upgrade, then add N->O 14:33:52 shardy: and for N to O, i 'm planning to start a WIP review depending on your prototype 14:33:53 but we need the basic proven process in place 14:34:30 yeah so M to N would be the first step, as we know it works 14:34:36 matbu: Yeah I guess we could go directly to the new model 14:34:40 shardy: afair, the mutlinode won't work with mitaka. so last week we decide to use oooq (at least for periodic jobs) 14:34:50 I'd have been happier if we had a green job with the old implementation, then switch over to the new approach 14:35:07 but that's probably more work e.g writing the special-case support for moving services under httpd 14:35:47 sshnaidm: we can make gating jobs with tripleo-ci and oooq ? 14:35:52 matbu: Ok, sounds like this will have to be done after we get the initial N->O upgrades working then 14:36:11 matbu, I think so 14:36:13 matbu: perhaps we can start with a small subset of services, then add coverage as we implement full upgrade support 14:36:23 shardy: yes, baby steps. 14:36:25 shardy: yes 14:36:31 e.g a multinode job which upgrades just keystone using the new upgrade patch (which can be broken up) 14:36:37 then we add things service by service 14:36:48 shardy: just depending on your review, then adding more and more services and HA 14:36:54 +1 14:36:57 I just posted that big patch as an example, we can easily break it up 14:37:05 Ok, that sounds like a good way forward, thanks! 14:37:28 shardy: EmilienM just a question about the upgrade proto 14:37:30 matbu: let me know if I can help getting that initial test in place - should I break up the patch to e.g do just keystone as a first step? 14:37:30 im finishing to locally wire up the major upgrade pacemaker scripts to a HA N -> O deployment, the thing is that im getting some unbound variables out and to fix/redeploy them takes some time 14:37:38 shardy: do we plan to landed it to ocata-2 ? 14:37:55 matbu: possibly - let me break up the patch and see if reviews happen 14:38:08 matbu: yes, it would be great 14:38:11 a passing CI job would probably help give folks confidence in the approach 14:38:23 shardy: k 14:38:51 ccamacho: that's why i created pcs ansible module, it's more robust than the bash scripts i think 14:38:51 like shardy, we really need to make progress on this topic, even with a very few services 14:39:19 shardy +1 14:39:19 shardy: btw i got a almost fully upgraded deployment with pacemaker , marios 14:39:21 matbu: ++ had a quick first pass at those they look like a good idea to me 14:39:24 matbu yeahp, But I wanted to start with some working and then push it forward to your patch 14:39:41 matbu: i mean the https://review.openstack.org/#/c/403397/ ansible module 14:39:50 marios: yep 14:39:52 matbu: great! Did you exclude the services that moved under httpd? 14:40:01 I've still not had time to figure out a generic fix to those 14:40:16 shardy: yep, i exclude some service 14:40:27 matbu: Ok, sounds good, thanks for the update 14:40:29 shardy: i wanted to see how handle pacemaker 14:40:41 and if it was working consistensly 14:41:08 shardy: marios btw i feel the approach really better for debug and for rerunning failling step 14:41:50 shardy: but i was wondering to maybe split each playbook per steps ? 14:41:53 matbu: are we getting better error output from ansible compared to bash you mean? plus yeah by definition the steps are more discrete 14:42:01 (maybe not the right place to discusss about that) 14:42:02 marios: no 14:42:06 err, matbu ^ 14:42:18 matbu: we thought we could use tags 14:42:38 matbu: rather, having a playbook per role 14:42:56 EmilienM: yes but we will have a big huge playbook , harder to read than some small pieces of step 14:43:11 matbu: I'd prefer to have one playbook and run it via --tags stepN 14:43:22 but it's just a personnal feeling :) 14:43:28 matbu: as that's exactly how we run it via heat, but open to discussion 14:43:45 maybe we can discuss about it later 14:43:48 outside the meeting I mean 14:43:49 yup 14:43:49 shardy: yep k 14:43:52 thx 14:43:53 EmilienM: yes 14:43:56 anything else about CI ? 14:44:11 #topic specs 14:44:13 #link https://review.openstack.org/#/q/project:openstack/tripleo-specs+status:open 14:44:42 nothing to be added on my side, except, keep reviewing it when you have cycles :) 14:45:11 do we have anything to talk about specs this week? 14:45:26 quick reminder: ocata specs need to be merged before ocata-2 release 14:46:01 could I get a +W on https://review.openstack.org/#/c/388162/ 14:46:45 Something that came up on #tripleo before regaring spec reviews: Are sub project core reviewers supposed to 2+ specs related to their projects, even if the spec spans multiple projects? 14:46:54 owalsh: I'll approve it shortly if no more feedback 14:47:04 *regarding 14:47:34 EmilienM: thanks 14:47:37 florianf: I think it's up to reviewers to feel confortable or not to +2 or +1 14:48:23 we have 10 min left, I'll open the discussion 14:48:24 OK, so the normal rule for sub org cors doesn't apply here (only +2 your own project's repos ) 14:48:27 #topic open discussion 14:49:20 if you have any question or feedback, please go ahead now 14:49:20 I'd like you all to see the CI status page and to give your feedbacks/RFE, I'll write more detailed ML about it later: http://status-tripleoci.rhcloud.com/ 14:49:26 ya, I think +2 spec related to a specialty is fine... we probably would not want to merge any spec that spanned multiple specialties without a representative from each group +2 though 14:49:46 trown: excellent, sounds good. 14:50:12 sshnaidm: nice. I would try to merge tripleo.org/cistatus.html and your page 14:50:16 so we make things easier to find 14:50:18 * jpich wonders if that's worth an update to https://wiki.openstack.org/wiki/TripleO/SpecReviews to be explicit about it 14:50:47 sshnaidm: your dashboard is really awesome 14:50:49 jpich: It probably is, just to clarify... 14:51:13 jpich: it sounds like a good idea 14:51:21 florianf: I'll copy-paste some of the statements from the meeting there after 14:51:35 Ok will do 14:51:36 jpich: maybe move https://wiki.openstack.org/wiki/TripleO/SpecReviews to tripleo-specs as a policy? 14:51:49 Yeah I was thinking about it, it would be easier to find 14:51:54 so we can have a group discussion and leave Wiki (FYI OpenStack wiki will die one day) 14:52:10 Everything does one day 14:52:20 I'll take the policy patch as an action item :) 14:52:42 #action jpich to propose a policy for TripleO Spec Reviews 14:52:45 jpich: thanks a lot 14:52:52 do we have anything else for this week? 14:52:54 No problems 14:53:39 thanks everyone, keep rocking! see you on #tripleo 14:53:47 #endmeeting