14:00:17 #startmeeting tripleo 14:00:17 #topic agenda 14:00:17 * Review past action items 14:00:17 * One off agenda items 14:00:17 * Squad status 14:00:17 * Bugs & Blueprints 14:00:18 * Projects releases or stable backports 14:00:18 Meeting started Tue Mar 13 14:00:17 2018 UTC and is due to finish in 60 minutes. The chair is mwhahaha. Information about MeetBot at http://wiki.debian.org/MeetBot. 14:00:18 * Specs 14:00:19 * open discussion 14:00:19 Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 14:00:20 Anyone can use the #link, #action and #info commands, not just the moderatorǃ 14:00:20 Hi everyone! who is around today? 14:00:21 The meeting name has been set to 'tripleo' 14:00:29 o/ 14:00:30 o/ 14:00:31 o/ 14:00:36 o/ 14:00:38 o/ 14:00:44 o/ hey 14:00:48 o/ 14:01:14 \o 14:01:23 o/ 14:01:31 o/ 14:01:40 o/ 14:01:42 o/ 14:01:43 o/ 14:02:06 o/ 14:02:06 o/ 14:02:17 ok looks like we've got a decent number of people 14:02:19 let's begin 14:02:25 o/ 14:02:29 #topic review past action items 14:02:41 sshnaidm to raise ML post about reducing risk with quickstart and quickstart-extras patches 14:02:46 o/ 14:02:46 o/ 14:02:49 \o 14:02:57 would be nice to get that on the CI squad agenda maybe 14:03:01 mwhahaha, let's cancel it for now 14:03:04 k 14:03:06 o/ 14:03:18 EmilienM & mwhahaha to sync with ccamacho about patch missing from queens? - DONE was figured out last meeting as tripleoclient had been branched earlier than everything else. I believe the patches have been backported. 14:03:38 yes 14:03:38 ccamacho: are we missing anymore patches or have they all merged? 14:04:04 mwhahaha I think most of them are +2 +A 14:04:08 k 14:04:10 just pending for merge 14:04:22 mwhahaha: i was doing that and there a re still a few to go as ccamacho said 14:04:25 mwhahaha: sec 14:04:43 o/ 14:04:45 marios jistr matbu, any other 14:04:45 https://review.openstack.org/#/q/branch:stable/queens+topic:update/workflow-stable/queens mwhahaha here 14:05:03 mwhahaha: (also linked in upgrade squad status fyi) 14:05:11 marios: cool thanks 14:05:21 o/ 14:05:22 EmilienM to release stable tripleo this week - DONE 14:05:39 mwhahaha to send a note about blueprints & owners to the ML - DONE 14:05:39 #link http://lists.openstack.org/pipermail/openstack-dev/2018-March/128217.html 14:05:50 Please note that there is a due date on blueprints 14:06:14 April 3, 2018 14:06:29 any blueprint not updated with an assignee and not approved will be moved out 14:06:36 ++ 14:06:39 sounds reasonable 14:06:48 that's ~3 weeks before Rocky-1 14:06:57 so we should have a good idea what we're working on 14:07:25 EmilienM & and others to send ML posts about PTG 14:07:26 Seen some notes about the sessions. Please make sure that if you have some notes that you send them to the ML. 14:07:41 is the due date for policies too ? 14:08:02 panda: no 14:08:07 panda: not specs, just blueprints 14:08:12 ok 14:08:28 panda: no specific due date on policies 14:08:51 moving on to the one off items which it appears we have some this week 14:08:54 #topic one off agenda items 14:08:54 #link https://etherpad.openstack.org/p/tripleo-meeting-items 14:09:00 (gfidente) can we enable Ceph testing in OVB? featureset001? 14:09:18 no 14:09:33 we try to keep OVB jobs strictly minimum 14:09:48 so they are fast and test what we actually need 14:09:55 gfidente: what's the reasoning behind it? 14:09:56 ceph is already well tested on scenario001 and scenario004 14:10:06 EmilienM: i wouldn't call it well tested 14:10:08 EmilienM yeah but to answer mwhahaha's question 14:10:08 OVB is useful to test ironic/mistral/nova 14:10:15 URGENT TRIPLEO TASKS NEED ATTENTION 14:10:15 https://bugs.launchpad.net/tripleo/+bug/1754036 14:10:16 Launchpad bug 1754036 in tripleo "fs020, tempest, image corrupted after upload to glance (checksum mismatch)" [Critical,Triaged] 14:10:27 it's tested, but not that well 14:10:29 EmilienM exactly, the access.yaml workflow, we only test the codepath which uses "deployed server" right now (multinode) 14:10:34 mwhahaha: well, it's not that bad 14:10:49 we test autoscaling on scenario001 with ceph backend for gnocchi, it was a lot of work 14:10:59 gfidente: we do test that path on 001 with --config-download 14:11:04 EmilienM mwhahaha in OVB we could test the other part of the workflow enable_ssh_via_nova 14:11:10 and we test Ceph Radow GW on scenario004 to replace swift, and use it as image backend in glance, same thing, lot of work 14:11:23 slagle yeah what I am saying is that we test only that in scenarios 14:11:48 what is the gap you are trying to cover w/ ceph + ovb? 14:11:50 EmilienM mwhahaha slagle yes guys I am pretty happy about scenarios, I am just proposing OVB to extend coverage of access.yaml 14:12:25 let me find a link to the specific piece of code 14:12:27 you could do like Juan and create another OVB jobs (new featureset) maybe 14:12:50 https://github.com/openstack/tripleo-common/blob/master/workbooks/access.yaml#L89-L144 14:12:54 but I'm not in favor of adding things on fs001/035. We've spent crazy time on reducing their runtime. 14:13:23 Juan created an OVB job for testing IPsec, and it's running only when some files are touched, iiuc 14:13:25 so maybe a new ovb fs for ceph-anisble & access.yaml? 14:13:36 Jose Luis Franco proposed openstack-infra/tripleo-ci master: Add multinode-overcloud-update playbook to run list https://review.openstack.org/547058 14:13:37 gfidente: tripleoclient calls enable-ssh-admin already, even with ovb 14:13:39 gfidente: would that work fo ryou? 14:13:59 gfidente: that uses access.yaml 14:14:01 gfidente, https://etherpad.openstack.org/p/quickstart-featuresets fyi 14:14:09 just saying, that is already tested, what you're missing is ovb+ceph 14:14:15 slagle I know it does, but in access.yaml there are two codepaths for deployed server vs regular nodes https://github.com/openstack/tripleo-common/blob/master/workbooks/access.yaml#L37 14:14:28 gfidente: yes. and they're both already tested 14:14:38 create_admin_via_nova vs create_admin_via_ssh 14:14:39 o/ 14:14:42 slagle oh wait let's go back to this after the meeting then 14:14:49 Marios Andreou proposed openstack/python-tripleoclient master: Wire up a --skip-tags option for the overcloud upgrade run cli https://review.openstack.org/551976 14:14:50 Marios Andreou proposed openstack/python-tripleoclient master: Remove nargs="+" from --nodes and --roles expect "comma, string" https://review.openstack.org/552528 14:14:57 if you say both are tested maybe this happens in another configuration and it's still good enough 14:15:07 maybe add as action item for next meeting? 14:15:40 gfidente: what you want is ceph+ovb+config-download, in a new fetureset 14:15:59 slagle oh I was thinking to use scenarios for config-download 14:16:11 not ovb 14:16:33 ovb featureset001 already uses config-download 14:16:41 I think we'll switch scenarios to run config-download very soon, isn't? 14:16:51 which is why access.yaml is already covered for create_admin_via_nova 14:16:53 EmilienMyeah so again I am happy about that and scenarios coverage 14:17:04 my goal was extend coverage for the workflows in access.yaml 14:17:33 * EmilienM has to disappear 2 min 14:17:46 slagle oh so you're saying _via_nova is called in featureset001 because it's OVB 14:18:06 ? 14:18:27 (from queens, given the tripleoclient change) 14:18:27 ? 14:18:30 this is what i'm saying 14:18:46 ok not worth the effort of another featureset just to cover pike then 14:18:59 works for me 14:19:09 ok sounds like we answered that one 14:19:12 thanks gfidente 14:19:16 thank you! :D 14:19:27 HA on two controllers, any plans to support that by TripleO ? (repeating question) 14:19:31 from unknown contributor 14:19:36 so yes this is from me 14:19:51 I keep seeing that question from customers 14:19:59 bandini, rasca, Ng: 2 node HA? 14:20:04 and I wonder if there are any plans to implement it 14:20:11 yes I know how it sounds, 2 node HA 14:20:30 mwhahaha, there's no HA with 2 nodes 14:20:32 it's not weird, i've seen it done :D but usually active-passive 14:20:33 but maybe some extra monitor on compute ? 14:20:49 rasca: I know 14:20:52 :) 14:20:54 Merged openstack/instack-undercloud stable/newton: [CVE-2018-1000115] memcached: restrict to TCP & localhost https://review.openstack.org/551393 14:20:55 Merged openstack/instack-undercloud stable/ocata: [CVE-2018-1000115] memcached: restrict to TCP & localhost https://review.openstack.org/551390 14:21:10 radeks: so sounds like currently no plans 14:21:20 understood 14:21:24 thanks! 14:21:33 (beagles) Neutron agent "side car containers" patches up for review - while the related bug (https://bugs.launchpad.net/tripleo/+bug/1749209) isn't marked as critical this is a pretty major issue. I want to amp up the visibility here in case others are affected (upgrades, other networking related projects, etc). 14:21:34 Launchpad bug 1749209 in tripleo "Stopping neutron agent containers also brings down dataplane services" [High,In progress] - Assigned to Brent Eagles (beagles) 14:21:40 mwhahaha, radeks, we've spent some time about that topic, and the outcomes of the discussion needs to be discussed more 14:22:02 https://review.openstack.org/549855 Add docker packages to dhcp and l3 agent containers - tripleo-common 14:22:02 https://review.openstack.org/550224 Adding wrapper scripts for neutron agent subprocesses - puppet-tripleo 14:22:02 https://review.openstack.org/550823 Generate and mount wrappers for dnsmasq and keepalived​ - tripleo-heat-templates​ 14:22:06 so for those that are interested, this is a case of an openstack component launching it's own containers 14:22:07 rasca: I am happy to discuss :) 14:22:20 neutron needs to do this to maintain the dataplane 14:22:48 it's weird so I'd like any interested/curious to have a look and provide comments 14:22:55 rasca: we could probably put Galera monitor and Pacmeaker agent on one of the computes 14:23:10 radeks, wow 14:23:16 rasca: with the composable roles this is possible 14:23:26 just an idea 14:23:27 radeks, yes, but let's bring the popcorns then :D 14:23:35 :) 14:23:59 but seriously the related bug is a pretty big deal so any help with expertise and reviews is definitely welcome 14:24:05 radeks, there are a lot of things that a 2 node cluster implies 14:24:09 Hamdy Khader proposed openstack/tripleo-heat-templates master: Add environment file for Mellanox SDN https://review.openstack.org/550507 14:24:19 Hamdy Khader proposed openstack/puppet-tripleo master: Enable networking-mlnx ml2 drivers https://review.openstack.org/550505 14:24:30 radeks, starting for example on the fence aspect 14:24:39 beagles: ok is this something we'll need to backport to queens when it lands? 14:24:44 * mwhahaha assumes yes 14:24:45 rasca: I am not going to argue, I know what are the problems 14:24:48 mwhahaha, yup 14:24:58 beagles: fyi as promised we're exploring the rolling pre-upgrade tasks here https://review.openstack.org/#/c/552073 14:25:05 jistr, ack thanks! 14:25:20 rasca: but you probably see the same kind of question popping up from time to time 14:25:28 * jistr testing it right now 14:25:30 radeks, sure. My point is: there are plenty of workaround you can use to have it, but then when it comes to production we need to be careful 14:26:15 beagles: what cleans up the containers that get launched? 14:26:20 radeks, so if we want to call it HA we need 3 nodes, otherwise it's a workaround HA solution. IMHO unsupportable (too many variables) 14:26:21 rasca: I know but if customers are willing to take a risk and there are customer who are willing to take it 14:26:24 beagles: very interested in this: so far this dynamic launch of containers for services is scoped only to controller nodes? 14:26:35 tbarron, at the moment 14:26:57 radeks, risk == no support? 14:27:11 rasca: yeahh it's a good question 14:27:16 mwhahaha, ummm... crap that's a good point. Just because the container gets killed doesn't mean it goes away 14:27:16 don't know 14:27:23 Matt Young proposed openstack/tripleo-quickstart-extras master: queens: Add failing barbican tempest test to skip list https://review.openstack.org/552529 14:27:36 mwhahaha, I'll see what can be done 14:27:37 beagles: we should ensure that any implementation is not tied to the controller role(s), composable roles and composable HA enable a lot of flexibility in that regard so it'd be good to maintain it 14:27:40 beagles: for manila we're looking to scale out services (ganesha gateways, ceph mds daemons) per tenant rather than sharing them among all tenants 14:28:09 beagles: but that means scale *out* -- can't just pile them up on controllers 14:28:19 i don't necessarily think it's a good idea to have containers launching more containers 14:28:22 shardy, right.. I don't think this will get in the way there but we can test it 14:29:05 radeks, I agree. The central discussion is about two questions. Does it works? It can. Should we support? I guess no. 14:29:15 tbarron, that's a bigger architectural type consideration - this is just getting the processes out of the agent's so things continue to function as expected 14:29:30 beagles, shardy: fwiw the pre_upgrade_rolling_tasks part should support such composable approach. We should probably take the approach of "run these actions on whichever roles have L3 agent". 14:29:32 tbarron, when i see that I think "kubernetes?" :) 14:29:42 beagles: ack, and that's what we're thinking too 14:30:06 jistr: ack I think with per-service groups in tripleo-ansible-inventory we can already do that pretty easily? 14:30:09 jistr: we should get that with your https://review.openstack.org/#/c/552073/ and then add the tasks to the specific files 14:30:25 jistr: i mean for l3agent in this case 14:30:53 shardy: yes it's quite a small change as far as the upgrade framework is concerned. Depends how well will that fit on the Neutron side of things. 14:31:21 marios: yea 14:31:47 Juan Antonio Osorio Robles proposed openstack/instack-undercloud master: Enable TLS by default https://review.openstack.org/552382 14:32:09 any other thoughts on the neutron container fun? :D 14:32:30 mwhahaha: link to etherpad with discussion on upgrade status 14:32:51 #link https://etherpad.openstack.org/p/tripleo-upgrade-squad-status 14:32:52 that one? 14:33:05 Last community call Mar 09 https://bugs.launchpad.net/neutron/+bug/1738768 Dataplane downtime when containers are stopped/restarted https://etherpad.openstack.org/p/tripleo-upgrade-squad-meeting 14:33:06 Launchpad bug 1738768 in tripleo "Dataplane downtime when containers are stopped/restarted" [High,In progress] - Assigned to Brent Eagles (beagles) 14:33:09 mwhahaha: yeah 14:33:35 #link https://etherpad.openstack.org/p/tripleo-upgrade-squad-meeting 14:33:37 Juan Antonio Osorio Robles proposed openstack/instack-undercloud master: Make user-provided cert take priority over autogenerated one https://review.openstack.org/552381 14:33:38 (meeting vs. status) 14:33:53 jistr: ;) 14:34:26 ok anything else? 14:34:34 mwhahaha: Can we recheck yet? 14:34:55 * mwhahaha is unsure 14:35:06 Juan Antonio Osorio Robles proposed openstack/instack-undercloud master: Enable TLS by default https://review.openstack.org/552382 14:35:20 dpeacock: i think so, will check after the meeting and let you know 14:35:35 last item from the one offs 14:35:36 (gcerami) Request for feedback http://lists.openstack.org/pipermail/openstack-dev/2018-March/128181.html https://review.openstack.org/548005 14:35:38 Honza Pokorny proposed openstack/tripleo-ui master: Refactor RolesActions to use startWorkflow https://review.openstack.org/552532 14:36:33 yep, just wanted to draw attention to the devmode replacement proposal here http://lists.openstack.org/pipermail/openstack-dev/2018-March/128181.html , since it's something most of the developers are asking, please understand if this would work for you. 14:37:07 Ronelle Landy proposed openstack/tripleo-quickstart-extras master: Fix overcloud_nodes check failing branch FF https://review.openstack.org/551369 14:37:47 panda: thanks for raising this 14:38:06 any questions on the devmode improvements? 14:39:16 sounds like nope, ok moving on 14:39:20 for any question or feedback there's the CI community meeting schedule right after this meeting 14:39:30 I'll be there, thanks. 14:39:42 #topic Squad status 14:39:42 ci 14:39:42 #link https://etherpad.openstack.org/p/tripleo-ci-squad-meeting 14:39:42 upgrade 14:39:42 #link https://etherpad.openstack.org/p/tripleo-upgrade-squad-status 14:39:43 containers 14:39:43 #link https://etherpad.openstack.org/p/tripleo-containers-squad-status 14:39:43 config-download 14:39:44 #link https://etherpad.openstack.org/p/tripleo-config-download-squad-status 14:39:45 integration 14:39:45 #link https://etherpad.openstack.org/p/tripleo-integration-squad-status 14:39:46 ui/cli 14:39:46 #link https://etherpad.openstack.org/p/tripleo-ui-cli-squad-status 14:39:47 validations 14:39:47 #link https://etherpad.openstack.org/p/tripleo-validations-squad-status 14:39:47 networking 14:39:48 #link https://etherpad.openstack.org/p/tripleo-networking-squad-status 14:39:49 workflows 14:39:49 #link https://etherpad.openstack.org/p/tripleo-workflows-squad-status 14:40:59 thanks everyone for updating the status etherpads 14:41:10 looks like there's some good info in there, please take a look 14:41:19 #topic bugs & blueprints 14:41:19 #link https://launchpad.net/tripleo/+milestone/rocky-1 14:41:19 For Rocky we currently have 63 (+1) blueprints and about 605 (+19) open bugs. 601 rocky-1, 1 rocky-2, 3 rocky-3. 14:41:40 so we appear to continue to be adding more bugs. I know last week was a bit weird due to rdo cloud issues 14:42:11 as a reminder about the blueprints from earlier, please see my note about making sure you've properly updated any blueprints you plan on landing 14:42:29 We've got 63 right now and we probably can't land them all 14:42:49 I'll add in a placeholder for the s cycle since I don't think the name has been decided yet 14:43:06 any bugs or blueprints that need attention? 14:43:43 mwhahaha: i have two blueprints we can discuss now or after the meeting if you like 14:43:59 fultonj: now should be fine, what's up? 14:44:05 https://blueprints.launchpad.net/tripleo/+spec/bluestore https://blueprints.launchpad.net/tripleo/+spec/osds-by-rule 14:44:18 mwhahaha: i talked to EmilienM about them 14:44:51 i basically wanted to get them approved 14:45:08 fultonj: ok are they just some basic changes to expose support via ceph-ansible? 14:45:10 they're somewhat simple as it's a matter of enabling tripleo to use a ceph-ansible feature 14:45:16 mwhahaha: yes 14:45:34 but those changes to expose that support would reference the blueprints and we'd know why 14:45:42 e.g. new tht params 14:45:53 fultonj: fair enough, i don't see any reason not to approve these 14:45:58 fultonj: i'll update them after the meeting 14:46:00 mwhahaha: ok, thanks 14:46:32 fultonj: speaking of ceph ansible have we moved it to an external task yet? 14:46:42 mwhahaha: not yet 14:46:53 there's a review up that we're debugging 14:46:56 * fultonj gets review 14:47:06 Jose Luis Franco proposed openstack/tripleo-upgrade master: New major upgrade workflow implementation. https://review.openstack.org/548336 14:47:07 Jose Luis Franco proposed openstack/tripleo-upgrade master: [DNM]: Apply workaround for P->Q upgrades. https://review.openstack.org/549220 14:47:07 https://review.openstack.org/#/c/546966/ 14:47:29 ok if i remember correctly that's an issue for switching to config-download by default 14:47:40 please let us know if you need any help 14:47:40 i think we're pretty close might be done in a week or two depending in interupts 14:47:47 sounds good 14:47:55 mwhahaha: thanks we want to land it asap 14:48:13 cool moving on 14:48:14 #topic projects releases or stable backports 14:48:22 any backports people need eyes on? 14:48:35 we just did a stable release last week so we're not due for one yet 14:48:37 Jose Luis Franco proposed openstack/tripleo-upgrade stable/queens: New major upgrade workflow implementation. https://review.openstack.org/552082 14:48:38 Jose Luis Franco proposed openstack/tripleo-upgrade stable/queens: [DNM]: Apply workaround for P->Q upgrades. https://review.openstack.org/552084 14:48:47 Brent Eagles proposed openstack/puppet-tripleo master: Adding wrapper scripts for neutron agent subprocesses https://review.openstack.org/550224 14:49:02 https://review.openstack.org/#/c/552018/ 14:49:08 queens backport ^ 14:49:59 Dan Prince proposed openstack/tripleo-heat-templates master: Configure firewall rules for Congress https://review.openstack.org/552535 14:49:59 * mwhahaha adds to the list 14:50:05 i'll take a look when the recheck comes back 14:50:19 thanks 14:50:29 moving on 14:50:32 #topic specs 14:50:33 #link https://review.openstack.org/#/q/project:openstack/tripleo-specs+status:open 14:50:50 probably a similar thing to the blueprints, we need to review/land them 14:51:09 please take a look at the open ones when you have time 14:51:18 anyone have any spec questions/issues? 14:52:18 sounds like nope 14:52:20 #topic open discussion 14:52:23 any other business? 14:52:29 mwhahaha: radeks: (and apologies if someone answered in the mean time) - 2 node HA has some solutions, but it's likely >1yr away given our current roadmap 14:53:16 mwhahaha: we will have a tripleo community meeting today in 7 minutes 14:53:19 Ng thanks. sounds like radeks , rasca & company should chat about some options 14:53:23 https://bluejeans.com/7071866728 14:53:29 for whoever is interested 14:53:37 * mwhahaha can make it today, yay daylight savings 14:53:56 alright thanks everyone 14:53:59 #endmeeting