14:00:05 #startmeeting tripleo 14:00:06 #topic agenda 14:00:06 * Review past action items 14:00:06 * One off agenda items 14:00:06 * Squad status 14:00:06 * Bugs & Blueprints 14:00:06 Meeting started Tue Dec 5 14:00:05 2017 UTC and is due to finish in 60 minutes. The chair is mwhahaha. Information about MeetBot at http://wiki.debian.org/MeetBot. 14:00:06 * Projects releases or stable backports 14:00:07 * Specs 14:00:07 * open discussion 14:00:07 Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 14:00:08 Anyone can use the #link, #action and #info commands, not just the moderatorǃ 14:00:08 Hi everyone! who is around today? 14:00:10 The meeting name has been set to 'tripleo' 14:00:10 0/ 14:00:12 o/ 14:00:16 o/ 14:00:16 o/ 14:00:18 o/ 14:00:21 o/ 14:00:38 o/ 14:00:44 o/ 14:00:45 o/ 14:00:47 o/ 14:00:50 gchamoul: Hi! akrivoka and I have created an etherpad with patches we'd like to give priority among the many which haven't been reviewed yet. feel free to add some of your patches you consider most important: http://etherpad.corp.redhat.com/validations-reviews 14:01:00 o/ 14:01:02 o/ 14:01:08 hi 14:01:39 o/ 14:01:39 o/ 14:01:40 florianf: gchamoul: let's use this one instead :) https://etherpad.openstack.org/p/validations-reviews 14:01:40 o/ 14:02:02 o/ 14:02:08 o/ 14:02:14 #topic review past action items 14:02:14 none 14:02:24 moving on 14:02:32 #topic one off agenda items 14:02:32 #link https://etherpad.openstack.org/p/tripleo-meeting-items 14:02:43 o/ 14:02:43 (mwhahaha) Queens-m2 is here and gone. Queens-m3 is next (Jan 22 - Jan 26). 14:02:58 Milestone 2 is this week. We've already cut the release so at this point it's now more book keeping to understand what was released and where we're are for the final milestone. It should be noted that we were going to stop allowing the starting of new features at milestone 2 this release so if you're feature isn't started then now is the time to considering moving it to Rocky. I believe we can still make some 14:02:58 progress on some of the work if we can ensure that the feature is off by default and does not impact the existing features. 14:03:05 If you have an open feature, please review the outstanding work and be able to explain why we should allow your feature to continue forward for Queens. There are some features which we will most likely still need allow new work. For example the upgrades workflows and things related to fast forward upgrades. Beyond that, we should be focusing on bugs as we have nearly 540 open bugs. 14:03:22 I'm going to also send an ML note about this as well. 14:03:37 It's time to start stablizing 14:03:43 tl;dr no new things from now 14:03:44 o/ 14:04:06 akrivoka: :-) 14:04:35 any questions? 14:04:35 florianf, akrivoka: ack 14:04:39 I agree with Alex, we should stabilize what we started in queens, stop starting, start stopping, with exceptions for FFU and upgrades 14:04:44 I think integration squad has a couple of things in progress, which landed only partially 14:04:56 that ceph rbd mirror and nfs-ganesha 14:05:07 gfidente: I think that's fine, since it's a feature in progress I guess 14:05:15 it can be finished in m3 14:05:16 gfidente: which is fine but we'll need to raise those and explain the impact and make sure we're not adding regressions 14:05:17 o/ 14:05:50 iiuc the message is: from now, don't expect patches to land if they do something really new 14:05:55 I was wondering if we planed to swith to the --config-download workflow for default or if we'll have to wait for rocky 14:05:57 gfidente: it would also be beneficial to land those sooner than later, so let us know if you need reviews 14:06:09 mwhahaha EmilienM ok thanks guys 14:06:13 shardy: really good question 14:06:15 rbdmirror should be good, just needs an update 14:06:22 IMO it'd be good to align on one deploy workflow, as it's a bit of a headache trying to keep the two approaches in sync 14:06:25 nfs-ganesha news some more iterations 14:06:31 * social o/ 14:06:53 shardy: AFIK it doesn't work yet with OVB (we tried yesterday) - maybe a simple thing, but we haven't tested it in CI 14:07:07 unless there are blocker issues, I'd like to switch to it by default, with a flag to re-enable the "classic" heat based workflow for Queens 14:07:07 shardy: I agree 14:07:22 then remove the heat software deployment bits potentially in Rocky 14:07:33 ++ for config-download 14:07:38 if we can, let's do it like ASAP 14:07:44 It's not something we should do in january 14:07:46 EmilienM: Ok, maybe we could have a WIP patch switching the default to flush out any remaining issues? 14:07:52 i'd rather we have time to flush out all the issues 14:08:07 slagle: thoughts? 14:08:10 mwhahaha: yup agreed, if we're doing it we need to land it asap 14:08:45 o/ 14:08:52 Will we be tagging releases early on milestone week now, in general? They used to be done on Thursdays during milestones, from what I recall 14:09:03 shardy is config-download currently interpolating with the ExternalResource ? 14:09:26 gfidente: yes it works with the external_deploy_tasks, but not the mistral workflow resources 14:09:27 sorry not interpolating, interleaving 14:09:40 gfidente: what's the status of reworking the ceph integration to external_deploy_tasks? 14:09:50 shardy auch, like zero 14:09:51 jpich: the release team got really approve happy, but we shouldn't expect the end of the week. that being said we'll allow a few things to land this week while we do the updates to BPs and such 14:09:58 gfidente: :( 14:10:04 I thought it was discussed a while ago 14:10:14 discussed yes, at ptg 14:10:15 URGENT TRIPLEO TASKS NEED ATTENTION 14:10:16 https://bugs.launchpad.net/tripleo/+bug/1736284 14:10:17 https://bugs.launchpad.net/tripleo/+bug/1736411 14:10:17 Launchpad bug 1736284 in tripleo "Cannot commit transaction AddPortCommand exceptions cause tempest network_basic_ops to incur ssh timeouts" [Critical,Triaged] 14:10:18 Launchpad bug 1736411 in tripleo "CI: scenario002 job fails when not finding centos-binary-barbican-keystone-listener container" [Critical,Triaged] 14:10:23 mwhahaha: Noted 14:10:26 Thanks 14:10:26 in theory it should be fairly easy as it's just moving the ansible playbook run from mistral to t-h-t 14:10:28 shardy did not do the actual work though 14:10:40 gfidente: that's true, I did not ;) 14:10:42 I think jistr did some investigation? 14:10:54 shardy yeah I did not even, just bringing up needs to be done 14:10:58 gfidente: maybe we can take a look and figure out how much work it will be 14:11:18 shardy yeah sounds like we should 14:11:27 we probably need to take some actions 14:11:57 #action EmilienM and slagle look at switching config-download as default 14:12:00 #action EmilienM to talk with slagle about switching default deployments to use config-download 14:12:04 argggg 14:12:05 gfidente, shardy, if you guys were going to do this soon (like today) we could kill two birds with one stone if I could look-on 14:12:05 #undo 14:12:09 * mwhahaha is faster 14:12:18 mwhahaha: mine was longer 14:12:23 undo didn't work? 14:12:24 beagles: ah yeah we need the same for octavia 14:12:24 lol 14:12:33 HI all , i'm new to triplo controller HA. 14:12:37 #action gfidente and shardy look at ceph-ansible and config-download 14:12:41 I think I've got the gist of it and am in the process of trying it out but I think the general ideas are the same 14:12:48 s_b_34343: in a meeting right now, we'll answer all your questions shortly 14:13:02 EmilienM: sorry got distracted with something else 14:13:08 we can discuss post-meeting 14:13:10 beagles: yeah I think it's logically the same as the mistral approach, but generating the inventory may differ a bit 14:13:13 how shoul i start will be needing doc 14:13:13 https://github.com/openstack/tripleo-heat-templates/blob/master/extraconfig/services/kubernetes-master.yaml 14:13:19 shardy, ack 14:13:20 slagle: perfect 14:13:20 that is a working example for kubespray 14:13:30 to understand controller ha working for triplo 14:13:33 also 14:13:34 #link https://releases.openstack.org/queens/schedule.html 14:13:36 shardy ack 14:13:37 ok moving on 14:13:39 Gael Chamoulaud proposed openstack/tripleo-validations stable/pike: Remove yum warning in ceph-ansible validation https://review.openstack.org/525618 14:13:44 (akrivoka) Validations patches in need of reviews 14:13:48 #link https://etherpad.openstack.org/p/validations-reviews 14:13:49 hi 14:13:57 it's been a bit of a challange lately getting validations patches merged, as we have very few cores reviewing regularly 14:14:02 we have some patches that have been up for a long time, so just wanted to ask people to review them so we can get them merged 14:14:08 (see link above) 14:14:10 akrivoka: nice work on validations, I'm doing some reviews sometimes, it's really good 14:14:11 that's pretty much it 14:14:20 akrivoka: thanks 14:14:20 : ok will wait 14:14:24 EmilienM: thanks, much appreciated! 14:14:29 #action team to review open validation patches 14:14:34 +1 for EmilienM 14:14:37 (gcerami) getting a list of containers built by kolla 14:14:52 quick question 14:14:58 before we switch to containers 14:15:06 akrivoka: where these validations are tested? 14:15:32 I know tripleo-ci-centos-7-containers-multinode is supposed to run them? 14:15:46 EmilienM: not sure tbh 14:15:47 akrivoka: can you show me logs on where validations are run? 14:15:49 http://logs.openstack.org/79/525179/1/check/tripleo-ci-centos-7-containers-multinode/a72e345/logs/ 14:16:03 maybe florianf knows? 14:16:14 akrivoka: your team needs to figure out how your code is tested, imho 14:16:21 akrivoka: and if you need help from CI team, please ask 14:16:45 EmilienM: validations runs are enabled only on a few list of scenarios 14:16:57 ok, I will look into it 14:17:00 gchamoul: which ones? can I see logs? 14:17:11 we can take it offline though 14:17:20 EmilienM: let me check the list for you 14:17:23 #action EmilienM, akrivoka, gchamoul to check where validations are tested in tripleo ci 14:17:28 gchamoul: ok, post-meeting 14:17:30 mwhahaha: go ahead 14:17:38 EmilienM: only manually afaik 14:17:46 well gcerami is not here 14:17:55 panda: ^ 14:17:58 mwhahaha, he's panda 14:18:09 EmilienM: I planned to add the right set of jobs into tripleo-validations 14:18:16 hi, right now we are using a single static file as a list of containers to upload after promotion. WE were planning to make it pe release and get the list from kolla/parsed_containers.txt on each build, but thre are some reviews that make that list obsolete, What is now the best way to get a list of built containers so we know what to upload after a promotion ? 14:18:22 ah brain not working yet 14:18:57 I guess that's a question for dprince or mandre -^ 14:19:48 it's root cause of current https://bugs.launchpad.net/tripleo/+bug/1736411 14:19:50 Launchpad bug 1736411 in tripleo "CI: scenario002 job fails when not finding centos-binary-barbican-keystone-listener container" [Critical,Triaged] 14:20:18 Jiri Tomasek proposed openstack/tripleo-ui master: Use ListView from patternfly-react for NodesListView https://review.openstack.org/505611 14:20:19 Jiri Tomasek proposed openstack/tripleo-ui master: Use ListView from patternfly-react for NodeDrives list https://review.openstack.org/505612 14:20:23 panda: I think you can use the new 'container image prepare' commands to generate an upload yaml file dynamically 14:20:38 panda: in python-tripleoclient. Would this help solve your case? 14:20:51 dprince: do we need to install undercloud first ? 14:20:55 Adriano Petrich proposed openstack/tripleo-common master: Validate parameters before updating https://review.openstack.org/511249 14:21:04 panda: no, the command just requires python-tripleoclient 14:21:11 jtomasek, I've updated ^^ now it returns the flattened params 14:21:18 panda: you could even install it in a .venv 14:21:25 dprince, does it generate per release? 14:21:28 panda: the overcloud_containers.yaml.j2 file in tripleo-common has the list of all the images used in tripleo deployments 14:21:29 apetrich: sweet 14:21:47 :) 14:22:05 sshnaidm|rover: it an. It has parameters for the container names (prefix/suffix). Also the tag, and namespace can be fully controlled 14:22:14 jistr: the PSK change worked with updates. 14:22:30 Carlos Camacho proposed openstack/tripleo-heat-templates master: Do not merge - Check job multinode-upgrades-master https://review.openstack.org/525625 14:22:40 mandre: taht would require tripleo-common to be fetched too ? 14:22:57 jaosorior: ack great :) (mtg here...) 14:23:01 sorry 14:23:09 panda,sshnaidm|rover: openstack overcloud container image prepare --namespace=trunk.registry.rdoproject.org/master --images-file=rdo-images.yaml --push-destination=172.19.0.2:8787 --tag tripleo-ci-testing 14:23:10 dprince, I mean in pike set of containers could be different then in queens, isn't it? 14:23:45 sshnaidm|rover: then you'd use the Pike version of tripleoclient. It has the same command and could manage the differences via the template in python-tripleoclient 14:24:07 panda: right, or a curl since it's just a single file 14:24:12 ( gfidente, shardy: sorry i missed the mention earlier, let's sync after the meeting. We have Kubespray running in CI and green so that's at least somewhat of a "proof" that porting ceph-ansible to external_deploy_tasks should be achievable.) 14:24:14 sshnaidm|rover: they already are different. 14:24:25 dprince, great 14:24:44 yeah i think we can try the command solution 14:25:04 thanks all! 14:25:22 thanks panda 14:25:49 cool thanks 14:26:00 moving on 14:26:09 #topic Squad status 14:26:10 ci 14:26:10 #link https://etherpad.openstack.org/p/tripleo-ci-squad-meeting 14:26:10 upgrade 14:26:10 #link https://etherpad.openstack.org/p/tripleo-upgrade-squad-status 14:26:10 containers 14:26:10 #link https://etherpad.openstack.org/p/tripleo-containers-squad-status 14:26:11 integration 14:26:11 #link https://etherpad.openstack.org/p/tripleo-integration-squad-status 14:26:12 ui/cli 14:26:12 #link https://etherpad.openstack.org/p/tripleo-ui-cli-squad-status 14:26:13 validations 14:26:13 #link https://etherpad.openstack.org/p/tripleo-validations-squad-status 14:26:14 networking 14:26:14 #link https://etherpad.openstack.org/p/tripleo-networking-squad-status 14:26:15 workflows 14:26:15 #link https://etherpad.openstack.org/p/tripleo-workflows-squad-status 14:26:40 workflows missing updates 14:27:00 I want to thank beagles for starting the networking squad meeting yesterday. very informative 14:27:23 everyone updated their status, thanks! 14:27:31 any other status related items folks would like to raise 14:28:10 I was wondering if a weekly small summary on the ML would help 14:28:19 some other teams like Keystone, Nova, do it 14:28:39 but it would take someone willing to write-up something 14:28:47 One for tripleo or per squad? 14:29:03 for tripleo maybe? I'm not sure, it's a proposal 14:29:11 like "a week in tripleo", that you can read in less than 5 min 14:29:23 I had considered doing this for the workflow squad, but there hasn't been much to share. (We have only had two meetings so far) 14:29:34 EmilienM: are you volunteering to put it together? :D 14:29:40 i mean it's a good idea but it's yet more work 14:29:43 Attila Darazs proposed openstack/tripleo-quickstart master: Use cachable facts to allow multiple runs https://review.openstack.org/525237 14:29:45 ok... got it... I'll start something then :D 14:29:49 lol 14:29:55 but in french 14:30:01 sure no one reads the ML anyway 14:30:06 there is also chicken egg of... will people read it 14:30:20 trown: I'll give it a first try and ask who read it 14:30:26 I guess it's more for folks not already knee deep in tripleo 14:30:34 to get the highlights 14:30:48 or lowlights :p 14:30:50 let's try it and it could just be a basic summary out of the status etherpads 14:30:59 #action EmilienM starts to send a weekly newsletter about tripleo 14:31:13 cool, moving on then 14:31:17 or a summary of weshay's bad jokes 14:31:18 #topic bugs & blueprints 14:31:18 #link https://launchpad.net/tripleo/+milestone/queens-3 14:31:18 For Queens we currently have 71 (+1) blueprints and about 540 (-1) open bugs. 540 queens-3. 14:31:22 haha 14:31:31 but we need more than 5 min reading 14:31:50 Carlos Camacho proposed openstack/tripleo-heat-templates master: Do not merge - Check job multinode-upgrades-master https://review.openstack.org/525625 14:31:52 EmilienM, you had a good one today.. I see forward progress 14:32:03 i'm probably going to move the <= medium bugs out to a rocky-1 milestone just so we can see what high/critical bugs we have for milestone 3 14:32:18 please take a look at the high/critical bugs as we've previously mentioned now is a time for stablization 14:32:28 mwhahaha: do you need help in creating milestones and moving things around? 14:32:32 also i see some alert bugs poping up, let's make sure to take a look at them 14:32:45 EmilienM: no i'll get it today. i moved all the m2 bugs to m3 last night 14:32:47 Carlos Camacho proposed openstack/tripleo-heat-templates master: Do not merge - Check job multinode-upgrades-master https://review.openstack.org/525625 14:32:53 ok 14:33:01 the 2 alerts are unassigned 14:33:10 https://bugs.launchpad.net/tripleo/+bug/1736284 14:33:10 Launchpad bug 1736284 in tripleo "Cannot commit transaction AddPortCommand exceptions cause tempest network_basic_ops to incur ssh timeouts" [Critical,Triaged] 14:33:14 who is working on this one? 14:33:21 that one looks like the one we just fixed 14:33:27 i hadn't looked at the logs yet 14:33:34 but we'd see the same stuff with the ovsdbapp thing 14:33:43 I pinged Ihar 14:33:53 k 14:33:57 same for https://bugs.launchpad.net/tripleo/+bug/1736411 14:33:57 Launchpad bug 1736411 in tripleo "CI: scenario002 job fails when not finding centos-binary-barbican-keystone-listener container" [Critical,Triaged] 14:34:12 that one i assume is a missing container from promotion? 14:34:15 jaosorior: can you take this one? or find someone actively working on it? 14:34:18 related to panda's request previously? 14:34:28 most probably - but we need an owner 14:34:47 EmilienM, I'm actually working on it 14:34:50 EmilienM: it's the image issue I was complaining about. 14:34:54 EmilienM: sshnaidm|rover is working on it AFAIK. 14:35:14 sshnaidm|rover: so if you're working on it, please assign it to you 14:35:26 folks, please keep launchpad updated 14:35:28 EmilienM, assigned 14:35:32 good. 14:35:50 cool thanks, any other bug related items? 14:37:14 #topic projects releases or stable backports 14:37:20 so m2 went out yesterday 14:37:33 EmilienM: when is the next round of stable releases? next week? 14:38:49 yeah I think so 14:38:57 or I can do it end of this week 14:39:10 the release team would probably prefer next week 14:39:16 sounds good for me 14:39:20 ok 14:39:25 any other backports people need reviews on? 14:40:43 we said we keep stable/newton until January and then we see 14:40:47 (just a heads-up) 14:40:51 k 14:41:02 the # of backports into that branch is decreasing a lot 14:41:06 (good sign) 14:41:20 I think we can move on 14:41:55 #topic specs 14:41:55 #link https://review.openstack.org/#/q/project:openstack/tripleo-specs+status:open 14:42:06 since it's m2, we should be moving any open specs to rocky 14:42:45 maybe we can give an extra-very-last week 14:42:53 that's fine 14:42:58 and warn people by IRC, email, etc 14:43:05 i'll include that in my email note about m2/m3 14:43:20 I see 5 potential specs that has +2 already 14:43:26 good 14:44:16 Also i'll be going through the blueprints and pinging folks and moving to future releases 14:44:22 anyway on to bonus time 14:44:23 #topic open discussion 14:44:35 any random topics that people want to chat about? 14:45:04 did we update the list of cores? 14:45:08 I've seen nominations lately 14:45:23 rlandy was the latest iirc 14:45:28 trown: ^ 14:45:58 i believe the rest have been included, rlandy should be added if not already 14:46:53 I went ahead and added her to https://review.openstack.org/#/admin/groups/190,members 14:47:00 k 14:47:22 sounds like we're done 14:47:30 ok thanks everyone 14:47:33 #endmeeting