14:00:19 #startmeeting tripleo 14:00:19 #topic agenda 14:00:19 * Review past action items 14:00:19 * One off agenda items 14:00:19 * Squad status 14:00:20 * Bugs & Blueprints 14:00:20 Meeting started Tue Feb 20 14:00:19 2018 UTC and is due to finish in 60 minutes. The chair is mwhahaha. Information about MeetBot at http://wiki.debian.org/MeetBot. 14:00:20 * Projects releases or stable backports 14:00:21 * Specs 14:00:21 * open discussion 14:00:21 Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 14:00:22 Anyone can use the #link, #action and #info commands, not just the moderatorǃ 14:00:22 Hi everyone! who is around today? 14:00:23 The meeting name has been set to 'tripleo' 14:00:26 o/ 14:00:28 o/ 14:00:28 o/ 14:00:28 o/ 14:00:31 o/ 14:00:36 o/ 14:00:38 o/ 14:00:41 o/ 14:00:43 o/ 14:01:02 o/ 14:01:19 o/ 14:01:37 o/ 14:01:37 o/ 14:02:02 o/ 14:02:10 o/ 14:02:27 alright got a busy meeting today. lots of previous action items to review. let's do this 14:02:34 #topic review past action items 14:02:34 o/ 14:02:41 tbarron or EmilienM to send a note to ML to start discussion on python3 testing - DONE 14:02:41 #link http://lists.openstack.org/pipermail/openstack-dev/2018-February/127334.html 14:02:55 thanks tbarron for sending the note 14:03:01 slagle to schedule to deep-dive about config-download - DONE 14:03:01 #link http://lists.openstack.org/pipermail/openstack-dev/2018-February/127529.html 14:03:20 reminder to add that to your calendar to learn more about config-download 14:03:35 sshnaidm to raise ML post about reducing risk with quickstart and quickstart-extras patches 14:03:39 o/ 14:04:25 mwhahaha: better late than never 14:04:42 I didn't see a note about quickstart so we'll postpone that one 14:04:47 #action sshnaidm to raise ML post about reducing risk with quickstart and quickstart-extras patches - postponed 14:04:52 o/ 14:04:54 sshnaidm, bogdando to figure out how to handle the fallout of https://review.openstack.org/#/c/517444/ 14:05:03 i think we're in the clear on that one, the ci was resolved 14:05:20 \o/ 14:05:27 mwhahaha, it wasn't reverted and we fixed everything there 14:05:43 yup 14:05:49 beagles to raise bug around neutron issues with containers - DONE 14:05:50 #link https://bugs.launchpad.net/tripleo/+bug/1749209 14:05:50 Launchpad bug 1749209 in tripleo "Stopping neutron agent containers also brings down dataplane services" [Critical,Triaged] - Assigned to Brent Eagles (beagles) 14:06:35 EmilienM to open a blueprint for realign puppet/services and docker/services for Rocky - DONE 14:06:35 #link https://blueprints.launchpad.net/tripleo/+spec/services-yaml-flattening 14:06:39 therve, hi 14:06:56 hopefully there will be some discussions around the neutron stuff as well as the services during the PTG 14:07:11 shardy to change the default deploy to include docker and add deprecation notice 14:07:13 indeed 14:07:28 shardy: were you able to post the docker as default? RC1 is this week so we need to get that in 14:07:54 Brad P. Crochet proposed openstack/tripleo-common master: Adds workflow to list deployment plans https://review.openstack.org/546154 14:07:58 Adriano Petrich proposed openstack/tripleo-common master: Move password generation to deployment phase https://review.openstack.org/542143 14:08:44 Sergii Golovatiuk proposed openstack/tripleo-heat-templates master: Refactor get-occ-config.sh https://review.openstack.org/546156 14:09:48 so we need to figure out that switch ASAP 14:10:19 URGENT TRIPLEO TASKS NEED ATTENTION 14:10:20 https://bugs.launchpad.net/tripleo/+bug/1749641 14:10:20 Launchpad bug 1749641 in tripleo "Overcloud deployment failing in promotion jobs at glance-manage db_sync" [Critical,In progress] - Assigned to Christian Schwede (cschwede) 14:10:21 do we have any volunteers to get that addressed before RC1? 14:11:12 tbh, I'm not sure we'll release RC1 this week 14:11:23 well i think we need to cut something 14:11:34 and maybe follow up with an rc2 14:12:01 out of current #topic but I don't think we should branch until we get promotion 14:12:19 Matthias Runge proposed openstack/tripleo-quickstart-extras master: Update openstack-virtual-baremetal https://review.openstack.org/546072 14:12:39 I can try to push a patch for the default docker thing 14:12:40 let's table that for now, i don't necessarily agree 14:12:46 Changing the default should be about adding a couple of lines to plan-environment.yaml, last time I chatted about this with dprince... What was the plan for the deprecation notice, a release note? 14:12:58 jpich: just a release note 14:13:06 jpich: i had already sent an ML note back in sept 14:13:12 or october 14:13:46 Ok, I can give it a shot after the meeting 14:13:54 thanks 14:13:55 yatin proposed openstack-infra/tripleo-ci master: Add bits to support stable/queens https://review.openstack.org/546062 14:14:11 #action jpich, EmilienM to change the default deploy to include docker 14:14:20 #topic one off agenda items 14:14:20 #link https://etherpad.openstack.org/p/tripleo-meeting-items 14:14:20 ykarel, Hi 14:14:32 (rasca) Follow-up about tripleo-quickstart-validation as a standalone repo under the tripleo umbrella http://lists.openstack.org/pipermail/openstack-dev/2018-February/127436.html 14:14:46 therve, you need environment for fs001 failure? 14:14:48 mwhahaha: Sorry, I started testing locally but didn't yet push it, will do so today 14:15:11 ok, so I opened the thread on the ML and would like to discuss it 14:15:22 since I'll not be at the PTG 14:15:24 therve, i have one, currently overcloud deploy is in progress 14:15:46 at least it would be good to know that this will be discussed 14:16:05 ykarel, OK, it'd be great to access it 14:16:11 rasca: it might be a good topic to bring up at the PTG during our CI slot 14:16:15 therve, your public key? 14:16:28 you can pm me 14:16:29 rasca: i'm not sure tripleo-validations should be doing any configuration or destructive testing 14:16:36 jpich: a release note would be fine with me 14:16:53 rasca: and the instance-ha configuration should probably be part of the deployment 14:17:15 dprince: Sounds like shardy is on top of it all :) 14:17:16 mwhahaha, yes, but not its testing 14:17:33 rasca: so can we leverage tempest for testing? 14:17:45 jpich: I'll work with you and EmilienM to get it done 14:18:27 Ryan Brady proposed openstack/python-tripleoclient master: Changes Delete Plan to Call Workflow https://review.openstack.org/539045 14:18:38 mwhahaha, I talked briefly with tosky about how to use tempest, but the aim here is to have a pluggable thing in which you can do whatever disruptive thing you want to check HA 14:18:52 rasca: I agree with mwhahaha. So far tripleo-validations are non-invasive. 14:19:05 rasca: tempest is our testing framework and it's plugable... 14:19:20 mwhahaha, florianf, because of this in my email I asked for a separate project 14:19:21 they're supposed to check things, but not change anything on the system. 14:19:43 mwhahaha: tempest is not for everything 14:20:01 i don't want yet another testing framework 14:20:14 mwhahaha: invasive, non-blackbox test operations (configuration change, killing of services) are better served with something else 14:20:23 mwhahaha: did you see the proposal for HA testing discussed with the *QA* team? 14:20:27 Sergii Golovatiuk proposed openstack/tripleo-heat-templates master: Get number of hosts from array and iterate over them https://review.openstack.org/546161 14:20:38 it's not about tempest, but some glue (ansible?) that can call any existing validation 14:20:55 each project with its own scope and strength; that's the base of openstack 14:21:01 mwhahaha, I understand we don't want one, but there's no place in which this kind of tests can fit 14:21:04 I'd just rather push for yet another tool if possible 14:21:35 Yurii Prokulevych proposed openstack/tripleo-upgrade master: Don't hardcode overcloud stack name. https://review.openstack.org/546162 14:21:36 note, there is also #openstack-self-healing 14:21:45 this seems like a major deficiency that should be solved in the community rather than just in tripleo as destructive testing should probably be a thing for OpenStack as a whole 14:21:46 and a SIG for it 14:21:54 mwhahaha, and in addition to that there also other things that this project does at the moment 14:22:17 *there are 14:22:18 wrt: HA testing upstream work 14:22:26 rasca: ok that's fine, i'm just saying validations doesn't make sense and I'd really prefer we not go of and build our own thing if possible 14:22:46 mwhahaha, yeah that's why I opened the discussion 14:22:51 if tempest doesn't make sense, fine. but we should document why 14:23:02 not just say "tempest is not for everything" 14:23:12 it 14:23:13 say "tempest is for A, B, C" 14:23:16 it's in the scope of tempest 14:23:26 I suggest to check few documents: 14:23:34 rally plugins could also help for injecting destructive things in the middle of a benchmark 14:23:42 my 2c 14:23:58 bogdando, instance ha test inside the project is based on rally 14:24:14 - the report from the QA discussions after the Queens PTG, check the section about "HA Testing": http://lists.openstack.org/pipermail/openstack-dev/2017-September/122295.html 14:24:22 bogdando, https://github.com/redhat-openstack/tripleo-quickstart-utils/tree/master/rally 14:24:32 - the link to the specific discussion about the HA proposal: https://etherpad.openstack.org/p/qa-queens-ptg-destructive-testing 14:25:00 - the link to the work-in-progress (now stalled) upstream spec about this (references in the above etherpad): https://review.openstack.org/#/c/443504/ 14:25:01 ack. Just thinking of possible ways to cooperate with that SIG openstack self healing 14:25:16 #link https://wiki.openstack.org/wiki/Self_healing_SIG 14:25:27 yatin proposed openstack-infra/tripleo-ci master: Add bits to support stable/queens https://review.openstack.org/546062 14:25:33 the reason why I've proposed this in the end, is also that we've been using it from over one year now in the internal pipelines and it helped a lot in discovering painful bugs 14:25:41 Sergii Golovatiuk proposed openstack/tripleo-heat-templates master: Get number of hosts from array and iterate over them https://review.openstack.org/546161 14:25:49 can I #link also the document that I posted, or only a session chair can do it? 14:25:58 tosky: you can #link 14:25:58 anyone can link 14:26:01 ack 14:26:07 #link http://lists.openstack.org/pipermail/openstack-dev/2017-September/122295.html 14:26:11 #link https://etherpad.openstack.org/p/qa-queens-ptg-destructive-testing 14:26:13 rasca: I agree we should be testing, but let's try and leverage existing stuff for it. 14:26:25 rasca: i'm also concerned about that set of roles in the redhat-openstack org 14:26:41 mwhahaha, what do you mean? 14:27:02 rasca: we need to discuss what that is and if that belongs under the proper tripleo umbrella 14:27:25 rasca: we should not be creating things under redhat-openstack if meant for usage by end users 14:27:47 mwhahaha, the reason why the repo is there are plenty 14:28:01 rasca: and we should have an open discussion about that 14:28:13 mwhahaha: my point above is that most of the things currently mentioned by rasca are not strictly tripleo-specific 14:28:30 and I think that there is space to move this to be a generic QA stuff, not just tripleo 14:28:49 tosky: right. I agree which is why I asked about other community tools 14:28:57 because there should be some stuff that we share 14:29:08 and not write our own if we don't have to 14:29:37 tosky: bogdando is the above rally pieces is the plugin itself na? 14:29:41 mwhahaha: sure; in fact an ansible-based framework (as per last discussion in the proposal) would potentially be able to call any existing tool for validation after changing the settings 14:29:48 mwhahaha, that was not the intention while this repo was created, but again, this is a long story 14:29:56 https://review.openstack.org/#/c/545824/ moves the rally-plugins to a seperate repo 14:30:17 chandankumar: tbh, I'm not much familiar with rally plugin systems, can't answer that 14:30:48 https://github.com/redhat-openstack/tripleo-quickstart-utils/tree/master/rally from this i was asking 14:30:53 rasca: are you going to the PTG? 14:31:01 my experience with it was limitied to creating a wrapper being invoked via the rally scrappy plugin. And that was a wrapper to call Jepsen :) 14:31:13 mwhahaha, no I am not so lucky :( 14:31:18 so basically I never touched nor rally itself not its plugins 14:31:31 rasca: is there anyone from your team who could talka bout this topic? 14:31:42 Derek Higgins proposed openstack/tripleo-quickstart master: Add featureset044 for Ironic in the Overcloud https://review.openstack.org/509829 14:32:06 mwhahaha, bandini, dciabrin_ and Ng are aware of the topic 14:32:13 and will be there 14:32:24 rasca: ok it might be beneficial for them to sit in on that QA topic as well 14:32:33 tosky: i like the idea of ansible-based framework, 14:32:58 #action team to provide feedback on http://lists.openstack.org/pipermail/openstack-dev/2018-February/127436.html 14:33:22 bogdando, tosky, chandankumar please provide some feedback/ideas via the ML for others as well 14:33:35 thanks everybody 14:33:40 thanks rasca 14:33:43 (mwhahaha) PTG schedule 14:33:48 #link http://lists.openstack.org/pipermail/openstack-dev/2018-February/127548.html 14:34:06 EmilienM and I put together a basic schedule. we need folks to step up to run the sessions 14:34:37 we tried to group similar topics together so that we can have some better conversations 14:34:42 mwhahaha: how/where do we do that? 14:34:48 if you have any additional topics or see anything mission let me know 14:35:05 slagle: on the ML or IRC, I guess 14:35:25 or in the etherpad, I check from time to time if any updates 14:35:34 mwhahaha: we will have an upgrade/update one some combination of me and chem will run it i'll add something there 14:35:35 #link https://etherpad.openstack.org/p/tripleo-ptg-rocky 14:36:10 marios: thanks 14:36:15 any other questions about the ptg? 14:36:27 mwhahaha: some brief notes at https://etherpad.openstack.org/p/tripleo-ptg-rocky-upgrades 14:36:32 Pradeep Kilambi proposed openstack/tripleo-heat-templates master: FFU: Introduce gnocchi fast-forward-upgrade tasks https://review.openstack.org/540385 14:36:56 Pradeep Kilambi proposed openstack/tripleo-heat-templates master: FFU: Introduce Aodh fast-forward-upgrade tasks https://review.openstack.org/539307 14:37:43 Marios Andreou proposed openstack/python-tripleoclient master: Split out the minor update and major upgrade classes https://review.openstack.org/545026 14:37:43 Marios Andreou proposed openstack/python-tripleoclient master: Adds Overcloud Upgrade class for openstack overcloud upgrade https://review.openstack.org/545365 14:37:59 marios: ok i've added it to the calendar 14:38:12 moving on 14:38:13 (mwhahaha) Queens RC1 14:38:14 mwhahaha: thanks i added to https://etherpad.openstack.org/p/tripleo-ptg-rocky line 55 14:38:23 mrunge, hi 14:38:23 so RC1 should be this week 14:38:26 we need to cut something 14:38:33 ykarel: hey 14:38:42 in my experience, cutting for just cutting wasn't helpful 14:38:44 Given the state of promotions I'm tempted to block anything not to fix a promotion issue 14:38:47 because if we cut we have to branch 14:38:52 mrunge, did you checked why overcloud deploy failed 14:38:58 and given the current state, branching doesn't make sense now 14:39:00 EmilienM: but we're currently not blocked on tripleo stuff right? it's just the glance issue? 14:39:06 mrunge, was it because of hash in openstack-virtual-baremetal 14:39:19 also; before we cut, we need stable/queens gate working 14:39:25 which isn't the case now, because we don't have promotion 14:39:35 so if we cut, we won't be able to land any code in stable/queens, 14:39:38 ykarel: I couldn't figure out why it fails. I've seen one success as well 14:39:54 ykarel, mrunge : we're in weekly meeting 14:39:59 mrunge, do you have logs 14:40:04 EmilienM, ok sorry 14:40:08 (in case you missed it) 14:40:19 I'm sorry. and yes, we apparently missed that 14:40:35 * ykarel me too 14:40:37 EmilienM: I'm still leaning towards not merging anything else until we get a promotion 14:41:01 the situation that can happen is 14:41:13 we cut, branch this week, while stable/queens gate isn't working yet 14:41:31 and then we realize we need to patch repo x and repo y to unblock blockers, but we can't, since gate is hosed 14:41:32 what's the stable/queens gate pending on 14:41:43 rdo promotion 14:41:59 right so let's not merge anythign else until we get a promotion so we don't have moving targets 14:42:07 afik, we manually created a symlink frm master but not even sure that works 14:42:08 are we just waiting on the glance issue or is there anything else 14:42:31 questions for rlandy|rover / arxcruz|ruck ^ 14:42:55 EmilienM: there's the glance issue, and mistral download config issue 14:43:04 we are following a couple others 14:43:14 therve: btw, ykarel has a env up for test the mistral download config issue 14:43:22 cschwede: where are we at with pushing https://review.openstack.org/#/c/544792/ 14:43:26 all on the promotion board 14:43:39 the glance one is the most common failure 14:43:52 and I'm not sure the glance fix (https://review.openstack.org/#/c/544792/) is going to land anytime soon if it lands 14:44:07 mwhahaha: jokke_ is looking into this, as well as croelandt 14:44:12 can we pin glance then 14:44:25 we get hit by oci runtime every now and again 14:44:31 note that there is a parent patch too that needs to be merged 14:44:37 also isn't that a rocky change 14:44:42 mwhahaha: EmilienM there's also a workaround in the mysql to set a config flag 14:44:45 cschwede: good point 14:44:46 so the glance thing shouldn't affect stable/queens 14:44:54 for glance, we need a mitigation 14:45:07 https://review.openstack.org/#/c/433934/ was for rocky, that shouldn't be affecting stable/queens 14:45:21 arxcruz|ruck, Yep on it 14:45:31 mwhahaha: so there is another glance issue? 14:45:43 cschwede: not that's the db thing 14:45:56 cschwede: that change landed on master but shouldn't be in queens 14:46:08 hmm 14:46:50 anyway we need to figure this out like today 14:47:03 it's not in stable/queens 14:47:11 #action EmilienM to sync with releases team to make sure us missing rc1 is not a problem 14:47:19 I can do that 14:47:37 #action mwhahaha & others need to come up with a plan to get stable/queens gate working ASAP so we can release 14:48:02 moving on 14:48:12 #topic Squad status 14:48:12 ci 14:48:12 #link https://etherpad.openstack.org/p/tripleo-ci-squad-meeting 14:48:12 upgrade 14:48:12 #link https://etherpad.openstack.org/p/tripleo-upgrade-squad-status 14:48:13 containers 14:48:13 #link https://etherpad.openstack.org/p/tripleo-containers-squad-status 14:48:14 config-download 14:48:14 #link https://etherpad.openstack.org/p/tripleo-config-download-squad-status 14:48:15 integration 14:48:15 #link https://etherpad.openstack.org/p/tripleo-integration-squad-status 14:48:16 ui/cli 14:48:16 #link https://etherpad.openstack.org/p/tripleo-ui-cli-squad-status 14:48:17 validations 14:48:17 #link https://etherpad.openstack.org/p/tripleo-validations-squad-status 14:48:18 networking 14:48:18 #link https://etherpad.openstack.org/p/tripleo-networking-squad-status 14:48:19 workflows 14:48:19 #link https://etherpad.openstack.org/p/tripleo-workflows-squad-status 14:48:48 * mwhahaha pokes UI team for updates 14:49:20 #topic bugs & blueprints 14:49:20 #link https://launchpad.net/tripleo/+milestone/queens-rc1 14:49:20 For Queens we currently have 33 (+0) blueprints and about 629 (+23) open bugs. 582 queens-rc1, 45 rocky-1, 1 rocky-2, 1 rocky-3. 14:49:29 so we should be focusing on the critical bugs for rc1 14:49:36 i'll move any non critical bugs out 14:49:44 we should not be merging anything that's not critical at this point 14:50:09 #link https://launchpad.net/tripleo/+milestone/rocky-1 14:50:09 As a reminder the PTG is coming up so please make sure to update your blueprints for the PTG. 14:50:16 any questions around bugs/blueprints? 14:50:52 #topic projects releases or stable backports 14:51:11 as previously mentioned work to get rc1 needs attention 14:51:25 EmilienM: any stable things that need addressing? 14:51:35 not this week! 14:51:36 Bogdan Dobrelya proposed openstack/tripleo-quickstart-extras master: undercloud: add missing TLS environments when preparing containers https://review.openstack.org/545444 14:51:37 Bogdan Dobrelya proposed openstack/tripleo-quickstart-extras master: Don't touch Ironic config when undercloud is containerized https://review.openstack.org/546055 14:51:45 #topic specs 14:51:45 #link https://review.openstack.org/#/q/project:openstack/tripleo-specs+status:open 14:51:59 also if you have a spec you want to talk about, make sure it's in good shape 14:52:10 #topic open discussion 14:52:18 any other topics? 14:53:22 sounds like nope 14:53:25 thanks everyone 14:53:27 #endmeeting