14:00:22 #startmeeting tripleo 14:00:27 Meeting started Tue Dec 13 14:00:22 2016 UTC and is due to finish in 60 minutes. The chair is EmilienM. Information about MeetBot at http://wiki.debian.org/MeetBot. 14:00:28 Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 14:00:31 The meeting name has been set to 'tripleo' 14:00:33 #topic agenda 14:00:35 * one off agenda items 14:00:37 * bugs 14:00:39 * Projects releases or stable backports 14:00:41 * CI 14:00:43 * Specs 14:00:45 * open discussion 14:00:47 who is around today? 14:00:50 o/ 14:00:50 o/ 14:00:51 .o/ 14:00:53 o/ 14:00:53 hey 14:00:53 o/ 14:00:54 Hi! 14:00:55 \o 14:01:02 o/ 14:01:07 hiya \o 14:01:21 o/ 14:01:28 o/ 14:01:38 o/ 14:01:42 what a nice gang here 14:01:43 o/ 14:01:59 oh you said that before I o/ 14:02:22 o/ 14:02:43 we had some actions 14:02:52 team to review tripleo spec reviews policy https://review.openstack.org/#/c/406264/ 14:02:55 jpich to breakdown https://review.openstack.org/#/c/406264/ 14:02:55 o/ 14:02:57 beagles to abandon old reviews and propose a policy to document it 14:02:59 EmilienM to test tripleo with centos7.3 and see how it goes 14:03:01 EmilienM to send update on ML about releases ocata-2 and newton 14:03:24 o/ 14:03:33 o/ 14:03:33 o/ 14:03:45 I need to postpone my actions 14:03:48 I did a cleanup but got sidetracked off of policy edit.. .will get to that this week it seems 14:03:48 o/ 14:03:56 Policy broken down, the straight up migration from the wiki has merged, the new tidbit is at https://review.openstack.org/#/c/407533/ 14:04:02 #action EmilienM to test tripleo with centos7.3 and see how it goes 14:04:10 #action EmilienM to send update on ML about releases ocata-2 and newton 14:04:21 o/ 14:04:30 beagles: thanks, are you doing all tripleo repos? 14:05:06 jpich: approved 14:05:17 EmilienM: almost all, I skipped diskimage-builder - some unfamiliar names on there 14:05:24 EmilienM: aren't we doing a yum update as part of the image build? 14:05:30 EmilienM: Cool, thanks 14:05:31 EmilienM: but will revisit all the same 14:05:51 all my local boxes (including undercloud) are yum updated and therefore running centos7.3 14:05:59 EmilienM: I'll reach out to those that seem still active if they are still around before abandoning them 14:06:24 shardy: right, that's why I wanted to see how it works, I've heard some issues with nova and qemu 14:06:25 EmilienM: wrt 7.3 testing... we are doing it now because we run yum update :) 14:06:26 #action beagles propose policy edit for periodically abandoning old reviews 14:06:31 jpena sent an email about it 14:06:42 trown: ok nice 14:06:48 EmilienM: Ok, I guess I assumed we did a yum update in CI too 14:06:51 EmilienM: so everywhere that is using centos and runs yum update is using 7.3 14:06:54 beagles: you can skip dib, it's fine 14:07:00 EmilienM: okay 14:07:18 shardy, we do 14:07:22 EmilienM: overcloud images might be the only place that is not tested yet in tripleo-ci, because I don't thing we run update there 14:07:28 and we don't have any trouble with the qemu/nova thing? 14:07:35 Ok, so we're already testing 7.3 then, which is what I thought 14:07:50 trown, https://github.com/openstack-infra/tripleo-ci/blob/3bd847ccd63d209c13091dd4f5d8f3de3102aadf/scripts/common_functions.sh#L106-L106 14:08:21 ok let's start the agenda 14:08:28 sshnaidm: nice 14:08:30 #topic off agenda items 14:08:34 #link https://etherpad.openstack.org/p/tripleo-meeting-items 14:08:41 ccamacho: shoot 14:09:03 EmilienM: I've had several problems with nova, but none related to centos 7.3 AFAIK, maybe there are issues I've not hit yet tho 14:09:03 we are going to send some TripleO cheatsheets to FOSDEM and devconf 14:09:22 feedback/reviews are welcomed there https://github.com/ccamacho/tripleo-cheatsheet 14:09:28 nice 14:09:49 that cheatsheet made it to reddit :) 14:09:53 shardy: are they reported somewhere? 14:10:07 ccamacho, thats awesome 14:10:17 ccamacho: nice work here :) 14:10:25 but they are WIP 14:10:29 ccamacho: do you need us to take some actions? review them, etc? 14:10:41 mostly give feedback for adding content there 14:10:44 EmilienM: yes https://bugs.launchpad.net/tripleo/+bug/1649341 14:10:44 Launchpad bug 1649341 in puppet-nova "Undercloud upgrade fails with "Cell mappings are not created, but required for Ocata"" [Critical,In progress] - Assigned to Alex Schultz (alex-schultz) 14:10:44 ccamacho: looks great weshay noway :) 14:11:06 o/ 14:11:08 becasue we should send them for printing at the end of the year 14:11:20 EmilienM: I then updated everything to delorean-current and nova then has RPC problems, so trunk may be broken, but I don't think it's because of centos 14:11:21 shardy: oh this one, mwhahaha is on it iirc 14:11:37 EmilienM: Yeah we worked through it yesterday 14:11:40 ccamacho: do you want to use ML to share infos here and get wide feedback? 14:11:52 awesome, Ill send then an ML 14:12:03 shardy: thanks. I'll try to help Alex also, we might have found some chicken&egg problem here 14:12:08 ccamacho: thanks 14:12:10 openstack-dev [TripleO] as usual 14:12:11 ccamacho: will these be associated with some talk? 14:12:13 bandini: go ahead 14:12:16 thanks 14:12:44 shardy I dont think so, they are general cheatsheets for people jumping into TripleO 14:12:52 so yeah I'd like some high-level feedback on https://review.openstack.org/#/c/408726/ which fixes https://bugs.launchpad.net/tripleo/+bug/1615983 14:12:53 Launchpad bug 1615983 in tripleo "if $::hostname == boostrap_node checks are fragile" [High,In progress] - Assigned to Michele Baldessari (michele) 14:13:06 I am not entirely happy with the approach I have taken as it is a lot of code churn 14:13:11 so am looking for other ideas 14:13:21 maybe shardy or dprince have some thoughts there? 14:13:27 bandini: is this for ocata? 14:13:30 so the idea is to build an array of nodes for each service 14:13:34 and take the first one for the bootstrap? 14:13:37 bandini: i mean rather, it isn't meant to be for newton? (me checks bug) 14:13:48 marios: ocata man... we don't backport features ;-) 14:14:01 ccamacho: Ok, I'll provide feedback on the ML - I wonder if targetting quickstart might be easier as the deploying cheat-sheet is pretty complex for an unassisted new user 14:14:02 bandini: fwiw will have a look added myself to reviews, sorry completely missed that one 14:14:07 marios: this is for Composable HA 14:14:09 well this is a bug, but it is extremely unlikely to hit newton 14:14:13 EmilienM: thanks 14:14:24 we hit it 100% with composable ha though as soon as we put a pacemaker resource in two different roles 14:14:24 bandini: well, not sure we can backpor that one 14:14:34 it changes the interface quite a bit 14:14:34 shardy awesome we can include also a OOOQ cheatsheet 14:14:36 EmilienM: yeah I don't think we need to backport anything to newton 14:14:40 (and, we're trying to move away from instack-virt-setup) 14:14:57 EmilienM: plus I'd first like to get consensus on the approach, maybe there are cleaner alternatives I have not thought of 14:15:38 anyway, if anyone can throw some cluebones my way on that review, that would be awesome ;) 14:15:51 bandini: I had a quick look and it looks good to me 14:16:03 bandini: but i'll spend more time. This thing looks critical 14:16:31 #action HA & puppet experts to look https://review.openstack.org/#/c/408726/ 14:16:33 yeah the more composable we get the more we will need to fix that one way or the other 14:17:05 bandini: thanks! 14:17:12 anything else before we go to regular agenda? 14:17:18 bandini: I think you may be onto the right track 14:17:18 I am good, thanks 14:17:33 yes I agree. Though we haven't test it yet 14:17:33 mwhahaha quick question, testing upstream composable upgrades, it hangs ... :( and I only found an connection error on heat as we are not setting up the transport_url ( https://github.com/openstack/puppet-heat/commit/b865dffaa68602f520d3b6fe6bfbe8f6f8a30cf1#diff-0e9e2411cdc9fbc9f7df5a11085ba2a9 ) do you think this can break the deployment? Currently testing https://review.openstack.org/#/c/410171/ but hitting an 14:17:44 bandini: one suggestion would be could we keep the mechanism in t-h-t such that we can use a similar approach for containers 14:17:50 and I would like to see a CI job that test it, since slagle did the 3-nodes CI job working 14:18:24 dprince: oh I see, I will look into that 14:18:30 bandini: with containers we aren't using Puppet for the db_sync, rather we would execute the db_sync by using docker-cmd to run a quick db_sync using the container itself 14:18:34 ccamacho: a bit out of topic now, can we stick on agenda? 14:18:45 EmilienM sure 14:18:53 oh I see 14:19:25 dprince, bandini: yeah but if we move this back into tht, we should avoid reintroducing a hard-coded 0 reference again 14:19:59 perhaps we can look into using yaql first() or something, as the lookup on the ResourceGroup ends up being by name, not by index 14:20:17 shardy: ack, noted 14:20:18 we fixed that by moving the lookup into puppet-tripleo, where it is by index 14:20:34 and export some hiera for the first() occurence? 14:20:37 shardy: would be nice, but I'd give the generic mechanism (works across containers and baremetal) priority. But ++ on trying first() w/ yaql 14:20:50 EmilienM: Yeah something like that 14:21:11 EmilienM: yes, I'd say keep the puppet modules dumber. Just assume they are told whether to sync or not to sync by heat 14:21:14 dprince: ack, I'm totally fine with moving it back to tht, but I'd rather avoid the hard-coded "0" issue we had before 14:21:18 ccamacho: I'll take a look today and get back to you 14:21:35 dprince: ++ 14:21:42 mwhahaha thanks man 14:22:11 are we done with $topic? 14:22:45 #topic bugs 14:22:50 #link https://launchpad.net/tripleo/+milestone/ocata-2 14:23:17 we have a huge number of bugs and I would like to drop a note about that 14:23:26 16 Confirmed, 50 Triaged, 41 In Progress 14:24:04 by the end of ocata-3, we'll freeze features in TripleO and will move efforts into fixing bugs and making things more stable until we release the final ocata 14:24:43 the only exception will be composable upgrades I guess 14:24:50 any feedback on that ^ is welcome 14:25:07 EmilienM: end of january for O3 right? yes i fully expect we'll be backporting to stable/ocata for at least the last bits 14:25:19 marios: yes 14:25:39 marios: don't think about branching now, we'll wait a bit more than last time 14:25:44 so we reduce the # of backports 14:25:53 EmilienM: shardy matbu chem ccamacho and anyone else that is interested we were thinking of having an upgrades squad syncup this week to talk about how to proceed with that 14:26:06 marios: that's excellent 14:26:09 marios++ 14:26:28 marios: feel free to send an email when it's done, with a summary of your progress 14:26:30 marios: +1 14:26:36 EmilienM: ack 14:26:40 marios: cool 14:26:42 something on openstack-dev [tripleo] [upgrade] blabla 14:26:45 marios: thanks man 14:26:47 marios: irc mtg ? 14:26:58 Hello, can anybody help me? I'm trying to publish installation guide for our service to docs.openstack.org 14:27:07 matbu: possibly... whatever works. personally i think a call may be better. 14:27:22 tikitavi: we're in the middle of a meeting, please ask on #openstack-dev or #openstack-doc 14:27:27 marios: ack, up to you 14:27:36 do we have outstanding bugs this week? 14:27:47 one I see is https://bugs.launchpad.net/bugs/1649341 14:27:50 Launchpad bug 1649341 in puppet-nova "Undercloud upgrade fails with "Cell mappings are not created, but required for Ocata"" [Critical,In progress] - Assigned to Alex Schultz (alex-schultz) 14:27:54 but shardy / mwhahaha are already working on it 14:28:02 is there anything we have missed? 14:29:06 #topic Projects releases or stable backports 14:29:18 so I missed it last week but I need to send an email 14:29:35 this week, we'll release ocata-2 14:29:46 i'll take care of moving bugs/blueprints to ocata-3 14:30:14 also, if a blueprint has zero or slow progress, I'll contact the assignee and propose a postpone to pike-1 14:30:40 I plan to release a new Newton version by next week also, when ocata-2 will be done 14:31:05 #link https://releases.openstack.org/ocata/schedule.html 14:31:17 bookmark this link if you need any info about release ^ 14:31:49 but after January 27th, we'll stop implementing features in TripleO 14:32:12 and work more on bug fixing and stabilization 14:32:29 I think that's the plan. Any feedback or questioN? 14:33:20 seems reasonable 14:33:27 +1 sounds good to me 14:33:30 it means, after ocata-3, we have 5 weeks to stabilize things 14:33:41 (if we use the trailing cycle) 14:34:10 marios: for composable upgrades, we'll still accept patches, as it's the most critical thing in this cycle. 14:34:18 +1 14:34:24 EmilienM: ack thanks for the clarification 14:34:41 #action EmilienM to start postponing bugs / blueprints to ocata-3 or pike-1 14:34:58 #topic CI 14:35:50 I would like CI folks to review https://review.openstack.org/#/q/topic:3-node please 14:36:21 slagle: would it be possible to have a 3-node CI job that test composable HA? or do we need more nodes? 14:37:16 also, I have an update about ovb jobs and recent problems 14:37:16 nice 14:37:41 derekh, sshnaidm and panda worked on rh1 issues and it seems things are working again 14:37:50 we need to merge this patch once it passes CI https://review.openstack.org/#/c/410131/ 14:38:08 #action CI folks to review https://review.openstack.org/#/q/topic:3-node and https://review.openstack.org/#/c/410131/ 14:38:36 EmilienM, I see ha job passed, so should be ok 14:38:39 ping test on ha job for the ovb fix just passed 14:38:44 bandini: could we run a 2-nodes overcloud and test composable HA? 14:38:51 or do we need more nodes? 14:39:31 EmilienM: I think we can, I need to have a think about it 14:39:53 bandini: ok thx, please let me know post meeting, it's really important that we have coverage for that 14:40:02 EmilienM: agreed 14:40:09 thx 14:40:20 any other info about CI? 14:41:07 #topic Specs 14:41:36 #link https://review.openstack.org/#/q/project:openstack/tripleo-specs+status:open 14:42:39 I would like to remind that specs not approved by ocata-2 (this week) won't be implemented in ocata and postponed to pike-1 14:43:14 the GUI and validations specs look in good shape, I'll review them this week 14:43:26 thanks :-) 14:44:23 #link https://bugs.launchpad.net/tripleo/+milestone/pike-1 14:44:38 yeah, we already creating the milestones :D 14:44:44 Note folks can target things to pike now, so we track them on the roadmap even if they aren't expected to land for Ocata 14:45:10 do we need to discuss about a spec in specific? 14:45:53 let's open the discussion 14:45:57 #topic open discussion 14:46:01 There has been some ML discussion about fully composable networks 14:46:03 feel free to bring any topic or question 14:46:04 http://lists.openstack.org/pipermail/openstack-dev/2016-December/108779.html 14:46:28 Interested to get feedback on the ideas in that thread - I think we might consider it as a feature for pike 14:46:34 matbu: what is the progress on upgrade job? 14:46:51 shardy: composable everything! :-) 14:47:05 EmilienM: hehe :) 14:47:26 EmilienM: I think the upgrades job patch is getting pretty close but there are still some issues to be worked out 14:47:44 I'm trying to get it working locally, so we can iterate faster with the testing 14:47:50 shardy: nice 14:48:15 will share notes when I have it functional 14:48:21 shardy: it would be nice to start a specs for the composable networks, maybe trozet can start it 14:48:26 EmilienM: i think we are realyl close, i'll make an update of the current review (i'm testing the change atm) 14:48:37 EmilienM: agreed - it's been discussed a few times but I don't think there is a spec 14:48:38 he's not online here but I'll check with him if he's interested 14:49:00 EmilienM: the job was green yesterday twice 14:49:09 EmilienM: as mentioned in the ML thread, we may end up partially implementing it as a side-effect of those two bugs related to composable roles and network isolation 14:50:20 matbu: how can we make a pingtest? 14:50:39 EmilienM: we can run pingtest when we've added the services pingtest requires 14:50:40 shardy: can we create a basic pingtest that create keystone resources? 14:50:42 EmilienM: hm actually we need a specific pingtest 14:50:56 EmilienM: but for now I was planning to add a basic sanity smoke test which uses e.g openstackclient 14:51:09 shardy: sounds good enough 14:51:31 shardy: as we only deploy keystone for now 14:51:41 matbu, shardy: how many time takes the job? 14:51:48 are we hitting timeout? 14:51:51 EmilienM: it took 59 14:51:54 minutes 14:51:58 yup, then we'll add e.g glance and create/delete an image, eventually moving over to the pingtest stack instead 14:52:00 in total? 14:52:05 yep 14:52:10 wow, that's good 14:52:13 this is really fast 14:52:16 yeah, next step is glance :D 14:52:23 * EmilienM dancing 14:52:28 well i wrong i think it's 49 minutes 14:52:40 I think that's not actually testing the full upgrade yet, as it's deploying master tht, but yeah it looks pretty good 14:53:11 and I think we can combine the upgrade with converge steps (just for CI) when we land https://review.openstack.org/#/c/409857/ 14:53:17 shardy: oh, you mean we don't upgrade everything else? 14:53:27 shardy: we don't run yum update I mean? 14:53:30 shardy: it does the converge now 14:54:17 EmilienM: No it does run the yum update, it's just not using the right tht yet 14:55:14 ok 14:55:26 folks, 5 min left, do we have anything else to discuss this week? 14:56:28 if I don't freeze, I'll see you next week! thanks for your time and have fun 14:56:33 #endmeeting