14:00:03 <EmilienM> #startmeeting tripleo 14:00:04 <openstack> Meeting started Tue Sep 13 14:00:03 2016 UTC and is due to finish in 60 minutes. The chair is EmilienM. Information about MeetBot at http://wiki.debian.org/MeetBot. 14:00:05 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 14:00:07 <openstack> The meeting name has been set to 'tripleo' 14:00:07 <d0ugal> Hey! 14:00:11 <EmilienM> #topic rollcall 14:00:15 <slagle> hi 14:00:18 <ccamacho> /o/ hey guys! 14:00:18 <shardy_mtg> o/ 14:00:19 <EmilienM> yoyoyo, who is here this week? 14:00:22 <d0ugal> o/ 14:00:28 <adarazs> o/ 14:00:31 <skramaja> hi 14:00:37 <coolsvap> o/ 14:00:44 <jrist> o/ 14:00:48 <mwhahaha> hi 14:00:50 <trown> o/ 14:00:51 <sshnaidm> o/ 14:00:54 <derekh> o/ 14:01:07 <marios> \o 14:01:14 <EmilienM> #topic agenda 14:01:20 <EmilienM> #link https://wiki.openstack.org/wiki/Meetings/TripleO 14:01:29 <EmilienM> #link https://etherpad.openstack.org/p/tripleo-meeting-items 14:01:34 <shadower> hey 14:01:44 <EmilienM> any off item should be added now on the etherpad, please :-) 14:02:21 <EmilienM> * one off agenda items 14:02:21 <sshnaidm> EmilienM, added the rdo outdated packages issue 14:02:24 <EmilienM> * bugs 14:02:26 <EmilienM> * Projects releases or stable backports 14:02:28 <EmilienM> * CI 14:02:30 <EmilienM> * Specs 14:02:32 <EmilienM> * open discussion 14:02:34 <EmilienM> sshnaidm: ack 14:02:36 <EmilienM> sshnaidm: should it be addressed during CI topic 14:02:41 <EmilienM> ? 14:03:04 <sshnaidm> EmilienM, yeah 14:03:09 <weshay> o/ 14:03:10 <EmilienM> ok 14:03:13 <matbu> hello 14:03:15 <EmilienM> so let's get started with bugs 14:03:17 <EmilienM> #topic bugs 14:03:40 <EmilienM> we are currently dealing with a Galera issue on HA jobs 14:03:43 <EmilienM> https://bugs.launchpad.net/tripleo/+bug/1622755 14:03:44 <openstack> Launchpad bug 1622755 in tripleo "tripleo-heat-templates: support recent version of the galera WSREP provider" [High,In progress] - Assigned to Emilien Macchi (emilienm) 14:04:36 <EmilienM> well, this is a CI bug :-) because we still have EPEL until we have a new DIB, I guess we can talk about it during CI topic 14:04:40 <marios> EmilienM: one upgrade blocker https://bugs.launchpad.net/tripleo/+bug/1620696 matbu is looking into that but fyi... seems related to service_enabled from roles+composable services 14:04:41 <openstack> Launchpad bug 1620696 in tripleo "M/N upgrades UPDATE_FAILED .enabled_services.list_join: Incorrect arguments to "list_join" should be: "list_join" : [ " ", [ "str1", "str2"]]" [Undecided,Confirmed] - Assigned to mbu (mat-bultel) 14:04:49 <EmilienM> #link https://launchpad.net/tripleo/+milestone/newton-rc1 14:05:13 <matbu> marios: EmilienM +1 14:05:22 <matbu> i have a fix for heat coming soon 14:05:38 <EmilienM> ok, please make sure the launchpad is updated 14:05:44 <matbu> ya 14:05:52 <EmilienM> I put the severity on critical since it blocks upgrades 14:06:00 <EmilienM> feel free to decrease it if needed 14:06:06 <matbu> ack 14:06:33 <EmilienM> matbu: we are really late in the cycle to have a patch in Heat merged and released tbh 14:06:52 <EmilienM> matbu: please sync with heat folks about the bug and fix 14:06:53 <marios> EmilienM: but is for upgrades so we have exception for backport to stable/newton 14:06:55 <matbu> EmilienM: hmm yep i imagine 14:06:56 <shardy_mtg> matbu: can you link the heat patch please? 14:07:08 <shardy_mtg> I can help with reviews and chasing other heat folks 14:07:10 <matbu> shardy_mtg: yep i'll 14:07:10 <marios> EmilienM: i mean if it is tht side for wexample or somewhere else we have control over 14:07:25 <EmilienM> marios: it sounds like in Heat, not in THT 14:07:36 <marios> EmilienM: right, it wasn't clear matbu was looking at both 14:07:47 <EmilienM> do we have any other outstanding bug? 14:07:48 <matbu> shardy_mtg: if you have few minutes today or tomorow we can discuss about that 14:08:05 <EmilienM> the list of WIP bugs in https://launchpad.net/tripleo/+milestone/newton-rc1 is still huge 14:08:37 <EmilienM> i'll repeat what I already said many times (don't hate me) but please update the bugs you're assigned to if needed 14:09:11 <rhallisey-brb> hi. Sorry I'm a little late 14:09:32 <EmilienM> I'll look at the list today and retarget bugs to rc2 if fix is not about to land this week 14:09:39 <EmilienM> shardy_mtg: does it sound good? ^ 14:10:07 <shardy_mtg> EmilienM: yes, sounds good, thanks 14:10:10 <EmilienM> k 14:10:27 <EmilienM> #action EmilienM to review rc1 bugs and start rc2 target 14:10:32 <shardy_mtg> I'll be available Thursday to help with that (assuming we're planning to release RC1 on Thursday) 14:10:44 <EmilienM> i'll have to use an etherpad fo rthat, I don't think rc2 exist in launchpad since we didn't release rc1 yet 14:10:51 <pabelanger> EmilienM: why is a new DIB release needed? Trying to search in review.o.o now 14:10:54 <EmilienM> i'll check with Doug 14:10:58 <shardy_mtg> EmilienM: we can just create it 14:10:59 <shardy_mtg> sec 14:11:05 <EmilienM> pabelanger: please hold on 14:11:12 <EmilienM> pabelanger: we'll talk about it during CI topic 14:11:35 <EmilienM> anything else about bugs today? 14:11:53 <pabelanger> EmilienM: sure 14:12:00 <shardy_mtg> https://launchpad.net/tripleo/+milestone/newton-rc2 14:12:01 <shardy_mtg> done 14:12:11 <EmilienM> shardy_mtg++ I'll work on it after the mtg 14:12:18 <EmilienM> ok let's move on then 14:12:28 <EmilienM> #topic Projects releases or stable backports 14:12:36 <EmilienM> so I have 2 informations for the team 14:13:23 <EmilienM> 1) TripleO RC1 is about to be released this week, so please review critical patches, and keep focus on rc1 work. Release is super close from now, we need everyone's focus on rc1 14:14:00 <EmilienM> 2) tripleoclient was released and branched 2 weeks ago, which means we had to backport patches from master to stable/newton 14:14:12 <EmilienM> when RC1 is about to be created in tripleo, branches will be created 14:14:20 <EmilienM> and we'll have to backport everything we need until final newton is out 14:14:39 <EmilienM> in other words: when RC1 is out, your stuff merged in master after might needs to be backported if bugfix 14:14:59 <EmilienM> we count on you to carry your backports 14:15:19 <shardy_mtg> can everyone tag newton-backport-potential on bugs so we can try to keep track 14:15:27 <slagle> shardy_mtg: +1 14:15:36 <shardy_mtg> (and propose the backport yourself) 14:15:38 <slagle> was just about to mention, i'd like us to require launchpad bugs for backports 14:15:41 <EmilienM> shardy_mtg: also i'll try to update blueprints status on https://launchpad.net/tripleo/+milestone/newton-rc1 14:16:10 <EmilienM> slagle: yes, I like the idea. I would even make it official in our specs repo and / or in governance with the follow-stable tag 14:16:30 <shardy> I think we've enforced that for all backports to mitaka, so +1 14:16:43 <slagle> EmilienM: right, we don't yet have the tag, but we do want it. so it would be good to follow that policy 14:16:48 <shardy> we should use the normal stable backport rules, except for any remaining FFE patches which don't make RC1 14:16:53 <EmilienM> slagle: big +1 14:17:15 <shardy> I applied for the tag, but it was denied because we allowed features for liberty 14:17:21 <EmilienM> #action EmilienM to investigate usage of tag-stable:follows-policy in TripleO 14:17:25 <shardy> we can re-apply when that branch goes EOL 14:17:32 <EmilienM> shardy: oh great 14:17:37 <EmilienM> shardy: do you have the patch handy? 14:17:42 <shardy> EmilienM: yup, sec 14:18:10 <EmilienM> do we have anything else about release management? 14:18:22 <EmilienM> any question / remark about rc1? 14:18:23 <shardy> EmilienM: https://review.openstack.org/#/c/314485/ 14:18:30 <marios> EmilienM: shardy upgrades exception for backports to stable still right? 14:18:31 <EmilienM> shardy: thx, /me bookmarks 14:18:54 <EmilienM> marios: if it has a launchpad, I think so 14:18:59 <shardy> marios: Yeah I think we have to consider upgrade issues all bugs 14:19:03 <EmilienM> to me, an upgrade issue is a bug 14:19:06 <marios> EmilienM: ok just calling it out since we are discussing thanks 14:19:13 <EmilienM> marios++ 14:19:14 <shardy> but we can't change any interfaces e.g mistral/composable services 14:19:17 <EmilienM> thanks for making it clear :) 14:19:54 <EmilienM> shardy: I'll proceed to rc2 on Thursday 14:20:10 <EmilienM> I'll start tomorrow afternoon to make sure we don't miss anything critical 14:20:29 <EmilienM> and I'll proceed to actual release on Thu and send communication about the change and what to consider (backports, etc) 14:20:41 <shardy> EmilienM: +1, thanks 14:20:42 <EmilienM> #action EmilienM to proceed RC1 release on Thursday 14:21:02 <EmilienM> on a side note: we won't release Puppet modules RC1, and wait end of September to branch them 14:21:21 <EmilienM> next topic? 14:21:30 <EmilienM> #topic CI 14:21:33 <EmilienM> pabelanger: shoot 14:21:52 <EmilienM> I'll give my updates: 14:22:04 <pabelanger> was just curious about the DIB release that tripleo needs? 14:22:15 <EmilienM> HA job is still broken, even if we thought it was fixed in tripleo-ci by removing EPEL 14:22:16 <pabelanger> looks to be around disabling EPEL? 14:22:26 <EmilienM> the problem is that the version of DIB in our images is still deploying EPEL 14:22:28 <EmilienM> so we hit https://bugs.launchpad.net/tripleo/+bug/1622755 14:22:29 <openstack> Launchpad bug 1622755 in tripleo "tripleo-heat-templates: support recent version of the galera WSREP provider" [High,In progress] - Assigned to Emilien Macchi (emilienm) 14:22:43 <EmilienM> I proposed this mysql change in puppet-tripleo: https://review.openstack.org/#/c/369369/ 14:22:54 <EmilienM> I need to wait for CI and also see if anyone with ipv6 deployment can test it 14:22:56 <bnemec> Something seems wrong with our repo setup if we aren't getting latest dib. 14:23:11 <EmilienM> pabelanger: yes exactly 14:23:23 <EmilienM> pabelanger: we're trying to remove EPEL completely from tripleo 14:23:25 <trown> bnemec: it is because DIB is tracked in U-C 14:23:29 <pabelanger> Have we confirmed this isn't going to break openstack-infra? We we do depend on EPEL 14:23:43 <derekh> bnemec: RDO no longer build the latest DIB, its now only build on release 14:23:57 <derekh> bnemec: along with other packages including tripleoclient 14:24:00 <trown> in general RDO no longer builds master for things in U-C 14:24:13 <trown> because most projects in there do not keep master working 14:24:16 <EmilienM> trown is true, we had so much problems with clients 14:24:28 <pabelanger> I'm not opposed with the removal of it, it would be nice if somebody could confirm that openstack-infra won't break after the new version is released to pip 14:24:31 <EmilienM> last example with openstackclient 14:24:39 <EmilienM> pabelanger: we haven't I guess 14:24:45 <bnemec> So our includepkgs list is basically useless because we aren't getting current master packages for stuff anyway? 14:24:48 <EmilienM> pabelanger: any wayt to test it in CI? 14:25:04 <sshnaidm> bnemec, yes, we have now very outdated packages instead of latest ones 14:25:08 <derekh> bnemec: correct (from some packages at least) 14:25:10 <pabelanger> EmilienM: yes, because if images are broken in nodepool, you'll like see that patch reverted right away 14:25:11 <EmilienM> bnemec: we're getting packages from "current", correct 14:25:19 <derekh> ha passes here btw : https://review.openstack.org/#/c/369329/1 14:25:29 <bnemec> EmilienM: But they're not actually current. 14:25:34 <pabelanger> EmilienM: project-config, you can use the tools/build-image script to create a centos-minimal image 14:25:37 <EmilienM> bnemec: yes :( 14:25:54 <pabelanger> If I drop, it is because I am on a train 14:26:01 <bnemec> pabelanger: We should make sure infra is explicitly including the epel element in their images. That's how this should have worked in the first place. 14:26:03 <sshnaidm> derekh, EmilienM this one also passes: https://review.openstack.org/#/c/369325/ 14:26:09 <EmilienM> derekh: do we really need https://review.openstack.org/#/c/369303/ ? 14:26:09 <jrist> pabelanger: do you realize how funny that sounds? :) 14:26:12 <bnemec> pabelanger: Can you point us in the right direction for that? 14:26:41 <EmilienM> oh dear https://review.openstack.org/#/c/369325/ is much better 14:26:55 <EmilienM> are we good to land it? 14:27:09 <EmilienM> as we keep cached images 14:27:12 <pabelanger> bnemec: I think the first step would be to build an image with tools/build_images.sh, and see if it fails. 14:27:17 <shardy> EmilienM: Hey, sorry one last RC1 question - if puppet modules won't branch this week, should we align the TripleO branch with when they do branch? 14:27:22 <trown> bnemec: derekh, actually there is a dlrn instance building master everything... we should use that for our includepkgs repo... just getting link now 14:27:26 <shardy> particularly if it makes e.g backport overhead less? 14:27:29 <EmilienM> bnemec, you ok with https://review.openstack.org/#/c/369325/ ? 14:27:39 <shardy> I was originally thinking both would branch RC1 this week 14:27:58 <pabelanger> it not, great. If it does, we need to figure out how to move forward. It could be adding epel to our DIB elements in project-config, or updating system-config to do it. 14:28:11 <EmilienM> shardy: that's a good question, indeed 14:28:12 <pabelanger> However, we do need epel on our images, because of devstack things 14:28:15 <bnemec> EmilienM: I think so. 14:28:28 <EmilienM> sshnaidm: approved 14:28:43 <bnemec> pabelanger: That's fine, we just needed to stop forcing epel. It's no problem to include it in images by choice. 14:28:54 <trown> derekh: bnemec: https://trunk.rdoproject.org/centos7-master-head/current/ I will put up some patches after the meeting 14:29:00 <shardy> EmilienM: if Doug is OK with it, I'd be fine with extending the RC1 window to match puppet but I'll leave the final decision to you 14:29:02 <pabelanger> Also, I want to bring tripleo-test-cloud-rh2 back online for the purpose to test CI refactoring. 14:29:18 <EmilienM> so we need to backport the tripleoclient patch that drops EPEL 14:29:25 <derekh> pabelanger: bnemec epel isn't explicitly added in infra 14:29:27 <EmilienM> shardy: Ok I'll check with him later today 14:29:29 <derekh> 2016-09-13 14:08:54,954 INFO nodepool.image.build.centos-7: + /usr/local/share/diskimage-builder/lib/common-functions:arg_to_elements:197 : echo 'Building elements: base centos-minimal vm simple-init openstack-repos nodepool-base cache-devstack puppet cache-bindep growroot infra-package-needs stackviz' 14:29:35 <slagle> pabelanger: bnemec : i say we just propose a patch to infra to excplicitly include epel 14:29:37 <derekh> pabelanger: bnemec from http://nodepool.openstack.org/dib.centos-7.log 14:29:38 <pabelanger> bnemec: Right, I know there is some history there. As along as all parties are fine, I think we can more forward. However, we just need to confirm -infra doesn't break 14:29:41 <EmilienM> #action EmilienM to check if we can slip RC1 and do it in same time as Puppet OpenStack 14:29:56 <slagle> i can do that, it's rather straightforward 14:29:56 <derekh> EmilienM: 369325 sounds good to me 14:30:00 <bnemec> pabelanger: Sure 14:30:03 <bnemec> slagle: +1 14:30:10 <EmilienM> sshnaidm, derekh: kudos guys for your help this morning, thanks a lot 14:30:20 <derekh> EmilienM: np 14:30:25 <EmilienM> ha job is back on stage! 14:30:32 <pabelanger> slagle: yes, that is fine. Just want to make sure that is done before we cut the DIB release. 14:30:38 <EmilienM> next step is to bring liberty and mitaka jobs back too 14:30:54 <slagle> pabelanger: yep, sounds good 14:31:18 <bnemec> pabelanger: I like the idea of testing the nodepool change in rh2. 14:31:22 <EmilienM> #action backport https://review.openstack.org/#/c/369276/ to mitaka and liberty 14:32:11 <EmilienM> matbu: any update about upgrade jobs? 14:32:19 <EmilienM> panatl: any update about ipv6 job? 14:32:24 <EmilienM> err, panda 14:32:31 <pabelanger> bnemec: Yes, I just need somebody with admin credentials in tripleo-test-cloud-rh2 to create some new projects for me. 14:32:38 <matbu> EmilienM: UC major upgrade and update has been merged 14:32:38 <pabelanger> we can talk more about that in #tripleo 14:32:51 <EmilienM> matbu: yup, /me interested by OC ;-) 14:32:52 <bnemec> pabelanger: Sounds good 14:33:02 <matbu> i hit some timeout on the OC upgrade ( during the undercloud install :/) 14:33:12 <matbu> probably a missconfig with the repo 14:33:37 <matbu> i hope to figure out this asap 14:33:43 <panda> EmilienM: at the moment ha ipv6 is blocked by an error in puppet-tripleo/swift add_devices. THere is a regexp that deals only with IPv6 and I'm still trying to find a solution 14:33:46 <EmilienM> matbu: ok, keep us updated on ML the situation if you want 14:33:51 <EmilienM> so we can define blockers and work on it 14:33:55 <matbu> EmilienM: ack 14:34:01 <panda> s/IPv6/IPv4 14:34:04 <EmilienM> panda: did you file a bug in puppet-swift? 14:34:20 <EmilienM> sounds like a critical bug to me, as we know it worked before. 14:34:35 <panda> EmilienM: no, filing now 14:34:50 <EmilienM> panda: thanks, let us know bug URl so we can make triage and assign it probably 14:35:14 <EmilienM> pabelanger: you good with CI topic? 14:35:23 <EmilienM> is there anything else about CI this week? 14:35:43 <pabelanger> EmilienM: yes, thanks 14:36:01 <EmilienM> #topic one off agenda items 14:36:12 <EmilienM> #link https://etherpad.openstack.org/p/tripleo-ui-newton-reviews 14:36:16 <EmilienM> jrist: I guess it was you? ^ 14:36:35 <jrist> yeah 14:36:46 <jrist> just want to bring attention to some reviews is all. nothing special 14:36:58 <jrist> the big one is capabilities map 14:37:05 <EmilienM> jrist: are they all urgent and critical? 14:37:15 <jrist> they are our high priority ones yes 14:37:24 <EmilienM> #action team to review tripleo-ui patches from https://etherpad.openstack.org/p/tripleo-ui-newton-reviews 14:37:29 <EmilienM> jrist: ack 14:37:31 <jrist> thx 14:37:33 <EmilienM> sure 14:37:41 <EmilienM> anything else about tripleo-ui? 14:38:10 <EmilienM> I guess not 14:38:13 <EmilienM> #topic Specs 14:38:45 <EmilienM> #link https://review.openstack.org/#/q/project:openstack/tripleo-specs+status:open 14:38:51 <jrist> EmilienM: nothing pressing at the moment that I know of :) it is coming along nicely 14:39:17 <EmilienM> team, please vote for this policy change: https://review.openstack.org/#/c/363960/ 14:39:55 <EmilienM> also, we need to defer all specs to Ocata i guess 14:40:40 <EmilienM> i'll update the open specs and ask to defer them 14:40:59 <EmilienM> #action EmilienM to -1 all specs targeted but not implemented for newton and defer them to Ocata 14:41:12 <EmilienM> questions / feedback about specs? 14:41:44 <EmilienM> ok folks keep it up, last topic and we can all go back to the beach 14:41:51 <EmilienM> #topic open discussion 14:42:29 <EmilienM> if you have any question or something to say, it's time now :) 14:43:17 <thrash> EmilienM: I'm guessing mistral and zaqar as composable services are unlikely to make it? 14:43:29 <thrash> And an ffe would be required for that? 14:43:36 <shardy> thrash: were FFE's requested? 14:44:02 <EmilienM> thrash: is it critical to have zaqar/mistral on overcloud for newton ? (/me asking stupid question) 14:44:03 <shardy> Yeah, we should've had an FFE a couple of weeks back and got them tracked for rc1 14:44:05 <thrash> shardy: no. Wasn't sure since they were part of the composable services, but not already in the overcloud 14:44:31 <thrash> shardy: EmilienM I don't think it's critical for newton. Really just me asking for my own clarity. 14:44:36 <EmilienM> oki 14:44:38 <shardy> Ok, I'd personally be fine with landing them if they're ready, but if they're not then I guess they will slip 14:44:51 <trown> wanted to make a brief announcement re: quickstart 14:44:52 <EmilienM> I guess they won't make it if CI is not green before Thursday 14:46:08 <trown> if folks have recently tried it and found that "devmode" was broken, I apologize... it has been in a bit of a rough state as the instack.qcow2 produced in July has slowly rotted, however https://review.openstack.org/#/c/368810/ should get devmode working again by using the tripleo-ci produced overcloud image as an undercloud image 14:46:35 <trown> so if anyone has had recent failures using devmode and would like to give that a go it would be greatly appreciated. 14:47:26 <EmilienM> I would like to remind everyone that Summit is soon and etherpad for topics is here: https://etherpad.openstack.org/p/ocata-tripleo feel free to add more 14:47:35 <EmilienM> I'll try to arrange a bit the etherpad 14:48:10 <EmilienM> trown: maybe using ML for the record would be nice too :) 14:48:18 <EmilienM> in case someone missed that meeting 14:48:22 <marios> EmilienM: #shardy: i'd love to get the manila-netap stuff landed but theres a few review in play (puppet-tripleo/tht .. tbarron is testing maybe we get it green by thursday .... https://review.openstack.org/#/c/354019/ and related changes/dependencies 14:48:33 <trown> EmilienM: hmm, good point, I will send something to ML when that patch merges 14:48:34 <EmilienM> marios: i'll review it 14:48:46 <saneax> trown I will try, and give feedback 14:49:05 <EmilienM> do we have anything for this week folks? 14:49:06 <trown> thanks saneax 14:49:14 <EmilienM> anything else* 14:49:49 <EmilienM> thanks for your time, let's get back to work now :-P 14:49:55 <EmilienM> #endmeeting