14:00:22 <mwhahaha> #startmeeting tripleo 14:00:22 <openstack> Meeting started Tue Sep 26 14:00:22 2017 UTC and is due to finish in 60 minutes. The chair is mwhahaha. Information about MeetBot at http://wiki.debian.org/MeetBot. 14:00:22 <mwhahaha> #topic agenda 14:00:23 <mwhahaha> * review past action items 14:00:23 <mwhahaha> * one off agenda items 14:00:23 <mwhahaha> * bugs & blueprints 14:00:23 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 14:00:24 <mwhahaha> * Projects releases or stable backports 14:00:24 <mwhahaha> * CI 14:00:24 <mwhahaha> * Specs 14:00:24 <mwhahaha> * open discussion 14:00:25 <mwhahaha> Anyone can use the #link, #action and #info commands, not just the moderatorǃ 14:00:26 <openstack> The meeting name has been set to 'tripleo' 14:00:35 <mwhahaha> Hi everyone! who is around today? 14:00:38 <trown> o/ 14:00:38 <slagle> Hi 14:00:47 <cdearborn_> \o 14:00:47 <jrist> o/ 14:00:52 <jrist> two meetings at once! 14:00:59 <fultonj> o/ 14:01:01 <thrash> o/ 14:01:02 <ccamacho> hi! 14:01:09 <mwhahaha> multitask! 14:01:14 <mwhahaha> at least it's not 3 14:01:14 <owalsh> o/ 14:01:15 <abishop> o/ 14:01:15 <chem`> o/ 14:01:35 <jpich> o/ 14:01:40 <EmilienM> o/ 14:01:42 <jfrancoa> o/ 14:02:06 <oidgar> o/ 14:02:29 <mwhahaha> ok lets go 14:02:29 <atoth> o/ 14:02:33 <marios> o/ 14:02:34 <mwhahaha> #topic review past action items 14:02:39 <mwhahaha> shardy to look at how to reduce # of services deployed on ovb (continued) 14:02:59 <mwhahaha> he's not here, so i guess we'll postpone that one again 14:03:03 <matbu> o/ 14:03:10 <mwhahaha> #action shardy to look at how to reduce # of services deployed on ovb (continued) 14:03:22 <mwhahaha> That's it on the previous action items 14:03:29 <mwhahaha> #topic one off agenda items 14:03:35 <mwhahaha> #link https://etherpad.openstack.org/p/tripleo-meeting-items 14:03:52 <gchamoul> o/ 14:04:07 <mwhahaha> Newton EOL & upstream support phases 14:04:17 <mwhahaha> So newton is going EOL next month 14:04:30 <EmilienM> ouch 14:04:32 <mwhahaha> so FYI if you're trying to propose anything to newton, it's unlikely that we should be merging anything 14:04:54 <mwhahaha> Additionally the release team is pushing back on some of the items we are backporting because they are not following the support phases 14:04:56 <mwhahaha> #link https://docs.openstack.org/project-team-guide/stable-branches.html#support-phases 14:04:58 <marios> mwhahaha: we have an upgrades related stable/newton one sec fetching link 14:05:12 <EmilienM> in my experience we're kind of free to EOL when we want, in a reasonable timeframe 14:05:20 <EmilienM> e.g. a few weeks after 14:05:27 <mwhahaha> please be aware that when we backporting stuff we need to make sure they are appropriately critical 14:05:43 <marios> mwhahaha: #link https://review.openstack.org/#/c/474967/ reviews please so we can land this in time for Newton EOL ? 14:05:46 * EmilienM hides 14:05:48 <mwhahaha> we follow stable-policy starting in Ocata so we need to be using their policy 14:05:54 <chem> mwhahaha: we have this one that should gets in https://review.openstack.org/#/c/507188/ 14:06:04 <mwhahaha> marios: k i'll take a look 14:06:06 <chem> mwhahaha: currently testing that it's correct 14:06:26 <mwhahaha> chem i'll have to review that one 14:06:30 <EmilienM> we should get a gerrit query to review them all 14:06:48 <EmilienM> (abandon these who won't make it and highlight the ones we need) 14:06:54 <chem> mwhahaha: it fixes a "regression" 14:07:15 <mwhahaha> chem: yea but this late in the support cycle "regession" isn't necessarily what we should be fixing 14:08:10 <EmilienM> FWIW, my gertty says we have 14 patches in newton 14:08:26 <mwhahaha> ok let's take a look at them this week 14:08:36 <EmilienM> #action review newton backports in gerrit 14:08:52 <mwhahaha> sounds good 14:08:54 <EmilienM> mwhahaha: I'll help on that front 14:09:10 <mwhahaha> so any other open items folks would like to talk about? 14:09:45 <mwhahaha> k moving on 14:09:50 <mwhahaha> #topic bugs & blueprints 14:09:50 <mwhahaha> #link https://launchpad.net/tripleo/+milestone/queens-1 14:09:50 <mwhahaha> We currently have 60 blueprints and about 470 open bugs. Please take some time to review your blueprint status and make sure it is properly up to date. 14:10:14 <ooolpbot> URGENT TRIPLEO TASKS NEED ATTENTION 14:10:14 <ooolpbot> https://bugs.launchpad.net/tripleo/+bug/1718387 14:10:16 <ooolpbot> https://bugs.launchpad.net/tripleo/+bug/1719373 14:10:17 <ooolpbot> https://bugs.launchpad.net/tripleo/+bug/1719525 14:10:17 <openstack> Launchpad bug 1718387 in tripleo "ping test is periodically failing for the gate-tripleo-ci-centos-7-nonha-multinode-oooq " [Critical,Triaged] - Assigned to Marios Andreou (marios-b) 14:10:18 <openstack> Launchpad bug 1719373 in tripleo "tripleo ocata ceilometer-db-upgrade fails, as gnocchi cant reach swift" [Critical,Triaged] - Assigned to Pradeep Kilambi (pkilambi) 14:10:19 <openstack> Launchpad bug 1719525 in tripleo "CI: scenario002 periodic job fails with Zaqar error "404 not found" in undercloud" [Critical,Triaged] 14:10:29 <mwhahaha> Anyone have any additional bugs they want to raise? 14:10:48 <EmilienM> do we need to discuss about the alerts? 14:10:56 <EmilienM> sounds like one is not assigned 14:11:01 <marios> for 1718387 i posted an elastic recheck query so we could see how frequent it is https://review.openstack.org/505574 but didn't get attention yet 14:11:28 <mwhahaha> isn't 1719525 a dupe of the swift one from 2 weeks ago 14:11:30 <sshnaidm> this one: https://bugs.launchpad.net/tripleo/+bug/1719525 I opened today, didn't look into it too much.. 14:11:31 <openstack> Launchpad bug 1719525 in tripleo "CI: scenario002 periodic job fails with Zaqar error "404 not found" in undercloud" [Critical,Triaged] 14:11:39 <sshnaidm> mwhahaha, ok 14:11:45 <EmilienM> and the 2 others "Triaged" but not "In Progress" 14:11:53 <mwhahaha> that's the swift issue that therve fixed for master (i think the backport was pending) 14:11:59 <therve> mwhahaha, Yes I think so too 14:12:10 <sshnaidm> mwhahaha, therve so it's back? 14:12:19 <therve> sshnaidm, Not fixed in pikee 14:12:21 <therve> pike 14:12:21 <EmilienM> marios: how often 1718387 is hit? 14:12:26 <therve> https://review.openstack.org/#/c/505700/ 14:12:41 <marios> i filed 1718387 last week as i wasn't sure how often it was happening (it passed on recheck for the change I saw it on originally) EmilienM exactly i don't know thats why i posted the query 14:12:53 <EmilienM> marios: you probably want to ping pabelanger about the query 14:12:54 <mwhahaha> sshnaidm: was that a pike or master periodic job? 14:13:20 <sshnaidm> mwhahaha, sorry, it was master gate job 14:13:26 <sshnaidm> just fixed the name 14:13:32 <marios> EmilienM: ack will do (the change i wanted to land, and where i first saw it and filed the bug, passed on the next run, so it isn't reliable/wasn't at least) 14:13:33 <sshnaidm> not periodic 14:13:40 <mwhahaha> therve: can you take a look again 14:13:42 <sshnaidm> therve, it's master 14:14:00 <therve> Urg 14:14:04 <therve> I guess 14:14:51 <mwhahaha> k any other bugs? 14:15:03 <mwhahaha> perhaps some not related to CI as we have a section for that 14:15:37 <mwhahaha> ok moving on 14:15:38 <mwhahaha> #topic projects releases or stable backports 14:15:54 <mwhahaha> EmilienM is working on getting stable releases out for newton/ocata/pike 14:16:06 <EmilienM> mwhahaha: I just remember that shardy is off today for OpenStackDaysUK, he's presenting TripleO ! 14:16:20 <mwhahaha> anyone have any additional backports they want to raise (besides the previously mentioned newton one) 14:16:24 <EmilienM> mwhahaha: yes I'm working on it, and I made a mistake to approve https://review.openstack.org/#/c/492245/ 14:16:37 <EmilienM> mwhahaha: it sounds like a feature and we merged it 14:16:49 <mwhahaha> EmilienM: so we weren't following stable policy in newton 14:16:51 <EmilienM> but our drivers keep pushing things in stable branches, it's hard to say no 14:17:00 <EmilienM> mwhahaha: right, I made a mistake 14:17:12 <mwhahaha> EmilienM: or is that something we can point out to the release team for newton 14:17:32 <EmilienM> mwhahaha: yeah I'm already with doug on #openstack-release, trying to figure that out 14:17:36 <mwhahaha> k 14:17:41 <EmilienM> I think it's fine this time 14:17:50 <EmilienM> but in the future I (we) need to be more careful 14:17:56 <mwhahaha> this is something we need to keep an eye on for >=ocata 14:18:17 <EmilienM> yes 14:19:29 <mwhahaha> any other release/backport stuff? 14:20:00 <atoth> one question on it 14:20:03 <EmilienM> (done for me) 14:20:26 <amoralej> we need https://review.openstack.org/#/c/507495/ to unblock ocata promotion 14:20:39 <atoth> thedid you do anything that created the backport for the lbaas bug fix or was that done automatically somehow? 14:20:49 <matbu> i'll need to backport in pike those two reviews (when will be merged) : https://review.openstack.org/#/q/topic:minor-update-workflow+(status:open+OR+status:merged) 14:21:02 <mwhahaha> amoralej: there's already a patch for that 14:21:22 <mwhahaha> amoralej: i thought that was already merged 14:21:23 <amoralej> mwhahaha, that's a cherry pick from patch in master 14:21:34 <mwhahaha> amoralej: i rolled it in to the backport let me see 14:22:04 <owalsh> amoralej: IIRC *something* in nova or libvirt uses machine-id 14:22:23 <slagle> matbu: nice progress on moving the logic to tripleo-common 14:22:26 <amoralej> mwhahaha, yes, it's in https://review.openstack.org/#/c/489618/ 14:22:50 <mwhahaha> amoralej: i thought i fixed it in the backport but oh well +A 14:22:52 <EmilienM> slagle: is that something we should backport to pike? 14:23:18 <EmilienM> I'm afraid we don't have the choice for containerized deployments? 14:23:34 <amoralej> mwhahaha, i don't see it in https://review.openstack.org/#/c/493684 14:23:36 <matbu> slagle: thanks, i just need to fix the UTs and its sound like its working good according to folks from upgrade dfg 14:23:41 <slagle> EmilienM: i believe we have to? marios/ matbu can confirm 14:23:48 <mwhahaha> amoralej: https://review.openstack.org/#/c/493684/2/elements/remove-machine-id/post-install.d/70-remove-machine-id 14:23:50 <matbu> yep 14:23:53 <mwhahaha> amoralej: see the comment, i rolled it in 14:24:03 <social> slagle: yeah those patches are on right track 14:24:12 <mwhahaha> amoralej: we can pull in that patch (it won't hurt) but that's not the fix 14:24:26 <amoralej> mwhahaha, i don't follow you 14:24:32 <mwhahaha> amoralej: see my comment on that file 14:24:39 <mwhahaha> amoralej: see new file more 100755 14:24:47 <amoralej> the problem is not the mode 14:24:49 <slagle> matbu: fyi, i have soem patches based on cofing download, that i will rebase on top of your updates: https://review.openstack.org/#/q/status:open++branch:master+topic:config-download 14:24:57 <amoralej> but removing the file instead of empty it 14:25:02 <mwhahaha> amoralej: oh yea 14:25:08 <matbu> slagle: ack cool 14:25:10 <mwhahaha> amoralej: sorry thought it was the other patch 14:25:17 <trown> ya and if the file is missing journal cant start 14:25:24 <amoralej> exactly 14:25:35 <owalsh> amoralej: nova will not like an empty machine-id - https://github.com/openstack/nova/blob/328eb58a4a408706c75ba7493df24f6dcc472bb2/nova/virt/libvirt/driver.py#L3826 14:25:41 <amoralej> and it seems that if it doesn't exist systemd doesn't recreate it properly 14:25:55 <mwhahaha> amoralej: yea, pesky systemd 14:26:04 <amoralej> :) 14:26:12 <mwhahaha> owalsh: systemd pukes if the file is missing but if it's empty it'll populate it 14:26:26 <mwhahaha> owalsh: which is prior to the nova startup 14:26:31 <amoralej> yes 14:26:44 <owalsh> mwhahaha: ack, ok. maybe that's the problem I'm remembering :-) 14:27:03 <mwhahaha> yea it's a really weird interaction 14:27:21 <amoralej> mwhahaha, i'll propose the cherry-pick to newton also, as the element is being added there too 14:27:21 <mwhahaha> we'll need to also backport that to newton i think as well but that's on my plate 14:27:42 <mwhahaha> amoralej: ok please do 14:27:47 <trown> thanks amoralej for tracking that down! 14:28:12 <trown> was filed as a quickstart bug not collecting /var/log/messages and amoralej was very fast to find the actual cause 14:28:46 <EmilienM> amoralej: thanks! 14:28:47 <mwhahaha> ok any other backport things? 14:30:28 <mwhahaha> amoralej: https://review.openstack.org/#/c/505305/ is the previous patch for newton (is not merged yet). i guess we can just abandon that and i'll downstream it since newton will be EOL soon 14:30:35 <mwhahaha> #topic CI 14:30:49 <mwhahaha> how's CI today? 14:31:30 <amoralej> mwhahaha, ok, no problem from my sid 14:31:38 <amoralej> just remember to abandon https://review.openstack.org/#/c/505310 also 14:31:41 <weshay> <joke> A man walks into a ci system 14:31:50 <amoralej> what was the bug fixed by that? 14:31:53 <weshay> bnemec++ 14:32:02 <weshay> fixed a lot of the timeouts 14:32:21 <mwhahaha> amoralej: it breaks a bunch of stuff because all the machine-ids are identical for all nodes 14:32:30 <mwhahaha> amoralej: it was an really old customer bug from like rhos7 14:33:05 <mwhahaha> amoralej: with some angry folks asking why it took 2+ years to clear a file 14:33:15 <amoralej> ok, then we could do the three backports, it shouldn't be hard 14:33:35 <mwhahaha> amoralej: i was only planning to backport to 10. i'll handle it 14:33:43 <openstackgerrit> Alfredo Moralejo proposed openstack/tripleo-puppet-elements stable/newton: Set remove-machine-id to executable https://review.openstack.org/507561 14:33:51 <amoralej> mwhahaha, ^ the missing one 14:33:54 <mwhahaha> k 14:34:00 <mwhahaha> ok any other CI related problems? 14:34:53 <mwhahaha> ok moving on 14:34:54 <gchamoul> no big blocker today AFAIK 14:35:01 <mwhahaha> #topic specs 14:35:01 <mwhahaha> #link https://review.openstack.org/#/q/project:openstack/tripleo-specs+status:open 14:35:11 <mwhahaha> please take some time to review queens specs 14:35:18 <mwhahaha> also if you have any old specs, please retarget 14:35:32 <mwhahaha> anyone want to highlight a specific spec? 14:36:45 <mwhahaha> ok moving on 14:36:46 <mwhahaha> #topic open discussion 14:37:13 <mwhahaha> speak now or forever (until next week) hold your peace 14:37:19 <EmilienM> :-O 14:37:38 <EmilienM> (except on Fridays :-P) 14:38:28 <mwhahaha> sounds like that's it for the week 14:38:40 <mwhahaha> thanks everyone 14:38:43 <mwhahaha> #endmeeting