14:00:23 #startmeeting tripleo 14:00:24 Meeting started Tue Jan 17 14:00:23 2017 UTC and is due to finish in 60 minutes. The chair is EmilienM. Information about MeetBot at http://wiki.debian.org/MeetBot. 14:00:25 Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 14:00:27 o/ 14:00:27 The meeting name has been set to 'tripleo' 14:00:29 #topic agenda 14:00:31 * review past action items 14:00:33 * one off agenda items 14:00:35 * bugs 14:00:37 * Projects releases or stable backports 14:00:39 * CI 14:00:41 * Specs 14:00:43 * open discussion 14:00:45 hello, who is around today? 14:00:49 o/ 14:00:50 o/ 14:00:52 o/ 14:00:53 o/ 14:00:55 o/ 14:00:58 o/ 14:01:00 o/ 14:01:05 o/ 14:01:10 hi! 14:01:32 I'll let people take a seat and review past action it in meantime 14:01:36 o/ 14:01:39 o/ 14:01:39 #topic review past action items 14:01:44 o/ 14:01:44 o/ 14:01:47 o/ 14:01:59 o/ 14:02:04 EmilienM to help debug upgrade job timeout wrt haproxy config update: done (job has been promoted to check-nv) 14:02:11 team to look http://lists.openstack.org/pipermail/openstack-dev/2017-January/109600.html and give feedback: done 14:02:21 o/ 14:02:21 team to look https://review.openstack.org/#/c/359060/ and https://review.openstack.org/#/c/415179/ + give feedback: done by gfidente 14:02:31 ci squad to move the tripleo-quickstart upgrade job into periodic pipeline: in progress 14:02:40 o/ 14:02:50 trown: or is it done ^ ? 14:03:01 team to review capture-environment-status-and-logs blueprint: 405001 and 405002: done by d0ugal 14:03:34 o/ 14:03:40 anything else? 14:04:00 #topic one off agenda items 14:04:02 EmilienM: I dont think so... I have not payed attention to that one though... sshnaidm panda? 14:04:05 #link https://etherpad.openstack.org/p/tripleo-meeting-items 14:04:17 trown, no, it's still on review afaik, matbu ? 14:04:25 arxcruz, panda: o/ 14:04:42 trown: nope 14:04:44 0/ 14:04:45 o/ 14:04:48 EmilienM: hey 14:04:48 o/ 14:04:52 panda: go ahead 14:05:16 matbu, ^ 14:05:30 so we have two patches https://review.openstack.org/420647 https://review.openstack.org/420620 in place to replace the current periodic tempest job 14:05:31 sshnaidm: review are still waiting for vote :( 14:05:41 EmilienM, at the moment it's running internally.. looking to see if oooq upgrade job in upstream is a requirement 14:05:54 the infra patch already got the approval from infra, but then we needed to add something more 14:06:20 panda: ok, something needs to be discussed or some actions taken? 14:06:23 EmilienM: sshnaidm i'll probably need to update the quickstart one, some services has been added last week 14:06:25 or it's just announcement 14:06:27 ? 14:06:29 we are currently running a test upstream using thiss change https://review.openstack.org/421272 to see the end results from this replacement 14:07:08 EmilienM: I proposed a discussion on the mailing list, just to see if there are any objections 14:07:21 matbu, EmilienM maybe it's worth to mark upgrade reviews as action items? 14:07:31 sshnaidm: +1 14:07:45 arxcruz: can confirm as he's looking at the tempest part, the full list of tempest test passes, with a few exceptions 14:07:55 i think tripleo cores should be involved in the quickstart reviews 14:08:02 3 failures last I checked in full tempest on master 14:08:03 the delay is really huge 14:08:07 EmilienM: confirmed 14:08:08 2 are known issues 14:08:23 #action team to review oooq periodic job https://review.openstack.org/420647 https://review.openstack.org/420620 14:08:36 which https://review.openstack.org/#/c/421018/ will add to skip file 14:08:41 so, if no one has anything to object, seems like tempest is back on the menu 14:08:51 arxcruz: do you want to add something for the tempestmail spec ? 14:09:01 matbu: have you documented how to review quickstart patches? 14:09:08 panda: I wrote the spec, but didn't finish the code yet on tripleo-ci/scripts 14:09:09 matbu: have you thought at a deep dive sessions? 14:09:20 +1 on deep dive 14:09:25 panda: EmilienM spec at https://review.openstack.org/#/c/420878/ 14:09:27 and it's out of topic now, sorry 14:09:35 EmilienM: the commit message is explicit i guess 14:09:53 panda, is it nonha intentionally? 14:10:03 sshnaidm: yes 14:10:04 folks, we're having 4 topics in same time 14:10:12 ya, I am really lost 14:10:33 ha 14:10:40 :/ 14:10:40 there are some tests that fail in ha, because the sync between the controllers 14:10:52 o/ 14:11:03 EmilienM: mark the active topic then, sorry I thought you gave the green light. 14:11:06 I think we started from upgrade job 14:11:28 ok, so nothing else for moving periodic job to use oooq? 14:11:51 ok, seems like we can move to the next topic 14:12:01 weshay, matbu: it's you now 14:12:30 EmilienM: changes, jobs, and spec to send email to proper group in case of failure, nothing else on my part 14:12:34 EmilienM, I'm looking for input whether or not we think the upgrade oooq job should be running in a check gate 14:13:00 weshay: is it a multinode or an ovb? 14:13:07 or experimental 14:13:13 panda: feel free to use #action to track the list of things todo 14:13:13 ovb 14:13:31 weshay: right now, we already have a multinode (non oooq) job in THT check -nv 14:13:34 EmilienM: weshay fyi the tripleo-ci experimental has been merged 14:13:42 and it's working properly now 14:13:56 matbu: please be precise and give the jobname 14:14:05 the "the tripleo-ci experimental" is not enough to understand which one 14:14:35 weshay: we just moved the brand new non-ooq upgrade job to the t-h-t check queue yesterday 14:14:40 EmilienM: gate-tripleo-ci-centos-7-multinode-upgrades-nv 14:14:51 weshay: can you clarify, this is the exact same steps done via quickstart, or something else? 14:15:07 matbu: gate-tripleo-ci-centos-7-multinode-upgrades-nv has moved to check pipeline for THT and experimental for tripleo-ci 14:15:12 shardy: yes, but with less services than we added recently 14:15:15 shardy, ok.. having coverage is the most important thing 14:15:42 weshay: I'm kinda thinking we should leave that one alone for now, as it blocked landing upgrades patches for nearly two months 14:15:59 shardy, ok.. this is the input I was looking for 14:15:59 +1 to just using the multinode upgrade we have 14:16:05 +1 too 14:16:11 and we can revisit in Pike 14:16:11 I'd be fine with us proving the oooq one via the experimental queue tho and proving we can switch them later 14:16:20 shardy: yes exactly 14:16:24 +1 on oooq experimental 14:16:37 ok.. we can do that.. by replacing one of the current oooq jobs in experimental w/ upgrade 14:17:01 seems like a good idea to me to ensure we're covered in the short term and further out 14:17:02 weshay: sounds good - I'd be keen to see it running, but wary of any further blockages re the upgrades patches 14:17:07 I think we should make the oooq upgrade job based on top of the recent oooq-multinode job 14:17:18 the ovb queue is too crowded to add more jobs 14:17:28 oh.. that would be awesome 14:17:32 trown: yes 14:17:33 matbu, you following this? 14:17:35 sounds like we have a plan 14:17:37 trown: +1 - the current upgrades job uses multinode, so that would be consistent 14:17:56 when we prove they're working exactly the same, we can switch them in the t-h-t check queue, but probably not before early pike 14:18:10 +1 14:18:10 +1 14:18:13 maybe we can create a blueprint for that 14:18:17 so we track the work 14:18:31 weshay: can you take some #action please? 14:18:38 so it's somewhere in logs :-) 14:19:40 #action weshay, matbu, sofer to create a tripleo blueprint for multinode nodepool oooq based upgrade job 14:19:49 excellent 14:19:53 weshay: anything else for this topic? 14:20:00 nada 14:20:03 thanks! 14:20:13 bandini, ccamacho : o/ 14:20:18 yo ;) 14:20:21 yoooo 14:20:22 hey! 14:20:24 So with https://review.openstack.org/#/c/359060/ we add an explicit nonha-arch environment file *and* default to the HA deployment. The corresponding CI patch https://review.openstack.org/#/c/415179/ makes sure the nonha jobs use -e nonha-arch.yaml. The CI patch is failing on the current unrelated mitaka nonha. I'd love to merge both patches at the same time. Are there any concerns? (Note that 14:20:26 I added release notes for THT here https://review.openstack.org/421226 and plan to send a mail on openstack-dev once this stuff merges) 14:21:05 is it safe to do it now? (end of ocata) 14:21:08 bandini, does it mean that we won't have a job testing the "default" deployment (i.e. one without any -e)? 14:21:56 dtantsur: well for ocata -e environment/puppet-pacemaker and default will mean the same thing. so yes we are and it is the ha job 14:22:07 dtantsur, yes but it will be deployed using the resources that previously were in the puppet-pacemaker template 14:22:17 EmilienM: I think it is, but wanted to get the input of the larger group 14:22:42 it is basically one of the only differences between downstream and upstream and has confused endless folks 14:22:42 ya... looks like the same coverage CI wise to me 14:22:54 bandini, the HA job deploys with ControllerCount=3, right? 14:22:55 just changes which job needs to be explicit about changing default 14:23:16 bandini if we keep the puppet-pacemaker.yaml release, as planned, I think it's safe to do 14:23:21 dtantsur: I believe so 14:23:25 anything pointing to the old file should continue to work as intended 14:23:34 yeahp gfidente until pike the template will be there 14:23:36 bandini: fwiw reviewed that today, imo can land... besides CI most people will be using puppet-pacemaker already anyway 14:23:38 gfidente: yeah I plan to kill that file for pike, not anytime sooner 14:23:42 we just need to maintain registry/nonha/puppet-pacemaker in good shape for a release right? 14:24:01 bandini: I think we can't kill the file until queens 14:24:01 why do we need nonha job then? 14:24:06 for introspection? 14:24:10 deprecate in O remove in Q 14:24:14 why not moving introspection to ha job then? 14:24:18 trown++ 14:24:32 trown: sounds good to me too 14:24:37 EmilienM: so my take on nonha is that we can kill that too eventually, but it seemed too much of a big change 14:24:41 trown: +1 14:24:48 two releases better than one 14:24:48 EmilienM I think we should do this step by step 14:24:51 bnemec: thoughts ^ ? 14:25:11 I don't see any blocker 14:25:16 any actions to take folks? 14:25:20 on dropping entirely nonha I am not convinced 14:25:42 I'd just let the reviews merge after appropriate +2, if folks agree 14:25:46 I think it's good to keep it in working state 14:25:48 gfidente: yeah that requires a longer discussion 14:25:56 EmilienM, ceph and undercloud-ssl are checked in nonha only 14:26:06 right 14:26:13 so for ceph I am adding HCI to the HA job 14:26:21 ceph is highly tested in scenario001 and scenario004 14:26:24 https://review.openstack.org/338088 14:26:28 gfidente: Yeah I agree, if nothing else it's useful for minimal testing by developers 14:26:30 so we don't need extra nodes 14:26:41 however I guess the question is can we justify bandwidth if that's the only reason 14:26:49 e.g CI bandwidth 14:26:55 EmilienM, and "no network isolation" there 14:26:57 ceph HCI that depends on https://review.openstack.org/#/c/419049/ though 14:27:03 sshnaidm: right 14:27:14 sshnaidm right if we land that we will test it with network isolation 14:27:20 bandini, ccamacho: any #action to take before we go to next topic? 14:27:39 shardy ack on bandwidth 14:27:42 EmilienM we are good 14:27:43 EmilienM: I think we're good. We'll let the reviews do the rest and merge once we're okay 14:27:47 we can revisit the topic of keeping nonha for later 14:27:51 yeah 14:27:55 thanks! 14:28:00 thanks everyone 14:28:03 #action team to review https://review.openstack.org/#/c/359060/ and https://review.openstack.org/#/c/415179/ 14:28:04 can I add one more question for CI topic? 14:28:13 gfidente: of course, go ahead now 14:28:26 if with jprovazn we wanted to have a scenario to test the manila/cephmds 14:28:37 shall we prefer hacking on an existing scenario or add a new one? 14:28:57 (cephmds is a new service we don't deploy by default, needed for cephfs, needed in turn by manila) 14:29:00 gfidente: that's a good question 14:29:20 so scenario001 is really about testing rbd with nova/cinder/glance/telemetry 14:29:29 cool, so we could add it there? 14:29:31 and scenario004 is about ceph rgw with glance 14:29:41 could it be in scenario004? 14:29:47 yes fine for me 14:29:53 thanks 14:30:05 #action gfidente to investigate cephmds + manila on scenario004 14:30:18 gfidente: thanks for adding more CI coverage, very nice :) 14:30:44 * dtantsur reminds himself that he should get to an ironic scenario eventually 14:30:47 you mean better than driving bling in the night 14:30:52 yes I learned that 14:31:05 :) 14:31:16 ok next topic 14:31:21 PTG 14:31:27 #link https://etherpad.openstack.org/p/tripleo-ptg-pike 14:31:35 we already have a few sessions, which is great! 14:31:50 I've set a deadline for submitting sessions, so we have enough time to organize ourselves 14:31:58 deadline: February 6th 14:32:06 any feedback on ^ is welcome 14:32:21 I haven't seen much sessions about CI 14:32:37 weshay, trown, adarazs : are you interested to add some? 14:33:05 EmilienM: I won't be there, so as you wish. :) 14:33:15 EmilienM: it might be useful though for people present. 14:33:21 I believe trown will have to lead some sessions, I am moving to colorado at that time 14:33:24 EmilienM: ya maybe we can discuss more in CI squad meeting on thursday and put something up based on that 14:33:42 sounds like a plan 14:33:52 #action trown & EmilienM to prepare CI sessions for PTG 14:33:55 #action trown bring up PTG CI sessions in thursday CI squad meeting 14:34:00 jinz 14:34:06 jinx even 14:34:08 other folks, please propose topics by the deadline 14:34:21 reminder: we have a room for 3 days, so don't be shy :) 14:34:33 #undo 14:34:34 Removing item from minutes: #action trown bring up PTG CI sessions in thursday CI squad meeting 14:34:42 #undo 14:34:43 Removing item from minutes: #action trown & EmilienM to prepare CI sessions for PTG 14:34:50 #action trown bring up PTG CI sessions in thursday CI squad meeting 14:34:52 voilĂ  14:34:55 :) 14:35:09 any question about PTG or feedback? 14:35:22 ok let's move 14:35:26 #topic bugs 14:35:36 #link https://launchpad.net/tripleo/+milestone/ocata-3 14:35:57 I sent an email about remaining bugs for tripleoclient before we release it next week 14:35:59 #link https://goo.gl/R2hO4Z 14:36:10 we still have 6 bugs open 14:36:37 tripleoclient reviewers, please look at them and make sure they need to be in ocata-3 or defer it to pike-1 14:36:58 other than tripleoclient, do we have outstanding bugs to discuss this week? 14:37:16 74 Triaged, 67 In Progress for ocata-3 14:37:40 it's huge :) 14:37:46 just a heads up on https://bugs.launchpad.net/tripleo/+bug/1656939... 14:37:46 Launchpad bug 1656939 in tripleo "Empty THT strings now being interpreted as actual values by puppet" [High,In progress] - Assigned to Brent Eagles (beagles) 14:37:51 I think I already asked last time, but any help on triage would be welcome 14:38:11 EmilienM: i just added one more (the tag was wiped earlier when i added puppet-nova) https://bugs.launchpad.net/tripleo/+bug/1656791 14:38:11 Launchpad bug 1656791 in tripleo "Mitaka to Newton undercloud upgrade needs "nova db online_data_migrations" after the undercloud upgrade" [High,In progress] - Assigned to Marios Andreou (marios-b) 14:38:13 I suggest we defer all low/medium unassigned bugs to pike-1, and move all high unassigned bugs to rc1 14:38:25 EmilienM: its the one you already reviewed the code for (its an upgrade related one) 14:38:29 #link https://review.openstack.org/#/c/420947/ 14:38:30 then we can focus on fixing the most serious issues prior to the release 14:38:35 beagles: ack, thanks for this one 14:38:37 this came to my attention yesterday. I expect that neutron might've been the only one affected but if in case you've done the "bad thing" mentioned in the bug in the past you might want to check on 14:38:54 shardy: sounds like a plan 14:39:33 #action EmilienM to move ocata-3 low/medium unassigned bugs to pike-1 14:39:35 +1 shardy 14:39:46 #action EmilienM to move ocata-3 high/critical unassigned bugs to ocata-rc1 14:40:09 marios: sounds good, it's in our radar for this week 14:40:16 marios: i'll help you to get this puppet-nova patch merged asap 14:40:50 mwhahaha: FYI https://review.openstack.org/#/c/421249/ 14:40:53 EmilienM: thanks will reach out to someone from nova to sanity check too 14:41:07 EmilienM: appreciate the reviews 14:41:15 marios: please 14:41:30 do we have other bugs to discuss this week? 14:42:29 #topic Projects releases or stable backports 14:42:44 reminder: tripleoclient will be release next week, on Thursday 14:42:58 #link https://releases.openstack.org/ocata/schedule.html 14:43:33 we'll also release ocata-3 14:43:51 Jan 27 is the target but it will likely happen on Jan 26 I think 14:44:26 No featureful patch should be landed after this point. Exceptions may be granted by the project PTL, on openstack-dev ML 14:44:57 I'm working on release notes with d0ugal 14:45:07 any help on adding release notes for Ocata is very welcome 14:45:18 ping me or d0ugal to sync 14:46:11 any question or feedback on releases? 14:46:44 #topic CI 14:46:51 #link http://tripleo.org/cistatus.html 14:46:59 #link http://status-tripleoci.rhcloud.com/ 14:47:13 #link https://dashboards.rdoproject.org/rdo-dev 14:47:22 panda, sshnaidm: any idea why we haven't promoted last night? 14:48:17 (if there is anything to discuss about CI, please raise it here) 14:48:32 bnemec managed to make Mitaka jobs passing again, see https://review.openstack.org/#/c/420156/ - bnemec: can you remove -W when you can? 14:48:37 EmilienM, " Ironic deployment timeout" in nonha job 14:48:45 sshnaidm: have we filed a bug? 14:48:51 EmilienM, I don't think so 14:49:10 sshnaidm: let's do it after meeting and see why we have that now. 14:49:14 EmilienM, I'll handle it, suspecting it's timeout issue we had in past 14:49:15 just want to bring up that we have a CI squad meeting on Thursdays to discuss ongoing improvements to CI... adarazs is the liason and did a great write up of last weeks meeting http://lists.openstack.org/pipermail/openstack-dev/2017-January/110034.html 14:49:22 sshnaidm: we can maybe ask dtantsur to look if it's ironic related 14:49:55 #link http://lists.openstack.org/pipermail/openstack-dev/2017-January/110034.html 14:49:59 trown: thanks. :) I think that status is still correct. 14:50:05 adarazs: yes, I also found it excellent 14:50:27 all right, looking forward to the next. :) 14:50:31 marios: do you keep doing weekly standups? I'm not sure I've seen notes posted on ML 14:50:39 chem: ^ 14:51:04 EmilienM: no we have been having more ad-hoc calls really 14:51:13 EmilienM: we always announce on the tripleo chan 14:51:37 EmilienM: i guess we should organise a weekly if that would be useful/if there is someone who is interested and is unable to follow the work 14:51:38 like I said, it's good to let squads do how they like, though having public notes on the ML would be helpful 14:52:07 EmilienM: marios hum, yes that's kind of a good idea 14:52:09 weekly or not, I think having public notes make to do things "in the open" 14:52:32 Just having an etherpad with the current status of upgrades would probably help - I think matbu started one 14:52:40 EmilienM: fwiw we have notes https://etherpad.openstack.org/p/tripleo-upgrades-squad 14:52:44 shardy: yep 14:52:46 excellent 14:52:47 EmilienM: last time we had a call was last friday for example 14:53:07 https://etherpad.openstack.org/p/composable-upgrade 14:53:07 I guess this etherpad is enough :-) thanks folks 14:53:17 #link https://etherpad.openstack.org/p/composable-upgrade 14:53:24 It might have been more appropriate for the bugs section rather than CI, but I noticed unit tests don't pass on stable/mitaka anymore so patches can't merge (the branch is still security supported). I put some debugging notes up at https://bugs.launchpad.net/tripleo/+bug/1656871 but am a bit stuck. I'd appreciate pointers on how to help resolve it, if anyone has any 14:53:24 Launchpad bug 1656871 in tripleo "Mitaka unit tests broken (CLI/OSC test dependencies)" [High,Triaged] 14:53:24 it needs more love :D 14:54:07 #action team to help jpich to debug https://bugs.launchpad.net/tripleo/+bug/1656871 and unblock stable/mitaka unit tests 14:54:23 jpich: thank you 14:54:41 do we have anything else about this CI week? 14:54:46 Thanks! 14:55:03 #topic specs 14:55:14 #link https://review.openstack.org/#/q/project:openstack/tripleo-specs+status:open 14:55:23 jpich: have you seen https://review.openstack.org/#/c/413792/ ? 14:55:37 that's a patch from bnemec which aims to fix several mitaka related issues 14:55:40 not yet landed 14:56:06 yeah it's a backport 14:56:29 we probably need to Depends-On https://review.openstack.org/#/c/420156/ 14:56:40 to check that it doesn't break OVB jobs 14:57:11 jpich: can you check with bnemec when he's around? 14:57:20 shardy: I suspect this is related to larger issues with the long-running jobs, that bug seems to be related to unit tests and dependencies at the pip level 14:57:28 EmilienM: Will do 14:58:06 a quick note on specs, more a reminder 14:58:25 if you submit a spec, please target it for Pike and not Ocata :-) 14:58:42 I know it makes sense but I've seen specs for Ocata coming up yesterday ;-) 14:58:54 lol, that's the spirit! 14:59:03 :) 14:59:10 i'll open the discussion 14:59:13 #topic open discussion 14:59:23 if you have any question or feedback, you have 45 seconds 14:59:35 or you can use #tripleo at any time 14:59:39 EmilienM: there as spec for adding tempest mail to tripleo-ci, bt arxcruz 14:59:48 panda: I've seen it 14:59:50 EmilienM: not sure it should be targeted to OPike also 14:59:56 panda: yes, Pike. 15:00:07 spec freeze was December 16th iirc 15:00:08 EmilienM: ok, thanks 15:00:22 thanks everyone! 15:00:24 #endmeeting