14:02:04 #startmeeting tripleo 14:02:04 Meeting started Tue Jan 26 14:02:04 2016 UTC and is due to finish in 60 minutes. The chair is dprince. Information about MeetBot at http://wiki.debian.org/MeetBot. 14:02:05 Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 14:02:07 The meeting name has been set to 'tripleo' 14:02:26 yp 14:02:29 yo 14:02:30 o/ 14:02:32 o/ 14:02:35 \o 14:02:36 o/ 14:02:41 hi everyone 14:02:53 hey 14:02:54 o/ 14:03:04 o/ 14:03:07 thanks for reminder dprince am also on a bluejeans call so forgot 14:03:21 hiya 14:03:23 marios: np 14:03:28 hey 14:03:30 \o 14:03:31 #topic agenda 14:03:31 * bugs 14:03:31 * Projects releases or stable backports 14:03:31 * CI 14:03:31 * Specs 14:03:33 * one off agenda items 14:03:36 * open discussion 14:03:59 There are no 'one off agenda' items this week... 14:04:12 anything else to add to the agenda? 14:04:25 dprince: I got 2 topics to discuss 14:04:35 I have some stuff for open discussion if there ends up being time 14:04:37 dprince: 1. critical alerts for bugs effecting tripleo 14:04:46 dprince: 2. using launchpad for wish list bugs 14:05:17 derekh: ack, we'll add those in.. 14:05:24 ty 14:05:39 trown: hopefully there is time for you topic too :) 14:05:43 lets go then 14:05:50 #topic bugs 14:06:09 derekh: would you like to just discuss your bugs stuff here? or later? 14:06:14 this seems to have started today, https://bugs.launchpad.net/tripleo/+bug/1538127 14:06:15 Launchpad bug 1538127 in tripleo "overcloud deploys timing out" [Critical,Triaged] 14:06:28 dprince: either is fine by me 14:07:03 on bug #1538127 I'm waiting from some logs to find the problem, will investigate after this call 14:07:05 fwiw we do not see this issue in RDOCI, I think we really need to work on getting you guys moved to a newer delorean pin 14:07:30 so that the tests in RDO are at least similar to the tests in tripleo 14:07:50 derekh: okay, so no data yet on the timeout. we'll have to follow up on this in #tripleo as it blocks our CI 14:08:08 right now in tripleo we are really testing the ability to deploy liberty 14:08:16 with mitaka tripleo 14:08:17 trown: yup, thanks a lot of the problems you been sorting out, we're closer now to moveing to a new repository https://review.openstack.org/#/c/229789/ 14:08:50 derekh: nice, you got one pass there :) 14:08:55 :) 14:09:22 dprince: the ha test only failed the ping test, if this was last week we'd have switched already ;-) 14:09:26 which is one more pass than the other jobs are getting right now 14:09:34 trown: yup 14:10:08 ok, may aswell talk about it here 14:10:14 critical alerts for bugs effecting ci 14:10:16 derekh: yep, go for it 14:10:20 Some time ago, I created (refreshed) a tripleo trello board, https://trello.com/tripleo 14:10:25 For a while I've been using it to track almost everything I've been doing for upstream tripleo 14:10:30 but nobody else did much, so I think I may aswell drop it 14:10:30 the one thing I think has been good and worked is the alerts I generate from it when ci is broken 14:10:30 I'm thinking maybe we keep that and maybe drive it from tripleo launchpad bugs that have the ci tag and are critical 14:10:30 what do people think? 14:11:03 +1 to the alerts 14:11:25 derekh: +1 personally I think launchpad is sufficient, although maintaining the alerts would be good 14:12:05 derekh: +1 14:12:16 so instead of opening a trello card (cause that hasn't been working I think), we insteand open a critical bug, with a psecial tag (maybe blocker or ci), then drive alerts from that 14:12:34 dprince: shardy trown ok, I'll make the switch when I get a chance over the next day or 2 14:12:56 dprince: done with that topic, unless there are objections/ better ideas/// 14:12:59 derekh: maybe just send an email out with information on the 'tag' we want to use for these things 14:13:07 dprince: will do 14:13:13 derekh: or update the CI wiki page too? 14:13:29 dprince: CI wiki page? 14:13:39 derekh: yeah, I think there is one 14:13:40 lol I thought the same 14:13:51 I have not seen this wiki 14:14:24 derekh: I'm not finding it. Ignore me 14:14:30 derekh: okay, wishlists? 14:14:40 dprince: np, will have a look for it just incase 14:14:56 wishlists, this came out of a discussion with some downstream tripleo consumers (QE), they would like to test new features as they get added to tripleo (or as soon as possible new features are added). 14:15:04 I propose we would use launchoad wish list bugs for new features, to allow them to be tracked 14:15:18 as with any other bug it would get automatically closed once the last patch is merged to add the feature. 14:15:23 thoughts? 14:15:49 small new features can be wishlists, sure 14:16:06 derekh: In Heat, we've adopted the "spec-lite" approach outlined by glance: 14:16:09 http://docs.openstack.org/developer/glance/contributing/blueprints.html#glance-spec-lite 14:16:13 but larger major features (with specs) are blueprints right 14:16:21 specs tell us we agree that something should be done but don't give us tracking to allow somebody know something has been finished 14:16:22 basically it is tracking small features via launchpad, with a tag of spec-lite 14:16:29 so I'm +1 on doing the same 14:17:28 dprince: blueprints would work too, I just said wishlist bugs as they are more lightweight I've never seen any tripleo blueprints 14:17:30 are we using blueprints at all? 14:17:32 spec light sounds fine to me, may require some occasional grooming to make the call on spec-light vs. blueprints 14:17:45 trown: I do sometimes, yes. But not heavily 14:18:02 * dprince did for the swift artifact URL's stuff 14:18:03 dprince: Yeah, that's part of the triage process, you set it invalid if you feel it must have a full spec 14:18:10 so maybe the proposal should be all RFEs tracked with blueprint or spec-light 14:18:42 I think the important part is having all features tracked 14:18:46 trown: +1 that will make it *much* easier to define what we've delivered each release 14:19:40 we may also want to consider adopting the automated release notes tools too, e.g reno 14:19:43 http://docs.openstack.org/developer/reno/ 14:19:50 ya, Ironic has been using that 14:19:59 derekh: okay, positive feedback on both your ideas so proceed I think 14:20:12 Either is fine with me, just learning about the spec lite process now, looks nice and light weight, then can be a full spec if needed 14:20:16 Now we have stable branches, we'll want to be able to anounce all the new features and have accurate release notes 14:20:20 dprince: cool, I'll summarize both to the list 14:21:17 shardy: cool, I haven't spent time on looking at reno. Will try to have a closer look 14:21:50 #topic CI 14:22:04 other than our CI being broken ATM is there anything else here? 14:23:16 I've been trying to look into some of our intermittent errors as time allows (I hadn't done this in a while), we need to get our success rate up a bit 14:23:44 besides the current blocker, we now have what seems to be a lot of intermittent error 14:24:21 derekh: okay, thanks for the update 14:24:24 dprince: one thing I'm proposing to move tripleo.sh back to tripleo-ci : https://review.openstack.org/#/c/272210/ 14:24:32 for the HA job, we probably need a beefier undercloud 14:24:32 Need to get that passing, but just so folks are aware 14:24:38 shardy: +1 14:24:57 dprince: reason is it handles both branches, and we're wasting effort backporting everything in tripleo-common 14:24:57 I had all kinds of intermittent failures in RDOCI on the HA job before bumping the undercloud stats 14:24:57 shardy: cool, thanks for pointing it out 14:25:31 I think the stack for the HA deploy is pretty resource intensive to create 14:25:47 trown: Yeah, running without swap and not much RAM is bound to cause spurious failures 14:25:53 we do have some swap now tho 14:26:10 would be good to see if it is being used, as obviously it'll hurt performance if it is 14:26:19 for reference I use 12G RAM 4CPU undercloud in RDO 14:26:31 shardy: true, but at some point we'll move on. Perhaps that just means we could branch tripleo-ci for the stable jobs though 14:26:38 trown: how do your CI runtimes compare? 14:27:00 dprince: I'd prefer if we made tripleo.sh handle all branches 14:27:06 shardy: I am doing something a bit different so it is apples to oranges, that is one of my topics for open discussion though 14:27:14 * derekh would love to just get to OVB already, then we can try all these things at a whim 14:27:39 derekh: What is blocking us right now? 14:27:40 dprince: well, it already does, I mean keep it that way 14:27:50 but we can debat that if/when the time comes ;) 14:27:53 debate 14:28:02 shardy: sure, sounds like a fine plan for now 14:28:11 +1 to moving tripleo.sh to tripleo-ci repo 14:28:21 * bnemec doesn't want to de-bat. They eat mosquitoes ;-) 14:28:24 makes it clear it is a dev CI tool 14:28:30 trown: +1 14:28:42 bnemec: I really want to test it on 10+ compute nodes so that 1. I can test a reproducable deployment and 2. make sure its ok when things scall up to 100+ simultanious jobs 14:28:50 i'm already seeing things i don't like with people using it as a non-dev tool :) 14:29:16 bnemec: I don't want to risk taking down what we have only to find out we can't get it to work 14:29:33 slagle: That's because it's where we put all of the hacks to work around things we aren't fixing in other projects. :-/ 14:29:44 derekh: Understood. Maybe we need to see if we can schedule some time in the scale lab? 14:30:04 bnemec, right and because of that the official docs frequently doesn't work 14:30:24 bnemec: yup, we tied at one stage, but could only gewt it one day a week, maybe we can push a bit harder 14:30:26 gfidente: Yeah, I've pretty much given up on the official docs. 14:30:44 bnemec, indeed, but that is what people will look at 14:31:05 gfidente: bnemec, I have an idea here that is the other topic I wanted to discuss in open discussion... hold tight :) 14:31:07 so I am thinking we should also avoid having hacks in tripleo.sh and deploy using tripleo.sh 14:31:39 we used to run Heat tests direct from docs using scripts in the markup 14:31:47 it would be possible to do the same I guess 14:31:56 derekh: Yeah, we need to do something. Has downstream not made any progress with it? I know they were setting up an environment too. 14:32:28 bnemec: last I heard they were waiting on HW to be rack, I'll see if I can get an update 14:32:49 Bleh, hardware. 14:34:30 okay, we skipped a topic so lets get it now 14:34:39 #topic Projects releases or stable backports 14:35:09 Have the backports finally slowed? 14:35:23 dprince: there's still a large backlog because of the CI problems 14:35:52 shardy: okay, but if we just moved over the tripleo.sh it would eliminate that issue? 14:35:56 dprince: the backports will be quite hot and heavy once ipv6 starts landing on master 14:35:59 dprince: yeah more or less what we needed to do the initial rebase for downstream tht, though as shardy says the 'few remaining' have grown into list because ci 14:36:23 dprince: No, that doesn't really fix anything other than making landing fixes a bit easier/quicker 14:36:31 since ipv6, as well as any other ssl improvements, must be backported to liberty 14:36:56 shardy: I see, so the backports are fixes to CI 14:37:23 https://review.openstack.org/#/c/272194/ and https://review.openstack.org/#/c/272191/ are the main ones I'm aware of 14:37:49 dprince: Yeah, we need to fix CI then there's a ton of stuff to recheck and review 14:38:01 slagle: okay, thanks for the update 14:38:12 same story for master I suppose, our review/merge rate is depressing there also 14:38:56 hopefully we can work on improving that via getting CI more stable 14:38:59 side note here for cores, can we make sure that any thing that we +W has recent as in from the last day or so CI results? 14:39:01 Yeah, I'm gonna say TripleO is struggling at best ATM... many reviews are blocked needing manual upgrade testing 14:39:21 and features are blocked still because of the pain of backporting to stable liberty? 14:39:38 we do not run the functional tests in the gate pipeline, so things can definitely slip through with outdated CI results 14:39:39 we seem to have upstream development backwards... 14:39:53 dprince: I think backporting to stable is working fine, but CI reliablity is preventing us landing things in a reasonably timely fashion 14:40:22 shardy: the backporting mechanism is fine. The fact that we are still backporting features halfway through the release is the problem :/ 14:40:50 it does feel a little silly that we are basically backporting 90% of patches 14:40:52 shardy: until this stops it is going to really make it difficult to get in potential new features (composable roles, split stack) I think 14:41:03 dprince: Yeah, I'm saying if it didn't take 3 months to merge to master and 2 months to merge to stable, we'd be in better shape 14:41:08 the reality is that 90% of tripleo devs are working on liberty support still 14:41:31 Or Kilo 14:41:37 bnemec: don't go there :) 14:41:43 :-) 14:41:51 slagle: I realize this, and I'm trying to be mindful not to cause conflicts. But there is a cost... 14:41:57 is there some proposal to change that? otherwise I think we probably just have to take that as a given 14:42:17 maybe we'll have a chance to discuss this next week 14:42:38 slagle: I'd be interested to discuss if there's an alternative 14:42:45 i'd bring up backwards compatibility again, but i'm afraid i'd get run out of the room :) 14:43:05 It is tough as so many features are getting backported, but we still can't really move forward on non-backportable changes on master 14:43:11 as dprince was referring to 14:43:13 exactly 14:43:22 not to mention actual bugs 14:43:31 "90% of tripleo devs are working on liberty support" 14:43:36 that is the actual problem :( 14:43:43 * dprince has stopped rebasing new code until it settles down 14:43:43 i can't deny it 14:43:49 at least its happening upstream now 14:44:07 derekh: good point, thanks for being positive :) 14:44:33 okay, so we've got one agenda topic left 14:44:36 #topic Specs 14:44:42 anything for Specs this week 14:46:20 The TripleO API thread continues upstream. Probably not worth bringing up here but it seems to be the most debated topic we have ATM with regards to Specs 14:46:48 that has been good reading :) 14:46:52 It is proving pretty contentious 14:47:52 It'd be great if we can figure out a way to step back, take away some of the emotion and solve the actual problem as a community 14:48:02 yes, it is 14:48:16 it seems a bit like we've veered into a flamey discussion over implementation, vs the requirement 14:48:35 FWIW the iterative feedback I've gotten from working w/ some of the UI guys on prototypes has been actually quite positive/constructive 14:49:35 yes, that's valid, but then i'm concerned when i see reponses that we don't expect people to use the API on its own 14:50:16 if we're just solving for the UI, then that is not what i was expecting 14:50:27 so maybe a level set / reset on what we're doing is important 14:50:46 I kinda think seeing a spec from the Mistral point-of-view is important 14:50:47 slagle: FWIW, most of my arguments are based in expressly *not* wanting to deliver and support a "solving for the UI" layer 14:50:49 slagle: people can use either API on its own. When I initially tried it I used the API via mistralclient for example. both solutions have APIs and they can be used outside of our python-tripleoclient UI tooling 14:51:05 I really want it to be something more generally useful, given the overhead of maintaining it 14:51:51 yes, the ability to use it either way is there 14:52:53 tzumainn: I'll post a spec as soon as I can 14:53:12 #topic open discussion 14:53:26 trown: you had a think you wanted to mention...? 14:53:32 thing 14:53:36 I have a couple :) 14:54:40 the first is that there is a testday for RDO Manager this Thurs/Fri, I know folks here are busy with lots of stuff, but anyone wanting to help out would be great 14:54:46 end of that topic 14:55:19 the other thing is more interesting 14:55:39 I started playing with generating docs from CI: https://review.gerrithub.io/#/c/261218/ 14:56:10 that is a POC that really just templates out one bit of RDOCI, but what I wanted to present here was more the idea 14:56:33 of being able to have the same thing that runs in CI templated out to docs as simply as running `tox -e docs` 14:57:11 trown: would we end up with tripleo.sh in the docs ? 14:57:15 trown: the end goal being to view docs during the code review process? 14:57:32 dprince: the end goal being no CI drift from docs 14:58:05 trown: I see, similar to how our devtest was self documenting... 14:58:05 derekh: I think we would need to template tripleo.sh similarly to what I do for RDOCI in that review 14:58:20 trown: ack 14:58:36 trown: cool, I like where you are going 14:58:46 Interesting, similar to what we did for heat, only we generated a script by processing an rst file with the docs in 14:58:47 derekh: so tripleo.sh.j2 becomes a jinja template that can generate tripleo.sh for CI, as well as docs from the same template 14:59:36 great, almost out of time this week. Thanks everyone 14:59:46 it is a pretty immature idea at this point, I just started hacking on it late last night, but wanted to see if there was interest 15:00:18 trown: sounds cool to me 15:00:30 #endmeeting