13:01:33 #startmeeting releaseteam 13:01:34 Meeting started Mon Aug 24 13:01:33 2015 UTC and is due to finish in 60 minutes. The chair is ttx. Information about MeetBot at http://wiki.debian.org/MeetBot. 13:01:35 Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 13:01:37 The meeting name has been set to 'releaseteam' 13:01:59 Had a number of topics ti discuss, with liberty-3 / feature freeze coming up next week 13:02:19 The first one is the Oslo libraries stable branches 13:02:28 following the recent thread 13:02:46 I keep on getting more confused, maybe we changed too many things 13:03:05 We had a plan out of that crazy session outside in YVR 13:03:11 I posted it on the thread 13:03:18 yeah, I'm reading that now 13:03:39 So I'm surprised that we don't need stable branches to achieve that plan now 13:03:51 I don't see dims on line right now 13:04:05 I don't think anyone said we didn't need them? 13:04:16 lifeless suggested we may not need requirements caps 13:05:12 he said "stable branches are orthogonal to constraints IMO. If the only reason for the stable branch is the capped requirements, then I would not make the branch." 13:05:18 because we'll have the constraints system to protect us at first, and we can add caps later for individual libs if we need to 13:05:29 pretty much "we don't need them" 13:05:34 yeah, but that's not the only reason 13:05:59 right, which means we need to take them into account in the constraints checks 13:06:22 as I pointed out later in the thread, landing changes in master becomes a mess if we don't have stable branches, because we have to test against the stable branch as well as master and then remove that testing when we have an incompatible change to make 13:06:56 I think for the constraints what we do is leave requirements in libraries uncapped for good and leave requirements in servers uncapped *for now* 13:07:07 create stable branches for everything (order to be determined) 13:07:16 agreed. I just worry about that "Enable master->stable cross-check" 13:07:29 which is very much to make contrsinats work in a stable-full world 13:07:37 constraints 13:07:37 dhellmann: I think it's just a trade off between managing backports vs. ensuring there are stable jobs on libs 13:07:37 and then as we release new versions of the libraries, test them with constraint changes in the stable branches, and if they don't work introduce a cap instead 13:08:14 sdague: the problem isn't making sure we have stable jobs, it's rushing to remove them when they start breaking our ability to land changes in master because we raise a minimum requirement somewhere 13:08:26 dhellmann: right, gotcha 13:08:31 sdague: which is sort of the reverse of what we have now, where master changes break stable 13:09:00 I think with the constraints system fully in place, we're safe to not have caps until we actually discover an incompatibility somewhere, at which point it's relatively easy to do that 13:09:11 it's not clear we have enough of the constraints work in place, yet, though 13:09:31 dhellmann: regarding "as we release new versions of the libraries, test them with constraint changes in the stable branches, and if they don't work introduce a cap instead" ... does that mean we may end up with stable branches being tested with library versions that are on master ? 13:09:35 the tox stuff, in particular 13:09:57 ttx: we could have that if we explicitly update the constraints to allow it, yes 13:10:00 dhellmann: right, I don't know exactly what lifeless has left 13:10:21 dhellmann: under the current rules, we are supposed to accept any constraint change that passes tests 13:10:38 stable branches would be frozen ? 13:10:40 ttx: fwiw, that's the way this always used to work. The idea of stable branches in libraries is relatively new. 13:11:08 sdague: yeah, he mentioned being blocked on some sort of governance change in his email to update the testing interface, but I think we can do that in parallel if not after the fact 13:11:41 sdague: right, but that means we may release a .Z version on a stable branch and never test with it 13:12:17 I would expect us to be making far fewer changes to the constraints on the stable branch than on master, but I wouldn't consider it frozen 13:12:31 and I would definitely expect us to update it if we need a stable release of a library 13:12:44 so, honestly, there are a lot of moving parts here. I'd suggest instead of everyone trying to keep it in their heads, get some bits written down with diagrams about the scenarios. 13:13:02 given the rate at which we update some of these libs, I don't think we want to tell a deployer the only place they can get a fix for 1.2.x is in 1.15.x 13:13:15 dhellmann: trying to see how that would work with the automated constraints bump suggestions. 13:13:27 we would disable that on stable ? 13:13:45 so, constraints shouldn't be landed in libraries 13:13:46 or we would apply totally different rules on master vs. stable requirements ? 13:13:48 ttx: maybe, that's a good question. We should get lifeless to chime in on this. I agree with sdague we need a good write-up 13:14:37 sdague: we did diagrams and scenarios. It's just that we may not be were we were supposed to be 13:14:49 ttx: url ? 13:15:26 whiteboards in YVR that were posted on the ML at some point 13:15:26 sdague: I think https://doughellmann.com/blog/2015/05/28/openstack-requirements-handling/ 13:15:42 those are the notes from the summit, at least 13:16:48 re-reading them, they're pretty raw 13:16:53 right, those are the notes. My suggestion is to turn them into a more detailed doc 13:17:01 yeah 13:17:24 because I think those notes can be interpretted differently, and some of the tricky specifics need to be fleshed out 13:17:45 mostly for clarity, it's just a lot to keep in mental ram 13:18:24 ttx: do you have time to start that, or should I put it on my list? Where do we want it to live? 13:19:29 dhellmann: my gut feeling is that we built a system that is orthogonal to stable branches (like lifeless said), and putting stable branches back into it is likely to take more than the week we have to figure it out 13:20:00 ttx: ok, I always understood the changes we were making to be on top of stable branches, so I'm surprised to hear that. 13:20:27 dhellmann: in particular, my impression is that the system is designed to work with a single channel of releases 13:20:31 if we don't create stable branches, we need to have test jobs for changes to master versions of all of the libraries running against stable versions of the server code 13:21:34 and if we have that, we need some sort of quick response plan for the day a team is blocked making changes because we've introduced an incompatible dependency in master so that the master version of the lib no longer works with the stable version of the app -- that will happen, it happens all the time 13:21:37 dhellmann: I think we need stable branches. Otherwise you need to branch whenever the library starts to implement changes that don't work with the previous release 13:21:44 right 13:21:55 either to mark that beforehand 13:21:56 ok, I thought you were saying we didn't need them 13:22:05 so, it's not a terrible thing to make libraries have that level of compatibility 13:22:07 easier* 13:22:23 that just means interfaces need a couple cycles to be dropped 13:22:34 dhellmann: no, I'm surprised to discover lifeless thinks we don't need them. 13:23:03 sdague: the issue is not usually with our code (though that does certainly come up), it's with wanting to add a dependency on a new thing (or a new version of a thing) we don't control, and having *that* break the stable branches 13:23:42 stablea branches are just about freezing the state every 6 months so that maintaining it working is easier 13:23:47 dhellmann: ok, so, perhaps documenting some cases of when that happened will make it more clear 13:25:04 dhellmann: ok, I'll start a thread on the stable/release/requirements process, beyond only Oslo 13:25:20 no point in discussing that without Robert 13:25:23 anyway, probably worth figuring out lifeless's plan here 13:25:24 yeah 13:25:25 yeh 13:25:54 Second topic I had is about intermediary-released thingies and RCs 13:26:16 Historically Swift was the only thing we did intermediary-releases for (beyond libs) 13:26:45 We would just push a tag there 13:26:56 but the "coordinated" swift end-of-cycle release still used RCs 13:27:10 we would create a release branch there and tag rc1 13:27:28 To be consistent I think we need to just tag there 13:27:39 the supposedly-final release 13:27:54 and bump Z in case we need to have another "candidate" 13:28:17 in summary... 13:28:43 for projects with intermediary releases, the fnal release candidates are just intermediary releases 13:29:14 and you just do another one on the stable/liberty release branch if it's too broken. 13:29:18 yes, I think that's what I was expecting. release:cycle-with-intermediary projects don't tag betas or RCs, just actual releases. 13:29:33 right. That means we need to communicate that to Swift 13:29:37 since that's a change to them. 13:29:44 I'll talk with notmyname about it 13:30:15 I think they never wanted to have the final release being different than others anyway, but better doublecheck 13:30:16 if he objects, I wouldn't have a problem with making that process change next cycle 13:30:36 dhellmann: it came up when I started updating the rc*.sh scripts 13:31:03 ok 13:31:20 #action ttx to reach out to notmyname about the need for RCs for the "final" swift release of the cycle 13:31:45 #action ttx to start a wider thread about the stable/release/requirements process, beyond only Oslo 13:32:29 dhellmann: my next topics were about stable branch releases and release notes. But there may be other "urgent" topics to discuss before ? 13:33:06 I don't have anything urgent, so let's keep going with your agenda 13:33:41 #topic Stable branch releases from liberty on 13:34:04 Current state of that is well summarized as options in : 13:34:37 http://lists.openstack.org/pipermail/openstack-dev/2015-August/072603.html 13:34:50 Tag-every-commit vs. Time to time tagging 13:35:15 Have any preference ? 13:35:26 At this stage I'm willing to opt for the less disruptive and less work 13:35:36 i.e. "Time to time tagging" 13:35:54 yeah, I think time-to-time makes sense 13:36:03 that can always devolve into every commit, if we need it to 13:36:06 I can see how it fails to reach the original objectives, so I sympathetize to Daviey 13:36:16 but yeah, easier to evolve from there 13:36:18 but I'm also inclined to see how far we can stretch that time between tags 13:36:24 good incremental step 13:36:34 maybe with projects or distros asking for tags when they feel they are needed 13:37:17 Also a good thing to have release automation fully in place and see how much it changes things 13:37:47 yeah 13:37:58 #agreed Release team leaning toward "Time to time tagging" for stable/liberty as a more realistic objective and good incremental step 13:38:14 oh, on stable branches we do post-versioning, right? 13:38:40 I think we currently do pre-versioning 13:39:00 but I don't think that's hard to change there 13:39:08 yeah, we just need to remove the version string from setup.cfg 13:39:36 anything else before we move to stable release notes ? 13:40:15 I'll take that as no 13:40:16 #topic Stable release notes 13:40:22 no, sorry, phone rang 13:40:26 So on this topic there might be more work involved 13:41:30 I like the idea of having the release notes in tree. The scheme lifeless described seemed a bit more complex than necessary 13:41:39 I think the two solutions are: lifeless's YAML directory for semi-continuous release note generation, and a single file in the root of the directory 13:41:41 could we not just have a relnotes subdir under doc/source and let sphinx build the final document for us? 13:42:01 oh, you're suggesting something in between now 13:42:18 yeah, we get the merge protection of lifeless' yaml proposal without having to build any new software 13:42:54 we could even have relnotes/$SERIES/*.rst and just keep the notes published forever 13:43:17 dhellmann: yeah, I was thinking that part (determining which release notes to build) was a bit over the top 13:43:58 dhellmann: are docs generated as part of the tarball creation ? 13:44:08 I think so, but I'm not certain 13:44:39 fungi: do you know if docs are built as part of tarball creation? ^^ 13:44:49 I can check really quick 13:45:25 I don't get them in an sdist 13:45:42 yeah, can't find them in built tarballs on tarballs.o.o either 13:47:04 ok, so that may not be optimal 13:47:34 So... we can still brainstorm solutions there. I guess plan B is to edit a single file (or let whoever triggers the time-to-time release update it before tagging) 13:48:08 so we have a backup plan if we can't come up with something good before stable/liberty is cut 13:48:47 right 13:49:12 another benefit of keeping them in tree is we could encourage reviewers to require them as patches are made 13:49:37 yeah, that's the main argument for it 13:49:40 of course it opens us up to having notes changed for clarity after the fact, and then those changes not being backported, but I think we can live with that 13:50:20 oh, I thought we were doing this because without official stable releases we didn't have anyone doing the gathering 13:50:25 How would you reconcile building release notes continusouly and post-versioning ? We would generate a RELEASENOTES-NEXT.rst that would contain untagged notes ? 13:51:31 if we go with the sphinx idea and group the notes by release series, we can have an N directory until N gets its name, and then that can be renamed. Or we could just use the letters to avoid having URLs change 13:52:06 then release notes would be published live in the docs for the project as soon as the patch lands, and would be included in a discoverable directory afterwards 13:52:35 if we want one big release notes file in the tarball, to complement the file published on docs.openstack.org, we could make pbr build that for us 13:53:17 it could do something with docutils to convert the rst to plain text and concatenate the results together or something 13:53:18 hmm, you need release notes per version, not per series 13:53:29 although you can group them in a single file per series I guess 13:53:35 oh, true 13:53:43 you need to at least write them to a version 13:54:02 hence my question on how to write to the "next" version in postver 13:54:14 we'll likely need some placeholder 13:54:30 I guess this is why lifeless' proposal was more complex :-) 13:54:35 hehe 13:54:59 He is often right, which makes it difficult to constantly disagree with him. 13:55:12 * dhellmann can't believe he has been building release notes management systems for nearly 20 years now 13:56:01 ok, so I should have a side releaseteam-meeting with lifeless on APAC-friendly time (my tomorrow morning) to clarify both things 13:56:15 and I should re-read his proposal 13:56:19 #topic Open discussion 13:57:14 you saw that I submitted all of the release tag changes? 13:57:16 Anything else ? I'm a bit panicked because it feels like we changed the engine to the train, I don't know how it works anymore, the train is full speed ahead and the next train station is next week 13:57:34 yes I saw that. I'll have a deep look soon 13:58:00 I'm a little worried, but I think the only thing we actually have to solve by next week is understanding the order to do the tagging/branching 13:58:03 also, the mechanic of the train leaves on the opposite side of the world 13:58:09 lives* 13:58:13 ttx: http://www.imdb.com/title/tt0477080/ ? :) 13:58:14 the release notes thing we can always do in the stable branches directly, we don't need that in master, yet 13:58:16 and I make funny english mistakes, too 13:58:46 ttx: I read it with a french accent, so I understood what you meant ;-) 13:59:11 what else is inducing panic? 13:59:49 stable branch / RC / cut order. I think we'll fumble it again this time 14:00:14 I'm missing too many pages of that book to be sure. 14:00:42 Can't wait until I reach firefighting stage and not panic anymore 14:00:52 #endmeeting