13:02:48 #startmeeting releaseteam 13:02:49 Meeting started Mon Oct 12 13:02:48 2015 UTC and is due to finish in 60 minutes. The chair is ttx. Information about MeetBot at http://wiki.debian.org/MeetBot. 13:02:50 Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 13:02:53 The meeting name has been set to 'releaseteam' 13:03:05 Last of our Liberty meetings, i'll give you the chair baton for the next ones 13:03:18 sounds good 13:03:18 (probably when back from Tokyo) 13:03:27 I don't have much 13:03:34 https://wiki.openstack.org/wiki/Release_Cycle_Management#Meeting 13:03:51 #topic Liberty release coordination 13:03:59 https://etherpad.openstack.org/p/liberty-final-release 13:04:04 We are on the last line 13:04:07 3 things 13:04:24 1/ Keep on track of things that may trigger a late respin 13:04:45 I'll be in touch today with PTLs that have stable/liberty changes up, just in case 13:05:05 but nothing has been mentioned yet 13:05:15 ok, good 13:05:22 There is https://review.openstack.org/#/c/232917/ 13:05:34 which if accepted would trigger requirements syncs all over 13:05:53 do we still need it? 13:05:53 My current take is to merge it first thing after release 13:05:59 if still needed 13:06:08 was that bumped ? 13:06:23 no, that is still the most current version of requests, I thought it was possible they had fixed it 13:07:07 not fixed yet. I'd say it's not a critical problem unless we have stuff to merge 13:07:24 we /could/ merge it and ignore the syncs 13:07:29 do you know if the distros are aware of the issue? 13:07:35 yeah, that's another approach 13:07:50 I guess at this point the main concern is to communicate that we know this version breaks us in some ways 13:07:55 we already ignore a number of syncs, like translations 13:08:06 We can make a release notes note 13:08:08 but it's not clear to me whether we're really *broken* or just broken because of the images used in the gate 13:08:15 ok 13:08:18 that's what we did to avoid such late blacklisting in the past 13:08:36 * dhellmann nods 13:08:38 maybe flaper87 can shine some light 13:08:58 whether we're really *broken* or just broken because of the images used in the gate 13:09:32 I'm fine approving the requirements blacklist pre-release, if we stand ready to -2 the resulting syncs 13:09:59 since those are often blindly approved 13:10:10 we need to catch them before they are 13:10:12 yeah, do you think we would catch them all before someone approved one? 13:10:27 we probably would, if we can predict when that triggers 13:10:51 release day is this thursday, right? 13:10:55 yes 13:11:03 so let's go through the other last things first 13:11:22 2/ Release notes 13:11:32 A number of people are chasing projects down so that they fill them 13:11:45 good 13:11:49 there is the weird question of scope, since there is no integrated release anymore 13:12:06 We said we'd chase down release:managed ones and make sure those are done 13:12:12 I'll hold off on the reno announcement until after the work on the wiki page has settled down 13:12:17 and anyone else is fine to add 13:12:25 3/ Release day. 13:12:46 On Thursday I consolidate the release pages on Launchpad, which takes nearly forever 13:12:52 and retag 13:12:56 (which doesn't take that long) 13:13:08 so Thursday is basically too late for respins 13:13:20 Wednesday is the very last moment for that 13:13:45 I'll try to get most of the launchpad foo done in the EU morning 13:14:12 I'll say that getting rid of that would remove a lot of work from release day. 13:14:20 when you say "consolidate" what does that mean? 13:14:23 but then I digged that hole 13:14:52 well, it's about moving everything that's been done on milestones to the final release page, by retargeting bugs and blueprints 13:14:57 ah 13:15:06 so that "the release" shows everything that's been done, instead of nothing 13:15:33 ah, right, I forgot you do that because we ended up not doing it for oslo 13:16:03 let me find some pointers 13:16:14 just in case 13:16:26 https://wiki.openstack.org/wiki/ReleaseTeam/How_To_Release#Final_release_2 13:17:15 Anything I'm missing ? 13:17:54 I think that's it. I couldn't tell just from scrollback whether we were waiting for any critical issues other than the requests requirement block 13:17:58 So.. back on the topic of the requests blacklist 13:18:02 * flaper87 reads backlog 13:18:03 it sounds like not? 13:18:10 two options 13:18:42 Plan A is to freeze that until post-release, secretly hoping requests will update their release anyway 13:18:53 Plan B is to approve the requierments change now and -2 the syncs 13:18:54 flaper87: we were trying to tell how much trouble the requests 2.8 release is causing us 13:19:26 Benefit of Plan B is that the requirements (and syncs) will be in, in case we urgently need them at the last minute for a late respin to pass tests 13:19:31 mmh, my current problem is this: https://review.openstack.org/#/c/232462/ 13:19:45 whether plan A can work depends on the nature of the issue -- if it's a packaging problem they may not fix it, because of the vendoring upstream vs. separate packages downstream 13:19:49 That backport for liberty is blocked by this requests issue 13:19:53 Drawback of Plan B is we need to catch the syncs before some trigger-happy PTL approved them 13:19:57 it's probably not critical and we could wait, FWIW 13:20:13 but even in that case we could retag the RC tag rather than HEAD, so it's not a critical issue 13:20:24 flaper87: is the problem related to a change in the behavior of requests? 13:20:35 yes 13:20:42 in both plans we would add a note to the RElease notes saying requests 2.8.0 sucks 13:20:45 There's a GH issue for it 13:20:48 lemme look it up 13:21:06 but, tl;dr: it is not behaving the same way 13:21:08 flaper87: ok, whenever requests comes up I hear about the packaging stuff so I wasn't sure if this was a real code change 13:21:27 flaper87: any other gate (other than python-glanceclient) broken by that ? 13:21:28 to be precise, the vendored urllib3 in requests is not behaving as it should 13:21:52 ttx: yup 13:21:54 2s 13:22:26 https://bugs.launchpad.net/ospurge/+bug/1503768 ospurge 13:22:26 Launchpad bug 1503768 in python-glanceclient "keystoneclient.exceptions.RequestTimeout with requests==2.8.0" [Undecided,New] - Assigned to Flavio Percoco (flaper87) 13:22:26 dhellmann: I think we can do plan B if we know when to catch the syncs. 13:22:52 but the exception in that bug is from keystoneclient so, I'd expect keystoneclient to be broken by this too 13:23:16 fungi: do you know when the requirements syncs are triggered ? Is it on merge or a periodic job ? 13:23:31 on merge 13:23:32 I believe it's periodic 13:23:33 ttx: I agree we can probably catch them. That won't solve flaper87's problem for glanceclient, but it will at least signal to distros that we have a known issue 13:23:37 u, I take that back 13:23:40 fungi: knows better 13:23:59 flaper87: it's just that the merge is periodic 13:24:05 ttx: lol 13:24:09 hah 13:24:20 well, the constraints proposals are periodic 13:24:28 fungi: does it take a while to trigger though, or is it relatively predictable ? 13:24:31 (in an actual, non-joking sense) 13:24:44 dhellmann: if only we knew when the next requests release will go out, I think we could avoid this. 13:25:03 I could approve it early tomorrow morning while the gate is calm and catch the syncs while most people are asleep 13:25:07 flaper87: have you talked with sigmavirus24_awa about that? he's part of that team, isn't he? 13:25:07 to be safe, we should probably skip that versoin but the uncapped requirement should be enough to install the newest and be done with it 13:25:19 ttx: it goes into the post pipeline, so if there's a resource starvation delaying post pipeline commits getting workers for their jobs then it will be however long that is plus something like 30 minutes for the job to run to completion 13:25:25 dhellmann: he's part of the team and I tried to reach out last week but TZs didn't work 13:25:25 or we could even temporarily disable the stable/liberty sync 13:25:35 flaper87: ack 13:25:41 fungi: oh, that is why I thought it's periodic 13:25:53 because when the gate is busy that can take quite a few hours 13:26:19 i should also check that the job is currently running properly. i'll have a look at a recent job log for it 13:26:28 one more reason to approve while gate is calm 13:26:48 dhellmann: so.. plan A or plan B ? 13:26:50 ttx: I do sort of like the idea of disabling the job for now, just to be safe, as long as we remember to turn it back on 13:27:01 * dimsum__ o/ (fashionably late to the party) 13:27:15 ttx: B 13:27:26 good morning, dims 13:27:41 dhellmann: I think the risk profile is low enough to avoid the extra work in disabling/reenabling 13:27:58 #link https://github.com/kennethreitz/requests/issues/2811 13:27:59 I'll approve it first thing tomorrow morning and block the syncs when they come through 13:28:02 That's the GH issue 13:28:03 ttx: yeah, if you do it early tomorrow I think that will be true 13:28:13 deal 13:28:35 flaper87: so the patch is merged but not released, it seems 13:28:47 #topic Open discussion 13:28:51 yup, I just double checked and requests' latest is still 2.8.0 13:28:51 Anything else ? 13:29:01 * ttx is late with slide work 13:29:27 I'm going to spend more time on that between now and thursday, too, since I'm delivering the talk thursday 13:29:52 I'll leave comments with the parts where folks had questions, so we can figure out whether to add more info or leave space for q&a at the end of the talk 13:30:04 fyi, log of most recent requirements update job: 13:30:05 #link http://logs.openstack.org/f8/f82ab92ed8ff1a278391a196a39382fbb4211583/post/propose-requirements-updates/7347d46/console.html.gz 13:30:22 most recent for stable/liberty i mean, of course 13:30:46 fungi: ty 13:31:08 that corresponds to the current branch tip of stable/liberty in the openstack/requirements repo, so barring more recent breakage than thursday when that ran, should be fine 13:31:42 ttx: I was going to send email today or tomorrow to start communicating with ptls for mitaka about planning and process changes. I'll hold off on bringing up reno until after thursday so they don't confuse that with the current release notes process. 13:34:00 ttx: it looks like there are some other library releases, too, so I'll probably handle those today 13:34:15 ack 13:35:23 dhellmann: ttx: ok with sending out a bunch of oslo library releases from master today? https://review.openstack.org/#/c/233326/ 13:35:46 dhellmann: ttx: ok with sending out a bunch of oslo library releases from master today? https://review.openstack.org/#/c/233326/ 13:36:00 dhellmann: ttx: https://review.openstack.org/#/c/232678/ and https://review.openstack.org/#/c/233256/ 13:36:10 dims__: yeah, that should be fine. I'll handle the others 13:36:21 yep 13:36:22 thanks i'll take care of those 3 items 13:37:19 ok, anything lese before we close ? 13:37:22 else* 13:37:27 that's all I have 13:37:32 alright then 13:37:35 #endmeeting