16:00:03 #startmeeting releaseteam 16:00:03 Meeting started Thu Feb 28 16:00:03 2019 UTC and is due to finish in 60 minutes. The chair is smcginnis. Information about MeetBot at http://wiki.debian.org/MeetBot. 16:00:04 Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 16:00:07 The meeting name has been set to 'releaseteam' 16:00:10 Ping list: smcginnis ttx dhellmann diablo_rojo hberaud evrardjp fungi armstrong 16:00:19 #link https://etherpad.openstack.org/p/stein-relmgt-tracking Agenda 16:00:19 o/ 16:00:22 aloha 16:00:33 Getting really far down in that tracking etherpad now. 16:00:51 o/ 16:01:40 o/ 16:01:52 o/ 16:02:10 #topic TripleO release issues 16:02:22 #link https://review.openstack.org/#/c/637857 16:02:30 #link https://review.openstack.org/#/c/637858/ 16:02:40 #link https://review.openstack.org/#/c/638459/ 16:02:40 Tasks update 16:02:51 Oops, a little too much grabbed on that last one. :) 16:03:00 Not sure who added this topic. 16:03:05 o/ 16:03:45 http://logs.openstack.org/57/637857/4/check/openstack-tox-validate/502eeab/job-output.txt.gz#_2019-02-22_10_12_23_715253 16:04:17 Looks like they need a release job defined and they have errors in their readme. 16:04:21 it looks like ttx might have added this? 16:04:42 ttx: This one yours? 16:04:44 see also last comments on https://review.openstack.org/#/c/637857 16:04:50 yes 16:05:10 It looks like this pile of things are blocked 16:05:58 First do fail due to a validation error 16:06:13 (missing release job for instack-undercloud) 16:06:22 but apparently fixing that is not the solution 16:06:30 see https://review.openstack.org/#/c/638459/ 16:07:08 I thought that was actually what was needed. 16:07:14 yeah, me too 16:07:24 Maybe we just need to let Andreas know that? 16:07:40 but sending Juan Antonio falling down the cracks between our teams is not the best we can do 16:07:56 "You have everything set up to release - the release repo check just does not know ;)" 16:08:14 https://github.com/openstack/tripleo-image-elements/blob/stable/queens/zuul.d/layout.yaml#L8 16:08:16 sorry it was kinda my fault to ping then 16:08:32 it's been laying around so I wanted some fresh eyes :) 16:08:33 I didn't think we could do that in-repo due to secrets. 16:08:34 yeah, we require the template so we don't have to know all of the other places jobs can be attached to projects 16:08:50 dhellmann: Can you comment on https://review.openstack.org/#/c/638459/ 16:08:51 maybe our repo check is a bit too restrictive. Since we did have a light agenda I suggested we look into it and give them an answer :) 16:08:57 the issue is that they have listed individual jobs in the release queue so ajaeger doesn't want to add the template 16:09:12 commented 16:09:41 I also restored the patch 16:09:53 So we either need to expand our validation checking for whatever combination folks try to add, or we enforce that they are consistent with how this is done. 16:09:56 I like consistency. 16:10:10 expanding our validation is unfortunately not easy 16:10:14 +2 for consistency 16:10:16 But, they do still need to get their README fixed too. 16:10:17 anybody know the backstory on why they aren't using the template today? 16:10:19 without reimplementing the full Zuul logic for it 16:10:44 like is it just historical baggage or are there jobs in the template they don't want to (or can't) run? 16:10:44 fwiw, this hasn't been a problem since I made the decision to require using the templates at the very beginning of writing the validation 16:10:50 Looks like that patch could be updated to also remove the individual jobs if they are adding the template. 16:11:04 couldn't the freeze job details help here? 16:11:06 there are some cross-dependencies listed there. maybe those are redundant? 16:11:28 Those seem... odd. 16:11:38 yes, that patch should drop the individual job def 16:12:53 i've lost track apparently. where are you looking at cross-dependencies? 16:13:10 https://review.openstack.org/#/c/638459/1/zuul.d/projects.yaml line 4523 and below 16:13:12 "these have to dupe the publish-to-pypi jobs 16:13:14 # because of the branch designation for the release job 16:13:16 " 16:13:40 not "cross" but dependencies 16:13:54 oh, i didn't think to look at context further down below the diff 16:14:12 ajaeger's objection to the patch seemed to be that the template was redundant 16:14:12 so the duplication was intentional 16:14:48 Which is valid. But our validation enforces that convention. 16:15:40 I suppose this wouldn't be an issue if the master branch wasn't closed? 16:15:52 I think that's the only gotcha in all that. 16:15:58 The issue here is that the template does not really match what they have. So we should add another template to match what they have, and add that to the validation test 16:16:25 why does it matter that they don't run jobs on master if they're not accepting patches for master anyway? 16:16:30 some publish-tripleo template mabe 16:16:33 ttx: in what way does it not match? 16:16:43 dhellmann: the branch exclusions 16:16:55 fungi: Yeah, there won't be any releases on master, so does it really matter? 16:17:02 I don't see any branch exclusions in this patch, are those in the template? 16:17:16 dhellmann: the lines just after the diff 16:17:41 those are the check jobs 16:17:42 I don't see any exclusions in the release jobs 16:17:45 fungi: maybe the check/gate could stay 16:18:00 yeah, those are the ones i was talking about, not the release jobs 16:18:19 oh, I see, those are release test jobs 16:18:27 just wondering why they even need to exclude master if they're not accepting changes for master 16:18:37 I suppose they had to exclude the job from master in order to close it and change the readme in the first place 16:18:41 they could probably remove all of that now 16:18:52 right, cleanup 16:18:56 wfm 16:19:12 I am confused there 16:19:36 ok, so remove all jobs and replace them with the template 16:19:38 This is getting into zuul configuration, so confusion is normal. :) 16:19:48 s/normal/required/ 16:19:53 I would hope not to me :p 16:19:57 here's what I think happened: 16:20:06 1. one day, a long time ago, this was a normal project. 16:20:08 but they don't want to run those jobs for master, which makes sense? 16:20:12 evrardjp: the complexity seems to be that they've retired the instack-undercloud repo, but because it wasn't retired until the current cycle they had to leave testing in place to accept fixes on stable branches 16:20:17 2. they closed master to new development, but kept the stable branches open 16:20:27 3. to do that, they had to disable some jobs on master 16:20:37 4. they did that by listing them all out explicitly instead of using the template 16:20:47 5. now the release validation is complaining because the template is not there 16:20:52 because they would need to override per branch in step 4 16:21:00 right 16:21:02 right 16:21:19 i agree with dhellmann's probable timeline there 16:21:27 me too, that's my understanding 16:21:36 I don't understand how cleaning would help :p 16:21:38 ok, should we propose the fix? Or go to write a lengthy explanation and expect someone else to push it? 16:21:39 So since they are not accepting changes to master, now that they've gotten past that final merge to master there is no need to have special handling. 16:21:41 the fix is to clean up the explicit job listings and apply the template 16:21:42 the jobs will run on master and fail, but we don't care 16:21:49 we're basically missing a step after retiring the master branch to put the jobs back the way they were 16:21:50 evrardjp : because it removes ajeager's objection to having redundant information in the config 16:21:57 smcginnis: oh ok 16:22:01 ttx: perhaps we should do it 16:22:05 ++ 16:22:10 This is an odd case where a repo was retired but they still care about stable. 16:22:10 I can push it if we agree 16:22:15 ttx: ++ 16:22:21 thanks ttx 16:22:24 ttx: That would be great. 16:22:29 (I'll just remove lines 4506 and below) 16:22:40 that will do it right 16:22:42 ttx: perhaps with a nice big comment in there with the template explaining why it's ok 16:22:45 yes 16:23:02 I'll explain in commit message 16:23:14 I still believe this is not the ultimate right way, but I am not sure we have the right tools now 16:23:39 evrardjp: What is you suggestion? 16:23:41 what would you do instead? 16:23:42 but hey, we all agree :) 16:24:07 I have nothing for now, but I would think changing the validation to use the future zuul's freeze api would help 16:24:31 Future zuul fixes everything. :) 16:24:32 because we could freeze the job info per branch, so it means for the release we are looking for we could know or not if that job exists 16:24:44 perhaps. the point of using the templates was to enforce an *interface* for releasing without enforcing details that tend to change a lot 16:24:58 fair :) 16:25:05 i.e., we added test jobs to that template this cycle 16:25:08 in any way, that's only about the future :D 16:25:20 fungi: should I keep a noop in check nd gate pipelines? 16:25:21 evrardjp: i might agree with you if we even had consensus on the patch series for those zuul features yet 16:25:23 sorry to digress here 16:25:32 ttx: nope, the noop job can go away 16:25:32 fungi: :D 16:25:34 I see they are in the process of merging a README fix, so the other issue should be resolved shortly too. 16:25:41 agreed 16:26:00 Any more on these? 16:26:22 #topic Tasks update 16:26:22 if they're adjusting the readme on master, then we shouldn't remove the jobs in 638459 until that's done 16:26:37 I hope they're only adjusting it in the releasable branches? 16:26:41 fungi: No, master and stable/rocky are done. They just hadn't fixed queens. 16:26:45 okay, in that case it's fine 16:27:15 diablo_rojo: Did you follow up with QA? 16:27:24 smcginnis, I did 16:27:37 sounded like he had it handled and didn't need anything from us 16:27:38 Done @ https://review.openstack.org/#/c/638459/ 16:27:55 diablo_rojo: Excellent, thanks. 16:27:58 And thanks ttx 16:28:02 smcginnis, no problem :) 16:28:12 fungi: Any updates on signing key? 16:29:18 The other task is for generating release requests for any missed c-w-i libs. 16:29:24 #link https://sks-keyservers.net/pks/lookup?op=vindex&search=0xcdc08088c3cb45a9be08332b2354069e5b504663&fingerprint=on OpenStack Infra (Train Cycle) 16:29:27 I can take that unless someone else want it. 16:29:42 fungi: Great, thanks! 16:29:46 i'm waiting for it to appear on the keyservers i'm pulling from so i can attest to it 16:30:11 so hopefully later today you'll see a signature from me on it there 16:30:19 but it's already signed by the stein cycle key 16:30:25 fungi: wondering why we still need the queue defined in https://review.openstack.org/#/c/638459/ but hey I'll add it 16:31:33 No one clamoring for doing generating the release patches, so I'll do it tomorrow once we're past the deadline and see what's left. 16:31:48 ttx: that's so the tripleo jobs all share a named change queue 16:31:54 ah ok 16:32:33 Looking ahead to next week's tasks, same thing for generating the client libs that are missed. 16:32:42 basically anything running those (so any changes for instack-undercloud) will end up in the tripleo queue even if they don't share jobs with other projects in that queue 16:33:01 I've put prometheanfire down in there as a reminder for requirements freeze. 16:33:06 smcginnis : perhaps we can sign up one of our new team members for that task next week, since they have more notice 16:33:19 dhellmann: I like that idea. :) 16:33:40 * dhellmann looks meaningfully at evrardjp and diablo_rojo 16:34:09 Ha ha so this is just generating another list? 16:34:15 It's a matter of listing what deliverables of --type=client-lib have not been released yet and generating release patches for any that have any commits. 16:34:30 right, not just a list but creating the patches to trigger the missing releases 16:34:31 diablo_rojo: and updating the docs I would say 16:34:36 Actually, at this point any regardless of commits as we need a stable/stein branchign point. 16:34:51 there are scripts to do it all, I think, and if not we have the parts to build the scripts 16:35:00 evrardjp, maybe we can work on this together again? 16:35:08 I would be fine to sync with diablo_rojo on that 16:35:09 yeah 16:35:09 The parts are there, but I never had time to start to automate it. 16:35:10 So we can fumble through it together? 16:35:28 I think we can figure that out. 16:35:29 fine for me 16:35:46 At this point it's ok to approve stein branch creations for libraries, right? 16:35:48 we'll ping for validation before sending a mass change 16:35:51 Cool, thanks. I've added your nicks to the task in the etherpad. 16:35:52 we'll help if you run into trouble, too 16:35:54 thank you! 16:36:09 ttx: it is, although I've been reminding folks that they can also wait to rc1 16:36:14 It will be a good exercise. 16:36:23 our docs say "allow ... but do not require" branches this week 16:36:39 If they're pretty sure they are done, no reason not to branch. 16:36:54 yeah, it's just a trade-off about potentially having to backport changes to fix bugs 16:37:01 But if there's some concern there might still be an important bug fix, branching just causes extra work at this point. 16:37:06 Since deadline is this week and we prefer not to release libs on Fridays... 16:37:06 right 16:37:40 Last task on the list is to run the aclissues checker. 16:37:41 the constraints list protects us from bad releases going into the gate 16:37:42 at least mostly 16:38:07 I can do that, unless someone else wants to try that 16:38:47 that feels like another case where it would be good to have someone "new" do it to verify the instructions 16:38:50 Anyone want to walk through with ttx as he does it to learn? Like maybe someone in a similar timezone? 16:39:01 :) 16:39:01 (subtle hint) 16:39:02 Wow that was targetted 16:39:08 hehe 16:39:11 I don't know, where is diablo_rojo this week? 16:39:20 I am fine with doing a session with ttx 16:39:24 I'm jetlagges so I'm currently UTC-3 16:39:26 dhellmann, physically? or mentall? 16:39:29 I think it's simply running a shell script 16:39:31 *mentally 16:39:32 iirc 16:39:42 diablo_rojo : temporally 16:39:56 evrardjp: Yeah, not a complicated task, but a few things to know. 16:39:57 dhellmann, in between waking up and going to take a nap 16:40:06 +1 for naps 16:40:33 evrardjp, we could add that to the list too and verify against each other 16:40:40 OK, I've put down ttx and evrardjp for that last task. 16:40:45 since we will both need to know how to do all these things anyway 16:40:48 Oh 16:40:51 * diablo_rojo shuts up 16:40:58 diablo_rojo: If you can join too, that's great. 16:41:16 smcginnis, might be a good idea 16:41:18 the more the merrier 16:41:41 Just trying to have a good transfer of knowledge where we can. 16:41:49 ++ 16:42:11 The other tasks in the process doc are reminders that I've put in the countdown email. 16:42:25 Just need to format them into actual content. 16:42:49 That should be it for tasks. 16:42:53 #topic Open 16:43:03 Anything else we should discuss in-meeting? 16:43:37 OK, we can get back to work then. 16:43:40 Thanks everyone! 16:43:41 there was some discussion of forum/ptg slots, how did that end up? 16:43:47 Oh, right. 16:44:13 bnemec: Had a topic submitted for the forum that will be a good time to try to get release change feedback from PTLs. 16:44:22 thanks smcginnis! 16:44:26 Anyone have a link for that? 16:44:39 I haven't actually submitted it yet. 16:45:43 OK. Let me know if you need anything from us, otherwise we can watch for the session and plan to be there. 16:46:03 Yeah, I was kind of waiting to submit it to see if there was interest. 16:46:08 Since there is, I'll go ahead. 16:46:13 ++ 16:46:53 OK, I guess that's it. Thanks everyone for participating in the release team. 16:46:59 #endmeeting