14:00:07 <dhellmann> #startmeeting releaseteam
14:00:08 <openstack> Meeting started Fri Jun 24 14:00:07 2016 UTC and is due to finish in 60 minutes.  The chair is dhellmann. Information about MeetBot at http://wiki.debian.org/MeetBot.
14:00:09 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
14:00:12 <openstack> The meeting name has been set to 'releaseteam'
14:00:18 <dhellmann> courtesy ping: ttx, dims, lifeless, tonyb, stevemar, fungi
14:00:25 <fungi> yuppp
14:00:32 <dhellmann> our agenda is under R-15 in https://etherpad.openstack.org/p/newton-relmgt-tracking
14:01:31 * dhellmann suspects he has trained ttx to expect this meeting to start late
14:01:43 <dhellmann> #topic automation update
14:01:44 <ttx> o/
14:01:56 <dhellmann> fungi, how're things looking?
14:02:02 <fungi> let's see
14:02:42 <fungi> #link https://review.openstack.org/333065 Add a node for artifact signing jobs
14:02:55 <fungi> that looks ready to merge (along with a cleanup it depends-on)
14:03:07 <dhellmann> \o/
14:03:36 <fungi> i've got the signing subkey distribution automation worked out and tested, next up is writing some more detailed documentation on generating/rotating/signing/revoking keys each cycle
14:04:03 <dhellmann> #link https://review.openstack.org/#/c/332447/ import the release scripts we need to run on the secure nodes
14:04:04 <fungi> though there's already some basic prose documentation in there about the intended process, just nothing cut-n-paste worthy yet
14:04:17 <dhellmann> k
14:04:35 <dhellmann> when 333065 lands, are we ready to start writing the job definition to do the tagging?
14:04:44 <fungi> but as soon as 333065 merges i can build the server in a matter of minutes
14:04:53 <fungi> pretty much, yep
14:04:57 <dhellmann> excellent
14:05:15 <dhellmann> I'll be at the TC training thing next week, but I'll plan to start working on that job the following week
14:05:27 <fungi> i'll want to do a couple final pre-flight checks on the server to make sure gpg --sign operations work as expected without having to tweak anything
14:05:42 <dhellmann> sure
14:05:49 <fungi> but i have confidence my prior testing will be consistent
14:05:58 <dhellmann> and the first version of the job will only actually do work for the openstack/release-test repo so we can get the kinks worked out
14:06:07 <fungi> perfect!
14:06:13 <dhellmann> we'll continue tagging everything else by hand in the mean time
14:06:14 <fungi> i didn't realize you had that repo
14:06:27 <dhellmann> yeah, we set that up late last cycle specifically for testing some of these scripts
14:07:09 <fungi> the other addition i'll want to make to that manifest in a folow-up patch (simple enough) is to add the gerrit ssh keys for the "proposal bot" account, or we can create a separate account specifically for tags
14:07:37 <fungi> but i wanted to make sure the gpg bits were working correctly before i went further on the puppet end
14:07:47 <dhellmann> makes sense
14:08:01 <dhellmann> and I can see pros/cons to setting up a separate account
14:08:06 <fungi> same
14:08:23 <dhellmann> should we work through that now, or is that a discussion for later?
14:08:52 <fungi> we can try to hack through to a decision quickly now if it won't derail your planned meeting agenda
14:09:06 <dhellmann> let's take 5 min
14:09:35 <dhellmann> aside from having to manage another set of keys and another identity in gerrit, what's the biggest con to setting up a "release bot" account?
14:09:41 <fungi> i think the main reason for a separate account is because the proposal bot currently has no special privileges, just "brand recognition"
14:10:15 <fungi> biggest con is probably having to make sure places we account for the proposal bot as a non-human actor would need updating for the second account
14:10:34 <dhellmann> in scripting? or in acls?
14:10:39 <fungi> for example, summit invites/electoral rolls, people's custom mail filters, whatever
14:11:09 <dhellmann> true
14:11:19 <fungi> if we want the proposal bot to push tags, it needs to be in a group which has push-signed-tag access to every release-managed repo
14:11:30 <dhellmann> oh, here's another: the release.sh script proposes a change to the requirements repo to update it for new versions of libs
14:11:38 <dhellmann> so that job is both tagging and proposing
14:12:17 * stevemar sneaks in
14:12:40 <fungi> true. for that case we could either 1) have the proposals associated with that use the tag bot account, or 2) write the job to use two accounts (one for the tag, one for the change)
14:13:07 <dhellmann> having them use the tag bot account seems reasonable
14:13:30 <fungi> the main risk is that the proposal bot account is used by lots of other jobs which don't need elevated privs, so granting it tag access to every repo is an increase in our risk profile
14:13:31 <dhellmann> otoh, if setting up a separate account is going to delay implementation I may be more inclined to just use the existing account
14:13:59 <dhellmann> yeah, the security side of it was one reason I was initially leaning toward having a new account
14:14:03 <dhellmann> well, that and branding
14:14:04 <fungi> it wouldn't really delay implementation. i can create additional gerrit accounts quite quickly/easily
14:14:09 <dhellmann> ok, cool
14:14:28 <fungi> the brand recognition is the only remaining sticking point really
14:14:31 <dhellmann> and those other things like summit invitations and elections we can deal with as they come up?
14:14:55 <fungi> right, those two tasks are handled by one script, and it's already extensible to allow us to specify a list of account ids to ignore
14:14:59 <fungi> just need to add one more to that list
14:15:03 <dhellmann> ok, so not that big of a deal
14:15:25 <dhellmann> so I think I'm on the side of a new account, unless there's a significant blocker for some reason.
14:15:57 <dhellmann> as much as I'd love to come up with a clever name, something like "OpenStack Release Bot" is probably better
14:16:04 <fungi> right, so it comes down to elevated privs for proposal bot plus tags aren't really being "proposed," vs people might wonder who this is pushing tags and what these update proposals are coming from this new account
14:16:35 <fungi> also worth noting, the signing key being used is (currently) named like "OpenStack Infra (Newton Cycle) <infra-root@openstack.org>
14:16:37 <fungi> "
14:16:53 <fungi> #link http://p80.pool.sks-keyservers.net/pks/lookup?op=vindex&search=0x64DBB05ACC5E7C28&fingerprint=on&exact=on newton cycle artifact signing key
14:16:54 <mordred> \o/
14:17:09 <dhellmann> ok, we should get the release team to sign that
14:17:48 <fungi> yep, also i want it signed by a majority of infra-root sysadmins before we start using it seriously
14:17:54 <dhellmann> yep
14:17:58 <dhellmann> ok, I think that's settled?
14:18:06 <fungi> i just need to write up the instructions on how they can validate the key fingerprint locally on our management bastion
14:18:17 <fungi> (part of the instructions i'm already working on)
14:18:31 <dhellmann> should we wait for that, too?
14:18:40 <fungi> not for testing, no
14:19:21 <dhellmann> I meant for signing
14:19:28 <fungi> so anyway, i guess the plan is to make a separate gerrit account "OpenStack Release Bot <infra-root@openstack.org>" and push tags and the associated proposals for release updates with it
14:19:51 <dhellmann> yes, that seems best
14:20:05 <dhellmann> #agreed requirements team spin out (dims)
14:20:06 <fungi> dhellmann: you might want to hold off havign the release team sign it until we have a bunch of infra-root signatures published for it, just so you can be comfortable signing. since you don't have an out-of-band mechanism to verify the key
14:20:08 <dhellmann> #undo
14:20:09 <openstack> Removing item from minutes: <ircmeeting.items.Agreed object at 0x7fc2af0cb1d0>
14:20:22 <dhellmann> #agreed make a separate gerrit account "OpenStack Release Bot <infra-root@openstack.org>" and push tags and the associated proposals for release updates with it
14:20:31 <dhellmann> fungi : good point
14:20:45 <fungi> that's all i have
14:20:50 <dhellmann> cool, thanks
14:20:53 <fungi> sorry to suck up a big chunk of the meeting
14:21:13 <dhellmann> oh, no worries, this is our major initiative this cycle so it's good to spend the time on it
14:21:15 <dhellmann> #topic requirements team spin out (dims)
14:21:29 <dhellmann> I didn't see dims raise his hand as present
14:21:51 <ttx> It feels like this is well in progress
14:22:05 <dhellmann> I know he had signed up a bunch of new folks for the review team, and was working on cajoling some of them to run for ptl
14:22:13 <dhellmann> I don't think it's blocked on anything at the moment
14:22:21 <dhellmann> #topic ACL dance process final review (ttx)
14:22:56 <ttx> I summarized the plan in the etherpad
14:23:07 <ttx> let me fetch that
14:23:23 <dhellmann> #link https://etherpad.openstack.org/p/newton-relmgt-plan
14:23:28 <ttx> https://etherpad.openstack.org/p/newton-relmgt-plan under Final Release Process Improvements
14:23:34 <ttx> item 9
14:23:51 <dhellmann> how much of that do you anticipate being able to script?
14:24:02 <ttx> dhellmann: see actions
14:24:15 <ttx> ideally everything
14:24:20 <dhellmann> k
14:24:30 <ttx> but we may do eth ACL patches be hand this time around
14:24:35 <ttx> by*
14:24:42 <dhellmann> ok
14:24:54 <ttx> i'll try to cover all those step
14:24:55 <ttx> s
14:25:23 <dhellmann> if we're not going to automate it to start, good instructions will help us build a script next cycle
14:25:38 <ttx> dhellmann: do you agree with the action plan ?
14:25:44 <dhellmann> yes, this plan looks good
14:25:59 <dhellmann> this is the sort of thing that I was thinking we'd want in a release manager's guide
14:26:18 <dhellmann> so when we start that, we can add the instructions to it
14:26:32 * dhellmann notes that's on his todo list still
14:26:44 <dhellmann> anything else on this?
14:27:00 <ttx> no, copying the action list to my todo
14:27:08 <dhellmann> ok, moving on then
14:27:09 <dhellmann> #topic Release model changes
14:27:15 <dhellmann> #link aodh https://review.openstack.org/#/c/331212/
14:27:22 <dhellmann> #link searchlight-ui https://review.openstack.org/#/c/333420/
14:27:34 <dhellmann> both of these make sense, and I've voted +1 on them
14:27:54 <ttx> yep so the first one makes aodh independent
14:28:15 <ttx> or just the client?
14:28:21 <ttx> oh, just the client
14:28:53 <ttx> dhellmann: IIRC there was some issue with making clients independent
14:29:02 <dhellmann> oh, hmm, I didn't look at that as closely as I should, I thought it was the whole project
14:29:12 <dhellmann> yeah, I don't think we want to do this
14:30:01 <fungi> what's the issue with making clients independent when servers are managed?
14:30:15 <fungi> just curious
14:30:21 <ttx> fungi: trying to remember
14:30:24 <fungi> heh
14:30:50 <dhellmann> they want to not create stable branches, which means we have to allow new versions from master into the server stable branch dependency list
14:30:51 <ttx> basically for oslo libs and clients, independent makes sense. yet we don't make them independent, because...
14:31:55 <fungi> oh, got it
14:31:58 <dhellmann> and that means that the new master release of the client, which may try to raise minimum versions of other dependencies, may cause issues in the stable branch test environments
14:32:00 <ttx> fungi: that's how it was and we changed it
14:32:04 <fungi> yeah, that becomes a mess
14:32:18 <dhellmann> now I think the telemetry team has been removing themselves from the requirements sync list, but I don't know if that's the case here
14:32:32 <fungi> i missed that release:independent meant that they _wouldn't_ have stable branches
14:32:40 <dhellmann> but it does mean that at some point we have to cap or exclude new versions
14:32:47 <ttx> fungi: if you do, that means you're with-intermediary really
14:32:51 <dhellmann> fungi : they can, but this patch says they don't plan to (that's the justification for the change)
14:32:57 <dhellmann> right
14:33:03 <fungi> right
14:33:23 <fungi> sorry to derail with questions
14:33:24 <ttx> dhellmann: I'll let you argue that one, my week is almost over
14:33:30 <ttx> fungi: no they help
14:33:32 <dhellmann> yeah, I'll post on the review after the meeting
14:33:34 <dhellmann> jd__ : ^^
14:33:46 <ttx> the other one is no brainer
14:33:47 <dhellmann> the searchlight-ui change is less controversial
14:33:49 <dhellmann> yeah
14:33:57 <dims> ttx : dhellmann : taking delivery of a sofa :)
14:34:01 <fungi> basically it might work _if_ they're not used by any other projects and we choose not to add them to global reqs in the future
14:34:01 <dhellmann> so under the new rules I think it's safe for you to approve that
14:34:03 <dims> so not really here :)
14:34:10 <ttx> yep
14:34:13 <dhellmann> dims : ack, np
14:34:37 <dhellmann> fungi : I don't think that applies to aodh
14:34:50 <dhellmann> but yeah, in general, a completely independent thing can do this
14:35:06 <jd__> fungi: that works if we don't have crazy requirements and if stable/* things just cap the client if needed
14:35:08 <jd__> IIUC
14:35:21 <fungi> and right, aodhclient is still in the global requirements list right now
14:35:25 <jd__> aodhclient is not widely used but it might be used by Heat at some point
14:36:14 <fungi> jd__: well, the challenge becomes how to apply security updates to a version of aodh used (and capped) by stable branch projects if there are any, without making stable branches for aodhclient
14:36:24 <jd__> fungi: right
14:36:38 <jd__> does that work if I said we decided not to implement security flaws?
14:36:45 <fungi> and there's no real control over that other than taking it out of global requirements and making sure it doesn't get added back
14:36:55 <dhellmann> yeah, I think we need aodhclient to go ahead and continue to create stable branches, even if you leave them largely unmaintained
14:37:24 <dhellmann> if you have no backports, that's no big deal
14:37:31 <dhellmann> but if you do, and there's no branch, that's all sorts of trouble
14:37:54 * jd__ nods
14:37:57 <fungi> and having it in global requirements means other projects _can_ depend on it without you necessarily even finding out
14:38:02 <dhellmann> right
14:38:18 <jd__> yeah that sounds fair enough
14:38:24 <fungi> so we don't know we've gotten into a sticky situation until it's too late and we have to try hacky workarounds
14:38:48 <jd__> we were mostly worried that people would keep on stable/* branch of aodhclient and except us to backport fixes and all, and we'll never do that unfortunately (unless we become very rich)
14:38:58 <fungi> (granted, our requirements management is already a pile of hacky workarounds, just not as bad as the ones we might have to fall back on)
14:39:13 <dhellmann> jd__ : it's not uncommon to have very inactive client stable branches
14:39:25 <fungi> i cite swift
14:39:31 <jd__> good to know
14:39:47 <fungi> in fact, most of the clients have very inactive stable branches
14:39:49 <dhellmann> fungi : I'd have to look, but I think most of them are pretty inactive
14:40:00 <dhellmann> yeah, we don't get a lot of call for stable client releases
14:40:10 <fungi> yeah, i wouldn't be surprised if the majority still have the branch tip as a release tag from before they branches
14:40:13 <fungi> branched
14:40:28 <jd__> cool, I guess I'll just abandon the patch :)
14:40:34 <dhellmann> so this is just a safety measure for when we need it, but jd__ I don't think you need to worry about downstream expectations of a lot of maintenance on your part
14:40:38 <dhellmann> ok, cool
14:40:44 * jd__ nods
14:40:48 <dhellmann> thanks!
14:41:01 * dhellmann makes a note to document the reasoning for this *somewhere*
14:41:09 <jd__> that'd be a good idea indeed :)
14:41:26 <ttx> dhellmann: PTG maybe in the section about choosing models ?
14:41:31 <dhellmann> ttx: good thinking
14:41:54 <dhellmann> ok, moving on
14:41:57 <dhellmann> #topic moving release tools
14:42:07 <dhellmann> #link remove from release-tools https://review.openstack.org/#/c/332448/
14:42:23 <dhellmann> the patch to add the scripts to project-config has already merged, so it should be safe to remove them from release-tools
14:42:38 <ttx> yeah, I'm resisting the move because we are still supposed to run the scripts by hand
14:42:51 <dhellmann> yes, but I want to make sure we aren't maintaining 2 copies
14:43:03 <ttx> so we'd run them from project-config ?
14:43:09 <dhellmann> and I also want to make sure the new copies have all of their dependencies set correctly
14:43:17 <fungi> add an extra step to the manual process to clone project-config and copy/link there?
14:43:24 <dhellmann> ttx: yes, I've done that for one tag this week and it seemed to work fine, but you  know how many edge cases there are
14:43:53 <ttx> dhellmann: hmm
14:44:18 <ttx> how are we getting the right deps in venv there.. let me check
14:44:23 <dhellmann> fungi : it should only be necessary to check it out once, as long as that repo is dept up to date
14:44:37 <dhellmann> ttx: there's a requirements.txt file in the jenkins/scripts/release-tools directory
14:44:57 <dhellmann> the scripts don't create their own virtualenv, because when they run on the secure node they need to use system packages
14:44:59 <ttx> ah.
14:45:05 <dhellmann> I'll need to get help setting that up when I define the job
14:45:47 <ttx> dhellmann: maybe update the instructions at the top of the README then
14:45:54 <dhellmann> so theoretically you *should* be able to do the same thing you do now
14:45:56 <dhellmann> good idea
14:46:34 <ttx> just need to create an additional venv under that project-config subdir
14:46:53 <dhellmann> ttx: I'm ok holding off on deleting the files until we get back from ann arbor next week, if you'd prefer that
14:46:57 <fungi> right, we don't want to be pip installing anything on that machine if we can help it
14:47:08 <fungi> even in a venv
14:47:26 <ttx> dhellmann: I'm fine approving the move if the instructions are clear enough for me to be able to survive it
14:47:29 <dhellmann> yeah, I'm fairly sure we established that all of these things are available in system packages, so we just need to use pip for manual runs
14:47:30 <fungi> (well, more that we don't want to be _using_ pip-installed anything on that machine with access to the private key)
14:47:37 <dhellmann> ttx: ok, I'll work on that
14:48:15 <dhellmann> fungi : yep
14:48:30 <dhellmann> we have a couple of other quick items
14:48:32 <fungi> reduces the number of parties we need to trust with the ability to backdoor us to our cloud provider, our distro, and our team
14:48:58 <dhellmann> when the time comes, I'll just need a pointer to an example of how to declare those dependencies properly. I guess using bindep?
14:49:53 <fungi> for this we're using puppet, but we can just put a list of package names into the puppet manifest
14:50:03 <dhellmann> sounds good
14:50:07 <dhellmann> #topic stop merging tags
14:50:12 <fungi> i'm happy to add them as long as you give me a list of the names of teh packages on ubuntu trusty
14:50:13 <dhellmann> #link https://review.openstack.org/#/c/333421/
14:50:33 <dhellmann> fungi : ok, I'll work that list out when I sit down to write the job
14:50:57 <dhellmann> so this is the project-config patch to remove the job for merging tags from stable branches into master
14:51:00 <fungi> so we have consensus on 333421?
14:51:15 <dhellmann> That was my impression of the ML discussion, yes.
14:51:58 <fungi> since mordred and ttx were the original drivers for that being added, and have both expressed consent, and i don't see any dissent, i'm happy to merge it now or whenever you want it to land
14:52:12 <dhellmann> now seems as good a time as any :-)
14:52:26 <fungi> fire in the hole
14:52:32 <dhellmann> we definitely want it before we start doing final tags this cycle
14:52:34 <dhellmann> cool
14:52:39 <dhellmann> one more item
14:52:45 <dhellmann> #topic pbr / reno integration
14:52:49 <dhellmann> #link https://review.openstack.org/#/c/330814/
14:53:07 <dhellmann> there's something weird going on with the integration job there, and I need to look into it
14:53:15 <dhellmann> but reviews are welcome in the mean time
14:53:50 <dhellmann> and that's it for the formal agenda
14:53:53 <dhellmann> #topic open discussion
14:54:24 <dhellmann> note for the record: all 3 of the release team are traveling next week
14:54:43 <dhellmann> I included that in the countdown email yesterday
14:55:13 <dhellmann> I'll try to do some releases monday morning before going to the airport, and I don't know what the break schedule for the leadership training thing is going to be like
14:55:52 <dhellmann> if there's nothing else we can stop a few minutes early
14:56:00 <fungi> i got nothin'
14:56:37 <dhellmann> ok, good time before my next meeting :-)
14:56:39 <dhellmann> thanks everyone!
14:56:51 <dhellmann> #endmeeting