19:02:09 <clarkb> #startmeeting infra
19:02:10 <openstack> Meeting started Tue Mar  3 19:02:09 2020 UTC and is due to finish in 60 minutes.  The chair is clarkb. Information about MeetBot at http://wiki.debian.org/MeetBot.
19:02:11 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
19:02:14 <openstack> The meeting name has been set to 'infra'
19:02:33 <clarkb> #link http://lists.openstack.org/pipermail/openstack-infra/2020-March/006605.html Our Agenda
19:03:01 <clarkb> #topic Announcements
19:03:28 <clarkb> fungi is largely afk this week as are ttx. They have meeting topics that we'll probably go through quickly as a result
19:03:34 <clarkb> #topic Actions from last meeting
19:03:47 <clarkb> #link http://eavesdrop.openstack.org/meetings/infra/2020/infra.2020-02-25-19.01.txt minutes from last meeting
19:04:07 <clarkb> We have one action from the last meeting. ttx was to write an xwiki spec. This happend and we'll talk about it in the next section of the meeting
19:04:09 <clarkb> #topic Specs
19:04:35 <clarkb> we'll start with the xwiki spec.
19:04:40 <clarkb> #link https://review.opendev.org/#/c/710057/ xwiki for wikis
19:05:05 <clarkb> This hasn't gotten any reviews yet, but there was good discussion on IRC (in the meeting and out) last week. If you've got thoughts on this your reviews are much appreciated
19:05:13 <clarkb> frickler and corvus in particular seemed to have some good initial thoughts
19:05:35 <clarkb> Next is the cleanup of python tooling on our CI images.
19:05:39 <clarkb> #link https://review.opendev.org/#/c/709579/ Cleaning up python dev tools on our CI images.
19:05:57 <clarkb> infra-root I think this one is ready to go up for approval
19:06:24 <clarkb> is there any objection to that? many of you have already reviewd it. Looks like we are missing fungi though.
19:06:35 * diablo_rojo sneaks into the back of the room
19:06:37 <clarkb> maybe hold off until fungi can ack or nack?
19:06:57 <clarkb> or do you think we have had sufficient input? ianw mordred you've both been involved in this one so your thoughts on it would be great
19:07:09 <clarkb> (as far as proceeding, I Think you've both acked the review)
19:08:45 <clarkb> hrm we may not have quorum here today (yet). I'll wait a couple more minutes to see if others are able to drop in
19:08:58 <ianw> personally it's probably sufficient, as we can stage it in without affecting prodcution
19:09:12 <clarkb> ianw: thats a good point.
19:09:30 <corvus> i'm here, just quiet
19:09:32 <fungi> yeah, sorry, i can try to take a look in a bit
19:09:57 <clarkb> ok why don't we say end of clarkb business thursday? that should give fungi time to fit review into travel schedule
19:10:18 <clarkb> I'm reluctant to prolong this as we just had another virtualenv issue this week
19:10:33 <clarkb> I think making progress on this spec will be a g ood thing for the transition from python2 to python3 as that happens around us
19:11:11 <clarkb> Next up is the website acitivty sites spec
19:11:12 <clarkb> #link https://review.opendev.org/#/c/709236/ Website activity stats
19:11:46 <clarkb> fungi: I think proper reivew on this one from you would be great. But its also less urgent imo
19:11:50 <fungi> i'm probably mostly okay with both of those since they were written as an upshot of discussions i was involved in, but i'll check to make sure they reflect what i remember
19:12:30 <clarkb> fungi: ok, if you manage to review this one by EOD thurday I'll approve it too if there aer no new concerns, but if not I think we can pick it up next week
19:12:40 <clarkb> Thank you to everyone that has reviewed the specs
19:13:07 <clarkb> #topic Priority Efforts
19:13:13 <clarkb> #topic OpenDev
19:13:36 <clarkb> There was some excitement around opendev governance recently. The initial change I pushed to openstack/governance merged, but then was reverted
19:13:52 <clarkb> mnaser has pushed a new change for that and is trying to solicit more feedback
19:13:54 <clarkb> #link https://review.opendev.org/#/c/710020/ Split OpenDev out of OpenStack governance.
19:14:30 <clarkb> it seems that there is new concern over what opendev would be post split and one of the suggestions is that it could be a pilot project under the OSF
19:14:36 <clarkb> #link http://lists.openstack.org/pipermail/openstack-infra/2020-March/006603.html OpenDev as OSF pilot project
19:15:07 <fungi> i'm still not convinced that's necessary, but willing to entertain if it makes some stakeholders more comfortable
19:15:13 <clarkb> jbryce wrote ^ to start that conversation. I'm not opposed initially, but have asked for more information on what would be required of us to make that happen as well as how we see the pilot project criteria applying to opendev
19:15:54 <clarkb> If you've got thoughts on that I'm happy to hear them either directly or on the mailing list. Will do my best to stay on top of that input
19:16:15 <corvus> i also don't object but
19:16:28 <corvus> i also am not sure we need to block on that
19:17:12 <corvus> i don't think we're being as successful as we would like to be in growing the opendev community apart from openstack-infra
19:17:24 <corvus> and i think further delay is not going to help that
19:17:25 <clarkb> corvus: I agree. Personally I thought what we had was sufficient with the understanding that we are making incremental changes
19:17:52 <clarkb> but I'm also not sure we can assert that more strongly that we have without using the "fork" work
19:17:53 <corvus> so it would be nice if we could make progress before everyone burns out on the idea
19:17:54 <clarkb> *word
19:17:59 <clarkb> ++
19:18:18 <clarkb> corvus: I think that is good feedback for mnaser and the rest of the TC
19:18:24 <clarkb> I'll try to pass it along
19:19:35 <corvus> 2 years is a long time :)
19:19:47 <mordred> o/
19:19:59 <mordred> (sorry - was letting myself be distracted by mailing list threads)
19:20:45 <clarkb> corvus: maybe you can send a short response to jbryce? Then I can make sure the TC is trackign that thread?
19:20:55 <clarkb> If not I can try to summarize your thoughts and send it instead
19:20:57 <mordred> I agree with the above re: don't object and also don't need to block
19:21:06 <corvus> clarkb: sure i'll reply
19:21:09 <clarkb> thank you
19:22:09 <clarkb> On the operational side of things we have master builds of gitea working now
19:22:29 <mordred> I started looking yesterday evening at the template diffs
19:22:34 <mordred> but then EODd
19:22:40 <clarkb> we need to check for template changes and probably don't want to deploy "master" because templates and other things can change under us. But if we wanted to we could likely deploy gitea with the commit cache in the near future
19:23:10 <mordred> might allow us to provide feedback to lunny
19:24:08 <clarkb> mordred: one thing I wondered about is how difficult it would be to deploy a specific gitea backend with a different docker image
19:24:19 <clarkb> mordred: I think that would require us to use a different docker compose config file template but should be doable.
19:24:38 <clarkb> then we could monitor memory use and measure performance improvement/impact
19:24:59 <mordred> clarkb: I mean - we could put in an ansible variable with an image tag
19:25:06 <mordred> and then set different host_var overrides
19:25:10 <mordred> so I thnik it would be pretty easy
19:25:12 <clarkb> ah ya
19:25:23 <clarkb> maybe doing that real soon now is worthwhile with a single backend
19:25:39 <clarkb> and if there are problems we can remove it from the haproxy config
19:26:08 <mordred> yeah. we might want to put in a sha pin to the override-checkout and add a different tag name on the output image
19:26:17 <mordred> so that we could bump the sha we're working with later
19:26:21 <clarkb> my biggest concern is the gitea team does seem to curate releases for when they think the software is ready. They also put together pretty comprehensive release notes. W'd lack that in this case
19:26:31 <mordred> yeah
19:27:44 <clarkb> Ok we can discuss further outside the meeting. Lets move on
19:27:51 <clarkb> #topic Update Configuration Management
19:28:18 <clarkb> mordred: are there updates on gerrit images? Possibly related to jeepyb and bazel and stuff
19:28:40 <mordred> well - it all broke again
19:29:00 <mordred> but this: https://review.opendev.org/#/c/711072/ should fix the builds
19:29:50 <mordred> other than that - next up on gerrit is manage-projects
19:30:05 <mordred> turns out I missed that when doing review-dev - so solving it is the next task
19:30:51 <mordred> I think we should split manage-projects from the service stuff so we can retire remote-puppet-git and instead just have a playbook that applies project-config changes to all the places
19:31:18 <mordred> but I'm not all the way to having a stepwise plan for doing any of that in a sane way
19:31:34 <clarkb> remote-puppet-git?
19:31:49 <mordred> yeah - that's the playbook that currently runs things on review
19:32:15 <clarkb> oh right
19:32:17 <mordred> that's spread across gerrit and gitea and does the gitea service stuff as well
19:32:33 <corvus> mordred: so we'd pull the project-management stuff out of puppet?
19:32:37 <clarkb> refactoring the puppet and ansible to be just cleaner ansible
19:33:20 <mordred> corvus: more that I think we shoudlnt' copy the structure in the ansible replacement
19:33:26 <corvus> ah ok
19:33:42 <mordred> and I *think* it might be easier to do that as part of adding manage-projects support
19:33:53 <mordred> to the review-dev ansible
19:34:14 <corvus> mordred: other than run manage-projects, what else is involved?
19:34:35 <mordred> we clone project-config, then we copy of a bunch of files around to different places
19:34:54 <corvus> for gerrit?  what files?
19:35:04 <corvus> the acls?
19:35:09 <mordred> projects.yaml, projects.ini, acls dir
19:35:36 <mordred> it's also possible it's easier than my brain is making it right now
19:35:57 <mordred> but the puppet-project-config indirection is ... challenging to follow
19:36:09 <corvus> mordred: well, i agree with "the review playbook should run manage projects" which seems the overall goal
19:37:00 <corvus> hrm, it has to interact with gitea though, right?
19:37:01 <mordred> well -I'm not actually convinced in the fullness of time that that should be the goal. I think having a service-review and a service-prjojects playbook might serve us better
19:37:03 <mordred> yeah
19:37:31 <clarkb> mordred: corvus: and order the playbooks to enforce the current ordering we get in a single playbook?
19:37:41 <mordred> so I think we really want service-review.yaml that deals with gerrit config managemnt, service-gitea that deals with teh same for gitea, and service-project-config that applies project-config to the gitea and gerrit services
19:37:55 <clarkb> aha
19:37:58 <clarkb> that makes sense to me
19:37:59 <mordred> BUT - that might be too big of a change for this effort
19:38:09 <mordred> or it might be the _easier_ way to do this effort
19:38:18 <corvus> mordred: it looks like remote_puppet_git is basically already that, it just runs puppet on review
19:38:32 <mordred> corvus: yeah - but puppet on review runs manage-projects
19:38:39 <corvus> yes
19:38:50 <corvus> mordred: okay, so service-review should not run manage-projects.  remote_puppet_git should be updated to run manage-projects.  then it should be renamed.
19:39:07 <mordred> corvus: yeah
19:39:51 <corvus> mordred: i think we could just do that as part of the switch
19:39:59 <corvus> to do it earlier means extracting that from puppet
19:40:07 <mordred> yeah. and actually - thanks, this has been helpful to my brain
19:40:18 <corvus> to do it as part of the switch may slightly entend the period where we can't create new projects, but i'm not worried about that
19:40:24 <mordred> yeah
19:40:27 <corvus> s/entend/extend/
19:40:44 <AJaeger> we have currently very few project creation requests...
19:41:15 <AJaeger> so, unless there's a deadline - like a summit - we can easily block a few days or even more than a week
19:41:23 <clarkb> ++
19:41:23 <mordred> cool
19:41:30 <clarkb> especially if we communicate it in advance
19:41:34 <clarkb> "do this now or wait a while"
19:41:54 <corvus> i'd expect the worst case to be like a day :)
19:42:10 <mordred> I also looked BRIEFLY at the idea of "replace manage-projects with some ansible like gitea"
19:42:28 <mordred> but I think that should be a followup - there's a few potential gotchas
19:42:48 <clarkb> I've still got test changes up for manage projects which can help us limp it along too
19:42:50 <mordred> there are a few things jeepyb is taking care of though that we no longer care about
19:42:51 <AJaeger> corvus: a day? not a problem...
19:42:55 <clarkb> and then transition that into testing for new tooling
19:43:34 <clarkb> as a time check we've got ~17 minutes left. Should we continue on to the next several subjects?
19:43:45 <mordred> yeah. I'm good on this one if y'all are
19:43:46 <clarkb> (I doubt we'll solve the refactor here)
19:43:51 <corvus> ++
19:43:56 <clarkb> #topic General Topics
19:44:24 <clarkb> fungi: I don't think there has been any new progress on the wiki puppetry. If you are still here can you confirm?
19:44:40 <fungi> nope
19:44:49 <clarkb> ok
19:44:52 <fungi> (no progress, and i'm not here to confirm)
19:45:02 <clarkb> ianw: static.openstack.org -> static.opendev.org transition
19:45:07 <clarkb> ianw: are we basically done at this point?
19:45:11 <ianw> just cleanup now
19:45:19 <ianw> are we still ok with removing logs.openstack.org dns
19:45:46 <clarkb> ianw: our rotation time period was 30 days prior to switching to swift, we transitioned to swift well over 30 days ago. I think I'm good
19:46:04 <clarkb> we could potentially have logs.openstack.org redirect to zuul.openstack.org/builds
19:46:09 <clarkb> but I doubt even that is necessary
19:46:13 <ianw> there's some puppet cleanup https://review.opendev.org/#/q/status:open+topic:static-services
19:46:32 <clarkb> #link https://review.opendev.org/#/q/status:open+topic:static-services cleanup static.openstack.org now that all services are hosted by static.opendev.org
19:46:33 <ianw> right, in the spec we would remove the dns, just checking we're still ok with it
19:46:49 <clarkb> I'm ok with it
19:46:50 <ianw> files.openstack.org and static-old.openstack.org are now idle
19:47:35 <AJaeger> let's remove them as well
19:48:27 <AJaeger> ianw: but first change https://opendev.org/zuul/zuul-jobs/src/branch/master/test-playbooks/base-roles/validate-host.yaml#L13 ;)
19:48:39 <AJaeger> that's the only reason to keep files.o.o
19:49:20 <ianw> heh well actually files.openstack.org is a CNAME now to static ... i should of said the host files02.openstack.org
19:49:28 <ianw> but agree we could change that for completeness
19:52:48 <clarkb> alright sounds like that may be it on static
19:52:51 <clarkb> Now for the last subject
19:52:57 <clarkb> IRC channel usage.
19:53:21 <clarkb> frickler brings up that we have redundant channels between #openstack-infra and #opendev and that we mirror bot events to both
19:53:47 <clarkb> thinking about what corvus said before about how we've not effectively been building the opendev community much outside of openstack, maybe we should just transition channels sooner than later?
19:53:52 <AJaeger> that's one reason why I'm staying out of #opendev - there're only bots with the same information I saw elsewhere already
19:54:05 <clarkb> frickler: ^ I'm not sure if you are still around today, but happy to hear if you have specific thoughts too
19:54:37 <AJaeger> With the opendev split, we split repos into openstack and opendev - should we do the same for notifications?
19:54:56 <frickler> I don't have a solution, but my main concern are duplicated bot msgs, yes
19:55:00 <corvus> if we're going to start using #opendev, i think we're going to need to really decide to do it as a team and make an effort
19:55:12 <AJaeger> So, whatever stays in openstack , goes to #openstack-infra, what moves out goes to #opendev?
19:55:20 <clarkb> corvus: ya and have a flag day and if questions pop up that are opdnev specific in -infra really push people to move
19:55:26 <clarkb> AJaeger: yes I think that makes sense
19:55:38 <clarkb> we might mirror project-config on the short term since it still handles both
19:55:40 <AJaeger> I volunteer to propose a change...
19:55:50 <AJaeger> (if there is merit)
19:55:52 <corvus> the main thing i worry about with channels is that if we're mising important information in one, then it becomes less useful
19:56:10 <corvus> i'm happy to have the list reduced, but we should be careful
19:56:22 <corvus> like openstack/project-config should probably go to both right now.
19:56:27 <clarkb> corvus: agreed
19:56:30 <AJaeger> I'm fine duplicating a few projects - but not *all* of them
19:56:32 <corvus> and if we remove zuul/* from one, we should remove it from the other too
19:56:35 <frickler> maybe we need to move completely, redirect #openstack-infra to #opendev
19:56:51 <frickler> like with other retired channels
19:57:10 <clarkb> frickler: maybe we can consider that option if -infra goes idle after an initial split?
19:57:25 <clarkb> I'm worried we can't accurately predict now if there would still be openstack specific chatter in -infra or not
19:57:32 <frickler> cause for most infra issues that pop up, I think its difficult to say whether they are opendev or not
19:57:38 <clarkb> this is true
19:57:47 <clarkb> usually we have to investigate before things fall one way or the other
19:58:07 <frickler> and "customers" can tell even less
19:58:28 <clarkb> I'll have to think on that a bit more I guess. My initial thought was definitely to keep both
19:58:46 <clarkb> and accept a period where we'll have to manage transition and point people in the correct location
19:58:55 <clarkb> but that is an excellent point about how this might be difficult
19:59:08 <corvus> but if every "my devstack-based job failed" question goes to #opendev, then i think that's going to turn off potential non-openstack contributors
19:59:12 <AJaeger> let's cleanup the channels - I'll first cleanup #opendev, then #openstack-infra...
19:59:18 <clarkb> corvus: ya
19:59:46 <clarkb> so maybe we start with two channels (its easier to start there and go to one if we decide that works than it is to redirect and then unsplit later)
19:59:57 <clarkb> frickler: ^ do yu think that is an acceptable incremental state to be in?
20:00:14 <clarkb> (also as a time check we are now at time. There is no meeting after us so we can finish this topic before ending the meeting)
20:00:45 <frickler> sure, let's try and see where we end at
20:01:01 <clarkb> AJaeger: ^ I think that means you can proceed with your planned changes
20:01:09 <AJaeger> ack
20:01:20 <clarkb> If there are other thoughts that come up on this subject feel free to discuss further in -infra or on the reviews
20:01:40 <clarkb> and with that I think we should let people eat dinner/lunch/breakfast
20:01:42 <clarkb> thank you all
20:01:44 <clarkb> #endmeeting