19:02:09 #startmeeting infra 19:02:10 Meeting started Tue Mar 3 19:02:09 2020 UTC and is due to finish in 60 minutes. The chair is clarkb. Information about MeetBot at http://wiki.debian.org/MeetBot. 19:02:11 Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 19:02:14 The meeting name has been set to 'infra' 19:02:33 #link http://lists.openstack.org/pipermail/openstack-infra/2020-March/006605.html Our Agenda 19:03:01 #topic Announcements 19:03:28 fungi is largely afk this week as are ttx. They have meeting topics that we'll probably go through quickly as a result 19:03:34 #topic Actions from last meeting 19:03:47 #link http://eavesdrop.openstack.org/meetings/infra/2020/infra.2020-02-25-19.01.txt minutes from last meeting 19:04:07 We have one action from the last meeting. ttx was to write an xwiki spec. This happend and we'll talk about it in the next section of the meeting 19:04:09 #topic Specs 19:04:35 we'll start with the xwiki spec. 19:04:40 #link https://review.opendev.org/#/c/710057/ xwiki for wikis 19:05:05 This hasn't gotten any reviews yet, but there was good discussion on IRC (in the meeting and out) last week. If you've got thoughts on this your reviews are much appreciated 19:05:13 frickler and corvus in particular seemed to have some good initial thoughts 19:05:35 Next is the cleanup of python tooling on our CI images. 19:05:39 #link https://review.opendev.org/#/c/709579/ Cleaning up python dev tools on our CI images. 19:05:57 infra-root I think this one is ready to go up for approval 19:06:24 is there any objection to that? many of you have already reviewd it. Looks like we are missing fungi though. 19:06:35 * diablo_rojo sneaks into the back of the room 19:06:37 maybe hold off until fungi can ack or nack? 19:06:57 or do you think we have had sufficient input? ianw mordred you've both been involved in this one so your thoughts on it would be great 19:07:09 (as far as proceeding, I Think you've both acked the review) 19:08:45 hrm we may not have quorum here today (yet). I'll wait a couple more minutes to see if others are able to drop in 19:08:58 personally it's probably sufficient, as we can stage it in without affecting prodcution 19:09:12 ianw: thats a good point. 19:09:30 i'm here, just quiet 19:09:32 yeah, sorry, i can try to take a look in a bit 19:09:57 ok why don't we say end of clarkb business thursday? that should give fungi time to fit review into travel schedule 19:10:18 I'm reluctant to prolong this as we just had another virtualenv issue this week 19:10:33 I think making progress on this spec will be a g ood thing for the transition from python2 to python3 as that happens around us 19:11:11 Next up is the website acitivty sites spec 19:11:12 #link https://review.opendev.org/#/c/709236/ Website activity stats 19:11:46 fungi: I think proper reivew on this one from you would be great. But its also less urgent imo 19:11:50 i'm probably mostly okay with both of those since they were written as an upshot of discussions i was involved in, but i'll check to make sure they reflect what i remember 19:12:30 fungi: ok, if you manage to review this one by EOD thurday I'll approve it too if there aer no new concerns, but if not I think we can pick it up next week 19:12:40 Thank you to everyone that has reviewed the specs 19:13:07 #topic Priority Efforts 19:13:13 #topic OpenDev 19:13:36 There was some excitement around opendev governance recently. The initial change I pushed to openstack/governance merged, but then was reverted 19:13:52 mnaser has pushed a new change for that and is trying to solicit more feedback 19:13:54 #link https://review.opendev.org/#/c/710020/ Split OpenDev out of OpenStack governance. 19:14:30 it seems that there is new concern over what opendev would be post split and one of the suggestions is that it could be a pilot project under the OSF 19:14:36 #link http://lists.openstack.org/pipermail/openstack-infra/2020-March/006603.html OpenDev as OSF pilot project 19:15:07 i'm still not convinced that's necessary, but willing to entertain if it makes some stakeholders more comfortable 19:15:13 jbryce wrote ^ to start that conversation. I'm not opposed initially, but have asked for more information on what would be required of us to make that happen as well as how we see the pilot project criteria applying to opendev 19:15:54 If you've got thoughts on that I'm happy to hear them either directly or on the mailing list. Will do my best to stay on top of that input 19:16:15 i also don't object but 19:16:28 i also am not sure we need to block on that 19:17:12 i don't think we're being as successful as we would like to be in growing the opendev community apart from openstack-infra 19:17:24 and i think further delay is not going to help that 19:17:25 corvus: I agree. Personally I thought what we had was sufficient with the understanding that we are making incremental changes 19:17:52 but I'm also not sure we can assert that more strongly that we have without using the "fork" work 19:17:53 so it would be nice if we could make progress before everyone burns out on the idea 19:17:54 *word 19:17:59 ++ 19:18:18 corvus: I think that is good feedback for mnaser and the rest of the TC 19:18:24 I'll try to pass it along 19:19:35 2 years is a long time :) 19:19:47 o/ 19:19:59 (sorry - was letting myself be distracted by mailing list threads) 19:20:45 corvus: maybe you can send a short response to jbryce? Then I can make sure the TC is trackign that thread? 19:20:55 If not I can try to summarize your thoughts and send it instead 19:20:57 I agree with the above re: don't object and also don't need to block 19:21:06 clarkb: sure i'll reply 19:21:09 thank you 19:22:09 On the operational side of things we have master builds of gitea working now 19:22:29 I started looking yesterday evening at the template diffs 19:22:34 but then EODd 19:22:40 we need to check for template changes and probably don't want to deploy "master" because templates and other things can change under us. But if we wanted to we could likely deploy gitea with the commit cache in the near future 19:23:10 might allow us to provide feedback to lunny 19:24:08 mordred: one thing I wondered about is how difficult it would be to deploy a specific gitea backend with a different docker image 19:24:19 mordred: I think that would require us to use a different docker compose config file template but should be doable. 19:24:38 then we could monitor memory use and measure performance improvement/impact 19:24:59 clarkb: I mean - we could put in an ansible variable with an image tag 19:25:06 and then set different host_var overrides 19:25:10 so I thnik it would be pretty easy 19:25:12 ah ya 19:25:23 maybe doing that real soon now is worthwhile with a single backend 19:25:39 and if there are problems we can remove it from the haproxy config 19:26:08 yeah. we might want to put in a sha pin to the override-checkout and add a different tag name on the output image 19:26:17 so that we could bump the sha we're working with later 19:26:21 my biggest concern is the gitea team does seem to curate releases for when they think the software is ready. They also put together pretty comprehensive release notes. W'd lack that in this case 19:26:31 yeah 19:27:44 Ok we can discuss further outside the meeting. Lets move on 19:27:51 #topic Update Configuration Management 19:28:18 mordred: are there updates on gerrit images? Possibly related to jeepyb and bazel and stuff 19:28:40 well - it all broke again 19:29:00 but this: https://review.opendev.org/#/c/711072/ should fix the builds 19:29:50 other than that - next up on gerrit is manage-projects 19:30:05 turns out I missed that when doing review-dev - so solving it is the next task 19:30:51 I think we should split manage-projects from the service stuff so we can retire remote-puppet-git and instead just have a playbook that applies project-config changes to all the places 19:31:18 but I'm not all the way to having a stepwise plan for doing any of that in a sane way 19:31:34 remote-puppet-git? 19:31:49 yeah - that's the playbook that currently runs things on review 19:32:15 oh right 19:32:17 that's spread across gerrit and gitea and does the gitea service stuff as well 19:32:33 mordred: so we'd pull the project-management stuff out of puppet? 19:32:37 refactoring the puppet and ansible to be just cleaner ansible 19:33:20 corvus: more that I think we shoudlnt' copy the structure in the ansible replacement 19:33:26 ah ok 19:33:42 and I *think* it might be easier to do that as part of adding manage-projects support 19:33:53 to the review-dev ansible 19:34:14 mordred: other than run manage-projects, what else is involved? 19:34:35 we clone project-config, then we copy of a bunch of files around to different places 19:34:54 for gerrit? what files? 19:35:04 the acls? 19:35:09 projects.yaml, projects.ini, acls dir 19:35:36 it's also possible it's easier than my brain is making it right now 19:35:57 but the puppet-project-config indirection is ... challenging to follow 19:36:09 mordred: well, i agree with "the review playbook should run manage projects" which seems the overall goal 19:37:00 hrm, it has to interact with gitea though, right? 19:37:01 well -I'm not actually convinced in the fullness of time that that should be the goal. I think having a service-review and a service-prjojects playbook might serve us better 19:37:03 yeah 19:37:31 mordred: corvus: and order the playbooks to enforce the current ordering we get in a single playbook? 19:37:41 so I think we really want service-review.yaml that deals with gerrit config managemnt, service-gitea that deals with teh same for gitea, and service-project-config that applies project-config to the gitea and gerrit services 19:37:55 aha 19:37:58 that makes sense to me 19:37:59 BUT - that might be too big of a change for this effort 19:38:09 or it might be the _easier_ way to do this effort 19:38:18 mordred: it looks like remote_puppet_git is basically already that, it just runs puppet on review 19:38:32 corvus: yeah - but puppet on review runs manage-projects 19:38:39 yes 19:38:50 mordred: okay, so service-review should not run manage-projects. remote_puppet_git should be updated to run manage-projects. then it should be renamed. 19:39:07 corvus: yeah 19:39:51 mordred: i think we could just do that as part of the switch 19:39:59 to do it earlier means extracting that from puppet 19:40:07 yeah. and actually - thanks, this has been helpful to my brain 19:40:18 to do it as part of the switch may slightly entend the period where we can't create new projects, but i'm not worried about that 19:40:24 yeah 19:40:27 s/entend/extend/ 19:40:44 we have currently very few project creation requests... 19:41:15 so, unless there's a deadline - like a summit - we can easily block a few days or even more than a week 19:41:23 ++ 19:41:23 cool 19:41:30 especially if we communicate it in advance 19:41:34 "do this now or wait a while" 19:41:54 i'd expect the worst case to be like a day :) 19:42:10 I also looked BRIEFLY at the idea of "replace manage-projects with some ansible like gitea" 19:42:28 but I think that should be a followup - there's a few potential gotchas 19:42:48 I've still got test changes up for manage projects which can help us limp it along too 19:42:50 there are a few things jeepyb is taking care of though that we no longer care about 19:42:51 corvus: a day? not a problem... 19:42:55 and then transition that into testing for new tooling 19:43:34 as a time check we've got ~17 minutes left. Should we continue on to the next several subjects? 19:43:45 yeah. I'm good on this one if y'all are 19:43:46 (I doubt we'll solve the refactor here) 19:43:51 ++ 19:43:56 #topic General Topics 19:44:24 fungi: I don't think there has been any new progress on the wiki puppetry. If you are still here can you confirm? 19:44:40 nope 19:44:49 ok 19:44:52 (no progress, and i'm not here to confirm) 19:45:02 ianw: static.openstack.org -> static.opendev.org transition 19:45:07 ianw: are we basically done at this point? 19:45:11 just cleanup now 19:45:19 are we still ok with removing logs.openstack.org dns 19:45:46 ianw: our rotation time period was 30 days prior to switching to swift, we transitioned to swift well over 30 days ago. I think I'm good 19:46:04 we could potentially have logs.openstack.org redirect to zuul.openstack.org/builds 19:46:09 but I doubt even that is necessary 19:46:13 there's some puppet cleanup https://review.opendev.org/#/q/status:open+topic:static-services 19:46:32 #link https://review.opendev.org/#/q/status:open+topic:static-services cleanup static.openstack.org now that all services are hosted by static.opendev.org 19:46:33 right, in the spec we would remove the dns, just checking we're still ok with it 19:46:49 I'm ok with it 19:46:50 files.openstack.org and static-old.openstack.org are now idle 19:47:35 let's remove them as well 19:48:27 ianw: but first change https://opendev.org/zuul/zuul-jobs/src/branch/master/test-playbooks/base-roles/validate-host.yaml#L13 ;) 19:48:39 that's the only reason to keep files.o.o 19:49:20 heh well actually files.openstack.org is a CNAME now to static ... i should of said the host files02.openstack.org 19:49:28 but agree we could change that for completeness 19:52:48 alright sounds like that may be it on static 19:52:51 Now for the last subject 19:52:57 IRC channel usage. 19:53:21 frickler brings up that we have redundant channels between #openstack-infra and #opendev and that we mirror bot events to both 19:53:47 thinking about what corvus said before about how we've not effectively been building the opendev community much outside of openstack, maybe we should just transition channels sooner than later? 19:53:52 that's one reason why I'm staying out of #opendev - there're only bots with the same information I saw elsewhere already 19:54:05 frickler: ^ I'm not sure if you are still around today, but happy to hear if you have specific thoughts too 19:54:37 With the opendev split, we split repos into openstack and opendev - should we do the same for notifications? 19:54:56 I don't have a solution, but my main concern are duplicated bot msgs, yes 19:55:00 if we're going to start using #opendev, i think we're going to need to really decide to do it as a team and make an effort 19:55:12 So, whatever stays in openstack , goes to #openstack-infra, what moves out goes to #opendev? 19:55:20 corvus: ya and have a flag day and if questions pop up that are opdnev specific in -infra really push people to move 19:55:26 AJaeger: yes I think that makes sense 19:55:38 we might mirror project-config on the short term since it still handles both 19:55:40 I volunteer to propose a change... 19:55:50 (if there is merit) 19:55:52 the main thing i worry about with channels is that if we're mising important information in one, then it becomes less useful 19:56:10 i'm happy to have the list reduced, but we should be careful 19:56:22 like openstack/project-config should probably go to both right now. 19:56:27 corvus: agreed 19:56:30 I'm fine duplicating a few projects - but not *all* of them 19:56:32 and if we remove zuul/* from one, we should remove it from the other too 19:56:35 maybe we need to move completely, redirect #openstack-infra to #opendev 19:56:51 like with other retired channels 19:57:10 frickler: maybe we can consider that option if -infra goes idle after an initial split? 19:57:25 I'm worried we can't accurately predict now if there would still be openstack specific chatter in -infra or not 19:57:32 cause for most infra issues that pop up, I think its difficult to say whether they are opendev or not 19:57:38 this is true 19:57:47 usually we have to investigate before things fall one way or the other 19:58:07 and "customers" can tell even less 19:58:28 I'll have to think on that a bit more I guess. My initial thought was definitely to keep both 19:58:46 and accept a period where we'll have to manage transition and point people in the correct location 19:58:55 but that is an excellent point about how this might be difficult 19:59:08 but if every "my devstack-based job failed" question goes to #opendev, then i think that's going to turn off potential non-openstack contributors 19:59:12 let's cleanup the channels - I'll first cleanup #opendev, then #openstack-infra... 19:59:18 corvus: ya 19:59:46 so maybe we start with two channels (its easier to start there and go to one if we decide that works than it is to redirect and then unsplit later) 19:59:57 frickler: ^ do yu think that is an acceptable incremental state to be in? 20:00:14 (also as a time check we are now at time. There is no meeting after us so we can finish this topic before ending the meeting) 20:00:45 sure, let's try and see where we end at 20:01:01 AJaeger: ^ I think that means you can proceed with your planned changes 20:01:09 ack 20:01:20 If there are other thoughts that come up on this subject feel free to discuss further in -infra or on the reviews 20:01:40 and with that I think we should let people eat dinner/lunch/breakfast 20:01:42 thank you all 20:01:44 #endmeeting