19:01:04 <clarkb> #startmeeting infra
19:01:06 <openstack> Meeting started Tue Mar 17 19:01:04 2020 UTC and is due to finish in 60 minutes.  The chair is clarkb. Information about MeetBot at http://wiki.debian.org/MeetBot.
19:01:07 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
19:01:09 <openstack> The meeting name has been set to 'infra'
19:01:10 <ianw> o/
19:01:13 <clarkb> I remember to use the meeting name this time :)
19:01:19 <clarkb> #link http://lists.openstack.org/pipermail/openstack-infra/2020-March/006609.html Our Agenda
19:01:27 <clarkb> #topic Announcements
19:02:15 <corvus> o/
19:02:31 <clarkb> First up is the plan to switch over irc channels today. This ended up not getting communicated outside our bubble very well, but I'm thinking maybe we can still make the change and kindly redirect people as necessary?
19:02:45 <clarkb> https://review.opendev.org/#/c/711106/ is the change to update the bot side of that if we think that is an ok plan
19:03:00 <clarkb> (re communication fails I got distracted by nodepool and world events things)
19:03:12 * AJaeger will approve that change once this topic is done
19:03:41 <AJaeger> clarkb: yes, we can tell people - sending an email would be great in addition
19:03:57 <clarkb> AJaeger: ya maybe doing it concurrently is sufficient
19:04:14 <clarkb> (does anyone object to that plan: switch, be nice to people in the wrong place, and send email today)
19:04:31 <mordred> ++
19:04:33 <mordred> sounds great
19:05:04 * mnaser suggests a jar where you throw 5 cents everytime you use the wrong channel
19:05:19 <clarkb> sounds like a plan. AJaeger I'm happy to write the email if you don't want to
19:05:25 <clarkb> (I know its late for you)
19:05:31 <AJaeger> clarkb: please do
19:05:47 <clarkb> Ok next up
19:05:51 <clarkb> #link http://lists.openstack.org/pipermail/foundation/2020-March/002852.html OSF email on 2020 events
19:05:59 <fungi> we've ramped up use in #opendev a lot in the past week anyway
19:06:14 <clarkb> I'm sure the foundation would love to hear feedback particularly as this starts to affect more and more of us in a tangible way
19:06:41 <fungi> and opendev+ptg is only a few months away now
19:07:30 <clarkb> from my side, my kids will be home a lot more and I'll likely need to ensure my wife doesn't spend all her time caring for them. This means scheduling may be weird for me especially until we find a routine
19:07:44 <clarkb> I expect its a similar situation for a growing number of us
19:08:00 <AJaeger> yeah, kids at home since yesterday...
19:08:30 <clarkb> something something patience
19:08:57 <clarkb> #topic Actions from last meeting
19:09:05 <clarkb> #link http://eavesdrop.openstack.org/meetings/infra/2020/infra.2020-03-10-19.01.txt minutes from last meeting
19:09:13 <clarkb> Theer were no actions
19:09:20 <clarkb> #topic Specs approval
19:09:25 <clarkb> #link https://review.opendev.org/#/c/710057/ xwiki for wikis
19:09:37 <clarkb> zbr: frickler I believe that ttx has updated that to try and address some of the concerns you raised
19:09:55 <clarkb> from my side I think as a POC this is fine, we'll learn a lot more about vaibility through this experiement and can take it from there
19:10:26 <frickler> yeah, I'm not completely conviced yet, but I also don't want to block it
19:10:27 <clarkb> this is also a problem space where we have interest in usage but less so in implementation so any help we can get there is a good thing imo
19:11:10 <clarkb> if you've got time to (re)review I know ttx will apprecaite it
19:11:46 <fungi> we've also asserted i the past that for some services which opendev doesn't want to provide directly (for a variety of possible reasons) partnering with a hosted solution we can recommend to users is a viable alternative
19:12:45 <fungi> for example, to replace zanata with another supported translation platform
19:12:56 <clarkb> ya
19:13:10 <clarkb> I'll give this another minute before moving to the next topic if there are last bits of input
19:13:15 <zbr> i would personally find the gitea wiki more practical, in the end what stops any project from using their own external SaaS wiki? what makes xwiki more special?
19:14:02 <clarkb> zbr: gitea wiki is not technically feasible currently
19:14:34 <clarkb> it requires us to run a single gitea instance instead of 8, that instance must then have writable git repos (somethign that may be problematic with gerrit forcep ushing to it), and we need user management
19:14:34 <mnaser> is the issue maintaining the wiki itself
19:14:42 <mnaser> or the spam and whatn ot
19:14:50 <mordred> both
19:14:55 <fungi> mnaser: i think that's an issue with any wiki
19:15:08 <clarkb> mnaser: our mediawiki deployment is in really sad shape and we also don't have many people admining it to deal with spam etc
19:15:10 <fungi> mnaser: are you asking about wiki.openstack.org deployment of mediawiki, or just in general?
19:15:19 <clarkb> zbr: the spec was updated to capture those issues with the gitea wiki option
19:15:38 <mnaser> i see there's official docker images for mediawiki and infra is doing a lot of docker-compose these days
19:15:55 <mnaser> i imagine the infrastructure to get a db and an image up is (mostly) there, i think, but i may be trivializing things
19:16:33 <clarkb> really I think the biggest issue is no one is saying "I'll make a new wiki deployment and find admins"
19:16:44 <clarkb> ttx is offering an alternative that attempts to address both problems
19:16:52 <fungi> the openstack infra team (apart from opendev) is basically stuck maintaining a deployment of mediawiki for a while either way, so i don't object to exploring replacing the current puppet with docker images if we can get the right plugins working in it, but that's probably a somewhat separate topic
19:16:56 <clarkb> (but we recognize its an experiement and we'll have to see how well it does)
19:17:11 <zbr> my personal impression is that wiki popularity went down since markdown editing on top of git repos become so easy, am I wrong? i see myself having to interact with wikis far less than 10 years ago. Bonus, I do not see much spam as PRs/CRs.
19:17:32 <mnaser> honestly -- i'd like to move us (vexxhost) towards doing more self-hosted things, so i can help with a solution towards "ill make a new wiki deployment" -- but i cant solve the "admins" part
19:17:45 <clarkb> zbr: its a common request from projects using opendev
19:18:07 <clarkb> there is a group of users for which using git is the wrong tool
19:18:08 <corvus> mnaser: yeah, i looked at that a little too, i think it's feasible, probably easier than what we're currently doing, but not entirely turn-key
19:18:10 <mnaser> i'd gladly write a helm chart that we can deploy (maybe in a single node with k3s) so that i can also leverage it
19:18:12 <fungi> yeah, right now i'm basically the only person checknig wiki.o.o edits and blocking spammers and mass deleting their pages
19:18:19 <clarkb> but they need something to share information and collaborate with a bit more reliably (and with history) than etherpad
19:18:24 <frickler> so the question is: will xwiki.com do spam moderation as part of the hosting, or will we still have to take care of that ourselves?
19:18:32 <mordred> zbr: yeah - I agree with you personally - I much prefer in-repo reviewed rst text
19:18:39 <fungi> as someone who doesn't really even use wiki.o.o these days i'd love for it to be someone else's problem
19:18:51 <clarkb> frickler: my understanding is they provide tools to do it, btu then maybe we have to tune them ourselves
19:19:02 <clarkb> frickler: and learning how well those tools work is a good outcome of this experiement
19:19:06 <zbr> clarkb: sure, but this request does not come with specific wiki platfrom in mind? i am asking because i observed people not being open at all to switch one engine with another, they are not created equal.
19:19:14 <corvus> to represent ttx's pov, he would *also* prefer that, and did an enormous amount of work to reduce the use of wikis by all osf-related projects, but still this need persists; the spec is a recognition of that.
19:19:24 <mordred> yup
19:19:49 <mnaser> i can get a helm chart+ansible playbooks that can deploy wikis tested via zuul
19:19:50 <clarkb> zbr: typically the request is just for a wiki not a specific wiki
19:19:51 <corvus> and no, i don't think anyone has strong feelings about a particular engine (other than it be open source)
19:20:10 <mnaser> i have strong feelings about using an external hosted service IMHO, we've historically always walked the path of self-hosting things
19:20:14 <mnaser> that's just me though
19:20:40 <fungi> i think the "who will police this" is punted, but the multi-tenant nature of xwiki at least allows each fiefdom to do its own policing
19:20:50 <mnaser> there's always the possibility of xwiki deciding one day it no longer wants to have a free offering, or they stop maintaining the project
19:21:00 <clarkb> mnaser: yup and that has happend to us in the past
19:21:05 <corvus> mnaser: i'd be supportive of revamping our self-hosting
19:21:17 <fungi> mnaser: and in such a case it's something we can run ourselves
19:21:22 <mnaser> fungi: actually, my idea is that ever "group" would have their own deployment of mediawiki rather than one big multitenant one
19:21:22 <clarkb> the risk is real, but the current state of things is riskier imo
19:21:30 <corvus> regarding self-policing -- if we have a containerized deployment, ... yeah, what mnaser just said
19:21:50 <corvus> like opendev's mailing list service
19:22:03 <corvus> (and like its git service briefly was, until we folded it back in)
19:22:14 <fungi> mnaser: this same point comes up with the translation platform. we need to move folks off zanata since it's unmaintained. the folks who make the replacement we're looking at also offer to run it free for open source projects (with some caveats) so we talked about taking advantage of that rather than running it ourselves
19:22:31 <clarkb> my other concern is that it can often feel like a very small number of us do the life support for these more periphery services
19:22:43 <clarkb> so even if someone spins up a deployment a bit of involvement beyond that would be nice
19:22:45 <mnaser> i could use a self-hosted wiki, so i would gladly write helm charts that do it (if we're okay with maybe adopting something like using k3s in a single node)
19:22:46 <fungi> we don't want to wind up in the situation we were in with transifex, which was not really open source and then decided to kick all non-paying users off their hosted platform
19:23:15 <mnaser> if i was to do this, i'd consume it internally here, and hopefully there will be other consumers
19:23:18 <corvus> mnaser: i'm not familiar with k3s; why not docker-compose?
19:24:07 <mnaser> corvus: we deploy all of our apps inside k8s right now, so i was hoping that would be a path we can share with infra so that we can use a common base
19:24:22 <mnaser> k3s is pretty much a single node k8s stripped down out of all the unnecessary things using local sqlite instead of etcd
19:24:36 <mordred> corvus: https://k3s.io/ fwiw
19:24:46 <corvus> mordred: yeah, i googled that :)
19:24:47 <mnaser> so that way, infra can consume the helm chart and run wikis, and i can run it in full sized k8s clusters
19:24:55 <corvus> mnaser: we've talked about moving opendev to k8s, but our aim is to think about that after we actually get containerized services
19:25:00 <clarkb> I don't think we should add the complexity of yet another system in order to run a system no one is caring for today
19:25:06 <corvus> we're like a few years behind schedule on that
19:25:06 <clarkb> (thats my initial reaction)
19:25:27 <clarkb> that is how these system end up unmaintained
19:25:28 <corvus> so i'm also hesitant to potentially throw a wrench in that process by moving the target
19:25:30 <fungi> also my employer is a big part of why i've ended up maintaining the wiki. basically osf staff get contacted with complaints, dmca takedown notices, and various other legal threats if nobody keeps on top of spam policing for wiki.openstack.org
19:25:55 <mnaser> i understand, i mean fwiw we're adopting a lot more of the opendev "stack" so im building a lot more helm charts to deploy those on k8s
19:26:05 <mnaser> (recent example is helm charts that are fully tested for lodgeit are a thing now)
19:26:26 <mnaser> so i was hoping this can start a little roadways towards that but i understand
19:27:09 <clarkb> I'm not hearing anyone say that an low overhead (for us) experiment is a problem though?
19:27:20 <corvus> clarkb: which experiment?
19:27:26 <clarkb> corvus: the xwiki POC experiment
19:28:19 <clarkb> thinking we can continue discussion there on the spec
19:28:26 <clarkb> but we should probably move forward with the meeting agenda
19:28:50 <corvus> agreed, i still don't object to that; i also don't object to looking into containerized mediawiki (both to help maintain our current system and to provide an alternative if we don't select xwiki).  i understand some of the mediawiki stack and would be happy to help with that.  i'm less excited about bringing in k3s.
19:29:31 <clarkb> alright lets move on then
19:29:34 <clarkb> #topic Priority Topics
19:29:40 <clarkb> #topic OpenDev
19:29:46 <clarkb> #link http://lists.openstack.org/pipermail/openstack-infra/2020-March/006603.html OpenDev as OSF pilot project
19:29:52 <clarkb> if anyone else has thoughts on ^ that thread is still there :)
19:30:17 <clarkb> I'll try to remind jbryce that update on process would be good, but I think its been a busy couple of weeks
19:30:18 <corvus> i apparently had the last word on that one
19:30:28 <clarkb> #link https://review.opendev.org/#/c/710020/ Split OpenDev out of OpenStack governance. This change merged.
19:30:37 <clarkb> thats in which means we can now land the next change
19:30:47 <clarkb> #link https://review.opendev.org/#/c/703488/ Update OpenDev docs with new Governance. Now we can merge this change.
19:30:58 <fungi> yay!
19:31:18 <clarkb> I've removed my WIP and will hit the +A button at the end of the meeting
19:32:00 <corvus> i'm pretty sure we've all agreed to it in some form, but that could use some more infra-root +2s just for the record :)
19:32:32 * fungi is relieved to see he hasn't forgotten to do that
19:32:35 <clarkb> corvus: ya I think most of us have acked it but then a new ps happened for depends on being updated
19:33:01 <clarkb> frickler: Shrews ^ I Think you may be the last missing active cores if you want to review it
19:33:11 <Shrews> yep, skimming now
19:33:15 <clarkb> tyty
19:34:24 <clarkb> on the service side of things I don't think there have been any major changes in the last week
19:34:34 <clarkb> anything to add re opendev or should we continue?
19:35:14 <corvus> oh
19:35:21 <corvus> are we talking about gerrit restart this meeting?
19:35:25 <clarkb> corvus: yes
19:35:42 <clarkb> was goign to lump that into the next topic since it was lsightly related to mordreds review container work
19:35:42 <corvus> now or later?
19:35:44 <corvus> k
19:35:47 <clarkb> #topic Update Config Management
19:35:53 <clarkb> now works :)
19:36:01 <fungi> gerrit restart!
19:36:11 * fungi still plans to be on hand
19:36:11 <corvus> we're still on for friday the 20?  did we send announcements?
19:36:22 <clarkb> last week we sort of penciled in friday 20th. I got very distracted and didn't send announcements
19:36:32 <corvus> i have no conflicts on the 20th :)
19:36:47 <corvus> or, like any time for the next 3 weeks :)
19:36:57 <mordred> I still have no conflicts on the 20th
19:37:01 <clarkb> I can send announcements today if we still think that is the plan
19:37:11 <clarkb> mordred: ^ are things ready for that on the technical side of the house?
19:37:12 <fungi> clarkb sounded like he had an appointment to drive a race car or something
19:37:36 <corvus> yeah, i'd have liked to do an announcement earlier, but hopefully folks will be understanding
19:37:39 <mordred> clarkb: no - not 100% - but, if they aren't, I don't think we should block on it (I'm still pretty confident they will be by friday)
19:37:40 <clarkb> fungi: ya I think I've decided that I need the differently applied concentration that the road trip to arizona would have afforded
19:37:42 <corvus> i think we should do it because the timing is really good
19:37:56 <clarkb> mordred: gotcha so do the rename and not containers if necessary
19:38:05 <mordred> yah. rename + containers if we're happy
19:38:07 <fungi> all sounds fine to me
19:38:28 <clarkb> in that case I can send announcements after the meeting to the various project lists as well as one for #openstack-infra
19:38:48 <corvus> i'd really like to do the containers; what needs doing there?
19:39:12 <corvus> reviews, or is there some mini-task i can help with?
19:40:14 <clarkb> mordred: ^
19:41:55 <mordred> corvus: I just need to do the manage-projects shift and not something else
19:42:12 <mordred> corvus: I'll definitely ping you for reviews when I've got the patch
19:42:18 <corvus> mordred: k, lemme know if/how i can  help, thx
19:42:23 <mordred> kk. will do
19:42:23 <clarkb> is it reasonable to potentially say "project additions won't work" for a few days if that doesn't get done before firday?
19:42:32 <clarkb> then make that the next week's priority ?
19:43:01 <mordred> maybe - but I think it'll get done and be fine
19:43:16 <clarkb> k
19:44:26 <clarkb> The other config management topic was one I added under general topics but realized putting the agenda together on my local machine that we can cover here
19:44:32 <clarkb> the nb01/4.opendev.org rebuild
19:44:40 <clarkb> we've learned that docker and podman do behave differently at times
19:44:59 <clarkb> sounds like that is causing us to shift towards docker for consistency
19:45:21 <clarkb> and long story short nb04.opendev.org will be a new server in the near future to avoid hostname conflicts
19:45:28 <clarkb> ianw: mordred ^ anything else to add to that?
19:45:45 <ianw> not really, yeah in progress ...
19:46:10 <mordred> yeah. that covers it
19:46:36 <clarkb> #topic General Topics
19:46:45 <clarkb> fungi: I don't suppose there are updates on the wiki puppetry?
19:46:54 <clarkb> (based on earlier discussion I don't think so)
19:47:06 <mordred> minor gitea bump: https://review.opendev.org/#/c/713517/
19:47:09 <fungi> sounds like the best update is that mnaser might try out the docker containers
19:47:34 <fungi> the biggest challenges there will probably be making sure to cover openid integration and spam control/prevention
19:48:13 <fungi> we have a ton of spam prevention mechanisms in place on the current deployment
19:48:30 <fungi> and the openid extension is kinda finicky (i'm still struggling with it)
19:49:09 <fungi> but no, i've made no new progress on it
19:49:20 <clarkb> thats good input for anyone looking at alternative deployment methods
19:50:02 <clarkb> ianw: next up is static.o.o
19:50:19 <clarkb> have we cleaned up the bits that need cleaning up? and/or should we remove this topic at this point?
19:50:34 <clarkb> (as a related side note, the goaccess report generation is working now with the longer timeouts)
19:50:44 <ianw> no we haven't; maybe give me an action to do that in this week and we can stop discussing it next week
19:51:09 <ianw> (cleanup static bits i mean)
19:51:09 <clarkb> #action ianw cleanup static.openstack.org and files02.openstack.org resources that are no longer needed (now served by static.opendev.org)
19:51:18 <clarkb> thanks!
19:51:41 <fungi> that entire replacement effort has gone really, really, amazingly well
19:51:48 <mordred> ++
19:51:51 <clarkb> we've covered nb04.opendev.org and talked about gerrit downtime
19:52:02 <clarkb> oh I'll need to know a time window for email announcement
19:52:17 <clarkb> I expect I'll not be around so will let someone who has volunteered give me a window :)
19:52:31 <mordred> I'm free all day
19:52:42 <fungi> my available hours are probably somewhere in the 13:00-00:00 utc range
19:53:32 <fungi> though i can go later if needed, earlier is a bit of a caffeination risk
19:53:56 <corvus> i'm generally up by, what... 1400 utc?
19:54:11 <clarkb> should we say 1500-1600 UTC then so that corvus can have breakfast?
19:54:21 <fungi> wfm
19:54:38 <corvus> if clarkb isn't aronud and i'm the latest, i'd say feel free to start at 1400 or even earlier, since i'm backup
19:54:41 <mordred> wfm
19:54:57 <clarkb> ok 1400-1500 it is
19:55:02 <corvus> but if clarkb secretly wants us to start at 1500 so he can drop in if he feels like it, that's cool too :)
19:55:29 <clarkb> lets go 1400 then if I decide I want to be around that isn't too terrible for me
19:55:35 <mordred> cool
19:55:36 <clarkb> and gives me more time after to drive fake cars
19:55:48 <clarkb> #topic Open Discussion
19:55:56 <clarkb> And now ~4 minutes for anything else
19:56:16 <fungi> and now for something completely different
19:56:22 <fungi> it's the toad elevating moment?
19:56:38 <fungi> owl stretching time?
19:56:47 <AJaeger> zuul-jobs repo has quite a lot of open reviews where I'm unsure, would appreciate if somebody could do a review round and push stuff forward or advise what should not go in or needs change
19:57:22 <clarkb> #info zuul-jobs could use reviews
19:58:01 <frickler> weird last minute idea: would we want an #opendev-meeting channel, too?
19:58:14 <clarkb> frickler: I was thinking we might shift to doing the meeting in channel
19:58:19 <fungi> yeah
19:58:24 <frickler> ah, I don't like that
19:59:06 <clarkb> (also the meeting name is channel independent so we can keep doing it here for a bit if necessary or try a few things and still have continuity of log collection)
19:59:31 <clarkb> and at some point call it opendev (and possibly do a file rename on the logs server)
19:59:52 <fungi> i guess there were two main reasons behind openstack's community having dedicated meeting channels... reduction in schedule overlap and it gets the meeting talk uot of the general discussion channel... the first doesn't really apply for us and teh second is a matter of personal taste
20:00:12 <clarkb> fungi: ya I think the second would be the concern, frickler is that your concern?
20:00:21 <frickler> the idea was that ppl might not like the openstack part of the name here
20:00:24 <clarkb> (also we are at time but there is no meeting after us so can close this topic quickly)
20:00:30 <frickler> and also yes the latter
20:00:39 <fungi> a similar topic is what to do with #openstack-infra-incident
20:00:45 <fungi> but we're out of time in here
20:00:58 <fungi> (if this were in #opendev we could just keep going!)
20:01:05 <clarkb> ya I think we can create new channels for both of those in order to keep regular discussions in the main channel
20:01:11 <clarkb> #endmeeting