19:01:13 #startmeeting infra 19:01:14 Meeting started Tue Feb 25 19:01:13 2020 UTC and is due to finish in 60 minutes. The chair is clarkb. Information about MeetBot at http://wiki.debian.org/MeetBot. 19:01:16 Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 19:01:18 o/ 19:01:18 The meeting name has been set to 'infra' 19:01:22 #link http://lists.openstack.org/pipermail/openstack-infra/2020-February/006602.html Our Agenda 19:01:28 o/ 19:02:08 #topic Announcements 19:03:09 I'll be out tomorrow (going to try my hand at fishing) and then thursday morning I visit the tax man. Mostly just a heads up that like last week this week is a weird schedule for me. I'm trying to get stuff out of the way for next week (when i think fungi will not be around) 19:03:27 #topic Actions from last meeting 19:03:33 #link http://eavesdrop.openstack.org/meetings/infra/2020/infra.2020-02-18-19.01.txt minutes from last meeting 19:04:00 This didn't get recorded as a formal action, but I volunteered to write a spec for the pip and virtualenv python situation with our test images. I did do that so lets dive right into specs 19:04:07 #topic Specs 19:04:08 o/ 19:04:24 #link https://review.opendev.org/#/c/709579/ Spec for cleaning up python dev tools on our CI images. 19:04:40 This comes with a variety of supporting documentation and changes. I'll post links for them in a sec 19:04:57 clarkb: thanks, i haven't had a chance to review it yet, but that looks very helpful 19:05:02 It would be great if those involved in untangling the virtualenv upgrade mess could read that over 19:05:25 What I'd like to do is get it into shape then start passing it around with the constituent projects using the CI system 19:05:47 basically overcommunicate the chagne since it has potential to be a painful change. Though the plan is to make it as unnoticeable as possible 19:05:55 #link https://etherpad.openstack.org/p/pTFF4U9Klz : clarkb writeup of major issues 19:06:00 #link https://review.opendev.org/707499 : discussion in comments 19:06:04 #link https://review.opendev.org/707513 : use venv for glean 19:06:10 #link https://review.opendev.org/707750 : use venv in project-config elements; drop pip-and-virtualenv inclusion from element and move to individual configs; add node type with no pip-and-virtualenv. this can be used for job testing. 19:06:25 * diablo_rojo sneaks in late 19:06:27 we should probably roughly agree to the plan in the spec before we start landing those early safer changes 19:06:51 I do think we can land the last two changes linked without buy in from users as the first is glean specific and the second does this in a testing capacity 19:07:25 i hope it won't be as painful nor drawn-out as the migration off thick test images 19:07:42 seems likely to have less overall impact at least 19:08:08 I think the end result will be fewer surprises for test jobs as our nodes will act a bit more like the ones you get from centos and ubuntu 19:08:13 ++ will go through today 19:08:21 even if it impacts more projects in total, it does so in more standardized ways we can work out solutions for 19:09:17 I look forward to your feedback. Thank you 19:10:23 Then we have coincidental overlap in the need for updating our simple 404 tracking (and website trackign in general). I had an afternoon with a couple foundation people where the subject came up and ianw had a change up to remove our script as it requires writing to afs volumes and complicates the files to static.o.o migration 19:10:54 This prompted me to look into options for tools around there. When I was debugging haproxy logs at one point I had hacked up a goaccess config to render graphs for me and it supports apache logs out of the bogx 19:11:05 This led to writing another different spec 19:11:10 #link https://review.opendev.org/#/c/709236/ Website activity stats 19:11:42 yeah, goaccess seems really promising. when i was looking into/testing solutions for this i didn't find it 19:11:44 I think the key here is to agree on goals more so than specific technical details. For example are we all comfortable with publishing the stats planned there 19:12:17 it also serves as a great first test for our new opendev governing body 19:12:41 ianw has had good feedback for implementation ideas and I think we can edit the spec as necessary to reflect that. What I don't want to do is surprise anyone with public stats reporting that might be too generous (though I think the plan is quite conservative and should avoid any issues) 19:12:51 did ianw push up a change to partially implement this? 19:13:07 in the mean time, if we'd like to restore the existing test and also do a little POC on zuul collecting these stats we have 19:13:10 (i thought i saw something like a change that logged into static.o.o and did something via a zuul job) 19:13:11 #link https://review.opendev.org/#/c/709382/ 19:13:24 corvus: ya it basically takes our old 404 only reporter and adopts it to the newer hosting setup 19:13:44 ah cool. so that's that part of the process -- then if we like goaccess, we use that mechanism to run it? 19:13:49 I think that is a good in between spot to avoid regressing on the 404 reporting functionality while we spin up the richer reporting 19:14:03 corvus: yup exactly. One of the things goaccess can do is 404 reporting 19:15:05 corvus: if you have time to review the spec it would be great to get your perspective with a zuul hat on 19:15:11 on it 19:15:31 i'm also going to wear my "don't share pii hat" 19:15:34 ++ 19:15:47 i stapled that hat to my head already 19:16:13 Anything else on specs or should we continue? 19:16:23 much of the plan there is a result of discussions we had in #openstack-infra about things we shouldn't report because of that 19:17:04 #topic Priority Efforts 19:17:07 #topic OpenDev 19:17:31 The openstack governance change has four TC Acks. I think we need 3 more votes in order to land that 19:18:06 don't be surprised if that lands in the near future :) 19:18:28 On the gitea side of things mordred has a change up to upgrade to 1.11.1 19:18:39 it is currently failing testing, I haven't had a chance to look at why yet though 19:18:40 #link https://review.opendev.org/#/c/705807/ Upgrade Gitea to 1.11.1 19:18:59 I did go through the release notes and called out a couple that I thought might impact us but on further investigation we should be fine (details in a comment on the change) 19:19:16 clarkb: I set up an autohold 19:19:19 because I can't tell from logs 19:19:34 hopefully I'll both figure it out - and figure out how to make it possible to figure out from logs :) 19:19:50 GEtting to 1.11 will be our stepping stone to 1.12 which has the git commit cache support 19:20:13 we should probably consider deploying 1.12 pre release, but we can figure that out once on 1.11 19:20:27 and that's what we hope will significantly speed up ui rendering for large repos? 19:20:32 fungi: yes 19:20:39 * fungi does a little dance 19:21:32 Anything else on OpenDev? 19:22:17 #topic Update Config Management 19:22:36 mordred: I believe there has been a slight change of direction in the way we build gerrit images. Want to fill us in? 19:23:05 yeah - so - the previous way was based on a bazel builder image 19:23:20 so that we could make an image that had things like bazel and java and whatnot pre-installed and just focus on building gerrit 19:24:02 UNFORTUNATELY - bazel is still a quick moving target, so the gerrit folks have a .bazelversion file in repo that contols which verison of bazel should be used - and we were constantly chasing job breakage when the file would not match our buiklder image version 19:24:38 so instead we're now using copies of roles corvus wrote for upstream gerrit which does the build on the build node using bazelisk - which installs the right version of bazel 19:24:46 and then the dockerfile just copies in the war file built from that 19:24:59 should be more future proof - as well as handling different bazel versions across different branches 19:25:02 \o/ 19:25:14 The biggest impact to us as individuals is we'll have to do the bazelisk pre step if building the images locally 19:25:24 yeah 19:25:43 but that honestly doesn't happen super often 19:25:52 mordred: did those chagnes land? put another way are there things that need review? 19:25:54 yeah. we could move that into a docker image build, but there'd be less direct upstream role reuse (but they're not that complicated so maybe it's not a big deal) 19:25:54 but also we're better aligned with how gerrit builds gerrit? 19:25:59 also - change up to add 3.1 to the list of gerrits we build 19:26:01 https://review.opendev.org/#/c/709295/ 19:26:16 clarkb: the changes other than 3.1 landed 19:26:17 in general, i'm in favor of going with the hybrid approach mordred has up now, and moving stuff into containers if we feel we need it. 19:26:25 corvus: ++ 19:26:30 #link https://review.opendev.org/#/c/709295/ Build a Gerrit 3.1 docker image 19:27:17 fungi: it's a little hard for me to say how "gerrit builds gerrit" 19:27:43 any other config management updates to call out? 19:28:05 oh - fwiw... 19:28:10 for some values of that (ie, how a dev might expect to build it on their workstation) i'd say yes. for other values (how they build release artifacts, container images, etc), i'd say this is different, but that we don't want to emulate that due to the high amount of gerritforce-ci-specific stuff involved. 19:28:27 we found an issue with LE certs and review.opendev.org today that isn't super important but thought I'd mention the root cause ... 19:28:39 corvus: ahh, thanks for the clarification 19:28:41 we weren't making the LE certs for review.o.o that we thought we were (we're not using them yet, so no biggie) 19:28:46 mordred: note I think it may still be haivng problems (needs further investigating) 19:28:56 oh, this was probably more for the opendev topic, but new gerrit talk reminded me, i pushed up a change to opendevify the welcome new contributor message: 19:28:58 #link https://review.opendev.org/708975 Overhaul default welcome message for OpenDev 19:29:02 and the reason is that we had host_vars in ansible for review01.opendev.org ... but the host is review01.openstack.org 19:29:13 just to keep in mind as we work on things in the future 19:29:36 clarkb: nod 19:29:47 mordred: lesson learned: transitions are hard? 19:30:02 dotting all the Ts and crossing all the Is is hard 19:30:06 clarkb: yah. as are names 19:30:35 #topic General Topics 19:30:46 ttx: did you want to start us off with the xwiki item? 19:30:52 Sure. Hi! 19:30:56 As you may know we've had demand for wiki space from a number of projects 19:31:03 So far our response has been to ask them to squat on wiki.openstack.org 19:31:11 But that is not a long-term solution. And we don't really want to set up more Mediawikis either. 19:31:18 So I've been looking into open source wiki solutions that: 19:31:29 - would have limited maintenance costs, like stronger spam prevention features 19:31:35 - would have more of a "structured wiki" approach, to limit stale pages 19:31:46 - would be pluggable into openstackID to avoid maintaining a new set of credentials (or increase our UbuntuOne tech debt). 19:31:55 I've experimented with XWiki (see xwiki.org) and found it satisfactory. 19:32:00 (or at least as satisfactory as a wiki can be) 19:32:14 It's an open source solution, and the fine folks at XWiki.com offer free hosting on their "Xwiki cloud" for open source projects. 19:32:26 It allows for "subwikis", which are basically wikis with a root document and search context which live under the same main wiki domain 19:32:44 My current thinking is to take them up on their hosting offer, set up the main wiki under wiki.opendev.org, and experiment with a subwiki for the "Open Infra Labs" initiative 19:33:00 If that experiment works well, we would extend the possibility of a subwiki for other projects. 19:33:23 If it works *really* well, we could envision to transition wiki.openstack.org content and finally abandon Mediawiki maintenance. 19:33:36 And if it works *too* well and we get past the open source offer hosting limits (10 subwikis) then we could always look at hosting a Xwiki instance ourselves. 19:33:51 You can see a demo at: https://opendevwiki.demo.xwiki.com/ 19:33:55 You can log in at the topright corner 19:34:02 ttx: the subwikis would be subdomains of opendev.org ? 19:34:05 Opendev gets the wiki at the top level, which is used to list the subwikis, but could also be used to describe more of Opendev if you want. 19:34:12 i wonder how this relates to discussions we've had about possibly leveraging the github-wiki-like feature in gitea 19:34:15 no, more of a subdirectory 19:34:36 But it looks like the root of a wiki. So if you search within a subwiki, it looks for local results 19:34:36 ttx: nod. but still contained within the wiki.opendev.org domain 19:34:42 gotcha 19:34:44 Each subwiki can have a different theme and panels, which I tried to show in the demo. 19:35:08 The demo has superlong URLs but I suspect they can collapse that a bit 19:35:15 fungi: so far the github-wiki-like things in gitea seem to assume people need commit access to the git repo to make wiki changes - last I checked 19:35:23 operationally I think it would run similarly to gerrit (its a monolithic java app with a db connection) 19:35:31 if we end up needing to run it ourselves 19:35:32 mordred: yeah, and is tied to specific repos i guess 19:35:35 yeah 19:35:39 https://www.xwiki.org/xwiki/bin/view/Documentation/UserGuide/Features/ContentOrganization/ explains subwikis vs. nested pages 19:35:49 Should we pursue this, I volunteer to conduct the experiment 19:36:04 I would be more than happy to not run a wiki 19:36:10 They agreed to allow us to use a custom domain, which would make transitioning in the future a lot easier 19:36:27 it also touches on uor still as-of-yet unresolved need for an opendev sso 19:36:32 but yes, if the hosting works for us, I'd rather continue to have them run it 19:36:34 s/uor/our/ 19:36:44 fungi: yeah. that's the biggest issue I can think of 19:36:48 and encourage them to have a successfl wiki hosting business rather than continue developing "Pro" plugins 19:36:57 fwiw, i didn't know about either the demand for wikis, or the openinfra labs project; hopefully with the new opendev governance model we can improve those communication channels 19:37:34 corvus: we had Airship and StarlingX ask for wikis, we just redirected them to wiki.openstack.org 19:37:44 most of the "demand for wikis" is that every time we say we're going to wind down wiki.openstack.org we get a lot of folks jumping up and down saying we can't take it away because they have no good alternative 19:38:02 OpenInfraLabs is the new name for the Massachussets Open Cloud, extended beyond MA 19:38:07 oh neat 19:38:18 ttx: fungi's thing is the main thing I'd be worried about that I think we'd need to come up with an answer for 19:38:35 mordred: auth? 19:38:38 yeah 19:38:46 It's new, but one of the first things they asked for wasa wiki (don't ask me why, I hate wikis) 19:38:51 because in a hosted system, transitioning auth would be difficult/impossible? 19:39:07 corvus: or someone else's problem we can just throw money at? :) 19:39:08 ttx: is it possible to allow arbitrary sso? so that users can choose what identity they log in with (github/openstackid/ubuntuone/etc) 19:39:54 that might be a reasonable intermediate step? That assumes people don't want to tie wiki edits back to CCLA or similar 19:40:06 mordred: yeah, a few possibilites there, and knowing the answers before we start would be good 19:40:09 clarkb: Not 100% sure. 19:40:19 corvus: or - at least list that slving that is likely to be step 0 19:40:32 By default when unconfigured it allows people to provide a OID URL in an ugly form 19:40:46 But once configured it just forwards to openstackID very nicely 19:41:00 depending on the answer, maybe we can proceed full steam ahead and clean up auth later, or maybe we have to implement the opendev multiplexer first 19:41:09 I'd be happy to say "yeah, let's go for it, must solve auth" 19:41:11 corvus: yeah 19:41:35 I'm not super-concerned with migrating the few users we'll have in the PoC 19:42:04 if there's sufficient admin-level access to dump the equivalent of the user auth ids, group memberships and acls, then we can in theory punt on that 19:42:04 ttx: so we believe our xwiki friends can migrate a set of users for us should we need such a thing to be done? 19:42:13 or fungi's thing 19:42:23 good point. i don't think it has to block poc, but we should explore the question during the poc and have an answer before prod 19:42:29 corvus: ++ 19:42:30 ++ 19:42:50 mordred: I think we can also live with users being duplicated during transition 19:43:09 yeah, that's a valid answer too :) 19:43:11 yeah. I mean - also - it'sa . wiki, so it's not like it needs tight auth integration with anything else we do 19:43:18 probably the biggest question for opendev is whether we're okay with the opendev name attached to that hosted service... i see this as the same question for the hosted zanata replacement we've been discussing as well 19:43:26 I mean, they have been helpful so far, but the "open source offer" comes with minimal support 19:43:56 i read that as "we'll host this for you and even offer some support" 19:44:05 which sounds a lot better than no support 19:44:18 but I'm trying to pay them back with logos and user studies, so maybe they will be happy with us. I know when the OSI calls them they are VERY helpful 19:44:28 (OSI being another xwiki cloud user) 19:44:37 yeah- we'd be a pretty high profile user for them 19:45:00 It's more a "no guarantee" offer than a "no help" 19:45:05 i also like that hosting it on a custom subdomain means we can be in control of a migration (which might even be seamless). i think that's an important point. also a question to explore during the poc is what a migration to self-hosting would look like (exploring data exports, etc) 19:45:22 as a time check we have several more topisc to get through (we should probably get to them when we have 10 minutes remaining) 19:45:28 corvus: yes, I would not have considered it if they said custom domain was not a possibility 19:45:30 and the xwiki project is under the ow2 consortium, who have been great friends to openstack in the past 19:45:42 another question for POC - SSL 19:46:00 mordred: ++ 19:46:06 yeah, i'm cool saying go forth and poc 19:46:09 ++ 19:46:09 I wonder if we can point an acme record at them and then they LE 19:46:14 but definitely somethign to check on 19:46:30 Not sure what they will ask on that. Did not want to go any further without your blessing 19:46:42 ttx: can you write up a spec with questions for the poc period? 19:46:49 don't block on the spec 19:46:50 If you say "OK" we'll tread carefully 19:46:53 it can be a very short spec, probably 19:46:56 they can probably do the file/webserver based auth for LE? 19:47:07 more of a place where we can collect questions, answers, and develop plans for if we go to prod 19:47:11 ianw: ah yup 19:47:14 ok, open questions on the poc 19:47:40 not sure what LE is 19:47:46 ttx: letsencrypt 19:47:49 hah 19:48:48 I'll stop hijacking the meeting :) 19:49:26 thanks ttx! 19:49:35 can we agree on action items for 1) start the poc; 2) start a spec to collect feedback? 19:49:36 action me on a spec, and get them to set up the instance, coordinate with clarkb on getting SSL done 19:49:49 #action ttx write xwiki spec to capture POC questions 19:49:55 i agree on 1&2, yes 19:50:11 #agreed ok for ttx to proceed with xwiki POC while we review the spec 19:50:18 ttx: yay thanks! 19:50:51 Next up is a note that I did request space at the vancouver PTG. It sounded like we'd have ~3.5 people there and that worked out in shanghai pretty well. I think my goal will be to have some official time and space as that allows others to find us easily then we can organize in a more ad hoc fashion for the other times which worked out in shanghai. 19:50:52 thanks all! 19:51:15 If you'll be in vancouver feel free to find us! 19:51:31 i'll wear something bright 19:51:49 you always wear somethign bright 19:52:06 I expect the weather will be excellent when we are there so maybe we can do something fun as a group too 19:52:20 fungi: any progress with the existing wiki server management? 19:52:22 (speaking of wikis) 19:52:33 clarkb: nope, nope, nopeski 19:52:56 And that takes us to the files.openstack.org and static.openstack.org migration to static.opendev.org 19:53:09 ianw: I think docs.opendev.org is the last remaining proper site then we have to add redirects? 19:53:14 link https://review.opendev.org/#/c/709402/1 Move docs.opendev.org to static.opendev.org 19:53:17 #link https://review.opendev.org/#/c/709402/1 Move docs.opendev.org to static.opendev.org 19:54:08 yes, that docs one should be quick 19:54:20 ianw: AJaeger and everyone else that helped: Thank you for pushing this along 19:54:20 there's a few things open 19:54:21 https://review.opendev.org/#/q/status:open+topic:static-services 19:54:52 files.openstack.org got caught up in this too, so we might as well push on moving that too. the only thing remaining there is the git redirectors, which have a change to migrate up 19:55:13 #link https://review.opendev.org/#/q/status:open+topic:static-services Finish up static.openstack.org migration 19:55:25 also we figured out yesterday that we're not providing backward-compatible redirects for projects outside the openstack namespace. i meant to try and find more of those besides osf/openstackid but haven't tried to match up the lists yet 19:55:52 we've discussed it before, but i'm not sure the idea of the service load balancer is getting traction, the reviews have been there for several months 19:55:56 for tarballs.o.o redirects i mean 19:56:08 if we want to just punt on it and move it into apache config, i don't mind 19:56:48 ianw: I expect that is fungi's preference. I think I've largely been convinced that apache would be fine particularly since we don't have to juggle ssl 19:57:07 my only real objection to the haproxy idea is that we're already running apache on the servers we'd be redirecting to, so could accomplish the same with some aliases in existing vhost configs 19:57:10 it's certainly simpler, and we can save the haproxy setup for something when we need more of the load-balancing features 19:57:45 and one less service to manage if apache is already required either way 19:58:33 And we are just about at time 19:58:36 i mean, i think using haproxy for redirects is probably a fine solution and would work 19:58:37 #topic Open Discussion 19:58:47 I'll open the floor if there is anything else in the last minute 19:58:57 ok, i'll work up a change to put them in apache, i'm not really fussed i just want it finished now :) 19:59:09 ++ to finishing 19:59:13 thanks ianw! 20:00:06 And that is all we had time for. Thank you everyone. 20:00:08 #endmeeting