19:01:12 #startmeeting infra 19:01:13 Meeting started Tue Sep 1 19:01:12 2020 UTC and is due to finish in 60 minutes. The chair is clarkb. Information about MeetBot at http://wiki.debian.org/MeetBot. 19:01:14 Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 19:01:16 The meeting name has been set to 'infra' 19:01:27 #link http://lists.opendev.org/pipermail/service-discuss/2020-August/000080.html Our Agenda 19:01:31 o/ 19:01:40 #topic Announcements 19:01:44 I didn't have any announcements 19:02:06 forum cfp is open 19:02:16 did that just happen today? 19:02:23 yep 19:02:28 4 hours ago says my email, thanks for the notice 19:02:34 I had missed that 19:03:04 looks like the deadline for submissions is september 14 so we have about 2 weeks to add proposals there 19:03:29 #topic Actions from last meeting 19:03:38 #link http://eavesdrop.openstack.org/meetings/infra/2020/infra.2020-08-25-19.01.txt minutes from last meeting 19:03:55 No actions from last meeting. It was pretty informal too since many were out on vacation or otherwise unable to attend 19:03:59 o/ 19:04:08 #topic Specs approval 19:04:24 I did approve fungi's identity broker service spec some time ago in the past 19:04:38 If folks want to start working on that there is a spec now :) 19:04:41 late last week i think 19:04:58 the days all run together anymore 19:05:03 i was "on vacation" so my memory of last week is hazy 19:05:31 #topic Priority Efforts 19:05:39 #topic Update Config Management 19:06:20 I didn't have much on this topic other than to call out that I'm investigating nodepool builder arm64 images again now that pyca/cryptography has arm64 wheel published on pypi 19:06:51 It looks like there are additional things we need wheels for (expected) and the openstack wheel mirror currently is sufficient to get the job to run under the hour timeout 19:07:26 all of that to say we haven't regressed on what we already knew and we have made some progress 19:07:41 yay progress! :) 19:08:12 did anyone else have config management changes to call out? Any new ansiblification or docker container work? 19:09:18 i didn't have any, i don't think 19:09:46 #topic OpenDev 19:09:56 #link https://review.opendev.org/#/c/748263/ Update opendev.org front page content 19:10:11 This is a change that ttx put together to try and capture the "why opendev" better 19:10:28 looks like it has received some reviews and CI is currently failing on a setuptools 50 issue 19:10:43 if ya'll have time to review that it would be great, I think it is a good improvement 19:11:41 On the Gerrit upgrade side of things Luca has confirmed that disk performance is incredibly important for the gerrit upgrade process. That means we do want to update review-test to mimic production with its cinder volumes 19:12:07 There is one or two small things I want to test on my local gerrit upgrade test setup, but then I'll be looking into that next 19:12:28 I think I'll start with trying to add a volume to the existing server 19:12:55 And finally: Non master repo HEAD support for new projects has landed. We can offer this to new projects now. 19:13:25 I believe that gitea, gerrit, jeepy, and gerritlib work is complete to make this possible. That said it won't surprise me if we discover there are unknown unknowns we have to address once this is attempted 19:14:22 Any other opendev related topics to bring up? 19:14:30 this is sufficient for creating a new project with a different default branch, but doesn't cover changing the default branch of an existing project, right? 19:14:36 fungi: correct 19:14:57 I am hoping we can work through those unknown unknowns with a new repo before we tackle the additional steps of converting a project 19:15:39 makes sense 19:15:45 I think the conversion process for an existing project will look something like: update projects.yaml to set the branch value then run an outof band playbook to update gitea and gerrit values. Fixing CI is an exercise left up to the reader 19:16:18 pabelanger confirms that zuul itself is fine when you do a conversion 19:16:28 but jobs content is likely going to need updates depending on the projects 19:16:44 (we also added a unittest to zuul to ensure we don't regress that functionality) 19:17:24 s/conversion/rename/ 19:18:23 #topic General Topics 19:18:36 #topic Setuptools 50 Release 19:19:03 ~Sunday pypa released a new version of setuptools which has had a number of regressions assocaited with it 19:19:44 in particular it seems that older python3 like 3.5 has import issues for setuptools when installing setuptools 50. The other issue we've seen is the change in instalation directory when building packages from source and not installing them from wheels 19:20:13 in some places we're settinga flag to use the distro distutils (its actually setuptools vendoring distutils that causes the problems) and in others we are pinning setuptools 19:20:31 clarkb: has the change in directory not been matched with the packages not being found in any instances you're aware of? 19:20:40 it does look like upstream is working on fixing the older python issues though the installation location isn't expected to change again (that was intentionaly for them) 19:21:13 it's also worth remembering that setuptools doesn't consider itself to be "vendoring" distutils, the idea is that the setuptools maintainers are taking over distutils maintenance so it can eventually be removed from stdlib 19:21:34 ianw: the ones I've seen were all https://github.com/pypa/setuptools/issues/2357 and the /usr/local/bin vs /usr/bin problems 19:21:51 (i.e. what i'm saying is that s,/usr/local/bin,/usr/bin, doesn't work, because the binary can't find it's packages anyway) 19:22:31 ianw: oh I haven't seen that I don't think (probably because we are either using distro distutils or pinning to 49) 19:22:48 its possible that if python3.5 issues are fixed upstream we'd see a lot more of that? 19:22:53 this is the error from that ttx change clarkb mentioned earlier: https://zuul.opendev.org/t/openstack/build/b6aaf1af991948ffa3062a4c3be94906 -- what error is this? 19:22:59 yeah, debian's patched distutils is necessary to work with the patched module import path in their interpreter 19:23:15 corvus: I believe that to be https://github.com/pypa/setuptools/issues/2357 its a xenial python3.5 job failing to import setuptools 19:23:46 yeah -- https://github.com/pypa/setuptools/issues/2350#issuecomment-684082760 19:24:13 basically if you're seeing your binaries moved; you've used "pip install -e" or possibly setup.py directly, or otherwise *not* built a wheel 19:24:34 because, of course, the paths are totally different and wheels continue to work while installing doesn't 19:24:34 clarkb: so iiuc, we're waiting on a setuptools release for that? 19:25:04 corvus: ya or we can pin setuptools there or use the disto distutils. I only just looked at that job a few minutes before the meeting so haven't had time to think about which approach may be best 19:25:20 there is a 50.0.1 from a couple hurs ago 19:25:22 *hours 19:25:27 so maybe we shoul drecheck and see if it owrks now? 19:25:40 done 19:25:47 (i wouldn't hold much hope for that, i was scrolling through the issues, there's quite a few, and remember someone said it didn't fix it) 19:26:29 but ya the issues have been pretty far ranging and exhibit in different ways. Wanted to call out the change in case anyone missed it or had failures they weren't sure about 19:26:37 I'm happy to help debug any that pop up 19:26:44 ++ 19:27:12 devstack has just used the environment variable ... 19:27:45 the reason I went with pinning on our image builds instead is they build on a ton of different platforms and I didn't want t othink through waht using distutils on each of them may mean 19:28:01 was easier to use setuptools 49 in that case an have the same setup across the board in the virtualenvs 19:29:14 Any other questions or thoughts on this topic? I don't expect we'll solve the problems here and now but wanted to call it out as an ongoing issue 19:29:46 clarkb: when do you think we'll lift the pin? 19:29:54 ie, what needs to happen for that 19:29:57 * ianw declines to offer furter thoughts :) 19:30:10 corvus: we need the older python3 situation to be fixed 19:30:23 corvus: upstream seems to acknowledge the bugs and is working on them so that should eventually happen 19:30:32 there are like 8 bugs at least though :/ 19:30:35 probably if 748263 passes, that's a sign we can unroll it? 19:30:51 "Indeed, the impact is undesirably huge. I had gone down the route of attempting to maintain compatibility and incorporate the debian-specific changes, but decided against that. However, that was before #2259, which expanded the scope of the hack. Perhaps we should re-open #2232 to address Debian-specific needs. 19:30:51 " 19:31:04 https://github.com/pypa/setuptools/issues/2352#issuecomment-684989322 19:31:18 so i think the issue is in flux (that's from the maintainer) 19:31:48 fungi: isn't 263 running on a pinned image? 19:32:07 corvus: we only pinned the venvs we preinstall on the image 19:32:11 oh gotcha 19:32:12 oh, i guess it failed before we rolled the images back, not after? 19:32:28 if the job content itself is doing setuptools things it may trip over the issue 19:32:46 but yeah, if that job is installing a new setuptools on its own, then it's independent of the image pin 19:33:04 I do expect upstream will figur ethings out 19:33:09 they sorted stuff out after the 48 release 19:33:18 it may just need a little time given the number of problems with 50 19:33:29 2020-09-01 14:00 was long after we rolled images back, so i think it must have pulled its own setuptools 19:33:53 but i haven't had time to dig into the build log yet 19:35:10 Making ask.openstack.org read only 19:35:17 er that needed a topic prefix 19:35:19 #topic Making ask.openstack.org read only 19:35:36 #link https://review.opendev.org/#/c/746497/ set ask.openstack.org to read only 19:35:49 this has had a fair bit of feedback at this point and no one has objected 19:36:05 I was hoping to give the meeting here one last chance to hold off, otherwise I planned to approve the change 19:36:08 seems we can go forth and disable 19:37:26 I didn't expect any objections. Let me know before the end of the meeting I Guess if you did have one and we can wait 19:37:32 #topic PTG Planning 19:37:49 October PTG registration is now open: https://www.openstack.org/ptg/ 19:37:59 I don't think I've done that myself yet so this serves as a personal reminder 19:38:10 I also haven't gotten to the planning document yet simply too many other things popping up lately 19:38:28 #link https://etherpad.opendev.org/opendev-ptg-planning-oct-2020 October PTG planning starts here 19:38:33 clarkb, I saw you filled out the ethercalc, but not the survey? 19:38:42 diablo_rojo: I did both 19:38:50 clarkb, oh cool. 19:38:51 Thanks! 19:38:59 I guess double check you received my response? 19:39:12 and any one should feel free to put ideas on that etherpad 19:39:30 clarkb, I don't see it in the spreadsheet, but it could have gotten missed by the integration. 19:40:05 i'm also checking to see if anyone thinks the openstack tact sig needs dedicated ptg slots, but my guess is we'll just tag along with opendev 19:40:20 diablo_rojo: let me know if I should fill it out again 19:40:42 clarkb, looks like its not in the formstack data either so, yes please :) 19:40:46 (or with the openstack qa team, depending on topics) 19:40:46 Maybe you didn't click submit? 19:41:11 maybe I filled out the previous survey link bceause my email seraching foo was bad 19:41:14 I'll resubmit 19:41:25 clarkb, thanks in advance 19:41:37 Here's the link if you need: https://openstackfoundation.formstack.com/forms/oct2020_vptg_survey 19:41:41 thanks 19:41:48 #topic Bup and Borg Backups 19:41:56 the original announcement linked old ethercalc and old survey, fwiw 19:42:16 ianw: anything we can do to help on this? Just making sure we don't forget about it or have it stuck on something we can help with 19:42:24 (I know there has been a lot of other happenings recently) 19:42:56 yeah it's just been me not having a chunk to bring it all up 19:43:01 a chunk of time 19:43:08 k 19:43:15 #topic Open Discussion 19:43:28 The floor is open 19:45:37 ask.openstack.org going once... 19:45:42 twice... 19:45:44 ;) 19:45:52 fungi: you want to hit the button or should I? 19:45:58 re ask.o.o 19:46:22 Sounds like that may be the meeting. Thank you everyone 19:46:27 push the button frank 19:46:27 Do it do it do it. 19:46:31 #endmeeting