19:01:02 #startmeeting infra 19:01:02 Meeting started Tue Oct 26 19:01:02 2021 UTC and is due to finish in 60 minutes. The chair is clarkb. Information about MeetBot at http://wiki.debian.org/MeetBot. 19:01:02 Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 19:01:02 The meeting name has been set to 'infra' 19:01:04 #link http://lists.opendev.org/pipermail/service-discuss/2021-October/000293.html Our Agenda 19:01:10 #topic Announcements 19:01:23 This wasn't on the agenda but I saw an email going by indicating fungi may be doing gpg stuff? 19:01:33 fungi: ^ do you need us to go and sign that key in the near future? 19:01:46 ooh, i'm actually in the midst of completing that now 19:02:13 i'm updating the process, so not exactly no 19:02:21 ok, will wait for those updates then :) 19:02:29 #topic Actions from last meeting 19:02:35 #link http://eavesdrop.openstack.org/meetings/infra/2021/infra.2021-10-19-19.01.txt minutes from last meeting 19:02:42 There were no actions recorded last meeting 19:02:42 with the sks keyserver network collapse, the next best option is to switch to the gnupg.org keyservers, but they don't publish thord-party signatures 19:02:54 er, third-party 19:03:16 #topic Specs 19:03:21 #link https://review.opendev.org/810990 Mailman 3 spec 19:03:34 This is still needing reviews. infra-root if you can find time for this it would be greatly appreciated 19:03:38 #action infra-root review Mailman 3 spec https://review.opendev.org/810990 19:04:56 Not sure there is much more to say on that other than we need reviews. 19:05:03 #topic Topics 19:05:09 #topic Improving OpenDev's CD throughput 19:05:40 ianw: I rebased your first chagne to fix a merge conflict I introduced with some of the gerrit upgrade cleanup work. This change is +1 now 19:05:45 ianw: I guess we should try and land it this week? 19:05:56 #link https://review.opendev.org/c/opendev/system-config/+/807672 is the change I'm talking about 19:06:10 i noticed some of the other dependent changes acquired a -1 somehow, i haven't dug into it yet 19:06:38 but yep, we can keep at it :) 19:06:39 ah maybe we should ensure the rebase I did didn't introduce any of those problems 19:06:49 I'll put this on my TODO list for this week though 19:07:12 #topic Gerrit account cleanups 19:07:52 I haven't done the bulk work on this that is on my TODO list but did do a one off to allow a user to log in again (they had problems and I cc'd funig and ianw on the last email since they have looked at gerrit account stuff before) 19:08:03 The log of the work I did for that user is on review02 in the typical location 19:08:32 essentially I retired an old account and cleaned up an email address from that account so the user can associate it with a newer account they appear to be using (based on openid external ids) 19:08:52 #topic Fedora 34 Booting Problems 19:09:05 ianw: ^ can you quickly fill us in on the fun around that? 19:09:58 at least one problem is that it seems the initramfs has stopped shipping the drivers to mount the root disk on Xen, i.e. RAX 19:10:26 there is a dracut fix for that, but it's hard to deploy 19:10:42 #link https://bugzilla.redhat.com/show_bug.cgi?id=2010058 fedora 34 initramfs lacking xen drivers 19:10:46 #link https://review.opendev.org/c/openstack/diskimage-builder/+/815409 19:11:00 is a generic fix for dracut-regnerate, which ... uses dracut to regenerate the initramfs 19:11:05 ianw: I left a comment on that dib change wondering about centos 7 19:11:19 #link https://review.opendev.org/c/openstack/diskimage-builder/+/815385 19:11:29 centos 7 is python2.7 platform and uses dracut? Or maybe dracut is newer than centos 7? 19:12:26 even centos7 has python3 now 19:12:46 ianw: will dracut run with it? 19:12:55 I thought that centos 3 was independent of all the normal system tools 19:13:22 it's not dracut we need python-yaml there for, it's to run the dracut-regenerate tool from bin/ which is written in python 19:13:46 oh! 19:14:26 yeah, a bit confusing 19:14:53 I've updated my review to a +2 thanks 19:15:37 Then we also need to install some special dracut testing package because they haven't landed it upstream yet? 19:16:08 yeah; and then figure out how to test it, or deploy it 19:16:50 the most expedient method would be to hand-edit in the change to the nodepool-builder container 19:16:58 my previous effort at that wasn't a shining example 19:17:43 Probbaly the best thing is to do a small local build without all the git repos and extra stuff. Then manually upload to rax and boot it. The hardest thing about that is the manually upload step since rax image upload process si wird 19:17:52 you're basically forced to use shade/openstacksdk 19:18:03 yes ... 19:18:08 what i was trying to avoid :) 19:19:45 i'm not sure if they're going to regenerate the default initramfs for f34 or what 19:20:18 You'd hope they will since it has anothre 7 months or so of life. Kind of annoying that fixing existing users is less important that addressing an unreleased release 19:20:43 if i could get into a rescue mode i could copy in the initramfs, but that doesn't seem to work either 19:21:25 ianw: I think you may have to override the rescue mode host image. By default it uses the image that is being rescued 19:22:25 yeah, i thought the idea was that it boots into a completely separate mini-vm that then has your old vm as a block device 19:22:53 ianw: it does, but it has to have an image to boot that separate VM with and by default it uses the same image I think 19:23:09 there is a flag to set a different image for the boot which might solve the problem 19:23:42 huh, ok ... well i'll have a poke but i wasn't aware of that flag 19:24:22 Anything else on this topic before we move on? 19:24:41 i guess i'll have to try the build method 19:25:02 no, it's just annoying and not particularly what any of us want to be doing :) 19:25:11 indeed, thank you for looking at it 19:25:24 #topic Begin planning for Gerrit 3.4 upgrade 19:26:07 We're on 3.3 now but 3.4 exists and we should start looking at an upgrade to that version. As with 3.2 -> 3.3 we can do a downgrade from 3.4 -> 3.3 if we need to 19:26:44 We are already building 3.4 images and have an upgrade job to test the 3.3 -> 3.4 upgrade. There appears to only be an index change between the two and that gets updated online after restarting on 3.4 19:27:16 #link https://www.gerritcodereview.com/3.4.html Release notes 19:27:48 I think it would be good if we can find time to read over release notes to find any scary things or things that we expect will need testing. Then hold a 3.4 test node and manually interact with it a little bit to look for any unexpected changes 19:28:03 We can also use that held node to test a downgrade to 3.3. 19:28:19 "HTML plugin support is removed" -- we don't have anything still hanging on to that do we? 19:28:36 ianw: I wonder if our theming is considered an html plugin? 19:28:49 I don't know how that plugs into gerrit 19:30:04 it's a polygerrit plugin 19:30:05 https://groups.google.com/g/repo-discuss/c/YJY8oQZZo44/m/nF1BtuPQAwAJ lists "zuul-status" as requiring updates to polymer 3 19:31:20 (our display is zuul-summary-status confusingly and isn't affected) 19:31:29 zuul-status isn't one that we use either 19:31:49 fungi: I guess we needt o double check that it is polymer 3 safe 19:32:00 we should be able to verify that with a held test instance (either it will work or not) 19:32:28 The other thing we should try to do is improve the automated testing with new stuff as we find it so that we're not needing to do extra checks for the 3.5 upgrade 19:32:28 i'm happy to take an action item to get a checklist started, setup a held node and investigate downgrade 19:32:50 #action ianw Start Gerrit 3.4 upgrade checklist, setup a held node, and investigate the downgrade 19:33:07 ++ yep anything we find that we can test we should :) 19:33:08 ianw: thanks. And ya at this point I think we just need one or two people to start looking at it a bit more in depth so that we can fix issues and then figure out a plan from there 19:33:47 i expect our test screenshots alone will be sufficient for confirming the theme is okay 19:34:02 but deeper interactive testing is indeed warranted 19:34:39 fungi: ya there are plenty of unknown unknowns and actually loading up the UI is a good way to discover that stuff 19:34:52 sounds like we've got a plan to make a plan. Anything else we want to talk about on this subject? 19:36:07 #topic Open Discussion 19:36:27 We reenabled the inmotion cloud today. Please keep an eye out for oddities there, but they made a number of changes whcih we hope will make for a happier cloud 19:36:36 i'll have changes up shortly for the process updates and new signing key clarkb mentioned earlier 19:36:36 thank you yuriys (not in here but still we appreciate it) 19:37:32 I started looking at mordred's pep 517 change in pbr today. I think it may not work properly based on some early testing. setup.py is being deprecated and we'll need an option going forward one way or another and having PBR support for pep517 is probably a good stepping stone to something richer 19:37:32 if anyone was following the wheel discussion in zuul yesterday i wrote up 19:37:36 #link https://review.opendev.org/c/zuul/zuul/+/815406 19:38:10 oh yes I need to followup on that /me adds to the todo list :) 19:38:26 ianw: it looks like you're proposing opendev wheel generation as the main solution? (as opposed to a nodepool dependencies image?) 19:39:13 corvus: i think it makes the most sense. then anything that uses the python base images could use the wheels safely 19:39:36 as far as implementation details go, that probably makes it more of an opendev spec? 19:40:05 true, if that's what we agree on 19:40:19 maybe it's sort of like the matrix spec: "zuul project says we want this; details to be hashed out with opendev"? 19:41:00 looking at the wheels actually used, i think pynacl is the big problem 19:41:35 i have offered there to help build manylinux arm64 wheels 19:42:26 ianw: thanks for writing that up; i'll give it a look (probably mostly through the lens i described above) 19:42:54 clarkb: is it possible to summarise the pbr/pep517/setup.py thing? 19:43:27 ianw: ya basically pypa has decided taht the setup.py packaging interface is deprecated and will go away in the future. Setuptools and Wheel are not going anywhere just the setup.py script interface 19:43:49 ianw: in short, setuptools has marked any invocations of `setup.py build` as well as setting setup_requires metadata or calling into easy_install as deprecated 19:43:53 ianw: this means that eventually you'll have to use some other tool like pip or build or poetry to manage python packages for your projects 19:44:29 the standard they are coalescing around supporting is the pep517/pep518 specifications where you put information about how to build your projects in pyproject.toml and then pip/build/poetry know what to do with them 19:44:35 setup.py isn't exactly deprecated, only calling it directly, as well as some things which they now feel should be done through pyproject.toml configuration 19:45:08 basically they would like to eventually be able to get rid of setuptools directly managing build requirements 19:45:38 i'd like to share a change that i did not push up for review: i wrote, but did not push, a change to split out zuul geard serving to a separate container. i wrote it to prep for running a second scheduler. but i realized that the way we have everything set up, it's fine for a second scheduler to also run gearman because nothing will use it (even the second scheduler). so i have that sitting on the shelf if we need it, but don't plan 19:45:39 to use it. i will hopefully soon push up a change to spin up a new scheduler host for zuul. 19:45:41 so expect you to call a wrapper like https://pypi.org/project/build instead, which will ask pip to satisfy your build backend's requirements first before it calls into setuptools 19:46:13 corvus: that makes sense 19:46:35 in a PBR context what is important is that we haev some way to run PBR then setuptools without going through setup.py 19:46:52 the pep517 spells out the high level interface for doing that and mordred wrote a change to pbr to integrate with it that way 19:47:21 through testing I'm not entirely sure it is working as expected so trying to dig into that, but I doubt this is going to be a full time thing for me until it is fixed. More of a poke at it as able to learn more about the python packaging changes 19:47:28 if someone else ends up digging into it I won't be sad 19:48:25 fungi: re setup.py not being fully deprecated I think the effect won't be much different 19:48:56 fungi: if pip won't run setup.py to build stuff for you then you're going to need to at the very least add a pyproject.toml that says run setuptools but then you can just setup.cfg 19:49:05 ok, thanks for the overview. i'd been vaguely aware of how it hangs together and even added some toml files, etc. but i'll have to bootstrap myself on the pbr level details :) 19:49:12 I Guess for complicated builds that do compiles against external libs you might still have a setup.py 19:49:16 well, they still expect you to be able to use setup.py to instrument setuptools, at least for a while 19:49:27 fungi: ya but for 99% of projects that is a setup.cfg :) 19:49:34 lots of non-pure-python projects need it for things like building c extensions 19:49:48 in fact one thing we might consider for a number of projects is dropping pbr entirely in favor of setuptools + setup.cfg + pyproject.tml 19:49:51 for pure python projects you're right 19:49:53 but one thing at a time 19:50:20 we'd need setuptools-scm to do the versioning, at a minimum 19:50:26 is it possible to get git versioning with that trio? 19:50:30 answered :) 19:50:41 and i don't think it generates authors files or changelogs either 19:51:02 it might, i'd need to revisit it 19:51:17 ya I'm not yet sure what all the things we'd lose are. But possibility to simplify exists at least 19:51:26 there are going to be lots of corner cases we've solved which likely aren't met by other tools out of the box 19:51:56 gutting bits of pbr in favor of other modules/mechanisms which exist bit at a time would probably make the most sense 19:52:05 heh, doesn't that imply that the replacement implements "build reasonableness". that sounds like a big assumption :) 19:52:25 and keeping pbr as a wrapper around those things for the transition 19:53:40 Alright anything else or can we go eat $meal now :) 19:55:11 #endmeeting