19:01:09 #startmeeting infra 19:01:09 Meeting started Tue Nov 9 19:01:09 2021 UTC and is due to finish in 60 minutes. The chair is clarkb. Information about MeetBot at http://wiki.debian.org/MeetBot. 19:01:09 Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 19:01:09 The meeting name has been set to 'infra' 19:01:17 #link http://lists.opendev.org/pipermail/service-discuss/2021-November/000295.html Our Agenda 19:01:45 #topic Announcements 19:02:05 I was hoping I could link to gerrit user summit stuff but I can't find any details on that yet. They must be running into planning issues. I can sympathize with that 19:02:52 o/ 19:03:01 #topic Actions from last meeting 19:03:08 #link http://eavesdrop.openstack.org/meetings/infra/2021/infra.2021-11-02-19.01.txt minutes from last meeting 19:03:19 We didn't record any new actions and the prior actions are all done or in progress. \o/ 19:03:25 #topic Specs 19:03:56 Just a quick note here that I approved fungi's mailman3 spec after a quick respin to address input from frickler. The updates were minor and didn't seem like anything that needed to be completely rereviewed 19:05:13 #topic Topics 19:05:20 #topic Improving OpenDev's CD throughput 19:05:31 this is still on my todo list. All the zuul and container fun has been distracting me :/ 19:05:46 ianw: have you had a chance to look at why the child changes are failing CI? (I'm mostly just curious) 19:06:06 no sorry, i've managed to be distracted on other things 19:06:14 I think we've all been in that boat recently 19:06:22 #topic Gerrit Account Cleanups 19:06:52 I was mostly going to skip this over except fungi found a story where someone had trouble with @ in their username. fungi has asked them to clarify if they were trying to use an email address as a username or if their username actually has an @ in it 19:07:07 Noting that here for the possibility there is another sort of cleanup we'll need to do in normalizing usernames 19:07:17 Not really actioanable at this point but an interesting possibility 19:08:00 #topic Zuul Multi Scheduler Setup 19:08:41 there are 2 schedulers 19:08:45 Zuul has made great progress on supporting multiple schedulers (removing the last remaining spof for a zuul install). Our OpenDev zuul is running two schedulers. One on zuul01.o.o and the other on zuul02.o.o. Zuul02 is the "primary" 19:08:55 What makes zuul02 the primary for us is that all web traffic hits it first 19:09:08 and gearman 19:09:13 and actually there's no web on zuul01 19:09:35 If things go really sideways I think we can stop zuul01's scheduler and restart the scheduler on zuul02 only 19:09:43 (cause we don't have a load balancer for it) 19:09:56 ++ 19:10:00 and, if necessary, clear out zk 19:10:10 and if things go really badly, clearing the zk state would be a good idea 19:10:16 It is worth noting that we have run into problems but we've been trying to work through them as they show up. corvus has been a great help with that. 19:10:41 So far we've had issues with retried jobs not behing handled properly. Nodepool requests getting stuck in a perpetually waiting state, and config errors not serializing properly 19:10:41 i'm thrilled that we haven't needed to downgrade again 19:10:49 i think we're at the point where the problems that have been cropping up have been minimal enough we can roll forward 19:11:33 There are a few more new issues showing up today that deserve followup after this meeting. Specifically johnsom's designate change error and elodilles lack of a zuul.tag var on release jobs. There is also a job runtime estimate problem that corvus has a fix up for 19:11:43 corvus: ^ fyi details for both of those other things are in #opendev 19:13:13 Please do report any weird behavior. So far a lot of weird behavior I have noticed ahs been tracked back to the multi scheduler setup and reporting those things has been very helpful because now they are fixed :) 19:13:42 And ya the recovery at this point can probably be achived by simple restarting onto one scheduler using zuul02 since the code seems quite stable in a single scheduler setup 19:14:36 Anything else to add to the zuul topic? or questions about the setup? 19:15:49 #topic User management on our systems 19:16:08 Last week I started pulling on a thread and noticed there were some improvements we could make to how we manage our users and uids 19:16:14 thank you to those who helped me make sense of it all 19:16:47 A number of changes have come out of that. Some potentially impactful if we aren't careful. I'd like to ask infra-root take a look over these changes so that we can start landing them when we are confident they are safe and are able to watch them go in 19:16:55 #link https://review.opendev.org/c/opendev/system-config/+/816869/ Be explicit about uid/gid ranges 19:17:24 This first change adjusts our configured ranges in adduser.conf and logins.def which different tools refer to when creating usrrs 19:17:38 The rough layout is: 0-999 system, 1000-1999 unallocated, 2000-2999 for infra-root users, 3000-9999 host level users, 10k - 64k container users that need uids on the host as well for bind mounts. 19:17:40 #link https://review.opendev.org/816869 Lower UID/GID range max to make way for containers 19:17:43 also that one 19:17:48 Thats the same one :) 19:18:04 er, yep sorry. i guess it had a different title 19:18:20 the idea there is we've put a number of container services on high uids like 10001 but then when we create say the letsencrypt group it gets created as gid 10002 19:19:05 fungi and I are thinking it would be better to not assign specific values to stuff that actually belongs to the system like our users and letsencrypt group and so on so we cap those at 9999 then we can eb explicit about container uids/gids and ensure those are non overlapping in the >10k space 19:19:35 well, our users are already statically assigned uids and gids 19:19:40 yup 19:19:44 "our" personal accounts i mean 19:19:59 #link https://review.opendev.org/c/opendev/system-config/+/816771 Clean up unused bootstrapping users 19:20:11 Is antoher user related change. This time to cleanup users that I think we don't need. 19:20:36 This is mostly a belts and suspenders spring cleaning move and one that scares me slightly since we might've accidentally used one of those users for something functional but as far as I can tell this isn't the case 19:20:49 well, it's also a security concern 19:20:55 right 19:21:05 I think it is important, but one we should take care with and review carefully 19:21:08 as on some systems those accounts come with provider-supplied authorized keys 19:21:27 #link https://review.opendev.org/c/opendev/system-config/+/816769/ Give gerritbot and matrix-gerritbot a shared user 19:21:51 This is a followon to 816869, and I'd like to have us work our way through giving most of our other containers similar treatment 19:22:04 lodgeit, refstack, hound, some other irc bots, etc 19:22:21 But I figure start slow make sure we've got things laid out the way we want first before we do a bunch of work that needs updating again later 19:22:31 The gerritbots seemed like a good example case 19:23:03 Then when that has been worked through we should also look into updating the uid for our mariadb containers to something other than 999 19:23:20 The mariadb uid was what sparked this whole thing off in the first place and will likely be the last thing to be addressed :) 19:23:48 I think for mariadb we can probably run it as the user for the services it supports in most cases. For review run it as the gerrit user, for etherpad run it as the etherpad user and so on 19:24:02 Since we don't run shared mariadbs between services we can do that safely 19:24:53 If y'all can review with a very critical eye I would appreciate it. I'm happy to do additiaonl testing (we already did some manual testing before settling on 816869) 19:25:29 yeah, it *seems* to work as hoped 19:25:44 part of the problem is that adduser and useradd rely on entirely different configs 19:26:05 and package maintscripts and ansible roles use who knows which one 19:26:26 I think I foudn some evidence that package scripts do use both. Or rather one package uses one and another packages uses the other 19:26:36 right 19:26:43 the evidence for this is that one will do gidmax-1 and the other will do gidmin+1 19:26:47 so at least having the two of them in sync should help 19:26:49 and we see evidence of both on our systems 19:28:02 Any other questions or concerns to bring up on this topic? 19:28:27 thanks for digging into it and explaining, i'll take a look at the changes too 19:30:11 #topic Open Discussion 19:30:40 I wasn't sure which other topics we would want to discuss so decided to trim the agenda down and let this portion of the meeting cover anything else 19:31:01 Gerrit3.4 upgrade stuff and fedora 35 work seem to be progressing, but not sure there is anything to share yet 19:31:13 I think there is a dib change I should review for containerfile stuff that I haven't been able to get to 19:31:35 #link https://review.opendev.org/c/openstack/diskimage-builder/+/817139 dib handle containerfile errors better 19:31:35 yeah, that was working but last night centos-9 mirrors were broken 19:31:41 speaking of 19:31:44 #link https://review.opendev.org/c/opendev/system-config/+/817136 19:32:13 i could use some help on fixing bitrot in the storyboard-webclient builds, if anyone has tips for how to update a yarn.lock 19:32:14 adds centos 9-stream mirrors, but i don't think we have space. my plan is to continue to remove debian-stretch 19:32:15 #link https://review.opendev.org/814053 [opendev/storyboard-webclient] Bindep cleanup 19:32:57 ianw: just left a comment on the centos-9-stream change 19:33:29 fungi: to update a yarn.lock you remove the lock and then reinstall iirc. That produces a new lock file and if testing succeeds with that you can merge it 19:33:53 reinstall what? 19:34:00 reinstall the javascript stuff using yarn 19:34:08 i've always just done "yarn upgrade" i think 19:34:17 https://classic.yarnpkg.com/en/docs/cli/install/ I think 19:34:19 oh, i can give that a try, thanks 19:34:28 oh, also i've had this up for a while, to hopefully make our system-config jobs a little more robust... 19:34:28 ianw: ah that might be the better method then. I guess upgrade ignores the lock and writes a new one? 19:34:30 #link https://review.opendev.org/813880 [opendev/system-config] Retry acme.sh cloning 19:34:47 but either way, the *real* problem is going to be every javascript library that has maintained the same name but rewritten itself completely (see prior discussions in #opendev yesterday :) 19:35:52 yeah, in this case the reason i need to do it is because one of the js packages needs updating to support python3 19:36:08 but i assume that will involve updates to a lot of other dependencies 19:36:23 ya you might have to fiddle with the requirements file equivalent to find something that produces a working set 19:36:32 when I've done this for zuul before it is a fun exercise 19:36:42 and this for making the rejects in our iptables rules slightly more expressive... 19:36:44 #link https://review.opendev.org/810013 [opendev/system-config] Switch IPv4 rejects from host-prohibit to admin 19:38:58 fungi: I've +2'd that one so you can approve it when you are able to watch it. Like the user changes has potential for wide spread pain if somehow it goes wrong (though again I don't expect any issues) 19:39:25 yep 19:39:38 i did some spot testing and confirmed it works as expected, at least 19:40:03 oh, i approved it in between, but yeah, i'll be around 19:40:08 cool 19:40:17 I'll give this meeting a few more minutes to bring anything else up 19:40:30 But then I need to review some zuul fixes and eat lunch :) 19:40:47 i'll be around too, of course 19:40:59 #link https://review.opendev.org/c/opendev/system-config/+/816766 19:41:09 is a minor one to expose the db in gerrit testing 19:41:24 i don't think we were noticing gerrit wasn't actually talking to the db correctly 19:41:26 oh, right, i meant to look at that one, thanks 19:41:33 ++ thats a good update to our testing for gerrit 19:42:06 ianw: you might consider toggling the state too since we had the issue with the non unique keys thing in the past that was only hit on the change of an already created row 19:42:47 I approved it as is as this is claerly better than what we had before 19:43:06 that's a good idea, can do that, just the same request with a DELETE 19:43:43 I guess a loop of PUT DELETE PUT might work 19:44:24 i wonder if a template to method: works 19:45:33 seems like it should? it's just a string, right? 19:48:06 Sounds like that is it. Thanks everyone! 19:48:09 #endmeeting