19:01:06 #startmeeting infra 19:01:07 Meeting started Tue Aug 25 19:01:06 2020 UTC and is due to finish in 60 minutes. The chair is clarkb. Information about MeetBot at http://wiki.debian.org/MeetBot. 19:01:08 Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 19:01:10 The meeting name has been set to 'infra' 19:01:19 #link http://lists.opendev.org/pipermail/service-discuss/2020-August/000079.html Our Agenda 19:01:28 o/ 19:01:33 As mentioned lets see how many manage to make it today before digging in too deep 19:02:00 #topic Announcements 19:02:04 There are no announcements 19:02:12 #topic Actions from last meeting 19:02:20 #link http://eavesdrop.openstack.org/meetings/infra/2020/infra.2020-08-18-19.01.txt minutes from last meeting 19:02:25 There were no actions from last meeting 19:02:52 Thats the bookkeeping done. I'll give it a few minutes to see if anyone else makes it then continue from there 19:05:26 ianw: zbr: may just be the three of us. Should we do a less formal agenda then? 19:05:36 sure 19:06:04 ++ 19:06:17 cool I'll start with the item I want to bring up then will open it to other items 19:06:21 #topic Non master repo HEAD support for new projects 19:06:31 #link https://review.opendev.org/741277 Needed in Gerritlib first as well as a Gerritlib release with this change. 19:06:40 #link https://review.opendev.org/741279 Can land once Gerritlib release is made with above change. 19:06:57 Calling this out as it came up in the board meeting today (though I didn't manage to attend so can't provide direct details) 19:07:41 I also plan to attend the diversity and inclusion working group meeting on monday to talk about the technical aspects of any potential changes to git branch naming 19:07:46 clarkb: maybe there is a chance to also get https://review.opendev.org/#/c/729966/ into the new release? 19:08:18 zbr: I can review that one too ya 19:08:55 i don't think i've looked over gerritlib code before, but it lgtm, and mnaser has looked too, i think it's as gtg as ever 19:09:02 ianw: thanks 19:09:34 its been good to be able to point to steady progress on this as people have higher level conversations around the topic 19:09:40 anyway thats all I had on this subject 19:09:52 ianw: zbr anything you'd like to bring up? 19:10:15 nope; lgtm. i'm sure there's things lurking if it gets changed, but just have to deal with it as it comes up 19:10:42 known unknowns, unknown unknowns and all that :) 19:10:57 indeed 19:11:35 #topic Github 3rd Party CI 19:11:50 ianw: anything new on this? I see commentary about arm64 focal openafs in the other channel :) 19:12:45 umm, since last week we dug deeper on the wheel generation issues, and found it wasn't exactly the toolchain, but actually patchelf, as part of the wheel construction, messing the alignment 19:13:22 #link https://github.com/pypa/manylinux/issues/735 19:13:45 that has stalled a bit as the changes need to make it through a few projects 19:14:02 and then the other angle has been pyca/cryptography pushing rust into the build 19:14:47 so zuul jobs now has an ensure-rust, so we're ok with our extant arm64 testing ... the wheel story, something about rust and musl libc and the builder that i lost track of a bit 19:15:01 i don't think it quite works even on x86 atm 19:15:38 and upstream switched from travis-ci.org to travis-ci.com as well, which somehow apparently gives them access to arm64 hardware in aws 19:16:22 so ... while a lot happened, in terms of "is there a .whl file for arm64" the answer is still no :) 19:17:32 oh, and the focal mirror ... yeah the bionic arm64 mirror turned itself off again too 19:18:02 getting the focal mirror up has been an exercise too, ipv4 broke in the cloud which kevinz had to fix 19:18:28 I know back in the early days debian was considered to be the most stable on arm64 (or at least I seem to recall that) 19:18:36 do we need to consider maybe running more debian? 19:19:00 not sure, the oops appears to be from openafs, and at least to an initial search, appears to be unique 19:19:50 one of these great things we seem to hit where the only occurrence of the error message on google is the error string in the actual code 19:19:59 nice 19:20:52 https://www.google.com/search?nonzero+refcount+in+shutdown_osisleep 19:20:54 :) 19:21:32 anything else on this subject? 19:21:54 no, sorry progress has been slow. it's breaking a lot of new ground in various places 19:22:00 no worries 19:22:08 #topic Gerrit Upgrade Thoughts/Ideas/Info 19:22:14 I continue to poke at Gerrit upgrade things 19:22:42 I looked at review-test and noticed that the git repos are currently on the root device for the review-test server which is different than production gerrit. In prod we hvae the git repos on a fast cinder volume 19:23:12 I think we may want to build a new review-test and better line up the git repo locations as that will likely be important for timing data as well as having enough head room to do the notedb migration 19:23:33 that got me thinking about how viable keeping in sync with prod is if we're doing snapshots and rolling back 19:23:56 Mostly wondering if we should pick a point in time and not worry about keeping in sync with prod to simplify (that should still give us a really good idea for disk usage and migration times) 19:24:14 Separately I started digging into what notedb means for replication configs 19:24:36 for the account and group data we'll already avoid replicating them with our exisitng config due to the replicatePermissions = false setting on our replication configs 19:24:44 I think that is a good thing at least to start 19:25:05 the change notedb data goes in refs/changes/XY/ABCDXY/meta though 19:25:38 which means we'll be replicating all of that and i odn't know of a way to construct a refspec that replicates res/changes/XY/ABCDXY/1 2 3 etc but not meta 19:26:15 I think it would be fine to replicate the metadata fwiw. Just worry about disk usage on the gitea mirrors if we do that (we can check repo sizes for that during testing with prod data I guess) 19:26:19 are you supposed to replicate that? like are there client tools that might show interesting things if you have it locally? 19:26:44 ianw: so far it seems that the only things that really consume it are other gerrit servers 19:26:49 for active standby setups 19:27:56 that got me diving into how the new notedb setup stores data 19:28:20 i wonder if it ends up that much bigger? it would really only be the change descriptions right? the refs/changes we are already doing? 19:28:37 ianw: its all of the code review comments 19:28:46 which ya for most change swon't be that large 19:29:14 the other potential issue there is the increase in refs 19:29:22 gitea had trouble with that before but has gotten much better 19:29:51 One thing I noticed digging around in the notedb storage details is that All-Users:refs/meta/external-ids may need special permissions changes 19:30:15 I need to ask luca about that one. They do special perms for refs/users but not the external-ids and I expect they want that 19:30:32 basically a long winded way of saying I've learned a lot about notedb recently 19:31:19 Hope to send another email to luca in near future with the questions that have popped up 19:31:23 #topic Open Discussion 19:33:04 If there is anything else feel free to bring it up now. I'll leave the floor open for about 5 minutes and if we get nothing call that the meeting 19:37:55 Alright. Thanks everyone! 19:38:01 I'll call that a meeting 19:38:06 #endmeeting