| *** cloudnull9 is now known as cloudnull | 00:44 | |
| opendevreview | Ian Wienand proposed opendev/system-config master: gerrit: fix Launchpad credentials write https://review.opendev.org/c/opendev/system-config/+/801227 | 00:54 |
|---|---|---|
| opendevreview | Ian Wienand proposed opendev/system-config master: [dmn] testing mariadb connector https://review.opendev.org/c/opendev/system-config/+/801410 | 01:43 |
| opendevreview | Ian Wienand proposed opendev/system-config master: Remove review01 references https://review.opendev.org/c/opendev/system-config/+/801412 | 01:58 |
| ianw | clarkb: https://gerrit-review.googlesource.com/c/gerrit/+/312602 fixes the mariadb connector issue. https://213.32.78.143 is up with the fix, but it doesn't do anything exciting; just doesn't give you 500 errors when looking at files :) | 06:23 |
| *** amoralej|off is now known as amoralej | 06:56 | |
| *** rpittau|afk is now known as rpittau | 07:06 | |
| *** odyssey4me is now known as Guest1488 | 08:33 | |
| *** ksambor is now known as ksambor|lunch | 10:01 | |
| *** ksambor|lunch is now known as ksambor | 10:53 | |
| *** odyssey4me is now known as Guest1498 | 11:38 | |
| *** amoralej is now known as amoralej|lunch | 13:10 | |
| *** amoralej|lunch is now known as amoralej | 14:05 | |
| opendevreview | Danni Shi proposed openstack/diskimage-builder master: Add a keylime-agent element and a tpm-emulator element https://review.opendev.org/c/openstack/diskimage-builder/+/789601 | 14:06 |
| clarkb | ianw: upstream gerrit just merged your mariadb connector fix \o/ It isn't clear to me if we need to push that patch to stable-3.3, stable-3.4, and master now though | 14:54 |
| clarkb | mordred: corvus: ^ do you grok their forward/backward porting system and do you know if they'll do that for you or not? | 14:54 |
| *** dviroel is now known as dviroel|lunch | 14:54 | |
| corvus | clarkb: my understanding is someone will do it if you don't; but if you do it's appreciated and may happen faster. basically just 'git merge' to each branch in series | 14:57 |
| corvus | so master -> stable$latest -> $stable$latest-1 etc | 14:57 |
| clarkb | corvus: oh ok so you push merge commits and not cherrypicks? I'll look into that after some breakfast since in theory I'll need to do it for my change too if/when it lands | 14:58 |
| corvus | correct, merges | 14:58 |
| clarkb | corvus: ya I think that must be correct and that is why their git history is always interesting to look at | 14:58 |
| corvus | clarkb: looks like they include 'git log --oneline' in the merge commit msgs too. look for the string "Merge branch" in git history for an example | 14:59 |
| corvus | i don't know if there's a tool to do that automatically, or they just copy/paste | 14:59 |
| clarkb | interesting. I wonder if they have a developr doc on this | 15:00 |
| clarkb | I'll dig around, this helps ensure I'm looking for the right stuff though | 15:00 |
| mordred | corvus: it's master -> stable$latest and not stable$oldest -> stable$nextoldest -> ... master? | 15:02 |
| clarkb | mordred: I think the arrows are git dag direction which implies $stable$latest-1 is oldest in that series | 15:03 |
| mordred | I'm sure you're right - just trying to remember last time I did this | 15:03 |
| mordred | ah. I was thinking of it in terms of order of operations for dev | 15:03 |
| mordred | I think they are "land on oldest applicable branch then merge forward" as opposed to openstack which is "land on master then cherry-pick backwards" | 15:04 |
| clarkb | ya and in theory that sounds great as it ensures you aren't forgetting to fix bugs, but it makes for really odd histories. I always find it difficult to determine if a particular patch is included in any release without checking it out locally (the gitiles type tools all seem to fail me) | 15:05 |
| clarkb | not the end of the world to fetch it locally though | 15:06 |
| corvus | mordred, er yeah, oldest to newest, sorry | 15:11 |
| fungi | seems like it merged initially to stable-3.2 | 15:16 |
| fungi | so i guess that's the oldest relevant branch | 15:16 |
| clarkb | fungi: well I think that is what ianw pushed to | 15:17 |
| clarkb | certainly old enough for us :) | 15:17 |
| fungi | heh | 15:19 |
| clarkb | "Fixes that are known to be needed for a particular release should be pushed for review on that release's stable branch. They will then be included into the master branch when the stable branch is merged back. | 15:20 |
| clarkb | reading elsewhere they say "start on the newest `stable-*` branch that you can run" as a general rule of thumb for improvements and bug fixes | 15:23 |
| clarkb | I've found at least one example in the git log where they made a change to master and then cherry-picked it backward in a way we are more familiar with | 15:25 |
| clarkb | (good to know that is an option) | 15:25 |
| clarkb | corvus: I'm noticing that their merge rollups don't seem to go through gerrit? | 15:28 |
| clarkb | Also looks like they may try to bundle up more than one change when merging forward. I guess that means we can wait for my change to land before worrying about doing ianws | 15:28 |
| *** dviroel|lunch is now known as dviroel | 16:00 | |
| *** marios is now known as marios|out | 16:22 | |
| *** rpittau is now known as rpittau|afk | 16:45 | |
| *** amoralej is now known as amoralej|off | 16:50 | |
| clarkb | infra-root I updated the maintenance.html on review01 to help make things more clearly for people who may have DNS hardcoded to the old server | 18:01 |
| fungi | oh, great idea | 18:03 |
| clarkb | the gerrit user cleanup notes I've been taking have been moved to the new server as well. I'll need to do a check over the other stuff there to see if I want to copy anything else (I knew I wanted those files) | 18:17 |
| *** cloudnull9 is now known as cloudnull | 18:19 | |
| fungi | for the gerrit upgrade, did we blow away the accountpatchreviewdb content? | 19:48 |
| clarkb | fungi: yes | 19:48 |
| fungi | or whatever it's CALLED | 19:48 |
| fungi | gah, capslock | 19:48 |
| fungi | has anyone even noticed? ;) | 19:49 |
| clarkb | not that I've heard | 19:49 |
| clarkb | I did explicitly test it afterwards just to be sure no new weird behavior popped up and that is how I found the issue with the mariadb connector | 19:49 |
| clarkb | dansmith has reported the occasional slow page load (I'm guessing that is related to caches which is why I mentioend we can tune those more now) but also says submitting changes seems to be snappier | 19:51 |
| clarkb | overall I think the server is running in a much happier way | 19:51 |
| ianw | i actually now think this issue with the connector ignoring dups is related to the failed import | 19:52 |
| clarkb | oh! ya it could be | 19:52 |
| clarkb | since you were using the gerrit built in migration tool ya? | 19:52 |
| ianw | yeah | 19:52 |
| clarkb | at this point I think its fine. The value is minimal and very ephemeral | 19:53 |
| clarkb | email sent about the yum-puppetlabs mirror | 19:55 |
| ianw | i think the main benefit is that we are masters of our own database and can easily point ansible where we need it to go | 19:55 |
| fungi | yes | 19:56 |
| fungi | oh, here's one significant hurdle with dropping stretch... https://opendev.org/opendev/base-jobs/src/branch/master/zuul.d/nodesets.yaml#L51 | 19:58 |
| fungi | though not a lot of references to debian-stable in job configs on master branches either, thankfully | 20:00 |
| clarkb | fungi: I guess step 0 is bumping that to buster with a note bullseye will follow shortly? | 20:02 |
| fungi | or getting rid of that alias | 20:03 |
| clarkb | or that | 20:04 |
| clarkb | I think yum-puppetlabs hit its quota limit a couple of months ago and no one noticed. That is why it hasn'ed updated since sometime in may | 20:06 |
| ianw | heh, yeah, those lines converge | 20:24 |
| fungi | okay, i've sent the dropping stretch proposal to openstack-discuss | 20:24 |
| opendevreview | Clark Boylan proposed opendev/system-config master: Trim yum-puppetlabs content https://review.opendev.org/c/opendev/system-config/+/801529 | 20:34 |
| clarkb | that is based on feedback from mwahahah on my email. I'll ask them to review it quickly too before we land anything | 20:35 |
| ianw | oh the other thing i noticed was https://bugs.launchpad.net/ubuntu/+source/glibc/+bug/1926918 had broken package installs on review02 | 21:38 |
| ianw | i'll poke around on our focal systems to see if they have a similar issue | 21:39 |
| clarkb | ianw: I started looking at that since you had it on the etherpad. Did it get fixed on review02 already? and now we have to address the others ptentially? what was the fix downgrading glibc? | 21:40 |
| ianw | the fix was a manual downgrade, which then allowed an upgrade past the broken libc package | 21:41 |
| ianw | i've checked a few focal systems and they don't seem affected | 21:42 |
| ianw | ok i think i understand the problem better now. 2.31-0ubuntu9.3 was released and withdrawn, so focal-updates only has 2.31-0ubuntu9.2 | 22:02 |
| ianw | the problem is if you're on a system that has 9.3 libc installed, then you try to install libc-dev (such as via build-essential, which we tried to do when we pulled in borg, because we need to build fuse from source) | 22:03 |
| ianw | many of our systems have 9.3 installed via automated updates, but most have pulled along libc-dev with them (e.g. executors that require that for building dkms, etc) | 22:04 |
| ianw | so if i can come up with something that finds hosts with 2.31-0ubuntu9.3 libc installed, but !libc-dev, they are in the potentially broken state | 22:06 |
| clarkb | ianw: should we remove 9.3 everywhere if it was withdrawn? | 22:07 |
| clarkb | (I'm not sure how painful that will be and maybe we juts wait for 9.4?) | 22:07 |
| ianw | yeah, i guess that when they put something back in focal-updates, they will make sure it has a higher version | 22:07 |
| ianw | ok, here is the list of servers that are potentially broken if they (or something they depend on) try to install libc6-dev currently -> https://paste.opendev.org/show/807610/ | 22:32 |
| ianw | the two options are a) downgrade them to 9.2, the latest release in -updates or b) ignore this, assuming they won't need that package until ubuntu upstream release a new libc >9.3 | 22:33 |
| clarkb | ianw: we should probably consider why 9.3 was pulled when evaluating ? | 22:34 |
| clarkb | if it is a major issue then probably worth downgrading | 22:34 |
| ianw | it think it was pulled by unattended-upgrades | 22:39 |
| clarkb | ianw: oh I meant why was it pulled from the repository | 22:39 |
| clarkb | presumably there was something wrong with it | 22:39 |
| clarkb | though I guess if it was really bad they would have released a 9.4 to replace it with the old code? | 22:44 |
| ianw | "caused regressions" from the bug ... trying to see if i can find exactly what it was | 22:45 |
| ianw | i think it might actually have to do with WSL (!) | 22:45 |
| ianw | https://bugs.launchpad.net/ubuntu/+source/glibc/+bug/1912652/comments/15 | 22:51 |
| ianw | The fixed version has been moved back to focal-proposed due to regressions caused by LP: #1914044. | 22:51 |
| ianw | [SRU] gstreamer fails with "cannot allocate memory in static TLS block" error on aarch64 Edit | 22:52 |
| ianw | https://bugs.launchpad.net/ubuntu/+source/glibc/+bug/1914044/comments/18 | 22:53 |
| ianw | This fix proved to be too intrusive to be released via focal-updates. It requires an immediate restart of services and a reboot | 22:53 |
| ianw | ... I'm preparing 2.31-0ubuntu9.4 _without_ this fix but including other fixes which are safe to release to Focal. | 22:54 |
| ianw | i think we have a good handle on what's actually wrong now, and i've condensed my research into https://bugs.launchpad.net/ubuntu/+source/glibc/+bug/1926918/comments/11 because this problem is all over and not explained terribly well | 23:08 |
| ianw | i think we just leave things alone, wait for 9.4 packages, and in the very unlikely case we need to pull in something that depends on libc6-dev be aware of the broken servers and our options | 23:09 |
| fungi | that sounds fine to me based on what you've quoted so far | 23:11 |
| fungi | any idea how long that's been in limbo waiting for the updated package? | 23:12 |
| ianw | the "i'm preparing" message was 2021-05-20 | 23:18 |
| clarkb | maybe we can drop a comment there and ask if this is still the plan? | 23:19 |
| ianw | i can't see any other alternatives. i'm very surprised deleting a package from a stable release was an option | 23:24 |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!