| clarkb | ianw: I think it is using dhcp in ovh so I assume that is how it works in the gate | 00:00 |
|---|---|---|
| clarkb | I suspect that if we disabled dhcp in the gate we'd get better coverage since that only works if glean is sucessfukl | 00:00 |
| clarkb | and ya I expected glean problems but then testing worked so I went with it :) | 00:00 |
| clarkb | the MINA maintainer reports that MINA ssh client will use sha2 even if there is no server-sig-algs exchange. The RFC says this is correct. I'm really surprised openssh hasn't done that yet | 00:03 |
| ianw | so the NetworkManager bit of glean is working, but not reading from configdrive? | 00:07 |
| clarkb | ianw: no I don't think any of it is beacuse rocky isn't recognized as a distro. It basically mounts and reads config drive then consults the distro name and exits | 00:07 |
| clarkb | ianw: hwoeve,r when I run it by hand it exits 0 but noops. The systemd logs show it exited 1 | 00:08 |
| clarkb | I think we start with getting https://review.opendev.org/c/opendev/glean/+/830532 into shape adding rocky to the test list of distros. And take it from there | 00:08 |
| clarkb | I can look at this more closely tomorrow maybe | 00:08 |
| ianw | sigh that is really something i would have liked the gate tests to hit | 00:09 |
| ianw | a simple thing i guess is to check the service status in that test | 00:09 |
| clarkb | ++ we can probably manipulate openstack in those jobs to disable dhcp and get better coverage of this | 00:09 |
| clarkb | oh ya that would be a good check too | 00:10 |
| ianw | https://github.com/python-distro/distro/blob/master/src/distro/distro.py has rocky, perhaps we should just import that? | 00:11 |
| ianw | it is vendored | 00:11 |
| opendevreview | Ian Wienand proposed opendev/glean master: distro: sync to latest upstream version https://review.opendev.org/c/opendev/glean/+/830536 | 00:15 |
| Clark[m] | Ya updating the distro lib is one piece then I'm glean/cmd.py we need to treat rocky as CentOS/rhel | 00:18 |
| ianw | we can see if ^^ passes -- it looks like nothing major changed, but who knows | 00:19 |
| *** dviroel is now known as dviroel|out | 00:20 | |
| ianw | it does give a warning : glean/cmd.py:1521: DeprecationWarning: distro.linux_distribution() is deprecated. It should only be used as a compatibility shim with Python's platform.linux_distribution(). Please use distro.id(), distro.version() and distro.name() instead. | 00:21 |
| ianw | https://github.com/python-distro/distro/issues/263 | 00:31 |
| ianw | which makes things a bit weird, because distro was supposed to be a drop-in replacment for platform.linux_distribution() but now ... warns you to not do that | 00:31 |
| opendevreview | Ian Wienand proposed opendev/glean master: Add Rocky Linux support https://review.opendev.org/c/opendev/glean/+/830539 | 00:51 |
| opendevreview | Ian Wienand proposed opendev/glean master: Remove rebuild-test-output.sh https://review.opendev.org/c/opendev/glean/+/830540 | 00:51 |
| opendevreview | Steve Baker proposed openstack/diskimage-builder master: Detect boot and EFI partitions in extract-image https://review.opendev.org/c/openstack/diskimage-builder/+/828617 | 02:28 |
| opendevreview | Ian Wienand proposed opendev/glean master: distro: sync to 3.6 https://review.opendev.org/c/opendev/glean/+/830536 | 03:29 |
| opendevreview | Ian Wienand proposed opendev/glean master: Add Rocky Linux support https://review.opendev.org/c/opendev/glean/+/830539 | 03:29 |
| opendevreview | Ian Wienand proposed opendev/glean master: Remove rebuild-test-output.sh https://review.opendev.org/c/opendev/glean/+/830540 | 03:29 |
| *** pojadhav|out is now known as pojadhav|ruck | 04:43 | |
| *** ysandeep|out is now known as ysandeep | 05:15 | |
| opendevreview | chandan kumar proposed openstack/diskimage-builder master: [DNM] RDO third party check job checking https://review.opendev.org/c/openstack/diskimage-builder/+/830550 | 05:57 |
| *** bhagyashris_ is now known as bhagyashris | 06:28 | |
| *** amoralej|off is now known as amoralej | 07:06 | |
| *** jpena|off is now known as jpena | 08:34 | |
| *** ysandeep is now known as ysandeep|lunch | 08:49 | |
| *** ysandeep|lunch is now known as ysandeep | 09:24 | |
| *** bhagyashris_ is now known as bhagyashris | 09:25 | |
| gthiemonge | Hi Folks, there's a weird pip issue in one of the periodic jobs in Octavia: https://zuul.openstack.org/build/d61e51ad70854af093bd53492c1ad7b0/log/job-output.txt#3584 | 09:28 |
| gthiemonge | pip writes that pyroute2 "has inconsistent version: filename has '0.6.5', but metadata has 'file-.VERSION'" | 09:28 |
| gthiemonge | any ideas? corrupted package/mirror? | 09:29 |
| frickler | gthiemonge: this is broken for more than a week, the last working job installed pyroute2\<0.6, was there a requirements update? https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_168/periodic/opendev.org/openstack/octavia/stable/wallaby/octavia-amphora-image-build/1687b54/job-output.txt | 10:09 |
| gthiemonge | frickler: yes, in the requirements repo: https://opendev.org/openstack/requirements/commit/d672ca24226a6037b428293e4d2dfbf25550dad5 | 10:10 |
| frickler | that matches the start of the failures. not sure why it would only fail for wallaby | 10:16 |
| frickler | comparing to the xena job, I see two differences: pyroute2 seems to be capped to 0.6.4 in xena and it uses newer pip | 10:17 |
| gthiemonge | frickler: Ok, I can reproduce this issue locally with ubuntu bionic (the job builds a bionic image), but bionic is not a tested runtime for wallaby, I need to update the job to focal | 11:02 |
| *** rlandy|out is now known as rlandy|ruck | 11:14 | |
| *** dviroel|out is now known as dviroel | 11:26 | |
| stephenfin | Looks like Gerrit isn't sending emails at the moment | 11:28 |
| stephenfin | Just emailed openstack-discuss about it | 11:28 |
| stephenfin | Last email I received was Tue, 22 Feb 2022 11:57:11 +0000 | 11:29 |
| stephenfin | *was dated | 11:29 |
| frickler | stephenfin: I received mail 30 mins ago, might be an issue with your provider, I can check logs in a bit | 11:48 |
| frickler | from what I can see, it affects @redhat.com email. 550 sip24.invaluement.mimecast.org Blocked by ivmSIP and/or ivmSIP/24 - see https://www.invaluement.com/lookup/?item=199.204.45.33. - https://community.mimecast.com/docs/DOC-1369#550 | 11:50 |
| * frickler is not going to send an email to them as their delisting process requires, maybe someone from redhat wants to do that or some other infra-root | 11:52 | |
| frickler | infra-root: slightly related, this is something we might want to clean up one way or other: 550 5.1.1 <hudson@openstack.org>: Email address could not be found, or was misspelled (G8) | 11:59 |
| stephenfin | frickler: Ack, I'll try find the relevant people to poke internally so. Cheers | 12:01 |
| *** amoralej is now known as amoralej|lunch | 13:09 | |
| *** amoralej|lunch is now known as amoralej | 14:02 | |
| NeilHanlon | appears the glean startup still failed for https://review.opendev.org/c/opendev/glean/+/830539 (logs: https://paste.opendev.org/show/bAglT06ZOzFr3aAkqMss/) - will take a look | 14:03 |
| fungi | mgagne_: i bounced you a couple more messages from that e-mail thread just now, to keep you in the loop | 15:13 |
| *** ysandeep is now known as ysandeep|dinner | 15:15 | |
| mgagne_ | fungi: thanks. | 15:31 |
| *** gmann is now known as gmann_afk | 15:59 | |
| *** pojadhav|ruck is now known as pojadhav|dinner | 16:01 | |
| *** ysandeep|dinner is now known as ysandeep | 16:12 | |
| *** gmann_afk is now known as gmann | 16:17 | |
| clarkb | frickler: gitea seems to ahve confirmed that is a bug in their image diff with 1.16. It was added to the 1.16.3 milestone then removed from that milestone. Maybe that means they will try to fix it in 1.16.1? | 16:23 |
| clarkb | I think we can wait a bit and see if there is any movement on that before we upgrade. Though it isn't the end of the world as it appears to just be a rendering issue | 16:24 |
| *** pojadhav|dinner is now known as pojadhav|ruck | 16:25 | |
| fungi | yeah, and it's only going to come up in rare cases where someone looks at a diff which removed an image file | 16:25 |
| clarkb | renamed or deleted image files yup | 16:25 |
| fungi | so i agree it doesn't seem like a blocker for upgrading, just something we'd like to see fixed as soon as possible | 16:25 |
| clarkb | I checked text files and they don't berak | 16:25 |
| stephenfin | frickler: So I checked with IT, and they said openstack.org admins need to be the ones to request removal from the blacklist | 16:43 |
| stephenfin | ...which seems odd, given no one asked to be put on it, but that's what I was told | 16:43 |
| fungi | stephenfin: neat. maybe one of the openstack.org admins will find time to do it, but if red hat figures out how to remove things its employees want to receive from its own blocklist then that will likely be a faster resolution | 16:45 |
| fungi | the incentives seem to be woefully out of balance there | 16:46 |
| stephenfin | I know :( That's pretty much what I said | 16:46 |
| fungi | this is also why i highly recommend not using corporate-controlled e-mail accounts for public participation in open source communities | 16:48 |
| fungi | they're typically managed by admins who are primarily concerned with being able to send and receive messages with business partners, not public systems or individuals, so priorities are not aligned | 16:49 |
| stephenfin | Yup, I have a personal email address I use for all other communities. This is likely as good a time as any to switch OpenStack'y things over to that now | 16:51 |
| NeilHanlon | fungi: wouldn't it be on the opendev side anyways? the blacklist says that `199.204.45.33` is part of a spam-sending /24, which resolves to review02.opendev.org | 16:53 |
| NeilHanlon | or rather, vexxhost probably needs to request the removal since it's in their block | 16:54 |
| clarkb | I think the point is you should be able to solve it from both sides. I subscribe to a few bad actor lists on my firewall that filters dns records and does firewall filtering too. I can and have had to override subsets of those rules because I wanted to get the python documentation of all things. | 17:00 |
| fungi | NeilHanlon: i personally don't care, catering to every random blocklist on the planet only furthers the idea that relying on those flawed systems is a good idea | 17:00 |
| fungi | we shouldn't need to seek permission from dozens of random companies on behalf of their customers | 17:00 |
| clarkb | PBS is another one that breaks due to their ad system | 17:01 |
| fungi | it takes the customers of those blocklists going back to them and complaining that they're blocking legitimate senders to actually reform anything | 17:01 |
| fungi | if red hat's employees find that red hat has made business decisions which render their company mail accounts less useful, then they should be pushing back on those choices made by their management. we didn't make that choice for them, and expecting us to solve it simply reassigns work from the people who should have responsibility for it | 17:03 |
| fungi | our responsibility is to keep our systems in good shape and make sure they only send legitimate messages to users, and i'm committed to doing that | 17:04 |
| fungi | spam filtering systems which don't allow their users to indicate what senders are legitimate are inherently flawed non-solutions | 17:09 |
| NeilHanlon | fair enough | 17:10 |
| NeilHanlon | yeah, i've had to do similar myself, clarkb | 17:10 |
| fungi | yep, my personal spam filtering solution does take blocklists into account, but only uses them for weighted scoring of messages and not as a hard reject, so there has to be something pretty wrong with a message before i auto-discard it, even if its sender is also on several blocklists | 17:21 |
| *** pojadhav|ruck is now known as pojadhav|out | 17:27 | |
| tbarron | /set plugins.var.python.irssinotifer.api_token 1c124cc4-55c5-41fc-94e2-e15c7578705c | 17:28 |
| Guest252 | heh | 17:31 |
| *** jpena is now known as jpena|off | 17:35 | |
| frickler | the other aspect of the whole spam story is that with our currently lack of bounce handling, I wouldn't be confident in claiming that review02.opendev.org isn't sending spam. not intentionally, but by what could be called lack of administrative oversight | 17:47 |
| frickler | I'm wondering how other gerrit installations handle this | 17:48 |
| *** amoralej is now known as amoralej|off | 18:07 | |
| *** ysandeep is now known as ysandeep|out | 18:20 | |
| fungi | yeah, i'm not sure what we'd do with the bounces either, i expect there's a fairly high volume where people have switched e-mail addresses and not updated their gerrit accounts | 18:49 |
| fungi | we could institute a policy of disabling accounts whose preferred addresses are bouncing back, though that seems a bit extreme | 18:49 |
| NeilHanlon | there is so-called 'variable envelope return path' that's used by some stuff (mailman3, e.g.) to help detect bounces. not sure if gerrit supports it, though | 18:55 |
| fungi | yeah, we do verp in some places | 18:57 |
| fungi | we can set up some solutions like that in the mta layer without gerrit needing to explicitly support it | 18:57 |
| clarkb | I think we already get the bounces | 18:57 |
| clarkb | they are in the infra-root inbox | 18:58 |
| clarkb | well not the inbox they get filtered elsewhere btui delivered there | 18:58 |
| fungi | oh, right, we shuffle them into a subdir | 18:58 |
| clarkb | but ya I'm not sure how we'd address that? just start killing accounts if they bounce email? | 18:58 |
| NeilHanlon | skip all the hard work and just delete accounts at random 😈 | 18:59 |
| fungi | can i just delete my account? then i can pretend the problem is solved! ;) | 19:00 |
| NeilHanlon | perfect! | 19:00 |
| clarkb | this is sort of why my long effort to clean up the notedb for conflicting accounts has fizzled out. YOu end up where there are no easy answers and then the easiest thing is to punt :) | 19:01 |
| clarkb | the difficult thing here is if their email address bounces that is our contact method.. I suppose what we could do is disable any accounts for which that is the only email address listed. For those that have additional email addresses we can reach out to those email addrs and ask them to remove the offending email address | 19:03 |
| clarkb | and after some time if we are ignored remove the offending email address ourselves. But that is a fair bit of work | 19:03 |
| fungi | yeah, so gerrit is currently using review@ for its from address and sender at the envelope layer (with reply-to of the individuals subscribed to the change), messages for which are sorted into INBOX.review | 19:05 |
| fungi | i made the mistake of trying to open that... | 19:06 |
| fungi | -- Mutt: Directory [=INBOX.], File mask: !^\.[^.] | 19:06 |
| fungi | Fetching message headers... 700/276063 (0%) | 19:06 |
| fungi | sorry, accidentally grabbed a second line into my clipboard | 19:06 |
| fungi | think i'm a gonna just abort that | 19:07 |
| fungi | i may have to put a bullet in ol' yeller here | 19:07 |
| fungi | mutt seems to ignore keyboard interrupt in this situation | 19:08 |
| clarkb | oh so those bounces are the ones that we're the reply to for? | 19:08 |
| clarkb | I guess that gives us an incomplete picture then and there are things to improve | 19:08 |
| fungi | bounces normally go to the envelope senter | 19:08 |
| fungi | sender | 19:08 |
| fungi | reply-to is for actual replies, not bounces | 19:08 |
| clarkb | fungi: how are we getting them if we aren't the sender? | 19:09 |
| fungi | we're the sender, just not the reply-to | 19:09 |
| clarkb | or at least I interpreted your initial message that review@ gets them all | 19:09 |
| clarkb | and review@ != infra-root ? | 19:09 |
| fungi | review@ is aliased to that | 19:09 |
| clarkb | aha that explains it, thanks | 19:09 |
| fungi | and we have delivery rules putting messages for that address into INBOX.review | 19:10 |
| fungi | which is presently well on its way to having 300k messages in it | 19:10 |
| clarkb | gotcha so we do get a pretty good picture and could take action it? the problem is that the choices for action to take are all largely suboptimal? | 19:11 |
| clarkb | oh you know what. These would almost have to be active accounts too | 19:11 |
| fungi | yes, and without clearing this and starting from inbox zero, it's going to be hard to guess how much of the bounce volume might be noise | 19:11 |
| clarkb | otherwise they wouldn't be generatign the events that send email in the first place. So these are active accounts with old email addresses | 19:11 |
| clarkb | I suspect a lot of it isn't noise given ^ | 19:12 |
| fungi | well, there are still reviews merging for people who actually died years ago, so it's not like there will be 0 | 19:12 |
| clarkb | ya its nonzero but the bulk of it should be active accounts I suspect | 19:12 |
| NeilHanlon | lets just email everyone and tell them to make sure their emails are up to da---wait.. | 19:13 |
| fungi | it's too bad we don't host this inbox on our servers. i could literally just move the messages aside or bulk delete them | 19:13 |
| clarkb | NeilHanlon: ya exactly :) | 19:14 |
| fungi | i haven't killed mutt yet, it's loaded 4% of that mailbox so far | 19:21 |
| fungi | though i wonder if it will oom on me | 19:21 |
| fungi | yeah, i think it's going to, just watching `top` sorted by memory utilization. if i don't kill it, the kernel will | 19:22 |
| NeilHanlon | :D | 19:23 |
| clarkb | ianw: NeilHanlon: I've been digging into zuul reviews this morning but will try to page in glean stuff after lunch since I expect we'll be able to have more direct conversation about stuff. ianw in particular it seems the rework of how glean is executed fizzled out? We need to address that since a glean release will include it? | 19:26 |
| opendevreview | Clark Boylan proposed openstack/diskimage-builder master: Force use of NetworkManager with glean on Rocky Linux https://review.opendev.org/c/openstack/diskimage-builder/+/830689 | 20:45 |
| opendevreview | Clark Boylan proposed opendev/glean master: Add Rocky Linux support https://review.opendev.org/c/opendev/glean/+/830539 | 20:46 |
| opendevreview | Clark Boylan proposed opendev/glean master: Remove rebuild-test-output.sh https://review.opendev.org/c/opendev/glean/+/830540 | 20:46 |
| clarkb | Lets see if that is happier | 20:46 |
| *** timburke_ is now known as timburke | 20:59 | |
| ianw | clarkb: i think that we did all that on the glean-early stuff, but maybe just didn't release it | 21:07 |
| clarkb | ianw: ya I think so too and it seems like the -src tests are testing it works? Do we need to make any changes on the dib simple init side to support it? | 21:08 |
| ianw | no i don't think so | 21:10 |
| ianw | stephenfin: so ultimately you got no help from redhat IT? can you send me a link to any issue etc. | 21:11 |
| ianw | my overnight inbox is weirdly empty | 21:11 |
| clarkb | ianw: the big indicator to me that we are probably fine is that the rocky test failed against those glean changes. This indicates it is doing stuff to test it and if the other distros pass then we are probably ok | 21:12 |
| ianw | ++ the NetworkManager bits you've added definitely seem on the right track | 21:13 |
| *** dviroel is now known as dviroel|out | 21:18 | |
| ianw | infra-root: while i agree with all the points about it being stupid, i have also sent a mail to try and get the server de-listed | 21:24 |
| clarkb | ianw: thanks | 21:24 |
| fungi | thanks ianw! | 21:28 |
| NeilHanlon | thank you ianw ! | 21:30 |
| clarkb | looks like the glean rocky change passes the rocky test now with NetworkManager in use | 22:07 |
| clarkb | I don't know that the ordering matters too much here since rocky isn't gating anything yet. But the depends-on is probably fine as is | 22:08 |
| clarkb | anyway school run, but I think we can work to land some of that stuff next | 22:12 |
| ianw | clarkb: 3.6 is a typo, it's v1.6.0 as you say. i don't know if we want to retest for that | 22:15 |
| priteau | Hello. A kayobe patch earlier went through gate jobs successfully but didn't merge, yet I cannot see anything blocking it: https://review.opendev.org/c/openstack/kayobe/+/827487 | 22:20 |
| Clark[m] | I'm not worried about it or would've -1'd | 22:20 |
| ianw | priteau: hrm, interesting, i agree nothing obviously stopping it merging | 22:24 |
| priteau | ianw: some event lost in zuul? | 22:29 |
| ianw | it's parent merged a long time ago https://review.opendev.org/c/openstack/kayobe/+/827486 | 22:29 |
| fungi | priteau: ianw: a later revision of its parent merged. gerrit won't allow 827486 to merge without a rebase | 22:33 |
| fungi | the orange-ish tint to the "(Merged)" in the relation chain list is the color-blind-useless clue the gerrit webui gives you for that | 22:34 |
| ianw | oh, doh, yep | 22:35 |
| priteau | I see, PS2 is the parent | 22:35 |
| priteau | thanks for the tip | 22:35 |
| fungi | hopefully when we get gerrit's mergeability checks turned back on, we'll also see the "can't merge" message listed in the ui for such changes | 22:36 |
| ianw | ++ | 22:37 |
| ianw | speaking of gerrit, i seem to be getting mail again | 22:37 |
| fungi | oh good | 22:38 |
| sshnaidm | fungi, hi, do you know why it may happen - I'm core in the repo, but I can't merge and +2 into stable branch in this repo, which I created before: https://review.opendev.org/admin/repos/openstack/ansible-collections-openstack,branches | 23:07 |
| sshnaidm | and I can't merge this patch https://review.opendev.org/c/openstack/ansible-collections-openstack/+/829638 | 23:08 |
| clarkb | sshnaidm: https://review.opendev.org/admin/groups/a4a320b642153bcd16df5f4c7548b22f9c84c540,members this group has permissions on the stable branches for that repo and appears empty | 23:09 |
| clarkb | sshnaidm: https://opendev.org/openstack/project-config/src/branch/master/gerrit/acls/openstack/ansible-collections-openstack.config#L12-L25 | 23:10 |
| sshnaidm | clarkb, interesting.. and how can I add there? (can I?) | 23:10 |
| clarkb | uh I think the stable team has historically managed that? I don't know if that is important for this repo though | 23:10 |
| clarkb | I guess in this case it isn't related to the openstack stable releases so maybe doesn't matter? | 23:11 |
| sshnaidm | clarkb, can I just replace ansible-collections-openstack-stable-maint by ansible-collections-openstack-core | 23:11 |
| clarkb | sshnaidm: yes that is one way to handle it | 23:11 |
| sshnaidm | cool, will add a patch.. | 23:11 |
| clarkb | Mostly I just awnt to make sure whatever we do doesn't get in the way of the stable team which has historically managed this stuff for the openstack stable stuff | 23:11 |
| clarkb | sshnaidm: I think you can just delete the block I highlighted to do what you describe | 23:12 |
| sshnaidm | the whole "refs/heads/stable/*" ? | 23:12 |
| sshnaidm | so that the main block apply for all? | 23:13 |
| clarkb | yes, but again we should double check with the stable team before doing something like that. I suspect it is fine since they care about the integrated stable release and this is separate | 23:13 |
| sshnaidm | it's separate and it's SIG repo | 23:14 |
| sshnaidm | and it didn't have any stable branch till prev week | 23:14 |
| clarkb | ya so should be fine to remove that block then | 23:14 |
| clarkb | and just rely on the refs/heads/* access block instead | 23:15 |
| opendevreview | Shnaidman Sagi (Sergey) proposed openstack/project-config master: Remove special config for stable branch in collections repo https://review.opendev.org/c/openstack/project-config/+/830700 | 23:16 |
| sshnaidm | clarkb, fungi please review ^ | 23:17 |
| *** rlandy|ruck is now known as rlandy|ruck|bbl | 23:18 | |
| clarkb | ok the glean stack has my +2's maybe fungi can take a look? | 23:25 |
| fungi | lookin' | 23:25 |
| fungi | what's the review topic? | 23:25 |
| clarkb | fungi: https://review.opendev.org/q/topic:distro-update | 23:26 |
| opendevreview | Merged openstack/diskimage-builder master: Force use of NetworkManager with glean on Rocky Linux https://review.opendev.org/c/openstack/diskimage-builder/+/830689 | 23:35 |
| opendevreview | Merged openstack/project-config master: Remove special config for stable branch in collections repo https://review.opendev.org/c/openstack/project-config/+/830700 | 23:39 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!