| tonyb | clarkb: Yup I'm still interested. Today is supposed to be a holiday for you. Would you prefer to do it tomorrow? | 00:06 |
|---|---|---|
| Clark[m] | Sure we can do tomorrow | 00:22 |
| tonyb | kk | 00:39 |
| opendevreview | Merged openstack/project-config master: Add netdata onfiguration repo to NebulOuS https://review.opendev.org/c/openstack/project-config/+/905293 | 05:44 |
| *** benj_9 is now known as benj_ | 07:41 | |
| opendevreview | Jan Marchel proposed openstack/project-config master: Add new projects to NebulOuS: ontology-server, brokerage-quality-assurance-server, service-level-agreement-generator https://review.opendev.org/c/openstack/project-config/+/905657 | 09:04 |
| opendevreview | Jan Marchel proposed openstack/project-config master: Add new projects to NebulOuS: ontology-server, brokerage-quality-assurance-server, service-level-agreement-generator https://review.opendev.org/c/openstack/project-config/+/905657 | 10:10 |
| opendevreview | Jan Marchel proposed openstack/project-config master: d new projects to NebulOuS: ontology-server, brokerage-quality-assurance-server, service-level-agreement-generator https://review.opendev.org/c/openstack/project-config/+/905657 | 10:16 |
| rtomczyk | We have some issues with neutron-vpnaas libreswan driver. What is the best way to get some community support on it ? | 10:35 |
| opendevreview | Jan Marchel proposed openstack/project-config master: Add new projects to NebulOuS: ontology-server, brokerage-quality-assurance-server, service-level-agreement-generator https://review.opendev.org/c/openstack/project-config/+/905657 | 10:36 |
| fungi | yoctozepto: i assume you were okay with 905293 merging? i should have left a note on it to hold approvals until you had time to confirm | 14:18 |
| fungi | not sure if rtomczyk was expecting an answer out of band, since they seem to have left the channel one minute after asking their question | 14:19 |
| fungi | if they're reading this log, https://lists.openstack.org/archives/list/openstack-discuss@lists.openstack.org/ is usually where people ask questions about neutron-vpnaas or other openstack components | 14:20 |
| fungi | or in the #openstack-neutron irc channel here on oftc | 14:22 |
| opendevreview | Merged openstack/project-config master: Add new projects to NebulOuS: ontology-server, brokerage-quality-assurance-server, service-level-agreement-generator https://review.opendev.org/c/openstack/project-config/+/905657 | 14:54 |
| opendevreview | Seongsoo Cho proposed openstack/project-config master: Add weblate api key in zuul secret https://review.opendev.org/c/openstack/project-config/+/905701 | 16:13 |
| clarkb | I plan to mention this in our meeting today during open discussion, but I've been thinking about trying OpenDev ptg time again. Less office hours for other projects to engage with us (thoguh we can probably do some of that) and more time to focus on some of the tech debt we've been digging up (afs mirroring and nodepool test node images, zuul sql db hosting, etc) | 17:06 |
| clarkb | Mentioning it now so that people have a bit of time to formulate opinions on that before the meeting | 17:06 |
| fungi | thanks! | 17:08 |
| opendevreview | Merged opendev/system-config master: Remove bullseye python3.11 image builds https://review.opendev.org/c/opendev/system-config/+/905018 | 19:51 |
| clarkb | I'm going to figure out lunch now then go for one last walk in the snow before it turns to ice. tonyb does sometime after 2300 UTC work for the linaro cloud cert stuff? | 19:59 |
| tonyb | clarkb: perfect. | 20:00 |
| tonyb | With reference to: https://review.opendev.org/c/openstack/project-config/+/905701 | 20:05 |
| tonyb | Is there a way infra-root can verify that secret has been encrypted correctly/will decrypt and be useful in jobs? | 20:07 |
| fungi | tonyb: https://docs.opendev.org/opendev/system-config/latest/zuul.html#secrets but we generally don't bother | 20:09 |
| tonyb | fungi: Thanks. I feel obliged in this case as it's the first time I've helped create said secret, so I'm being extra careful/supportive | 20:15 |
| fungi | tonyb: as for mutt access to the shared inbox, here's a redacted subset of my config with the relevant options. fair warning, i use mutt's folder-hook option to switch out multiple muttrc files depending on which mailbox i switch to, so this isn't a complete muttrc: https://paste.opendev.org/show/bBdZP6e9WlefjKjUmwo8/ | 20:16 |
| tonyb | fungi: Thanks. | 20:16 |
| tonyb | fungi: when I used mutt I did the same thing with folder-hooks | 20:16 |
| fungi | yeah, i have mine hooked to 3 different accounts at separate providers currently | 20:17 |
| fungi | much more convenient than running separate mutt processes for every account | 20:18 |
| tonyb | Yeah. | 20:20 |
| tonyb | It's super handy. | 20:21 |
| fungi | which, to be honest, is what i did before i learned about folder-hook | 20:24 |
| tonyb | It's the easy way to get started. I used to have separate configs/processes when I had two accounts. When I added a 3rd account I looked for a better way .... enter folder-hook | 20:26 |
| fungi | also saved a fair amount of resident memory on the 1gb vm where i run my mua, irc client and calender/reminder software | 20:27 |
| tonyb | hehe | 20:28 |
| tonyb | I noticed that jq isn't installed on several of the production servers. | 21:12 |
| tonyb | What's the best way to get that added on "all the servers" .... I'm assuming there is an existing task to do that $somewhere | 21:13 |
| tonyb | in the meantime any objection to me installing jq on zuul02 ? | 21:14 |
| tonyb | Like ... adding jq to: opendev/system-config/playbooks/roles/base/server/vars/Debian.yaml | 21:17 |
| SvenKieske | good evening from europe | 21:53 |
| SvenKieske | anybody has any idea why https://pypi.org does stop responding _during_ the TLS handshake when called via curl? this is on a single machine of a customer | 21:53 |
| SvenKieske | other curl TLS requests to other domains work from the very same machine | 21:54 |
| SvenKieske | it stops at: TLSv1.3 (OUT), TLS handshake, Client hello (1): | 21:54 |
| SvenKieske | so right at the start | 21:54 |
| SvenKieske | and sorry for the slightly off topic question, but I figured we have some experienced people here. | 21:55 |
| SvenKieske | I still think this is something local | 21:55 |
| SvenKieske | other server has no problem :/ | 21:55 |
| SvenKieske | there's a pfsense firewall involved..mhm let's ask google | 21:58 |
| clarkb | SvenKieske: is this system too old to do SNI? | 21:58 |
| clarkb | SvenKieske: pypi disabled non SNI stuff a while back | 21:59 |
| clarkb | but I would do `openssl s_client -connect pypi.org:443` and see what that says | 21:59 |
| SvenKieske | it's ubuntu 20.04 based | 22:00 |
| tonyb | SvenKieske: do the working and not working systems share the same networking config? Any tunnels or VPNs involved? | 22:00 |
| clarkb | 20.04 should be new enough to SNI | 22:00 |
| clarkb | Its xenial era that was a problem I think | 22:00 |
| SvenKieske | we currently suspect some MTU misconfiguration with the pfsense | 22:01 |
| tonyb | SvenKieske: that's kinda why I asked about the networking config | 22:02 |
| SvenKieske | yeah, thanks so far, currently working our way through pinging with no fragment header set :D | 22:09 |
| SvenKieske | originally we wanted to upgrade an openstack installation..oh well.. | 22:09 |
| tonyb | SvenKieske: you can't just set the MTU to something "low" on the problem machine to verify? | 22:10 |
| tonyb | SvenKieske: or enable PMTU discovery? | 22:10 |
| clarkb | I don't recall needing to do anything special for MTUs on my pfsense router | 22:12 |
| clarkb | tonyb: I forwarded you the email with background on linaro cloud cert renewal steps. I'm happy to simply answer questions via irc as you do that or hop on a call and do it more paired programming style | 22:18 |
| SvenKieske | I guess I could, it's a customers machine and I'm merely consulting/helping them | 22:18 |
| SvenKieske | tonyb: in my experience PMTU discovery rarely works, do you had success with it in the past? I must admit I didn't really try it that often after being discouraged by stuff I read about it and advice from older networking gurus :D | 22:19 |
| clarkb | the problem with PMTU that you often hit around openstack is that only L3 "devices" can participate | 22:20 |
| clarkb | OpenStack environments often have many many interfaces and "devices" that are either virtual l1 or l2 equivalents and they all have MTUs that can't be adjusted via PTMU | 22:21 |
| SvenKieske | another problem I have found multiple times is, that you need ICMP for automatic mtu discovery and network engineers seem to really like blocking ICMP everywhere. | 22:24 |
| SvenKieske | but whatever, I hope my problem is solved by this, thanks all for your help :) | 22:25 |
| fungi | yes, blocking icmp error responses will break pmtud and create mtu black holes. never block icmp | 22:30 |
| fungi | the rest of tcp/ip doesn't work without icmp. if you really don't want machines participating on the internet, unplug them | 22:31 |
| fungi | at least that's what i always told my customers as the "in-house security expert" when i worked for a hosting company | 22:32 |
| clarkb | following up on the stream multiple packages using too much disk thing we currently have 5 thunderbird packages mirrored | 22:37 |
| clarkb | eahc one is just over 100MB | 22:37 |
| clarkb | similar story with firefox | 22:38 |
| clarkb | this is for centos 9 stream | 22:38 |
| clarkb | currently waiting for the sort by size update request to load | 22:39 |
| clarkb | I can see having two copies a current and a pervious but it isn't clear to me why you would need 5 | 22:39 |
| clarkb | tonyb: also left some short notes on the wiki etherpad section | 22:45 |
| fungi | as did i | 22:59 |
| ianw | hrm, https://review.opendev.org/c/opendev/system-config/+/885557?tab=change-view-tab-header-zuul-results-summary doesn't look as i'd hope for the openafs installation | 23:13 |
| ianw | the 9-stream log shows that the kernel module build worked -> https://224b312f67b64e43bc51-0845cf503ed6ea2ba7b6515ac4a85fb6.ssl.cf1.rackcdn.com/885557/1/check/system-config-zuul-role-integration-centos-9-stream/6f209f3/dkms-make-logs/var/lib/dkms/openafs/1.8.10-1.el9/5.14.0-407.el9.x86_64/x86_64/log/make.log | 23:14 |
| ianw | but then in the messages https://224b312f67b64e43bc51-0845cf503ed6ea2ba7b6515ac4a85fb6.ssl.cf1.rackcdn.com/885557/1/check/system-config-zuul-role-integration-centos-9-stream/6f209f3/messages.txt | 23:14 |
| ianw | Jan 16 21:29:51 np0036427074 systemd[1]: Starting OpenAFS Client Service... | 23:14 |
| ianw | Jan 16 21:29:51 np0036427074 modprobe[82159]: modprobe: FATAL: Module openafs not found in directory /lib/modules/5.14.0-404.el9.x86_64 | 23:14 |
| ianw | oh! | 23:15 |
| ianw | CC [M] /var/lib/dkms/openafs/1.8.10-1.el9/build/src/libafs/MODLOAD-5.14.0-407.el9.x86_64-SP/afspag.mod.o | 23:16 |
| ianw | it build the module for kernel 5.14.0-407 ... but it can't find a module for 5.14.0-404 ... which suggests to me that this is runing 5.14.0-404 kernel but has headers for 507 ... | 23:16 |
| clarkb | ianw: possibly because we did a system update prior to building but without a reboot? | 23:17 |
| clarkb | tonyb: let me know what you think re cert renewal process. Happy to jump on a call but in about an hour and a half I'll have to step away for a bit for dinner | 23:24 |
| clarkb | but also happy to just have you work through it and use irc if you prefer. | 23:25 |
| ianw | we might be in a period where a new kernel has released but we haven't updated and rebooted the hosts. or, the image building is broken and the hosts aren't being updated because the new image isn't created :/ | 23:28 |
| ianw | the arm64 versions worked, so that's good at least | 23:29 |
| tonyb | clarkb: I'm available now(ish) to look at the cert updates. I don't have a preference for call vs IRC | 23:30 |
| tonyb | IRC makes it easier to multitask so lets go with that | 23:31 |
| clarkb | sounds good. Hopefully what I sent makes sense | 23:32 |
| clarkb | the school just sent an email but it is a newsletter not another day of cancellations | 23:32 |
| clarkb | the worst offender in centos stream 9 multiple packages for the same major release of software appaers to be openjdk 11. There are 10 packages ranging from 246M to 319M depending on the specific minor version | 23:38 |
| clarkb | dotnet sdks, firefox, thunderbird, libreoffice, grafana, texlive, gcc and on and on. The tail is actually quite long here | 23:39 |
| clarkb | I have a very strong suspicion this is why we're growing rapidly in those mirrosr though | 23:39 |
| clarkb | they seem to append to the distro and not clean up the older stuff | 23:40 |
| clarkb | I wonder if pruning that is feedback they would be receptive to. | 23:40 |
| clarkb | Even if they keep older packages around it seems like it would be reasonable to not do so super long term? | 23:41 |
| clarkb | trying to look in other locations (that was the AppStream location) | 23:42 |
| clarkb | under BaseOS the big offender is linux kernel related stuff. Firmware and debug modules and so on | 23:43 |
| clarkb | NeilHanlon: ^ you might know why that is done? | 23:44 |
| ianw | are we sure we're not running with rsync flags that don't prune? | 23:49 |
| clarkb | we pass --delete | 23:53 |
| clarkb | https://opendev.org/opendev/system-config/src/branch/master/playbooks/roles/mirror-update/files/centos-stream-mirror-update#L52 | 23:53 |
| clarkb | reading the rsync manpage if you try and sync foo/* then --delete doesn't work but we don't do that | 23:54 |
| clarkb | "Prior to rsync 2.6.7, this option would have no effect unless --recursive was enabled. Beginning with 2.6.7, deletions will also occur when --dirs (-d) is enabled, but only for directories whose contents are being copied." Maybe that? | 23:56 |
| clarkb | we do pass -r nevermind | 23:57 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!