| *** mlavalle has quit IRC | 00:05 | |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: [wip] Use containers for functional testing https://review.opendev.org/c/openstack/diskimage-builder/+/791888 | 00:08 |
|---|---|---|
| *** cloudnull has joined #opendev | 03:47 | |
| *** ykarel has joined #opendev | 04:24 | |
| *** ysandeep|away is now known as ysandeep | 05:16 | |
| *** ralonsoh has joined #opendev | 05:36 | |
| *** sboyron has joined #opendev | 05:47 | |
| *** marios has joined #opendev | 05:59 | |
| *** slaweq has joined #opendev | 06:33 | |
| *** vishalmanchanda has joined #opendev | 06:35 | |
| *** hashar has joined #opendev | 06:38 | |
| openstackgerrit | Matthias Runge proposed openstack/project-config master: Retire panko, python-pankoclient and puppet-panko https://review.opendev.org/c/openstack/project-config/+/791905 | 06:57 |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: [wip] Use containers for functional testing https://review.opendev.org/c/openstack/diskimage-builder/+/791888 | 07:00 |
| *** fressi has joined #opendev | 07:07 | |
| *** lucasagomes has joined #opendev | 07:16 | |
| *** andrewbonney has joined #opendev | 07:17 | |
| *** amoralej|off is now known as amoralej | 07:20 | |
| *** ralonsoh has quit IRC | 07:24 | |
| *** ralonsoh has joined #opendev | 07:24 | |
| *** rpittau|afk is now known as rpittau | 07:33 | |
| *** SWAT has quit IRC | 07:36 | |
| *** tosky has joined #opendev | 07:38 | |
| *** ysandeep is now known as ysandeep|lunch | 07:40 | |
| *** SWAT has joined #opendev | 07:43 | |
| *** ChanServ has quit IRC | 07:54 | |
| *** ChanServ has joined #opendev | 07:54 | |
| *** services. sets mode: +o ChanServ | 07:54 | |
| *** jpena|off is now known as jpena | 07:56 | |
| *** DSpider has joined #opendev | 08:30 | |
| *** DSpider has quit IRC | 08:31 | |
| *** ysandeep|lunch is now known as ysandeep | 09:13 | |
| *** vishalmanchanda has quit IRC | 09:14 | |
| *** seongsoocho has quit IRC | 09:16 | |
| *** mnasiadka has quit IRC | 09:17 | |
| *** seongsoocho has joined #opendev | 09:21 | |
| *** auristor has quit IRC | 09:27 | |
| *** auristor has joined #opendev | 09:28 | |
| *** seongsoocho has quit IRC | 09:32 | |
| *** whoami-rajat has joined #opendev | 09:33 | |
| *** seongsoocho has joined #opendev | 09:35 | |
| *** vishalmanchanda has joined #opendev | 09:38 | |
| *** mnasiadka has joined #opendev | 09:40 | |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: [wip] Use containers for functional testing https://review.opendev.org/c/openstack/diskimage-builder/+/791888 | 10:11 |
| *** dtantsur|afk is now known as dtantsur | 10:16 | |
| *** lpetrut has joined #opendev | 11:29 | |
| *** jpena is now known as jpena|lunch | 11:31 | |
| *** klonn has joined #opendev | 12:01 | |
| frickler | so #freenode is kinda exploding right now, will be interesting to see where this ends. maybe at least it will solve the spam issue ;) semi-serious question: should we consider running our own server? | 12:01 |
| *** amoralej is now known as amoralej|lunch | 12:09 | |
| dmsimard | frickler: there was a thread about threat to freenode on openstack-discuss | 12:21 |
| fungi | i already do, it's not hard (i've run irc servers, including the usual suite of service bots) for years, but previously we'd figured that oftc was a fine alternative | 12:21 |
| fungi | i already squat a bunch of our channel equivalents on oftc, since i'm always connected there anyway for other projects who communicate there | 12:22 |
| *** jpena|lunch is now known as jpena | 12:22 | |
| *** auristor has quit IRC | 12:24 | |
| *** auristor has joined #opendev | 12:25 | |
| yoctozepto | oftc is fine by me as well, already lurking and seems slightly stabler from my location here | 12:25 |
| yoctozepto | (compared to freenode) | 12:25 |
| *** ysandeep is now known as ysandeep|afk | 12:26 | |
| frickler | so my current take from watching #freenode is that the hostile takeover has already happened and we should consider taking some action soonish, see also https://gist.github.com/aaronmdjones/1a9a93ded5b7d162c3f58bdd66b8f491 . likely oftc seems the best way forward, though I'd also be willing to give libera.chat a chance | 12:31 |
| fungi | since nothing seems to be on fire here at the moment, i'm popping out for a morning walk. i'll check back in before i need to do some errands, but plan to be around for good before we get started on merging the ansible changes for the listservs | 12:37 |
| fungi | back in an hour-ish | 12:38 |
| *** ysandeep|afk is now known as ysandeep | 13:01 | |
| *** amoralej|lunch is now known as amoralej | 13:04 | |
| *** ricolin has joined #opendev | 13:08 | |
| *** ricolin has quit IRC | 13:29 | |
| *** klonn has quit IRC | 13:30 | |
| *** klonn has joined #opendev | 13:30 | |
| *** ysandeep is now known as ysandeep|mtg | 13:38 | |
| *** corvus has quit IRC | 13:53 | |
| *** tosky has quit IRC | 13:56 | |
| *** tosky has joined #opendev | 13:59 | |
| *** corvus has joined #opendev | 14:09 | |
| *** fressi has left #opendev | 14:10 | |
| *** corvus has quit IRC | 14:11 | |
| mordred | a large part of the cost to moving is going to be moving. if we're going to move away from freenode I'd strongly recommend we consider matrix instead of just a different irc network. we could run our own homeserver, pay element to run one for us, or just use one of the public ones | 14:17 |
| mordred | I'm currently paying element to run a homeserver for me and have been using it as my IRC bouncer for over a year now. i've been pleased with it overall | 14:18 |
| yoctozepto | oh, good to know | 14:19 |
| fungi | oftc is also bridged to matrix, apparently | 14:19 |
| yoctozepto | I have never used matrix yet | 14:19 |
| yoctozepto | might be time to learn it | 14:19 |
| fungi | i need to head out to run errands before we start on the mailman config management switch, back soon hopefully | 14:20 |
| mordred | fungi: yeah - you can use the various irc networks from matrix bridges - but I'd suggest, fwiw, that if we're going to move we *move* and not attempt to maintain a footprint in 2 places | 14:21 |
| fungi | so, let's say freenode falls over next week, who's rewriting all our irc bots to be matrix bots? | 14:23 |
| *** earne1 has joined #opendev | 14:24 | |
| *** ysandeep|mtg is now known as ysandeep | 14:25 | |
| mordred | fair point | 14:29 |
| mordred | not saying it's a slam dunk - just that it's worth considering as plans are considered. it's entirely possible that "rewrite the bots" is the thing that makes it a no go :) | 14:30 |
| JayF | I don't understand why folks want to migrate off IRC :( (admittedly I'm not familiar with matrix; but I can tell it's not IRC from the name :P ) | 14:30 |
| clarkb | fungi: fwiw I want to double check a few things between the held test server and production (mostly configs lining up) but then I guess I'll approve the ansible listserv change | 14:31 |
| clarkb | JayF: matrix maintains a lot of the properties that people enjoy about irc while adding in functionality that make people enjoy newer chat systems | 14:31 |
| JayF | the thing I enjoy about IRC is that I've used the same client and semantics in it for ... 20-ish years? | 14:32 |
| *** corvus has joined #opendev | 14:34 | |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Start building gerrit 3.4 https://review.opendev.org/c/opendev/system-config/+/792047 | 14:35 |
| clarkb | JayF: ya I tend to use weechat plugins to make other chat systems like IRC :) | 14:41 |
| openstackgerrit | Riccardo Pittau proposed opendev/glean master: [WIP] test dib on ubuntu focal and do not test xenial anymore https://review.opendev.org/c/opendev/glean/+/792184 | 14:42 |
| JayF | I mean, I happily use a Discord client (for personal uses only) and Slack (because my company requires it)... but I always prefered the simplicity of IRC | 14:42 |
| corvus | i'd be happy with oftc or matrix (i would probably switch from irssi to weechat to use matrix). i do run my own matrix homeserver, and have for years; or at least, i have a vm where synapse is running and every few years i debug why it's not connected. :) | 14:43 |
| clarkb | JayF: weeslack works pretty well for slack. discord actively punishes (and bans etc) users of third party clients so there aren't great options for it | 14:43 |
| JayF | clarkb: well... most third party slack clients fail at enough of slack that I need to use a slack client (including weeslack, which falls over when connected to large slack servers) | 14:44 |
| JayF | clarkb: losing things like embedded files and threading is just not an option for how we use it internally | 14:44 |
| JayF | and that is basically my fear about a higher level protocol being used instead of IRC: most "oh it works with IRC with a bridge" comes with MAJOR tradeoffs for the IRC users, and IME that support gets worse and worse over time as the newer protocol adds another 10 ways to post a dancing cat in chat | 14:45 |
| clarkb | JayF: the best thing about weeslack is the dancing cats don't render and you can opt into them :P | 14:45 |
| clarkb | but ya if you're conencted to very large servers I have heard others say that can be problematic | 14:46 |
| JayF | I mean, Rackspace had a few thousand folks on Slack, and it broke my client | 14:46 |
| JayF | no way I'd even consider hooking up to Verizon slack, even if the security folks would allow it | 14:46 |
| clarkb | fwiw I've also heard people say matrix doesn't work well in large setups. However, I suspect we're below those thresholds and don't need to worry about it | 14:46 |
| corvus | clarkb: i suspect the homeserver matters there -- if you run a homeserver on a slow dsl link, do not join #matrix (ask me how i know). but not likely an issue if using a well-hosted homeserver. | 14:50 |
| corvus | maybe we could ask mozilla folks if they've had scaling issues | 14:51 |
| clarkb | corvus: not a bad idea | 14:53 |
| *** hashar is now known as hasharAway | 14:54 | |
| clarkb | infra-root and cc fungi specifically since fungi indicated interest I've double checked that mm_cfg_local.py files, apache configs, and /etc/mailman/sites all hash to the same value (in the case of sites I did a sort first) | 14:54 |
| clarkb | I also re double checked that running newlist on a list that exists is an error | 14:54 |
| clarkb | The onyl differences I've noticed are that mm_cfg_local.py files and apache config files have slighyl different ownership/permissions | 14:55 |
| clarkb | however I don't think any of those matter for functionality | 14:55 |
| clarkb | anything else you think we should check or should we approve https://review.opendev.org/c/opendev/system-config/+/789622 ? | 14:55 |
| clarkb | fungi: ^ I added lists.openstack.org and lists.katacontainers.io to the emergency file. Maybe you want to go ahead and +A the change if you are ready? | 15:01 |
| hasharAway | JayF: hi, about migrating off of irc, I guess it is due to usability concern. It is super complicated for non tech folk, clients can be a madness to setup and you get to know some commands to register your nickname. There is also no backscroll available to catchup conversations that happened while offline | 15:01 |
| clarkb | then once it lands we can work through manual applications of it | 15:01 |
| *** lpetrut has quit IRC | 15:01 | |
| hasharAway | JayF: compared to Slack or Discord. Also where are the GIF of cats on irc? :D | 15:01 |
| JayF | hasharAway: folks who have those problems; I usually suggest using irccloud. | 15:02 |
| hasharAway | JayF: Wikimedia constantly has the discussion we end up having communities on various platform (Slack definitely, some on Discord for sure). WMF internally does use Slack here and there depending on the team. The technology folks are most here on freenode though | 15:02 |
| hasharAway | yeah irc cloud is nice solution, I personally highly recommend it. + They are involved in improving the irc protocol so that alone is worth spending a few buck for their service | 15:03 |
| *** rpittau is now known as rpittau|bbl | 15:03 | |
| hasharAway | I definitely love IRC, that was my 2nd exposure to the internet after Usenet .. | 15:03 |
| hasharAway | anyway, I came around to state that wikimedia volunteers (and staff I guess) are looking at the freenode thing. There is a discussion that started with the wide community at https://meta.wikimedia.org/wiki/Wikimedia_Forum#Freenode_%28IRC%29 | 15:04 |
| hasharAway | will surely take a few days to settle on something (stay with freenode or/and try to fix it, move to libera.chat fork by freenode staff, something else) | 15:05 |
| avass | clarkb: do discord ban third party clients? #nim uses a bot to bridge slack+discord+irc | 15:07 |
| clarkb | avass: they do | 15:08 |
| *** mlavalle has joined #opendev | 15:08 | |
| clarkb | avass: https://news.ycombinator.com/item?id=25214777 recent case of it too | 15:08 |
| clarkb | fungi: let me know when you are ready to do listserv things so that we can sync up | 15:14 |
| fungi | i'm back from errands, going to eat real quick while i finish catching up on irc drama | 15:19 |
| clarkb | fungi: cool, maybe double check my double checking assertions above to make sure you're comfortable with the file perms and ownership deltas | 15:20 |
| clarkb | mm_cfg_local.py is root:root instead of root:list and 444 vs 644 (that is test vs prod in that order) | 15:20 |
| fungi | yeah, the permissions on those files are fine, read-only for a root owner is slightly silly imo, but it's irrelevant so long as those particular files are readable by the users which need them we're good (and they're world-readable, so that ticks the box) | 15:26 |
| fungi | clarkb: i've approved 789622 now | 15:27 |
| clarkb | cool, I'll keep an eye on it though the merge should be a noop until we remove a host from the emergency file and manually run the playbook against that host | 15:28 |
| clarkb | fungi: I'll set up a root screen on bridge for that and we can both hang out on the listservs and run the playbook when ready | 15:29 |
| *** ykarel has quit IRC | 15:29 | |
| clarkb | I guess I should eat breakfast while I wait for it to merge | 15:29 |
| clarkb | oh! I also need to set one var in host/group vars. Will do that now | 15:29 |
| dtantsur | JayF: re "why against IRC". I *love* IRC, but I suspect it may be a serious show-stopped for newcomers, especially those without a strong FOSS background. | 15:31 |
| JayF | dtantsur: I've had success pointing those folks at irccloud. | 15:32 |
| dtantsur | I don't see how this is related | 15:32 |
| dtantsur | Setting up a semi-decent client is not an issue; coping with how IRC does things is | 15:32 |
| dtantsur | Not realizing that logging off makes it impossible to respond to you, for one | 15:32 |
| JayF | dtantsur: irccloud is a web-based IRC client that maintains a persistent presence | 15:33 |
| dtantsur | trying to send files through some non-standard means (maybe this infamous irccloud) | 15:33 |
| JayF | dtantsur: it's what I typically point newcomers to use to connect to open source channels | 15:33 |
| clarkb | fungi: ok host vars are updated if you want to check the git log there | 15:33 |
| dtantsur | "Stay connected for 2 hours while inactive" | 15:33 |
| dtantsur | and file sharing feature that people assume to work outside of irccloud | 15:34 |
| earne1 | On the topic of IRC, any plans to move this onto libera.chat? | 15:34 |
| dtantsur | earne1: likely OFTC or even another platform | 15:35 |
| clarkb | earne1: not yet, we already squat channels on oftc so more likely there | 15:35 |
| dtantsur | Outreachy uses Zulip and tbh I really enjoyed having threads | 15:35 |
| clarkb | dtantsur: I thought irccloud gave you a link to the files shared there | 15:36 |
| clarkb | dtantsur: I recently tried zulip and enjoyed the threading too, but the client situation is really painful | 15:36 |
| clarkb | I basically ended up just using the browser client and while it worked is nowhere near as functional as something like weechat | 15:36 |
| dtantsur | clarkb: dunno, weechat tries to display these file transfers somehow, but this *somehow* never worked to me | 15:36 |
| fungi | yeah, the console client for zulip has some *very* rough edges at this point | 15:36 |
| dtantsur | yeah, I'd be sad without weechat | 15:36 |
| dtantsur | I know there is a matrix plugin and a so-so slack plugin, maybe zulip too? | 15:37 |
| fungi | right, and no weechat plugin for zulip that i was able to find | 15:37 |
| dtantsur | le sigh | 15:37 |
| fungi | but yeah zulup-term is still in a painful state | 15:37 |
| dtantsur | to be clear, I've never tries matrix, but mozilla uses it, so it cannot be too bad :) | 15:37 |
| fungi | many of these newer chat protocols are built on top of web technologies and assume all users are going to be connecting through a web browser or a phone | 15:38 |
| fungi | matrix does seem to have a fairly broad client ecosystem at this point though, yeah | 15:38 |
| dtantsur | having phone support would be vary handy during face-to-face (*sob* *sob*) events btw | 15:39 |
| clarkb | dtantsur: I used connectbot + screen + weechat for many years | 15:39 |
| clarkb | the onyl downside is it drains your battery if you keep a connection open long term, but for polling its fine | 15:39 |
| clarkb | I think there are android clients using the weechat protocol too | 15:39 |
| clarkb | but I have never used them | 15:39 |
| mordred | the matrix phone client is very good | 15:40 |
| mordred | I've used the android weechat clients and they're ok-ish - but questionable security | 15:40 |
| clarkb | mordred: I thought they used client cert auth? It should be fairly robust | 15:40 |
| clarkb | (you have to keep your ca safe though) | 15:41 |
| mordred | but the nice thing matrix-wise is that these features are easily available to our entire community not just those of us in a position to run our own bouncer servers and whatnot | 15:41 |
| clarkb | mordred: definitely | 15:41 |
| clarkb | but its also a giant effort that no one has time for right now (or at least I don't have time for it) | 15:41 |
| mordred | yup | 15:41 |
| mordred | that's the downside | 15:41 |
| mordred | it's a shame - my thoughts are mostly that if the community is going to have to eat the cost of a move to somewhere, such a time is the best time to make a tech shift if we were ever going to do so | 15:42 |
| dtantsur | on the other hand, currently our huge community pretty much relies on volunteers | 15:42 |
| dtantsur | which is not exactly fair | 15:42 |
| fungi | right now our focus is split between keeping things we absolutely need running, and shutting down anything we don't absolutely need so we have enough people for the first thing | 15:42 |
| mordred | yup | 15:42 |
| dtantsur | given the Big Names that participate in OpenStack, getting some money for a hosted server should be doable. | 15:42 |
| * dtantsur wishful thinking | 15:43 | |
| mordred | if we did want to go matrix, just paying element to host one for us would likely be more tractible than deciding to start running one ourselves | 15:43 |
| JayF | I suspect the resources needed to run our own server are more human resources than server resources | 15:43 |
| mordred | yah | 15:43 |
| dtantsur | I mean paying for a server | 15:44 |
| dtantsur | I definitely don't suggest creating much more work for our infra team | 15:44 |
| corvus | i do like matrix, but honestly, the cost of switching to oftc is relatively small in terms of both technology and training. if we need to vacate freenode in short order, oftc seems like a tractable solution. matrix would be a nice thing to actually plan for. | 15:44 |
| dtantsur | yeah, I don't disagree | 15:45 |
| clarkb | corvus: ++ | 15:45 |
| dtantsur | (assuming OFTC is well funded) | 15:45 |
| JayF | OFTC has been around a long time, and is run by a foundation | 15:45 |
| JayF | It's the obvious choice if we don't wanna go to libera | 15:45 |
| *** gothicserpent has joined #opendev | 15:46 | |
| dtantsur | do we know if OFTC is okay hosting us? | 15:46 |
| dtantsur | and yes, IRC has a benefit of preventing me from spamming #openstack-ironic with cat gifs, that's for sure :) | 15:47 |
| JayF | It's been stated on the mailing list that we've been camping the openstack- channels on OFTC as a backup for a while | 15:47 |
| corvus | they seemed quite happy when i asked about it years ago | 15:47 |
| corvus | quite supportive in getting all those initial channels set up | 15:48 |
| dtantsur | sounds like a no-brainer to me | 15:48 |
| fungi | granted i wouldn't be surprised if a freenode mass exodus puts some new strain on their network | 15:48 |
| dtantsur | I would expect libra to receive the most migrations given that it's the advertised destination | 15:50 |
| openstackgerrit | Merged opendev/system-config master: Ansible mailman configs https://review.opendev.org/c/opendev/system-config/+/789622 | 15:55 |
| *** marios has quit IRC | 15:56 | |
| corvus | my 2 cents: i feel like we need to establish a timeline -- 1) are we moving? 2) how long can we give ourselves to move? if it's sufficiently long to plan for updating our docs for matrix, then let's seriously consider it. if it isn't, then let's move to oftc and then seriously consider moving to matrix. | 15:57 |
| corvus | mordred: does matrix.org has a bridge to oftc? | 15:58 |
| fungi | supposedly yes | 15:58 |
| corvus | https://github.com/matrix-org/matrix-appservice-irc/wiki/Bridged-IRC-networks says yes | 15:58 |
| fungi | i haven't double-checked it, but saw others say it did | 15:58 |
| *** lucasagomes has quit IRC | 15:59 | |
| fungi | and yes, i agree we're currently stuck at #1 are we moving? signs point to yes, but it may make sense to give it a few days for more information to help determine that | 16:02 |
| *** ysandeep is now known as ysandeep|away | 16:03 | |
| clarkb | corvus: judging from conversations on the openstack discuss list and in the openstack tc channel it seems that our users would like to move. I would put the answer to 1) as most likely | 16:03 |
| fungi | right now the information which has been circulated seems a bit one-sided, but since there doesn't seem to be much effort yet at a rebuttal it's worth assuming the worst conspiracy theories may be true | 16:03 |
| fungi | where "worst" are in the vein of the overlord unceremoniously turning off everything in an effort to drive users to some new platform in which they have a financial stake | 16:05 |
| dtantsur | without judging who is right and who is wrong, even all current freenode staff members have left, it's no longer a stable platform for us | 16:05 |
| dtantsur | s/even/if/ | 16:05 |
| dmsimard | If it has an impact, I am told that OFTC doesn't support SASL while freenode/libera.chat do | 16:05 |
| fungi | did all of the staff really leave? or is someone just saying all the staff left? which reports are credible? making quick decisions based on incomplete and/or unconfirmed information may be rash | 16:06 |
| tosky | fungi: if it's not all, at least a good part of the staff has left | 16:07 |
| dtantsur | we're not going to move this afternoon anyway. reaching out to OFTC to probe the ground may be a good early move anyway. | 16:07 |
| tosky | fungi: we received this from a trusted member of the KDE community, who is also one of the former freenode administrators: https://mail.kde.org/pipermail/kde-community/2021q2/006888.html | 16:08 |
| JayF | Lots of other projects have already moved this afternoon; I don't think we should assume waiting is the best path. Right now, every OpenStack contributor still has to maintain a working nickname on this network. These are literally the last set of channels I'm in on freenode (all the others just moved) | 16:09 |
| dtantsur | (or start looking into what it takes to move to matrix. or making up our mind at least) | 16:09 |
| clarkb | JayF: I mean "maintain a working nickname" is basically free if you've previously done the setup | 16:09 |
| JayF | except now that email/password are in the hands of $unknown_entities | 16:09 |
| JayF | and I can't drop the nick as long as we continue to use this service | 16:09 |
| JayF | at least two of the resigning staff strongly suggested dropping your nicks or resetting email/password to keep yours private | 16:10 |
| JayF | how do we know that services DB isn't going to be sold now? Or used for $nefarious_purposes? | 16:10 |
| clarkb | re email those are already largely public (if not because of the billions of exposed accounts indexed by haveibeenpwned then due to work on open source with git). And passwords should be unique already? | 16:11 |
| mordred | well - my email is a lost cause because of git | 16:11 |
| mordred | my password is specific to freenode, so losing it to someone only risks them having access to my freenode info - which admins of freenode already have | 16:11 |
| mordred | clarkb: jinx | 16:11 |
| JayF | I guess that's a decent point. Doesn't make me feel less itchy to get an account off a system that appears to have been taken over with hostility. | 16:12 |
| avass | I hope people use password managers :) | 16:13 |
| fungi | depends on what you mean by password managers, but i don't reuse passwords between different systems | 16:14 |
| avass | fungi: well yeah, pretty much just that | 16:14 |
| corvus | fuchs is a well-known friendly op who has helped us many times | 16:14 |
| corvus | or i should say Fuchs | 16:15 |
| clarkb | corvus: and is one of the staff that quit right? | 16:15 |
| fungi | yep, network effect indicates their account is posted at https://fuchsnet.ch/freenode-resign-letter.txt | 16:15 |
| fungi | account of what's transpired i mean | 16:15 |
| fungi | i don't know for sure that fuchsnet.ch is really under their control | 16:15 |
| dtantsur | I think if the news were false, we would already have received a global message through freenode itself | 16:16 |
| dtantsur | so far the only global message confirmed the news | 16:16 |
| cenn | #freenode is chaos rn, so.. at the least staff is not in control / has abandoned control | 16:17 |
| JayF | I've seen no evidence that indicates the story we've heard is not true, or close enough to true, to believe. In any event, a network can't survive without staff, and freenode just had the bulk of their staff resign. Which means the 'whodunit' aspect of this whole thing doesn't matter; freenode is no longer a suitable host. | 16:18 |
| fungi | and i suppose we can assume that if they unresign (freenode staff drama seems to be a semi-regular event after all) then we should still see it as a sign that their operations are unstable | 16:20 |
| dtantsur | exactly | 16:21 |
| tosky | fungi: I don't think they are going to unresign after that email; I don't think Fuchs would have sent that email to kde-community | 16:21 |
| tosky | fungi: again: Fuchs is a trusted member (also) of the KDE community, and I trust what's written there | 16:22 |
| fungi | i have to admit that from the perspective of the sorts of drama which permeate efnet and undernet this seems mild, but i've always felt oftc would have been a better place for the channels we've got here anyway | 16:22 |
| fungi | i just always get suspicious when people try to rush decisions | 16:24 |
| *** hamalq has joined #opendev | 16:25 | |
| clarkb | fungi: I'm almost ready to do lists.katacontainers.io. I have a root screen up if you want to join and check the command that is queued up. We'll run base first because it was skipped by the automatic run due to being in the emergency file | 16:28 |
| fungi | i've attached | 16:31 |
| fungi | also logged into both list servers to double-check stuff | 16:31 |
| clarkb | great | 16:31 |
| clarkb | I've removed lists.katacontainers.io from the emergency file. Will run the base playbook now | 16:32 |
| *** cgoncalves has quit IRC | 16:32 | |
| *** cgoncalves has joined #opendev | 16:33 | |
| clarkb | fungi: that lgtm very noopy | 16:33 |
| clarkb | fungi: ready for me to run service-lists.yaml? | 16:33 |
| fungi | yep | 16:34 |
| clarkb | ok proceeding | 16:34 |
| clarkb | fungi: the ansible side lgtm. It reloaded apache2 because the file permissions chagned on that fiel I think | 16:36 |
| fungi | yep, i can still browse the website | 16:37 |
| clarkb | fungi: I am able to browse the web archives for the kata-dev list | 16:37 |
| fungi | and archives as well | 16:37 |
| clarkb | fungi: it also skipped creating the lists because they arleady exist which is what we wanted | 16:37 |
| clarkb | should we proceed with lists.openstack.org? it will be significantly more verbose as there are a lot more lists, but it shoudl skip them all too | 16:38 |
| fungi | no services were restarted, so if configs were changed we wouldn't know right away | 16:38 |
| clarkb | fungi: do we want to restart the mailman service manually? | 16:38 |
| clarkb | apache was reloaded so those config checks should be done, its just mailman to worry about I think | 16:39 |
| fungi | yeah, and i guess exim was already being configured by ansible so no risk there | 16:40 |
| clarkb | correct | 16:40 |
| fungi | i'll restart the mailman services on it just to make sure they start | 16:40 |
| clarkb | ++ thanks | 16:41 |
| fungi | #status log Restarted mailman services on lists.katacontainers.io in order to verify configuration after config management changes | 16:41 |
| openstackstatus | fungi: finished logging | 16:42 |
| clarkb | fungi: should I remove lists.o.o from emergency now and run base against it? | 16:42 |
| fungi | yeah, i count 9 mailman processes on the server, so should be working | 16:42 |
| clarkb | fungi: oh wait we should double check the crontab on lists.kc.io first | 16:42 |
| fungi | i doubt the list traffic there is constant enough that we can reasonably wait to watch it process some posts | 16:43 |
| clarkb | ya | 16:43 |
| fungi | there are no crontabs in /var/spool/cron/crontabs/ | 16:43 |
| clarkb | the cron is multisite only | 16:43 |
| clarkb | so ya nothing to check on lists.kc.io for cron | 16:43 |
| clarkb | we'll want to check and clean that up on lists.o.o though | 16:43 |
| clarkb | I'm remove it from emergency.yaml and run base.yaml against lists.openstack.org | 16:44 |
| fungi | yep, ready and watching | 16:44 |
| fungi | after this is done, we can send something to service-discuss about it, which doubles as confirmation the plumbing is intact | 16:45 |
| *** rpittau|bbl is now known as rpittau | 16:45 | |
| clarkb | ++ | 16:46 |
| fungi | that looked smooth | 16:47 |
| clarkb | fungi: base.yaml lgtm are you ready for me to proceed to service-lists.yaml? Also maybe we rerun service-lists.yaml one mroe time without the --limit once we've done the two servers independently | 16:47 |
| fungi | yeah, great idea | 16:47 |
| fungi | and i'm ready when you are | 16:47 |
| clarkb | cool proceeding then | 16:47 |
| *** cgoncalves has quit IRC | 16:52 | |
| clarkb | fungi: the crontab looks fine actually. | 16:52 |
| clarkb | there is an extra puppet comment to remove but otherwise its good as is I think | 16:52 |
| *** cgoncalves has joined #opendev | 16:53 | |
| fungi | agreed, the autogenerated by puppet comment block can be cleaned up | 16:54 |
| *** hasharAway is now known as hashar | 16:55 | |
| clarkb | fungi: its done | 16:56 |
| clarkb | apache was reloaded | 16:56 |
| clarkb | I can still get http://lists.opendev.org | 16:56 |
| clarkb | fungi: do you want to restart the mailman services on lists.o.o too? | 16:56 |
| clarkb | (I think we should asking if you want to run the commands or if I should, note they are vhost specific units/init scripts) | 16:57 |
| fungi | clarkb: luckily they're partitioned by site, so yeah we can restart the lists.opendev.org site daemons and then test | 16:57 |
| clarkb | fungi: ++ | 16:57 |
| fungi | i'll do that after i check stuff on the fs | 16:57 |
| clarkb | ok | 16:58 |
| fungi | pipermail hasn't suddenly started listing hidden lists, so that's good | 16:59 |
| fungi | lists with non-public archives are also still correctly flagged | 17:00 |
| fungi | and trying to browse those private archives anonymously still returns a 404 not found | 17:01 |
| fungi | looking right so far | 17:01 |
| clarkb | yay | 17:01 |
| fungi | ready for me to restart mailman-opendev daemons? | 17:01 |
| clarkb | yes | 17:01 |
| fungi | stopping mailman-opendev the process count for mailman dropped from 45 to 36 as expected (9 processes per site) | 17:03 |
| fungi | starting it again brought the total back to 45 | 17:03 |
| clarkb | I see the new processes now too | 17:03 |
| clarkb | should I send an email to service-discuss now? | 17:03 |
| fungi | yes please | 17:03 |
| fungi | if i receive that, then we're probably safe to proceed with the rerunning of the playbooks without --limit | 17:04 |
| *** ricolin_ has joined #opendev | 17:04 | |
| clarkb | really quickly before sending I think http://lists.openstack.org/cgi-bin/mailman/listinfo may not be using the right template? | 17:05 |
| clarkb | that isn't fatal so I think we can proceed and try to figure that out | 17:05 |
| clarkb | email sent | 17:06 |
| clarkb | fungi: I think I got the email sent back to me from lists | 17:07 |
| clarkb | fungi: have you recieved it yet? | 17:08 |
| fungi | yep, received | 17:09 |
| clarkb | fungi: I guess we restart the others now? | 17:09 |
| *** jpena is now known as jpena|off | 17:10 | |
| fungi | as for the template, the individual list pages do use the custom openstack one but the main list index seems not to. i don't recall if it ever did though | 17:10 |
| *** ralonsoh has quit IRC | 17:11 | |
| fungi | i'll go ahead and restart all 5 of the sites (including lists.opendev.org) just to make sure things are sane after that last seemingly no-op playbook run | 17:11 |
| clarkb | ya I don't recall either. I can see we have a listinfo.html template in that dir | 17:11 |
| fungi | there are a bunch of defunct runners left over from my attempts to stop services | 17:12 |
| fungi | giving them a minute to maybe wake up | 17:12 |
| fungi | okay, they did eventually terminate | 17:12 |
| fungi | and started again, we're back at 45 daemons (9 kinds of runners for 5 sites) | 17:13 |
| clarkb | I think the listinfo.html we have is for list info on a single list | 17:13 |
| clarkb | and that is why we get the branded output for single lists | 17:13 |
| fungi | #status log Restarted mailman services on lists.openstack.org in order to verify configuration after config management changes | 17:13 |
| openstackstatus | fungi: finished logging | 17:13 |
| clarkb | I suspect that we may just not template the thing for the listing (is it fully cgi without a template?) | 17:13 |
| clarkb | fungi: should I rerun the service-lists.yaml playbook now without the limit specified? | 17:14 |
| clarkb | ya if I grep for strings on the listing page in the templates dir I don't get any hits | 17:15 |
| clarkb | fairly certain that the "issue" there is the cgi for listing the lists doesn't use a template the way we brand the other things, but I could be wrong about that and someone else should probably double check me | 17:16 |
| clarkb | fungi: I've got the no limit service-lists.yaml playbook queued up. Ready to run that when you are | 17:18 |
| fungi | each of the pages uses a separate template, there is a default but we can supply our own as an override | 17:18 |
| clarkb | fungi: right looking in the defaults dir I don't find what the template is for that page | 17:18 |
| fungi | ahh, because there's probably no need since it doesn't need to be copied per-list | 17:19 |
| fungi | yeah, ready for the unlimited run now | 17:20 |
| clarkb | that was based on grepping for 'Below is a listing' in the default templates dir | 17:20 |
| clarkb | ok running unlimited run now | 17:20 |
| clarkb | using synchronize to sync the files says it changed things when it doesn't change then :/ oh well | 17:23 |
| *** frigo has joined #opendev | 17:24 | |
| *** dtantsur is now known as dtantsur|afk | 17:25 | |
| clarkb | fungi: another weirdness is that using snychronize to copy the templates is preserving the owner uid and group gid from bridge | 17:27 |
| clarkb | I'm not sure how to address that :/ | 17:27 |
| clarkb | maybe synchronize is the wrong tool here, but file and copy don't really do what I want either which is to say take the entirety of a dir and put it over here and set attributes | 17:28 |
| clarkb | fungi: cool service-lists.yaml has finished without limit and it looked ok to me | 17:28 |
| clarkb | https://review.opendev.org/c/opendev/system-config/+/790524/2 should be ready for review if we are happy with these results | 17:29 |
| clarkb | I'll work on a change to clean up the old puppetry as well as see if I can sort out how to copy those files in a better way for permissions | 17:29 |
| clarkb | s/ready for review/ready for approval/ | 17:30 |
| clarkb | I have cleaned up the puppet comment in the crontab | 17:33 |
| *** rpittau is now known as rpittau|afk | 17:37 | |
| fungi | we've seen that behavior with zuul jobs too. basically have to set ownership/perms before sync | 17:37 |
| *** frigo has quit IRC | 17:40 | |
| openstackgerrit | Clark Boylan proposed opendev/system-config master: Cleanup mailman puppetry https://review.opendev.org/c/opendev/system-config/+/792220 | 17:44 |
| clarkb | infra-root I think https://review.opendev.org/c/opendev/system-config/+/790524/2 and https://review.opendev.org/c/opendev/system-config/+/792220 should be ready to go now | 17:44 |
| *** mkowalski has quit IRC | 17:49 | |
| openstackgerrit | Clark Boylan proposed opendev/system-config master: Fixup openstack mailman template perms https://review.opendev.org/c/opendev/system-config/+/792228 | 17:54 |
| clarkb | infra-root ^ and that one should fixup the perms thing with template files | 17:54 |
| clarkb | I need to go make lunch for the kids now, but I'll still be around today if we want to keep pushing on this stuff | 17:55 |
| clarkb | fungi: did you put mk-archives-index together for indexing retired lists? can you maybe double check on that too? | 18:01 |
| *** amoralej is now known as amoralej|off | 18:02 | |
| *** andrewbonney has quit IRC | 18:05 | |
| fungi | yeah, i'll take a look | 18:08 |
| fungi | it's mainly for keeping track of what lists we have archives for whether or not they're still active vs historic archives | 18:09 |
| *** vishalmanchanda has quit IRC | 18:22 | |
| openstackgerrit | Clark Boylan proposed opendev/system-config master: Fixup openstack mailman template perms https://review.opendev.org/c/opendev/system-config/+/792228 | 18:31 |
| clarkb | testing is like the best thing ever | 18:31 |
| clarkb | fungi: do you think we should approve https://review.opendev.org/c/opendev/system-config/+/790524/2 now? also reviews on the other two chagnes I just pushed up arem uch appreciated | 18:51 |
| fungi | i'll take a look | 18:51 |
| fungi | approved it now, and looking at the other two as well | 18:53 |
| clarkb | thanks! | 18:53 |
| *** mkowalski has joined #opendev | 18:54 | |
| *** mtreinish has quit IRC | 18:55 | |
| *** hashar has quit IRC | 18:55 | |
| *** mkowalski has quit IRC | 18:55 | |
| *** mkowalski has joined #opendev | 18:56 | |
| *** mtreinish has joined #opendev | 19:01 | |
| *** DSpider has joined #opendev | 19:06 | |
| openstackgerrit | Clark Boylan proposed opendev/system-config master: Remove system-config-legacy-logstash-filters job https://review.opendev.org/c/opendev/system-config/+/792247 | 19:07 |
| clarkb | infra-root ^ that is sort of a temp taking change | 19:08 |
| clarkb | we said we wouldn't be turning anything off (at least not yet) service wise in the ongoing discusson on openstack-discuss, but that job runs against all our changes to system-config and isn't quick | 19:08 |
| clarkb | I'm thinking that since the ELK stuff is fairly fixed in time now we can safely stop running that job | 19:08 |
| corvus | clarkb: i left comments on puppet removal and mm templates changes | 19:11 |
| clarkb | looking | 19:11 |
| openstackgerrit | Clark Boylan proposed opendev/system-config master: Cleanup mailman puppetry https://review.opendev.org/c/opendev/system-config/+/792220 | 19:13 |
| clarkb | corvus: ^ thanks those were definitely in need of fixing | 19:14 |
| clarkb | corvus: for the rsync opts one do you have a preference? I feel like the copy is easier to understand but far more verbose | 19:14 |
| clarkb | which makes me waffle on which I prefer | 19:14 |
| corvus | clarkb: did you see my inline comment? | 19:15 |
| corvus | maybe i should have used more words, sorry; am eating lunch at same time :) | 19:15 |
| corvus | my idea is that if we can use those extra rsync opts, then we get best of both worlds -- ownership is correct, without additional verbosity | 19:16 |
| openstackgerrit | Merged opendev/system-config master: Add infra-prod-service-lists job https://review.opendev.org/c/opendev/system-config/+/790524 | 19:16 |
| clarkb | ya it wasn't clear if you prefered that since I don't think you voted either way. I'll update it | 19:16 |
| clarkb | looks like https://review.opendev.org/c/opendev/system-config/+/790524 didn't cause the deploy job to run. It will run during our periodic run instead | 19:19 |
| clarkb | or if we land an update like 792228 | 19:19 |
| clarkb | I'm just reading up on rsync really quickly then can push that | 19:19 |
| *** whoami-rajat has quit IRC | 19:19 | |
| openstackgerrit | Clark Boylan proposed opendev/system-config master: Fixup openstack mailman template perms https://review.opendev.org/c/opendev/system-config/+/792228 | 19:22 |
| clarkb | thanks for the reviews | 19:22 |
| openstackgerrit | Ananya proposed opendev/elastic-recheck master: Fixing: TypeError: '<' not supported between instances of 'list' and 'str' https://review.opendev.org/c/opendev/elastic-recheck/+/792259 | 20:09 |
| openstackgerrit | Ananya proposed opendev/elastic-recheck master: Fixing '<' not supported between instances of 'list' and 'str' https://review.opendev.org/c/opendev/elastic-recheck/+/792259 | 20:10 |
| *** auristor has quit IRC | 20:10 | |
| *** auristor has joined #opendev | 20:10 | |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!