| opendevreview | Ian Wienand proposed zuul/zuul-jobs master: test-requirements: bump to Ansible 2.8 https://review.opendev.org/c/zuul/zuul-jobs/+/851334 | 00:17 |
|---|---|---|
| opendevreview | Ian Wienand proposed zuul/zuul-jobs master: linters: use Ansible 2.8 for ansible-lint https://review.opendev.org/c/zuul/zuul-jobs/+/851289 | 00:17 |
| opendevreview | Merged opendev/system-config master: Use rackspace mirror to sync centos stream repos https://review.opendev.org/c/opendev/system-config/+/852793 | 00:17 |
| *** dviroel|afk is now known as dviroel | 00:26 | |
| ianw | ^ one that deploys i'll do a manual sync to ensure sanity | 00:35 |
| ianw | "sanity" | 00:35 |
| opendevreview | Merged opendev/system-config master: Add IRC logging for #openstack-latinamerica https://review.opendev.org/c/opendev/system-config/+/852923 | 00:37 |
| *** rlandy|bbl is now known as rlandy|out | 01:11 | |
| Clark[m] | I think precreating the xapian dir fixed the issue based on the screenshots | 01:15 |
| Clark[m] | The archives page has listings now | 01:15 |
| fungi | looks like infra-prod-base failed on deploying 852923 | 01:46 |
| fungi | checking logs | 01:46 |
| fungi | looks like it may be tripping over the ua enablement task for the storyboard server? | 01:48 |
| fungi | it's running `ua enable esm-infra` which is resulting in an rc of 1 | 01:49 |
| *** ysandeep|out is now known as ysandeep | 01:53 | |
| fungi | this check should be passing: https://opendev.org/opendev/system-config/src/branch/master/playbooks/roles/base/repos/tasks/Debian.yaml#L44 | 01:55 |
| fungi | causing it to be skipped i mean | 01:55 |
| fungi | i see the expected string in `ua status` output on the server | 01:56 |
| fungi | maybe it was something transient? | 01:56 |
| fungi | i'll try to reenqueue it | 01:57 |
| fungi | did it again. i've temporarily added the sb server to the emergency disable list and reenqueued again | 02:21 |
| opendevreview | OpenStack Proposal Bot proposed openstack/project-config master: Normalize projects.yaml https://review.opendev.org/c/openstack/project-config/+/852931 | 02:23 |
| opendevreview | Ian Wienand proposed zuul/zuul-jobs master: linters: add ansible-lint job with Ansible 2.8 https://review.opendev.org/c/zuul/zuul-jobs/+/851289 | 02:33 |
| opendevreview | Ian Wienand proposed zuul/zuul-jobs master: zuul_azure_storage_upload: rename https://review.opendev.org/c/zuul/zuul-jobs/+/852932 | 02:33 |
| opendevreview | Merged openstack/project-config master: Normalize projects.yaml https://review.opendev.org/c/openstack/project-config/+/852931 | 02:46 |
| *** ysandeep is now known as ysandeep|breakfast | 03:14 | |
| *** ysandeep|breakfast is now known as ysandeep|afk | 03:35 | |
| opendevreview | Ian Wienand proposed zuul/zuul-jobs master: zuul_azure_storage_upload: rename https://review.opendev.org/c/zuul/zuul-jobs/+/852932 | 04:42 |
| opendevreview | Ian Wienand proposed zuul/zuul-jobs master: linters: add ansible-lint job with Ansible 2.8 https://review.opendev.org/c/zuul/zuul-jobs/+/851289 | 04:42 |
| opendevreview | Ian Wienand proposed zuul/zuul-jobs master: linters: add ansible-lint job with Ansible 2.8 https://review.opendev.org/c/zuul/zuul-jobs/+/851289 | 05:05 |
| *** ysandeep|afk is now known as ysandeep | 05:35 | |
| ianw | #status log resynced centos-stream mirror from RAX | 06:34 |
| opendevstatus | ianw: finished logging | 06:34 |
| opendevreview | Merged opendev/system-config master: system-config-run-borg-backup: add to gate https://review.opendev.org/c/opendev/system-config/+/852799 | 07:11 |
| opendevreview | Merged zuul/zuul-jobs master: Use RDO wallaby repo to install openvswitch in CS9 https://review.opendev.org/c/zuul/zuul-jobs/+/852176 | 07:25 |
| opendevreview | Merged zuul/zuul-jobs master: Add cli running example for s3 and swift log uploaders https://review.opendev.org/c/zuul/zuul-jobs/+/848135 | 07:25 |
| *** ysandeep is now known as ysandeep|lunch | 07:31 | |
| *** ysandeep|lunch is now known as ysandeep | 08:34 | |
| *** ysandeep is now known as ysandeep|bbl | 10:05 | |
| *** ysandeep|bbl is now known as ysandeep | 11:16 | |
| fungi | #status log Restarted meetbot container to pick up recently deployed channel addition in its config | 12:46 |
| opendevstatus | fungi: finished logging | 12:46 |
| *** ysandeep is now known as ysandeep|afk | 12:55 | |
| *** ysandeep|afk is now known as ysandeep | 13:52 | |
| *** ysandeep is now known as ysandeep|out | 14:04 | |
| *** Guest139 is now known as diablo_rojo_phone | 14:10 | |
| *** dviroel is now known as dviroel|out | 14:31 | |
| *** marios is now known as marios|out | 15:00 | |
| fungi | eventual feedback on our warehouse feature request to chunk up the lines in the pep 691 json responses seems to be that proxies should adapt to pypi, not the other way around: https://github.com/pypi/warehouse/issues/11919#issuecomment-1213239156 | 15:39 |
| priteau | Hello. This is probably a stupid question, but do you know why https://review.opendev.org/c/openstack/kayobe/+/850903 has not run gate jobs when W+1 was added two days ago and why check jobs are not running after the rebase I just did? | 16:09 |
| priteau | There is no unmerged parent change. | 16:11 |
| priteau | Could it be stuck due to the depends-on? | 16:11 |
| fungi | priteau: zuul doesn't know about stackhpc/ansible-role-os-images | 16:11 |
| fungi | priteau: https://opendev.org/openstack/project-config/src/branch/master/zuul/main.yaml#L1443-L1444 | 16:12 |
| fungi | you'll want to add it there | 16:12 |
| priteau | Ah thanks :) | 16:12 |
| opendevreview | Pierre Riteau proposed openstack/project-config master: Add StackHPC Ansible roles used by openstack/kayobe https://review.opendev.org/c/openstack/project-config/+/853003 | 16:18 |
| fungi | testing the currently held mm3 test node... | 16:31 |
| fungi | i signed up with my normal address, which is the default owner for the openstack-discuss ml, and was able to manage it after fishing the address verification url out of the outgoing exim queue | 16:32 |
| fungi | i used that to bulk subscribe a separate address to that ml, and set it as pre-verified in order to simplify testing | 16:33 |
| fungi | i then injected a message over raw tcp socket for the list from a third address, confirmed it was held for moderation, released it from moderation, and found an outgoing copy to the list subscriber | 16:34 |
| fungi | next, i similarly injected another message to the list from the subscribed address, and confirmed that it went through without getting held for moderation | 16:35 |
| fungi | so basic list functionality seems to be there | 16:35 |
| fungi | archival doesn't seem to be working yet, or is delayed perhaps | 16:38 |
| fungi | hyperkitty claims there are have been no posts to the list, when it should have two | 16:38 |
| dpanech | Hi, could someone add me as the 1st member of the cores group of this repo that I created yesterday? https://opendev.org/starlingx/jenkins-pipelines | 16:39 |
| fungi | dpanech: sure, just a moment | 16:39 |
| fungi | dpanech: i've added you to the starlingx-jenkins-pipelines-core group | 16:42 |
| dpanech | thank you! | 16:42 |
| fungi | any time! | 16:43 |
| dpanech | fungi: I think the storyboard didn't get created for this project, can you help? | 16:55 |
| fungi | dpanech: yes, sorry we've got a config deployment hiccup there, i'll get it added momentarily. thanks for the reminder | 16:56 |
| Clark[m] | fungi: anything in the logs for hyperkitty? Maybe we still have permissions issues | 16:56 |
| fungi | Clark[m]: i haven't checked that far yet, just managed to perform rudimentary testing so far | 16:57 |
| Clark[m] | fungi: it just occurred to me that you have have tested on the older of two holds. The newest one was put in place at the end of my day yesterday and should've fixed hyperkitty. I'll dig that one out to confirm in a couple minutes | 17:10 |
| fungi | oh! | 17:10 |
| fungi | that would 'splain it if so | 17:11 |
| fungi | i pulled the ipa out of scrollback but may have missed a newer one mentioned | 17:11 |
| fungi | i should be able to repeat the same exercises on another node pretty easily | 17:11 |
| fungi | dpanech: the redeploy to update the sb projects is queued up behind some other builds but should hopefully run within the hour | 17:13 |
| dpanech | ok thank you | 17:13 |
| fungi | i'll give you a heads up once it's there | 17:14 |
| clarkb | fungi: 166.78.116.22 produced these screenshots https://1c52e9e53db6a2085483-4be2f0dc7165e9d197e0c3b76f5ede77.ssl.cf1.rackcdn.com/851248/59/check/system-config-run-lists3/63fc8c9/bridge.openstack.org/screenshots/mm3-opendev-archives.png so I suspect hyperkitty may actually function there | 17:22 |
| clarkb | that was the result of precreating the xapian index dir so that it can be chowned by the container entry point | 17:22 |
| fungi | got it. i'll retest on that node in a bit. thanks! | 17:26 |
| clarkb | wow pypi's respones to that line break issue is "we won't change it because we don't change anything for users" its frustrating because adding linebreaks doesn't violate any specs or protocols its purely to make things better for users | 17:35 |
| fungi | yeah | 17:35 |
| clarkb | on top of that we're doing this to make their lives easier | 17:36 |
| clarkb | maybe we should stop proxying entirely since their position seems tobe that proxying isn't a valid use case | 17:37 |
| fungi | agreed, we already "solved" it for our proxy (for the most part) | 17:37 |
| clarkb | until they add a package with more than 10MB of data then we'll hvae to bump our limits again. | 17:37 |
| fungi | yep | 17:37 |
| clarkb | but ya I think its mostly annoyin gthat they won't consider a different still completely valid json representation to make it easier on users | 17:37 |
| fungi | at least we mostly know about it now | 17:38 |
| clarkb | becaues the current version is already spec compliant. Well they both are... | 17:38 |
| fungi | i think newlines (or any "unnecessary" whitespace) are considered a waste of bandwidth these days | 17:39 |
| fungi | even though compression likely addresses that | 17:39 |
| clarkb | its also inconsequential in the face of 10MB of sha text | 17:39 |
| clarkb | I'm going to write a response indicating the reason we proxy in the first place is to reduce load on pypi. We'r etrying to be good community citizens but they are making it difficult and that adding newlines is just as valid as not according to json | 17:40 |
| clarkb | response posted. I tried to be constructive | 17:48 |
| fungi | dpanech: https://storyboard.openstack.org/#!/project/starlingx/jenkins-pipelines is showing up for me now | 19:22 |
| fungi | looks like it already has a story from you though, so i guess you spotted it before i checked again | 19:23 |
| clarkb | fungi: assuming your testing on the newer node shows hyperkitty is working I think the last thing is sortoing out if we want different configs for our lists and how to configure the "special" ones that are private or broadcast only? | 19:51 |
| dpanech | fungi: its all good, thanks | 19:52 |
| clarkb | infra-root the last patchset to the mm3 chnage added a restriction for django admin access to localhost only. I just tested it and it seems to work. Do we think that is desireable? I don't expect we'll ever really need it as we'll interact with mailman instead so figured walling it off was a good idea | 20:00 |
| fungi | clarkb: yes, the django localhost restriction sounds great to me | 21:21 |
| clarkb | great, doing an ssh port forward and hitting it from my browser worked fine and accessing it directly failed. I think we're set on that then | 21:22 |
| fungi | as for the list configs, i think what we're going to actually want to test is importing mm2 configuration for the existing ones of various sorts and confirming they still work the way we intended (private archives stay private, hidden lists stay hidden, completely moderated lists are still completely moderated, closed subscription lists still don't allow random people to subscribe, and so on) | 21:23 |
| clarkb | I see, then we can build out configs for new lists using ansible or just have their owners configure them before they see use | 21:23 |
| fungi | i think the importer is supposed to be pretty thorough at this point, but if we find corner cases we'll want to address those during migration | 21:23 |
| fungi | and yeah, adding new lists is probably fine to do the way we always have (let the list owner configure it before they announce it's ready to use) | 21:24 |
| fungi | existing lists we'll want to import the configs for | 21:24 |
| clarkb | the other thing I need to look at is refining the backup exclusions. I suspect we can avoid backing up the xapian content and just make mailman regenerate that if necessary | 21:25 |
| fungi | yeah probably. we'll want to backup the archives/configs/database mainly | 21:26 |
| clarkb | ya right now just about everything shoul dbe backed up in the existing change. More about not backing up stuff that will caus eproblems if backed up frequently like xapian indexes | 21:27 |
| fungi | clarkb: testing on 166.78.116.22 and hyperkitty still shows 0 posts in the openstack-discuss archive when there should be 2 | 21:42 |
| clarkb | fungi: which list? | 21:43 |
| fungi | openstack-discuss | 21:43 |
| fungi | it definitely tried to deliver both posts to the list's test subscriber though, i can see those sitting in exim's queue outbound | 21:43 |
| clarkb | I agree nothing in the archive svia the web there | 21:43 |
| fungi | 1oMcK0-000ADY-Iw and 1oMcOW-000AF9-7l got forwarded to the subscriber | 21:44 |
| fungi | so it's probably something on the hyperkitty side of things, i'm guessing | 21:44 |
| clarkb | ERROR 2022-08-12 21:40:40,204 439 django.security.DisallowedHost Invalid HTTP_HOST header: '127.0.0.1:8000'. You may need to add '127.0.0.1' to ALLOWED_HOSTS. | 21:45 |
| clarkb | it may be internal api requests between the mailman servers tripping over ^? | 21:45 |
| fungi | interesting | 21:45 |
| fungi | and yeah, i just double-checked that list is configured for public archives using hyperkitty, according to postorius, so inter-service communication problems would make sense | 21:46 |
| clarkb | I'm going to hack that into allowed hosts and restart the service | 21:46 |
| clarkb | in fact: "localhost", # Archiving API from Mailman, keep it. exists in the file already so ya I bet adding 127.0.0.1 will help | 21:47 |
| clarkb | fungi: that didn't cause it to reindex which is maybe expected. Do yo uthink you can try sending another email to see if that triggers it to happen? | 21:49 |
| fungi | sure | 21:53 |
| fungi | i can in a sec | 21:53 |
| fungi | clarkb: bingo, that was it | 21:58 |
| fungi | three messages in the archive now | 21:59 |
| clarkb | it logged that it hadnled all three too | 21:59 |
| clarkb | ok let me make an update | 21:59 |
| clarkb | ok I know why this happened I used 127.0.0.1 instead of localhost in the docker compose env var urls because when you use localhost for mariadb it assumes a socket file not tcp. I switched it to 127.0.0.1 then switched everything for consistency | 22:00 |
| fungi | ooh | 22:00 |
| clarkb | I think that consistency is still a good thing so I'll modify the config to allow localhost and 127.0.0.1 both | 22:00 |
| clarkb | fungi: any concern with clearing out the two held nodes and holding a new one on the updated ps? | 22:01 |
| fungi | none for me. delete them all as far as i'm concerned, there's nothing important on them | 22:01 |
| clarkb | I wasn't sure if you wanted to do more testing of things | 22:03 |
| clarkb | bah there is a merge conflict /me fixes | 22:05 |
| opendevreview | Clark Boylan proposed opendev/system-config master: WIP Add a mailman3 list server https://review.opendev.org/c/opendev/system-config/+/851248 | 22:07 |
| fungi | i can test just about as easily on the next one | 22:07 |
| fungi | though i'm also about tested out for today | 22:08 |
| clarkb | holds have been rotated out ^ will grab another server we can poke at later | 22:08 |
| clarkb | and ya its just about weekend time for you | 22:09 |
| fungi | well, that but also i'm juggling a bunch of other stuff | 22:09 |
| clarkb | I think the job takes about 25 minutes to complete if you wanted to reapply your messages to that. But I thnk we tested that well enough for now and can pick this back up again later and probably look at migrated a list next | 22:10 |
| clarkb | *at migrating a list | 22:10 |
| fungi | yeah, that was really just making sure that default non-subscriber moderation works, subscriber non-moderation works, messages are being delivered to a subscriber, and archives are being created | 22:10 |
| fungi | and also basic list owner tasks (bulk subscribing, handling moderation approvals, looking at the list settings) | 22:11 |
| clarkb | I think if we didn't have preexisting lists to deal with this is about when we'd consider deploying to production :) | 22:12 |
| clarkb | it is getting close | 22:12 |
| fungi | yep | 22:14 |
| fungi | mainly we need to do import/migration testing and validation as a next step, i think | 22:14 |
| clarkb | ++ | 22:14 |
| fungi | and decide on a rollout plan (new server vs trying to somehow do in-place, domain-by-domain or all-at-once, et cetera) | 22:15 |
| fungi | i'm leaning toward a new server and migrating one domain at a time, starting with lists.opendev.org and letting it mellow for a bit before trying more | 22:15 |
| clarkb | definitely a new server due to the boot issues the old one has. Then probably an early domain (lists.opendev.org) make sure its happy then do the rest | 22:16 |
| clarkb | ++ | 22:16 |
| fungi | sounds like we agree | 22:16 |
| clarkb | we also need to figure out how much disk space xapian needs | 22:16 |
| clarkb | testing migration should give us a good idea though | 22:16 |
| fungi | yep, we'll want to remember to check that | 22:17 |
| clarkb | fungi: for when you get time, maybe you can automate your test email stuff and add it to the testing of the server? | 22:17 |
| clarkb | probably in system-config/playbooks/zuul/test-lists3.yaml | 22:17 |
| clarkb | I don't think that needs to be rushed since you already check the behavior by hand and we're unlikely to do much that impacts it at this point | 22:17 |
| fungi | oh, good idea. the tricky bits will probably be working out how to subscribe someone via the api, how to approve moderated messages through the api, and how to dump a raw smtp session into a socket | 22:18 |
| clarkb | I think the first thing is easy. Not sure about the second. And the third shouldn't be too bad with nc and file redirection | 22:18 |
| clarkb | for the first thing I think you do the same thing we do to add list owners but you set the role value to subscriber instead of owner | 22:19 |
| fungi | sounds reasonable, based on how i see the account relationships modeled in the webui at least | 22:23 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!
