| *** ysandeep|out is now known as ysandeep | 05:42 | |
| *** jpena|off is now known as jpena | 07:37 | |
| *** lbragstad2 is now known as lbragstad | 08:01 | |
| opendevreview | James Page proposed openstack/project-config master: charms: add new ceph subteam https://review.opendev.org/c/openstack/project-config/+/859219 | 08:17 |
|---|---|---|
| *** ysandeep is now known as ysandeep|afk | 08:25 | |
| *** marios is now known as marios|call | 09:01 | |
| *** soniya29 is now known as soniya29|afk | 09:30 | |
| *** ysandeep|afk is now known as ysandeep | 09:50 | |
| opendevreview | James Page proposed openstack/project-config master: charms: add new ceph subteam https://review.opendev.org/c/openstack/project-config/+/859219 | 10:00 |
| opendevreview | Merged openstack/project-config master: Update link to zuul gating docs https://review.opendev.org/c/openstack/project-config/+/859150 | 10:38 |
| *** ysandeep is now known as ysandeep|afk | 11:20 | |
| *** soniya29|afk is now known as soniya29 | 11:20 | |
| *** ysandeep|afk is now known as ysandeep | 11:38 | |
| *** marios|call is now known as marios | 12:01 | |
| *** ysandeep is now known as ysandeep|afk | 12:40 | |
| *** ysandeep|afk is now known as ysandeep | 13:08 | |
| *** dasm|off is now known as dasm | 13:28 | |
| *** ysandeep is now known as ysandeep|out | 14:10 | |
| *** dviroel is now known as dviroel|lunch | 15:10 | |
| *** marios is now known as marios|out | 15:29 | |
| clarkb | looks like sphinx made a few releases over the weekend . Ithink at least one of the aerlier ones broke nodepool doc building and appears better now | 15:31 |
| clarkb | fyi if you see any sphinx 5.2.0 errors | 15:31 |
| fungi | thanks! and yeah, i suspected that was the cause for those | 15:32 |
| fungi | clarkb: with "104.239.143.143 lists.openstack.org" in my /etc/hosts this url from my browser history is working (once i bypass the cert warning of course): https://lists.opendev.org/pipermail/service-discuss/2022-August/000353.html | 15:57 |
| clarkb | fungi: you had to copy the archive to the appropriate location too right? Any feedback on whether or not we can use a simpler dir structure for that? The need for parsing info out of the url sort of informed my decisions there but it seems clunky. However if it works... | 15:58 |
| fungi | clarkb: yeah, i'm updating the etherpad with the notes about that right now. i think the location is fine, though i did need to create the parent dir | 16:00 |
| fungi | which we probably should add to ansible | 16:01 |
| clarkb | ++ | 16:03 |
| fungi | i'll do a new revision with that shortly | 16:04 |
| mnasiadka | hello | 16:11 |
| mnasiadka | fungi: https://review.opendev.org/c/openstack/project-config/+/858554 - are we waiting here for something? | 16:11 |
| mnasiadka | ah, probably for https://review.opendev.org/c/openstack/diskimage-builder/+/858606 | 16:12 |
| clarkb | I can probably review those later today | 16:13 |
| fungi | well, also ianw may simply have not felt confident enough to approve it when he was about to step out the door on holiday | 16:13 |
| clarkb | ya I think if the testing looks good we can land the prod update and worst case we pause those builds. But I need to catch up on a few other things first | 16:13 |
| fungi | 858606 adds testing for it, which is necessary but can probably be approved in parallel. clarkb's additional eyes on those would be appreciated though, since we accidentally approved some broken configs in similar places in recent weeks | 16:14 |
| clarkb | first on my agenda is fixing nodepool functional testing log collection. I think my change late last week to gather libvirt logs exposed a bug in this | 16:15 |
| opendevreview | Merged zuul/zuul-jobs master: test-requirements: bump to Ansible 2.8 https://review.opendev.org/c/zuul/zuul-jobs/+/851334 | 16:16 |
| *** dviroel|lunch is now known as dviroel | 16:20 | |
| fungi | clarkb: actually, i take that back, my testing was flawed (i had initially tried overriding lists.openstack.org before realizing the initial change only adds a vhost for lists.opendev.org) | 16:26 |
| mnasiadka | clarkb, fungi: thanks, need this for Kolla RL9 arm images support, but not that urgent that we can't followup with arm support later ;) | 16:26 |
| fungi | currently getting a 403 forbidden from apache on the test server, but i can't figure out why apache isn't logging anything (not even successful requests) | 16:27 |
| clarkb | fungi: the initial change should have server aliases for all of the domains | 16:27 |
| clarkb | there is only one vhost that serves everything. | 16:27 |
| fungi | oh, indeed i see that now | 16:28 |
| fungi | but yeah, logging doesn't seem to be happening in either case | 16:28 |
| opendevreview | James E. Blair proposed zuul/zuul-jobs master: zuul_azure_storage_upload: rename https://review.opendev.org/c/zuul/zuul-jobs/+/852932 | 16:29 |
| clarkb | fungi: are you sure you aren't hitting the old server then? | 16:29 |
| fungi | tcpdump on the test server confirms i'm hitting it with my browser | 16:30 |
| fungi | maybe ${APACHE_LOG_DIR} isn't expanding the way we want it to | 16:30 |
| clarkb | fwiw I reproduce what you see | 16:30 |
| fungi | the current lists.o.o apache vhost configs do also use ${APACHE_LOG_DIR} | 16:31 |
| fungi | so it's probably not that | 16:31 |
| fungi | unless we're missing a module or config somewhere | 16:32 |
| clarkb | there are also logs in the log dir | 16:32 |
| fungi | yeah, the current ones are just zero-byte | 16:33 |
| clarkb | right, they also rotated just a few minutes ago | 16:33 |
| clarkb | but the older ones don't have any additional useful stuff either | 16:33 |
| fungi | d'oh! | 16:33 |
| fungi | disk filled up | 16:33 |
| * fungi sighs | 16:33 | |
| fungi | /dev/xvda1 37G 37G 0 100% / | 16:34 |
| fungi | /var/log/unbound.log is almost 17gb | 16:34 |
| fungi | i guess we set the verbosity very high on it for jobs | 16:35 |
| clarkb | fungi: :( also the archive files don't actually exist | 16:35 |
| clarkb | so apache is doing the correct thing | 16:35 |
| fungi | i'll truncate it and reboot the server | 16:35 |
| clarkb | we just don't see logs becuase the disk is full | 16:35 |
| fungi | right | 16:36 |
| opendevreview | Merged zuul/zuul-jobs master: zuul_azure_storage_upload: rename https://review.opendev.org/c/zuul/zuul-jobs/+/852932 | 16:36 |
| fungi | i overwrote /var/log/unbound.log with /dev/null and rebooted | 16:36 |
| fungi | i'll check the file paths for the archives in a sec, i probably just got the destination wrong | 16:37 |
| fungi | /dev/xvda1 37G 20G 16G 57% / | 16:37 |
| clarkb | fungi: they are symlinks | 16:37 |
| fungi | that looks better | 16:37 |
| clarkb | and the targets don't exist | 16:37 |
| fungi | oh! yep i see it | 16:38 |
| fungi | we need to rewrite the symlinks since they weren't created as relative | 16:38 |
| fungi | the way public vs private archives work in mm2 is that everything goes into the private archives, and then anything which is set to have a "public" archive gets a symlink to the private one | 16:39 |
| fungi | i'll need to noodle on the best way to redo the symlinks | 16:39 |
| clarkb | and will apache rules need fiddling to deal with symlinks to other dirs? | 16:40 |
| fungi | quite possibly | 16:40 |
| *** jpena is now known as jpena|off | 16:41 | |
| opendevreview | Merged zuul/zuul-jobs master: Fix CORS and endpoint in AWS log upload https://review.opendev.org/c/zuul/zuul-jobs/+/858726 | 16:51 |
| clarkb | re the review.opendev.org cert expiry warning that went out this morning. My browser sees a current cert. I suspect that we've got stale apache workers on the host | 16:56 |
| clarkb | yes there is one older one. I think we can manually restart apache to address this. | 16:57 |
| clarkb | now is probably not a good time for that, but I can do this later today | 16:57 |
| fungi | sounds like a plan, and yeah i saw the notification earlier and suspected that was the cause | 17:02 |
| clarkb | mnasiadka: one small thing on https://review.opendev.org/c/openstack/project-config/+/858554 we should go ahead and address before we build images since the turn around time on those is quite large. Otherwise I think we can land that once fixed | 17:15 |
| fungi | clarkb: okay, a rewrite of the symlinks was sufficient, now https://lists.opendev.org/pipermail/service-discuss/2022-August/000353.html returns content from the held server for me | 17:17 |
| clarkb | fungi: cool testing here too | 17:17 |
| clarkb | fungi: I get a 503 hitting the root of the server now, but agree that url works | 17:18 |
| clarkb | fungi: I think maybe containers weren't restarted when you rebooted the server? | 17:19 |
| clarkb | yes they are all exited | 17:19 |
| clarkb | I'm going to up them | 17:19 |
| *** dviroel is now known as dviroel|dr_appt | 17:20 | |
| clarkb | hrm its still broken | 17:22 |
| clarkb | nevermind now it works. I guess it may have needed time to spin up | 17:23 |
| clarkb | fungi: so ya that looks good to me | 17:23 |
| fungi | yeah, they take a few to come up | 17:23 |
| fungi | okay, so need to create /var/lib/mailman/web-data/mm2archives and need to make docker-compose start the containers at boot? | 17:23 |
| clarkb | well docker-compose should start them on boot automatically I thought | 17:24 |
| clarkb | it should only not start them on boot if you've deleted the containers. but maybe the restart policy comes into play here in a way I didn't know about | 17:24 |
| clarkb | ya apparently we need to add restart policies for this | 17:25 |
| fungi | okay, the additions to the migrate script example in the etherpad have been tested now and i've used that exact code to relocate and relink all the remaining sites/archives on the held server | 17:36 |
| fungi | clarkb: should i include that script in 851248 at this point? | 17:37 |
| clarkb | fungi: can't hurt | 17:39 |
| clarkb | will help us see the updates to the script more easily than in etherpad too (git diff is a better record of that) | 17:39 |
| fungi | yep | 17:40 |
| fungi | including | 17:40 |
| fungi | clarkb: in our other compose files we seem to just set restart: always (except the jitsi-meet ones which default to unless-stopped, and gerrit where we restart: no) | 17:44 |
| fungi | is always what we want for these, do you think? | 17:44 |
| opendevreview | Dmitriy Rabotyagov proposed opendev/system-config master: Add Ceph Quincy mirror https://review.opendev.org/c/opendev/system-config/+/859327 | 17:45 |
| noonedeadpunk | o/ | 17:45 |
| noonedeadpunk | I know that afs is quite limited in space, but decided to push patch first and decide on how feasible it is to land it later (always can abandon) | 17:46 |
| noonedeadpunk | For balancing used diskspace I'd suggest to drop ceph-nautilus, but it's currently referenced here: https://opendev.org/openstack/loci/src/branch/master/playbooks/vars.yaml#L66 Tough I'm quite unsure about how much sense bionic does make today... | 17:47 |
| noonedeadpunk | given that release has been set to U | 17:47 |
| fungi | https://grafana.opendev.org/d/9871b26303/afs says that the other ceph reprepro mirrors take less than 10gb each and in most cases closer to 1gb (though we give them a 50gb quota just in case) | 17:49 |
| fungi | afs01.dfw is around 85% of 4tb used, so adding this probably won't make that much of an impact | 17:50 |
| opendevreview | Dmitriy Rabotyagov proposed opendev/system-config master: Add Ceph Quincy mirror https://review.opendev.org/c/opendev/system-config/+/859327 | 17:50 |
| opendevreview | Jeremy Stanley proposed opendev/system-config master: Add a mailman3 list server https://review.opendev.org/c/opendev/system-config/+/851248 | 17:51 |
| fungi | noonedeadpunk: probably the bigger hurdle is someone finding time to create the volume for that. i can try to get to it later today if there's consensus | 17:52 |
| noonedeadpunk | fungi: well, I can't say there's some huge rush - it's nt blocking gates at least | 17:53 |
| fungi | k | 17:54 |
| noonedeadpunk | *not blocking | 17:54 |
| fungi | i voted +1 on it for now since it will break if it lands before the volume has been added for it | 17:54 |
| noonedeadpunk | ok, great, thanks! | 17:54 |
| opendevreview | James E. Blair proposed zuul/zuul-jobs master: Add prefix to azure log upload https://review.opendev.org/c/zuul/zuul-jobs/+/859331 | 17:55 |
| fungi | clarkb: running mysqldump on the held node (with the full production data set for all ml sites imported) identically to how we're doing backups in the cronjob installed on it seems to work without error for me. took 1m51.350s to complete | 18:10 |
| clarkb | fungi: excellent. I would've expected similar errors about packet sizes if we had gotten the config wrong | 18:11 |
| fungi | an outstanding work item is that i need to shorten nonmember_rejection_notice for marketing@lists.openinfra.dev and legal-discuss@lists.openstack.org, and also member_moderation_notice for openstack-stable-maint@lists.openstack.org | 18:13 |
| fungi | i've added it to the todo list at the bottom of the etherpad for now so i don't forget | 18:14 |
| clarkb | and then we should probably rerun the migration just to eb sure that doesn't expose anything new | 18:14 |
| clarkb | or do you think that is overkill? I guess worst case it confirms we've shortened sufficiently | 18:15 |
| fungi | nah, not overkill at all. we've fixed several things since the last import | 18:15 |
| opendevreview | Clark Boylan proposed zuul/zuul-jobs master: Clarify that stage-output doesn't actually write to zuul_output_dir https://review.opendev.org/c/zuul/zuul-jobs/+/859333 | 18:24 |
| opendevreview | Clark Boylan proposed zuul/zuul-jobs master: Clarify that stage-output doesn't actually write to zuul_output_dir https://review.opendev.org/c/zuul/zuul-jobs/+/859333 | 18:28 |
| opendevreview | Merged zuul/zuul-jobs master: Add prefix to azure log upload https://review.opendev.org/c/zuul/zuul-jobs/+/859331 | 18:33 |
| opendevreview | Merged openstack/diskimage-builder master: Add Rocky 9 ARM64 functional test https://review.opendev.org/c/openstack/diskimage-builder/+/858606 | 18:35 |
| clarkb | fungi: some linter doesn't like the construction of your for loop in the migrate script. However, I think it is faulty and we should ignore that | 19:13 |
| clarkb | the way you've written it is more readable :/ | 19:13 |
| clarkb | infra-root with ianw out this week and frickler and corvus not typically participating in the meetings I'm wondering if we should cancel tomorrow's meeting? Or maybe have a much condensed agenda and focus on jaeger and mm3 migration stuff? | 19:14 |
| fungi | i like readable | 19:47 |
| fungi | but i'll see what i can do | 19:47 |
| fungi | bashate is a bit naive as linters go | 19:47 |
| fungi | system-config-run-mirror-x86 also had a random disconnect on that buildset | 19:51 |
| fungi | 'E010 The "do" should be on same line as for' seems like a very short-sighted concern, as it's not solvable when your loop condition is longer than the line length limit | 19:54 |
| opendevreview | Jeremy Stanley proposed opendev/system-config master: Add a mailman3 list server https://review.opendev.org/c/opendev/system-config/+/851248 | 20:00 |
| opendevreview | Jeremy Stanley proposed opendev/system-config master: Ignore bashate E010 https://review.opendev.org/c/opendev/system-config/+/859344 | 20:00 |
| fungi | clarkb: 859311 needs to be rebased on the newer 859172 revision in order to exercise it, right? | 20:05 |
| clarkb | fungi: its the other way around | 20:06 |
| fungi | ohhh | 20:07 |
| clarkb | 859172 is based on 859311 | 20:07 |
| fungi | gah, i always read gerrit's "relation chain" upside down | 20:07 |
| fungi | 859172 doesn't seem to run a functional-openstack job though? what's it running that's based on that abstract job? | 20:08 |
| clarkb | I think it is because the image build job timed out | 20:09 |
| clarkb | there is a little note that two jobs were skipped | 20:09 |
| fungi | oh, okay that makes sense. i assumed it was being skipped because of file filters, but that makes more sense | 20:33 |
| clarkb | I'm thinking I'll do an abbreviated agenda for tomorrow's meeting due to having fewer people | 20:35 |
| fungi | wfm | 20:35 |
| fungi | thanks! | 20:36 |
| *** dasm is now known as dasm|off | 21:35 | |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!