| opendevreview | Ian Wienand proposed opendev/system-config master: [wip] setup trusted CA for host comms in testing https://review.opendev.org/c/opendev/system-config/+/848562 | 00:23 |
|---|---|---|
| opendevreview | Ian Wienand proposed opendev/system-config master: [wip] setup trusted CA for host comms in testing https://review.opendev.org/c/opendev/system-config/+/848562 | 01:03 |
| opendevreview | Merged opendev/system-config master: gerrit: trigger rebuild of images to promote 3.6 https://review.opendev.org/c/opendev/system-config/+/848676 | 01:33 |
| opendevreview | Ian Wienand proposed opendev/system-config master: [wip] setup trusted CA for host comms in testing https://review.opendev.org/c/opendev/system-config/+/848562 | 02:00 |
| ianw | we have a gerrit 3.6 tag now | 02:01 |
| opendevreview | wangxiyuan proposed openstack/diskimage-builder master: [WIP]Add openEuler 22.02 LTS support https://review.opendev.org/c/openstack/diskimage-builder/+/848680 | 02:08 |
| opendevreview | Ian Wienand proposed opendev/system-config master: [wip] setup trusted CA for host comms in testing https://review.opendev.org/c/opendev/system-config/+/848562 | 02:40 |
| opendevreview | Ian Wienand proposed opendev/system-config master: [wip] test https https://review.opendev.org/c/opendev/system-config/+/848685 | 03:31 |
| opendevreview | Ian Wienand proposed opendev/system-config master: [wip] test haproxy to gitea https://review.opendev.org/c/opendev/system-config/+/848687 | 03:37 |
| *** ysandeep|out is now known as ysandeep|ruck | 04:02 | |
| opendevreview | Ian Wienand proposed opendev/system-config master: [wip] setup trusted CA for host comms in testing https://review.opendev.org/c/opendev/system-config/+/848562 | 04:10 |
| opendevreview | Ian Wienand proposed opendev/system-config master: [wip] test https https://review.opendev.org/c/opendev/system-config/+/848685 | 04:10 |
| opendevreview | Ian Wienand proposed opendev/system-config master: [wip] test haproxy to gitea https://review.opendev.org/c/opendev/system-config/+/848687 | 04:10 |
| opendevreview | Ian Wienand proposed opendev/system-config master: [wip] setup trusted CA for host comms in testing https://review.opendev.org/c/opendev/system-config/+/848562 | 04:35 |
| opendevreview | Ian Wienand proposed opendev/system-config master: [wip] test https https://review.opendev.org/c/opendev/system-config/+/848685 | 04:35 |
| opendevreview | Ian Wienand proposed opendev/system-config master: [wip] test haproxy to gitea https://review.opendev.org/c/opendev/system-config/+/848687 | 04:35 |
| *** ysandeep|ruck is now known as ysandeep|ruck|afk | 04:38 | |
| opendevreview | Steve Baker proposed openstack/diskimage-builder master: Support LVM thin provisioning https://review.opendev.org/c/openstack/diskimage-builder/+/840144 | 04:56 |
| opendevreview | Steve Baker proposed openstack/diskimage-builder master: WIP Add thin provisioning support to growvols https://review.opendev.org/c/openstack/diskimage-builder/+/848688 | 04:56 |
| *** ysandeep|ruck|afk is now known as ysandeep|ruck | 05:08 | |
| opendevreview | Ian Wienand proposed opendev/system-config master: [wip] setup trusted CA for host comms in testing https://review.opendev.org/c/opendev/system-config/+/848562 | 05:36 |
| ianw | i'm kind of glad i pulled on this thread as i think that having a CA and SAN setup correctly will remove a handy bunch of testing hacks | 05:39 |
| *** ysandeep|ruck is now known as ysandeep|ruck|afk | 06:18 | |
| *** ysandeep|ruck|afk is now known as ysandeep|ruck | 07:11 | |
| *** kopecmartin_ is now known as kopecmartin | 07:39 | |
| opendevreview | wangxiyuan proposed opendev/system-config master: Update openEuler mirror repo https://review.opendev.org/c/opendev/system-config/+/848703 | 08:11 |
| *** ysandeep|ruck is now known as ysandeep|lunch | 08:15 | |
| opendevreview | wangxiyuan proposed openstack/diskimage-builder master: [WIP]Upgrade openEuler to 22.02 LTS https://review.opendev.org/c/openstack/diskimage-builder/+/848680 | 08:41 |
| *** ysandeep|lunch is now known as ysandeep|ruck | 10:16 | |
| *** rlandy|out is now known as rlandy | 10:24 | |
| *** arxcruz is now known as arxcruz|brb | 10:59 | |
| *** dviroel|biab is now known as dviroel | 11:29 | |
| *** arxcruz|brb is now known as arxcruz | 12:03 | |
| *** ysandeep|ruck is now known as ysandeep|brb | 12:52 | |
| *** ysandeep|brb is now known as ysandeep|ruck | 13:01 | |
| *** dasm|off is now known as dasm | 13:15 | |
| opendevreview | gnuoy proposed openstack/project-config master: Add ACL to allow collaboration with Trilio devs https://review.opendev.org/c/openstack/project-config/+/848746 | 14:07 |
| *** ysandeep|ruck is now known as ysandeep|out | 14:39 | |
| opendevreview | Merged opendev/system-config master: Redirect all Mailman sites from HTTP to HTTPS https://review.opendev.org/c/opendev/system-config/+/848319 | 14:50 |
| fungi | digging deeper, i don't think we actually need to run fix_url.py since it doesn't appear that any of the list configs override web_page_url anyway, but i'll do some more testing to confirm that observation once the deploy completes | 14:52 |
| *** dviroel is now known as dviroel|lunch | 15:07 | |
| fungi | okay, scratch that, it must get baked into something that config_list doesn't emit | 15:53 |
| clarkb | meaning we do need to run the script afterall? | 15:53 |
| fungi | the moderator interface for the openstack-discuss ml kept trying to submit the form to http and wouldn't accept my actions | 15:54 |
| fungi | so i ran fix_url on that list and it's working now | 15:54 |
| fungi | i'll do the same for the rest | 15:54 |
| clarkb | thanks! | 15:54 |
| fungi | done now for every list on all 7 list sites across both servers | 15:56 |
| fungi | #status log Moved all mailing list sites entirely to HTTPS | 15:57 |
| opendevstatus | fungi: finished logging | 15:57 |
| clarkb | woot! | 15:57 |
| fungi | i'll keep an eye out for any problems with the lists i moderate, and check some new posts to make sure headers have updated to reflect the updated urls | 15:58 |
| *** marios is now known as marios|out | 16:03 | |
| *** dviroel|lunch is now known as dviroel | 16:22 | |
| corvus | i'm going to start a rolling restart of zuul to pick up clarkb python image changes. we'll be running under py 3.10 | 16:36 |
| clarkb | In theory the nodepool systems have already automatically udpated to python3.10 | 16:38 |
| clarkb | And this is zuul catching up | 16:38 |
| corvus | that was a later merge -- did it happen before the last restart? | 16:38 |
| clarkb | corvus: the nodepool iamges auto update when the image updates now | 16:39 |
| clarkb | corvus: since we can pretty safely restart them whenever without interrupting jobs | 16:39 |
| corvus | oh i thought it was daily? | 16:39 |
| clarkb | corvus: the job runs hourly | 16:39 |
| clarkb | so ya I guess if it hasn't run the hourly yet then it may need to wait for a little bit, but not long | 16:40 |
| corvus | i think it was ~2 hours ago | 16:40 |
| clarkb | nl01's launcher was restarted about an hour ago | 16:40 |
| clarkb | and that image is 3 hours old | 16:40 |
| corvus | rolling restart of mergers/executors in process | 16:42 |
| clarkb | and an ubuntu xenial image is building on nb01 which was also restarted. | 16:42 |
| clarkb | 2022-07-05 16:41:31,101 ERROR nodepool.zk.ZooKeeper: Error loading json data from image build /nodepool/images/fedora-35/builds/0000007464 there is some unhappy json in the db but nodepool seemed to skip over it successfully when doing an image listing | 16:42 |
| clarkb | I'll try to look into that later today once I'm through meetings and other things | 16:42 |
| clarkb | The xenial build failed because the chroot'd install of os-testr hit 'distutils.errors.DistutilsError: Could not find suitable distribution for Requirement.parse('pbr>5.7.0')' I don't think that is related to our update of python | 16:47 |
| clarkb | since that is nested away from the new version of python | 16:47 |
| clarkb | dib itself seems to be running well and doing what I expect | 16:47 |
| fungi | current versions of pbr should also work with xenial's python anyway | 16:58 |
| clarkb | ya I think this is possibly the lack of sni ssl support talking to pypi | 16:59 |
| fungi | oh, yeah that makes sense | 17:00 |
| fungi | and pbr is just the first thing it tries to grab | 17:00 |
| fungi | via setup_requires | 17:01 |
| clarkb | ya | 17:01 |
| clarkb | corvus: looks like ze06 may have stopped if you want to start it earlier than the others | 18:31 |
| corvus | sure, why not? :) | 18:34 |
| corvus | 2022-07-05 18:36:28,495 INFO zuul.ansible_manager: python version = 3.10.4 (main, Apr 20 2022, 05:30:59) [GCC 10.2.1 20210110] | 18:36 |
| corvus | looks nominal so far | 18:38 |
| clarkb | excellent | 18:39 |
| corvus | https://zuul.opendev.org/t/openstack/build/0e53970785d6428d98a6d54ff615abb1 completed sucessfully on ze06 | 18:40 |
| corvus | i'll do the same for ze02 since i'm here | 18:42 |
| clarkb | fungi: I had to hard refresh the service-discuss archives page to get it to show me my agenda email. Not sure if that is related to https change (perhaps different cache settings?) | 18:51 |
| fungi | clarkb: thanks, when you visited the old url you got a blank page or an error or... | 19:02 |
| clarkb | fungi: I got the stale content without my email entry in the list of archived threads | 19:03 |
| clarkb | then I did a hard refresh instead of a normal refresh and it appeared. This was well after I sent the email. Not sure if that was preexisting behavior with caches or not | 19:03 |
| fungi | ahh, that's weird, it shouldn't have served that over http at all | 19:03 |
| clarkb | no it was all https | 19:03 |
| clarkb | https://lists.opendev.org/pipermail/service-discuss/2022-July/thread.html that url | 19:04 |
| opendevreview | Ian Wienand proposed opendev/system-config master: [wip] gitea-lb: test ssl connections during testing too https://review.opendev.org/c/opendev/system-config/+/848777 | 19:05 |
| fungi | yeah, if i visit that url with a fresh browser i get the current content over https. i don't have a stale browser to test with though | 19:06 |
| clarkb | ya this may require checking opportunistically when new emails arrive. My hunch is this was a thing before too but calling it out just in case caching for https works differently enough to make it happen | 19:08 |
| opendevreview | Ian Wienand proposed opendev/system-config master: letsencrypt: selfsigned testing certs - use common CA, setup SAN https://review.opendev.org/c/opendev/system-config/+/848562 | 19:28 |
| opendevreview | Ian Wienand proposed opendev/system-config master: [wip] gitea-lb: test ssl connections during testing too https://review.opendev.org/c/opendev/system-config/+/848777 | 19:28 |
| opendevreview | Ian Wienand proposed opendev/system-config master: [wip] paste : move testing host to paste99, remove https hacks https://review.opendev.org/c/opendev/system-config/+/848685 | 19:55 |
| ianw | ^ this is the overall gist; targeting the testing host as if it were production -- but also being careful to not actually confuse production with testing (e.g. using suffixes like "99") | 19:58 |
| fungi | i got 99 servers, but that ain't one | 20:03 |
| ianw | i also wonder if we could do a better job with the LE "i've renewed your cert" handlers now. they were in that big file because that was the only way i could convince ansible to find them, but i'm pretty sure that was a bug. it might be fixed now | 20:27 |
| ianw | s/were/are/ | 20:27 |
| corvus | ze01-06 all upgraded; 07-12 stopping now | 20:28 |
| ianw | interestingly i was trying to see that the LB is passing data during system-config testing. here's a sample of what it is doing -> https://paste.opendev.org/show/bBUMrJhSPOlHS1DRGhQv/ ... getting hit by various scanning bots | 20:51 |
| ianw | it's only up for 12 minutes or so | 20:51 |
| fungi | the internet is for self-propagating worms | 20:53 |
| opendevreview | Ian Wienand proposed opendev/system-config master: gitea-lb: test ssl connections during testing too https://review.opendev.org/c/opendev/system-config/+/848777 | 20:59 |
| opendevreview | Ian Wienand proposed opendev/system-config master: paste : move testing host to paste99, remove https hacks https://review.opendev.org/c/opendev/system-config/+/848685 | 20:59 |
| opendevreview | Ian Wienand proposed opendev/system-config master: [wip] test haproxy to gitea https://review.opendev.org/c/opendev/system-config/+/848687 | 20:59 |
| fungi | infra-root: heads up that rackspace has notified us of an upcoming block storage maintenance on august 25 impacting afs01.ord and backup01.ord | 21:46 |
| corvus | love it if they're on the same rust | 21:47 |
| fungi | i'll try to swap out those volumes tomorrow | 21:47 |
| *** dviroel is now known as dviroel|afk | 21:59 | |
| *** dasm is now known as dasm|off | 22:02 | |
| ianw | Server balance_git_http/gitea99.opendev.org is DOWN, reason: Layer4 connection problem, info: "No route to host", ... | 22:09 |
| ianw | anyone know if haproxy logs when this comes back? | 22:10 |
| ianw | ... oh, now i wonder ... is the /etc/hosts the same in the container as the host? i wonder if haproxy container can't resolve gitea99 | 22:11 |
| clarkb | I think it isn't mounted by default | 22:11 |
| clarkb | docker will use google dns iirc | 22:11 |
| ianw | https://docs.docker.com/config/containers/container-networking/#dns-services | 22:13 |
| ianw | "Custom hosts defined in /etc/hosts are not inherited. To pass additional hosts into your container, refer to add entries to container hosts file in the docker run reference documentation." | 22:13 |
| opendevreview | James E. Blair proposed opendev/system-config master: WIP: Build a nodepool image https://review.opendev.org/c/opendev/system-config/+/848792 | 22:14 |
| opendevreview | James E. Blair proposed opendev/system-config master: WIP: Build a nodepool image https://review.opendev.org/c/opendev/system-config/+/848792 | 22:16 |
| ianw | hrm we explicitly set | 22:34 |
| ianw | server gitea99.opendev.org 198.72.124.63:3080 check | 22:34 |
| clarkb | ianw: I think in production we also use the ip addresses to specify the backends | 22:35 |
| clarkb | that way we don't rely on dns | 22:35 |
| clarkb | corvus: looks like all executors and mergers are restarted on the new python3.10 zuul image | 22:36 |
| clarkb | just scheduler services remaining | 22:36 |
| ianw | yeah, i'm unconvinced that we pass data through the lb in the system-config tests | 22:36 |
| ianw | https://review.opendev.org/845316 splits the logs out, making it easier to see | 22:37 |
| ianw | https://zuul.opendev.org/t/openstack/build/f89e7a94e2674e36ad7208d78873cc0e/log/gitea-lb01.opendev.org/haproxy.log ... all comes back NOSRV | 22:38 |
| clarkb | I feel like this did work at one time because we used the tests to verify the tls checking? But that did have problems too? | 22:38 |
| ianw | istm that it is testing to see if it can see gitea99, but it can't. so it's half-testing the checking (checking that it's checking, but not checking that the check checked) | 22:43 |
| ianw | maybe? or the services aren't responding while things are loading or something | 22:44 |
| ianw | i think i'll have to hold a node to diagnose it. anyway, it would be good to get to a point where the haproxy log is split out and clearly shows traffic through it, that is the ultimate goal | 22:44 |
| corvus | restarting zuul01 | 22:45 |
| clarkb | corvus: any idea how long that restart takes now? iirc there were a few improvements made | 23:04 |
| corvus | it's done; i'll check | 23:06 |
| corvus | 2022-07-05 22:44:55,071 DEBUG zuul.Scheduler: Configured logging: 6.1.1.dev32 | 23:07 |
| corvus | 2022-07-05 22:47:46,203 INFO zuul.Scheduler: Config priming complete (duration: 164.639 seconds) | 23:07 |
| clarkb | wow, under 3 minutes? And it took about 20 before? amazing improvement | 23:07 |
| corvus | yeah, it's a wee bit better :) | 23:08 |
| corvus | restarting zuul02 now | 23:08 |
| corvus | #status log restarted all of zuul on 78b14ec3c196e7533ac2c72d95fba09c936e625a | 23:09 |
| opendevstatus | corvus: finished logging | 23:10 |
| ianw | ok, if i "curl https://gitea99.opendev.org:3081" from the load balancer it works, so the connection is alive | 23:11 |
| ianw | (under test) | 23:11 |
| ianw | hang on, it's looking for port 3080 ... | 23:16 |
| ianw | right ... after fixing the ports "curl --resolve opendev.org:127.0.0.1 https://opendev.org" works with the new bits (because gitea99.opendev.org has a SAN for opendev.org) | 23:23 |
| *** dviroel|afk is now known as dviroel | 23:27 | |
| fungi | that makes a bit more sense, yep | 23:29 |
| opendevreview | Ian Wienand proposed opendev/system-config master: gitea: fix loadbalancer forwarding in testing https://review.opendev.org/c/opendev/system-config/+/848793 | 23:34 |
| ianw | ^ i haven't put that ontop of any of the testing CA/ssl changes | 23:37 |
| opendevreview | Steve Baker proposed openstack/diskimage-builder master: Parse block device lvm lvs size attributes https://review.opendev.org/c/openstack/diskimage-builder/+/839829 | 23:41 |
| opendevreview | Steve Baker proposed openstack/diskimage-builder master: Do dmsetup remove device in rollback https://review.opendev.org/c/openstack/diskimage-builder/+/847860 | 23:41 |
| opendevreview | Steve Baker proposed openstack/diskimage-builder master: Support LVM thin provisioning https://review.opendev.org/c/openstack/diskimage-builder/+/840144 | 23:41 |
| opendevreview | Steve Baker proposed openstack/diskimage-builder master: Add thin provisioning support to growvols https://review.opendev.org/c/openstack/diskimage-builder/+/848688 | 23:41 |
| *** dviroel is now known as dviroel|out | 23:50 | |
| corvus | memory usage on zuul01/zuul02 looks good. i think it's worth keeping an eye on that over the next week or so because of the python version change. | 23:59 |
| clarkb | ++ | 23:59 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!