| fazledyn | Hi everyone. Glad to be here. | 11:29 |
|---|---|---|
| fazledyn | I had a query regarding a commit made on https://github.com/openstack/ironic-python-agent. | 11:30 |
| fazledyn | Can anyone kindly point me whom should I contact? | 11:31 |
| jrosser | fazledyn: you can try in #openstack-ironic | 11:39 |
| fazledyn | thanks! | 11:42 |
| *** eandersson0 is now known as eandersson | 11:50 | |
| opendevreview | Merged opendev/irc-meetings master: Edit QA team meeting info https://review.opendev.org/c/opendev/irc-meetings/+/899457 | 12:26 |
| fazledyn | I'm curious. Where does the meeting take place? | 13:15 |
| *** dhill is now known as Guest5547 | 13:16 | |
| fazledyn | Are there any open source alternatives to Zoom ? | 13:16 |
| frickler | fazledyn: if you refer to the QA team meeting, it is taking place in IRC, in the #openstack-qa channel, like most team meetings do | 13:26 |
| frickler | fazledyn: regarding audio/video meetings we have our own server, meetpad.opendev.org, running jitsi with etherpad associated for collaborative editing of a document | 13:26 |
| fazledyn | thanks for the info! @frickler | 13:40 |
| kevko | Hi, any advice please ? https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_e7d/899901/4/check/kolla-ansible-debian-podman/e7d215c/primary/logs/ansible/pull unable to download image from mirror ...but it is normally working locally :/ | 14:41 |
| SvenKieske | yeah I also just checked it, works fine for me locally (using the opendev mirror), the culprit seems to be: | 14:45 |
| SvenKieske | docker pull mirror.iad3.inmotion.opendev.org:4447/openstack.kolla/fluentd:master-debian-bookworm | 14:46 |
| SvenKieske | or in that case podman pull I guess | 14:46 |
| fungi | i can look whether there's any issues with the proxy in inmotion | 14:46 |
| fungi | this is the same thing you pinged about in #openstack-kolla a few minutes ago? | 14:47 |
| SvenKieske | yeah, sorry :) I'll also try to replicate this locally for podman | 14:47 |
| SvenKieske | maybe it _is_ podman related | 14:47 |
| SvenKieske | thinking about it: kevko: do we setup any proxy env var or something to reach that mirror? | 14:48 |
| fungi | doesn't look like there are any local filesystem issues on mirror.iad3.inmotion at least (plenty of space, no errors in dmesg) | 14:48 |
| SvenKieske | mhm, having trouble setting up podman in a local docker container..I hate nested container stuff with different engines :D | 14:50 |
| fungi | fwiw, browsing to https://mirror.iad3.inmotion.opendev.org:4447/repository/openstack.kolla/fluentd does show master-debian-bookworm available, but as i said it's just proxying all that from dockerhub anyway | 14:51 |
| fungi | infra-root: i've un-wip'd https://review.opendev.org/899300 for the mailman upgrade, and am getting ready to approve it at the top of the hour (best case it'll still be after the announced 15:30 utc when the upgrade actually occurs) | 14:54 |
| SvenKieske | oh another mailman upgrade? | 14:54 |
| fungi | SvenKieske: https://lists.opendev.org/archives/list/service-announce@lists.opendev.org/message/MCCGA5276DKUHPGZTATXU7M3VJGFGSXF/ | 14:55 |
| SvenKieske | kevko: did you test locally with podman or docker? maybe it was just a random network error? | 14:55 |
| SvenKieske | fungi: ah thanks for the pointer, I really should subscribe to service-announce I guess | 14:56 |
| fungi | it's very low-traffic, one or two messages a month | 14:56 |
| fungi | i've approved 899300 now and will keep tabs on the eventual deployment jobs | 15:00 |
| fungi | i'll #status log once the upgrade is completed | 15:01 |
| clarkb | fungi: I'm here waiting for the gerrit community meeting to happen but maybe it will occur at 9am and respect the EU DST time change | 15:03 |
| clarkb | or maybe it won't happen at all. Either way I'm around | 15:04 |
| clarkb | SvenKieske: Gerrit will be upgraded November 17. Another announcement you would've received :) | 15:05 |
| SvenKieske | clarkb: thx, upgrades on a friday, you are certainly fearless ;) | 15:06 |
| clarkb | its well tested and we try to do gerrit upgrades during quiet times. I expect that time to be quiet in particular as a large US holiday week starts that weekend | 15:07 |
| fungi | zuul seems to think 899300 is going to merge around 15:47-ish | 15:25 |
| fungi | hopefully the deploy builds will beat the top of the hour when our periodic deploys start | 15:26 |
| fungi | depends on how long promote takes, i guess | 15:26 |
| clarkb | fungi: we do promote in deploy so should be good | 15:36 |
| fungi | oh, right | 15:37 |
| fungi | is all this traceback spam expected? https://zuul.opendev.org/t/openstack/build/8de9660e25c445b187768ab6a7e1f713/console#3/3/9/bridge99.opendev.org | 15:41 |
| fungi | ModuleNotFoundError: No module named 'tzdata' | 15:41 |
| fungi | seems to be maybe something with ara, i guess? | 15:42 |
| clarkb | yes its an ara thing | 15:43 |
| clarkb | it was there when I debugged the ruamel.yaml thing but didn't appear to be fatal so I focused on the problematic issue | 15:44 |
| opendevreview | Merged opendev/system-config master: Upgrade to latest Mailman 3 releases https://review.opendev.org/c/opendev/system-config/+/899300 | 15:51 |
| fungi | and it's in deploy now | 15:51 |
| clarkb | hasn't touched the running containers yet | 15:53 |
| fungi | it's pulling images now | 15:53 |
| fungi | and now it's restarting the containers onto the new images | 15:55 |
| clarkb | I'm still getting 503s from apache but docker does report the containers are up | 15:57 |
| clarkb | I seem to recall running into this last time as startup is not quick? | 15:57 |
| fungi | yeah, it takes some time | 15:58 |
| fungi | there it goes | 16:01 |
| fungi | https://lists.opendev.org/mailman3/lists/ reports "Postorius Version 1.3.10" at the bottom now | 16:01 |
| fungi | https://lists.opendev.org/archives/ reports "Powered by HyperKitty version 1.3.8." at the bottom now | 16:01 |
| fungi | those match the updated state in https://opendev.org/opendev/system-config/src/branch/master/docker/mailman/web/requirements.txt | 16:02 |
| clarkb | fungi: some minor things in the web logs: https://paste.opendev.org/show/bc7jfeZCt97fZm0dCPKw/ | 16:02 |
| clarkb | fungi: also do we write logs to logfiles or is it all through the docker logging system? I don't remember and can't find any specific log fiels after a quick look | 16:04 |
| fungi | clarkb: in /var/lib/mailman/core/var/logs/ | 16:05 |
| clarkb | aha | 16:05 |
| clarkb | also looks like /var/lib/mailman/web-data/logs for the web server | 16:06 |
| fungi | also /var/lib/mailman/web-data/logs/ | 16:06 |
| fungi | yeah | 16:06 |
| fungi | i notice some old elements of the webui are cached, force refreshing pages gets everything loaded though | 16:07 |
| fungi | status log Completed upgrade of mailing list sites to Mailman 3.3.9 as announced in https://lists.opendev.org/archives/list/service-announce@lists.opendev.org/message/MCCGA5276DKUHPGZTATXU7M3VJGFGSXF/ | 16:08 |
| fungi | that look good? | 16:09 |
| clarkb | ++ | 16:09 |
| fungi | #status log Completed upgrade of mailing list sites to Mailman 3.3.9 as announced in https://lists.opendev.org/archives/list/service-announce@lists.opendev.org/message/MCCGA5276DKUHPGZTATXU7M3VJGFGSXF/ | 16:09 |
| opendevstatus | fungi: finished logging | 16:09 |
| fungi | SvenKieske: https://fosstodon.org/@opendevinfra/ is probably the other stream worth watching, in addition to the announce ml, if you aren't already | 16:10 |
| fungi | pcheli: i checked back in on your ci system's gerrit account, and it has 11 ssh connections open at the moment. looks like they're accumulating slowly, so if you can't figure out the cause you're probably going to end up blocked again at some point weeks down the road | 16:12 |
| fungi | clarkb: any feel for when we should merge the etherpad changes? would you want to do the config refresh and the version upgrade on separate days? | 16:14 |
| pcheli | fungi: thanks for this. We'll check our infra again. Still I see only 1 connection from ci server. | 16:15 |
| JayF | This would strongly imply a badly behaving intermediate firewall or proxy pcheli | 16:16 |
| clarkb | fungi: I don't think they need to be separate days. Just separated by enough time to have a good idea if one created issues vs another | 16:16 |
| clarkb | fungi: maybe we go ahead with the config update now. Restart on that and an hour later deploy the upgrade? | 16:17 |
| clarkb | I'm in the gerrit meeting now though so a bit distracted | 16:17 |
| SvenKieske | fungi: ah will use my private mastodon account for that I guess | 16:19 |
| fungi | clarkb: no problem. i can go ahead and approve the config change and keep an eye on it. probably won't deploy until the call ends anyway | 16:19 |
| SvenKieske | couldn't those two information channels be merged? | 16:20 |
| clarkb | fungi: ya and I can't remember if we auto restart etherpad. We may not so may require us to manually restart anyway | 16:20 |
| fungi | JayF: agreed, that was my initial suspicion as well. i've seen it a lot with firewalls that assume idle ssh connections can just be silently dropped from the state table without doing a courtesy rst or fin to close them. sometimes turning on ssh keepalives helps | 16:21 |
| fungi | SvenKieske: maybe, though we use the announcements list as a very low-volume way of broadcasting what's coming up, while we use the statusbot to do more immediate notifications about things that are occurring or have occurred | 16:22 |
| fungi | statusbot has multiple urgency levels (log, notice, alert) we use to decide how widely to broadcast things (log only goes to mastodon/wiki page, notice also notifies irc channels/matrix rooms, alert temporarily changes irc channel topics) | 16:24 |
| fungi | and now that i think about it, i don't believe we got matrix support added to statusbot yet, so just irc not matrix at the moment | 16:24 |
| kevko | SvenKieske sorry, I fell in sleep 💤 😂 | 16:36 |
| clarkb | I wouldn't want to emit statusbot notices to service-announce but we could possibly go in the other direction | 16:36 |
| clarkb | but also I'm unlikely to implement that as I am not a mastadon user (and never used twitter eitehr) | 16:37 |
| opendevreview | Merged opendev/system-config master: Update Etherpad settings from upstream https://review.opendev.org/c/opendev/system-config/+/899894 | 16:51 |
| fungi | deploy job is pulling new etherpad image now | 16:53 |
| fungi | job claims it downed/upped the container | 16:54 |
| fungi | yeah, nodejs process has a start time of about a minute ago | 16:55 |
| clarkb | https://etherpad.opendev.org/p/gerrit-upgrade-3.8 appears to still be there for me | 16:55 |
| clarkb | nothing looks obviously different | 16:56 |
| fungi | somehow https://etherpad.opendev.org/p/oct2023-ptg-os-security got reverted to a new pad | 16:56 |
| frickler | showing proper content for me | 16:57 |
| fungi | oh! | 16:57 |
| * fungi is an idiot | 16:57 | |
| clarkb | I see content there too | 16:57 |
| clarkb | you still have your /etc/hosts override in place? | 16:57 |
| fungi | i did. now it's fine | 16:57 |
| fungi | panic subsiding | 16:57 |
| fungi | lgtm now that i'm connecting to the actual server | 16:58 |
| clarkb | cool. Honestly seeing that existing pads are still there (db connection is good) and web ui looks the same (explicit defaults didn't change somethnig somehow) make me happy with the config update | 16:59 |
| clarkb | happy to proceed to the version update whenever others are happy too | 16:59 |
| fungi | sure, i can approve that one next. there's still time for other infra-root to object with a -2 before it merges | 17:03 |
| fungi | but we did test the held node with the new version fairly extensively yesterday | 17:03 |
| clarkb | I also confirmed the config file updated on disk on the server | 17:04 |
| clarkb | and appears to have done so before the services were restarted based on timestamps so ya all good from my end | 17:04 |
| clarkb | unrelated to all this Gerrit 3.9 should release day after thanksgiving so November 24 | 17:05 |
| fungi | infra-root: i've approved https://review.opendev.org/896454 to upgrade etherpad.opendev.org from etherpad 1.9.2 to 1.9.4, please -2 to prevent it merging if you want more time to test it out | 17:05 |
| kevko | fungi: regarding https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_b70/899901/4/check/kolla-ansible-rocky9-podman/b705d84/primary/logs/ansible/pull where can I check how mirror-int.dfw.rax.opendev.org:4447 is configured ? | 17:05 |
| clarkb | kevko: opendev/system-config/playbooks/roles/mirror/templates/mirror.vhost.j2 | 17:06 |
| kevko | thanks | 17:06 |
| clarkb | kevko: you can drop the -int from the fqdn to access it externally too | 17:06 |
| clarkb | we found that network performance was sigifnicantly better within the cloud over the internal networks so point ci nodes at the internal network interfaces but expose it publickly too | 17:06 |
| fungi | kevko: specifically, https://opendev.org/opendev/system-config/src/branch/master/playbooks/roles/mirror/templates/mirror.vhost.j2#L476-L529 | 17:07 |
| kevko | can i check apache log somewhere ? | 17:12 |
| fungi | kevko: no, but i can check them for you. what's the client ip address of the test node and the approximate time of the request? | 17:13 |
| fungi | looks like the "primary" node was probably connecting with its private 10.209.33.135 address | 17:16 |
| kevko | it is this zuul run https://zuul.opendev.org/t/openstack/build/b705d848c7bb4fba94db633270f8efc2 | 17:17 |
| kevko | this one ? https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_b70/899901/4/check/kolla-ansible-rocky9-podman/b705d84/zuul-info/host-info.primary.yaml | 17:17 |
| fungi | the only connections i see logged from that address were at 09:07:28 utc, so probably from another build | 17:19 |
| kevko | that's weird | 17:20 |
| fungi | i do see some connections to the 443 mirror port | 17:22 |
| fungi | looks like it was successfully downloading rockylinux packages from the mirror | 17:22 |
| kevko | i also download successfully what is failing in a log :D | 17:24 |
| kevko | why 443 and not 4447 | 17:25 |
| clarkb | <104.130.158.64> Failed to connect to the host via ssh <- isinteresting | 17:25 |
| clarkb | but maybe that is a quirk of ansible logging. | 17:25 |
| fungi | yeah, i checked the logs on a whim but there's never any connection from 104.130.158.64 | 17:26 |
| clarkb | kevko: 443 hosts things that allow us to change the root path. Docker in its infinite wisdom decided to not let that be possible so it gets a dedicated port | 17:26 |
| fungi | 10.209.33.135 does connect to 443 to fetch distro packages | 17:27 |
| kevko | you probably can't give an access for a while to check what is happening, can you ? | 17:28 |
| clarkb | kevko: looks like you are using some ansible module to do the pulls? | 17:28 |
| clarkb | kevko: I would look at the source for that module and determine if that error message can be triggered by other failures | 17:29 |
| clarkb | for example the one above | 17:29 |
| kevko | clarkb: yes, and it is not problem for docker pull ...it's problem for podman pull | 17:29 |
| clarkb | I think this is failing at an ansible level and never talking to the mirror | 17:29 |
| fungi | i've got about 400 lines of access logs from that build hitting the mirror's 443/tcp address. i can probably break it up into 4 pastes | 17:30 |
| clarkb | I don't know that that will help | 17:31 |
| clarkb | a random ansible module with who knows what implementation is returning an error and it never talks to the web server | 17:31 |
| clarkb | the logs we need are on the ansible side | 17:32 |
| clarkb | (this is why I'm so often in favor of running commands directly rather than using modules, they are so obtuse when they break) | 17:32 |
| fungi | actually, looks like it was fetching python packages through the proxied connection to pypi | 17:32 |
| fungi | https://paste.opendev.org/show/bmlOH9qd7TaF78cN9DAL/ https://paste.opendev.org/show/bXY7a7eZ2GecNPjD2vrA/ https://paste.opendev.org/show/byuLKNq5LSBeXnHtERdU/ https://paste.opendev.org/show/bMSHvGgEJQCnkoRQBgGq/ | 17:33 |
| clarkb | and github won't let me search their repos without signing in | 17:33 |
| kevko | clarkb: fungi: let me try it to pull with bash | 17:33 |
| fungi | so we have proof that the test node did connect to the mirror and fetch things through a caching proxy there, just not the caching proxy for quay | 17:33 |
| opendevreview | Merged opendev/system-config master: Upgrade Etherpad to 1.9.4 https://review.opendev.org/c/opendev/system-config/+/896454 | 17:34 |
| fungi | deploy job is pulling the etherpad 1.9.4 image now | 17:36 |
| clarkb | https://opendev.org/openstack/kolla-ansible/src/branch/master/ansible/module_utils/kolla_podman_worker.py#L452-L471 this is where the error comes from | 17:36 |
| fungi | nodejs process start time now 17:36 | 17:37 |
| fungi | i'm able to pull up ptg etherpads still | 17:37 |
| clarkb | I can pull up other etherpads I've been using like the gerrit 3.8 etherpad | 17:37 |
| fungi | #status log The Etherpad service on etherpad.opendev.org has been upgraded to the 1.9.4 release | 17:38 |
| opendevstatus | fungi: finished logging | 17:38 |
| fungi | kevko: looking for some of those container identifiers, i do see other test nodes fetching things with similar names through that proxy | 17:41 |
| clarkb | https://opendev.org/openstack/kolla-ansible/src/branch/master/ansible/module_utils/kolla_podman_worker.py#L260 seems to imply it only returns {} if it got a 404. But our server logs indicate it never even made a request so how can it be a 404? | 17:41 |
| fungi | 10.209.129.178 - - [2023-11-02 17:02:20.638] "HEAD /v2/openstack.kolla/neutron-l3-agent/manifests/master-rocky-9 HTTP/1.1" 200 - - "-" "docker/24.0.7 go/go1.20.10 git-commit/311b9ff kernel/5.14.0-284.30.1.el9_2.x86_64 os/linux arch/amd64 UpstreamClient(docker-sdk-python/6.1.3)" | 17:42 |
| clarkb | unless there is no exception and https://opendev.org/openstack/kolla-ansible/src/branch/master/ansible/module_utils/kolla_podman_worker.py#L257 returns {} | 17:42 |
| fungi | different test node of course | 17:42 |
| kevko | as i said ..this is not docker pull but podman pull | 17:42 |
| kevko | can you check agent = .*podman.* | 17:42 |
| kevko | instead of docker | 17:42 |
| fungi | no references to "podman" at all in the log | 17:43 |
| kevko | and that's weird :P | 17:43 |
| fungi | well, not if it's never actually connecting | 17:43 |
| fungi | i basically don't see any indication from the mirror server/proxy side of things that podman ever attempted to request anything through 4447/tcp | 17:44 |
| kevko | nah, ubuntu don't have mod_macro packaged ? :( | 17:44 |
| kevko | fungi: cool, thanks, it's also helfull information | 17:44 |
| fungi | however i do see evidence that the same build was successfully reaching the mirror/proxy to pull things from pypi, so connectivity itself seems to have been fine | 17:45 |
| kevko | fungi: we will see in few minutes | 17:45 |
| fungi | also looks like docker clients are pulling through that proxy successfully in other jobs, so the proxy seems okay | 17:46 |
| kevko | now job is running ..i've bypassed ansible pull with ssh primary "sudo podman pull same_address_as_from_log" | 17:46 |
| kevko | podman is specific somehow :P | 17:46 |
| kevko | running https://zuul.opendev.org/t/openstack/stream/56d7491a96f24b6ab320d8274bfb8aa5?logfile=console.log | 17:46 |
| clarkb | kevko: https://opendev.org/openstack/kolla-ansible/src/branch/master/ansible/module_utils/kolla_podman_worker.py#L457 returns a List[Image] type here https://github.com/containers/podman-py/blob/main/podman/domain/images_manager.py#L320-L323 List doesn't have .attrs | 17:47 |
| clarkb | I don't think that is the issue, but potentially is another issue? | 17:47 |
| clarkb | would occur if you request all tags | 17:48 |
| kevko | maybe ..thanks .i will check | 17:52 |
| clarkb | idea time: maybe it is requesting these resources over http and not https. TLS setup fails and we never get far enough to record the action in the vhost. There may be logs in the general server error log for this? | 17:53 |
| clarkb | kevko: fungi ^ | 17:53 |
| clarkb | if so we don't seem to record it in the logs I can see | 17:55 |
| fungi | yeah, that would never reach apache's logs | 17:55 |
| fungi | we could set up special iptables connection logging | 17:56 |
| fungi | but that's probably overkill | 17:56 |
| kevko | hmm, http vs https can be issue | 17:57 |
| clarkb | its also making the request to a podman socket which presumably has some running daemon like docker does which makes the actual request | 18:01 |
| clarkb | At least I think that is how this is setup. If so I would look for logs from whatever has opened that socket file | 18:01 |
| SvenKieske | clarkb: fungi: could you maybe assist in debugging a login issue in gerrit? mjk: is new to the openstack contributor party :) | 18:05 |
| SvenKieske | he's getting a 404 redirect error upon login, which I can't replicate, just logged in again and it works | 18:06 |
| SvenKieske | the gerrit upgrade has not begun earlier, has it? :D | 18:06 |
| mjk | hey team! let me know what info you need from me | 18:06 |
| SvenKieske | at least I hope someone in here can help from the gerrit side, the launchpad stuff is somewhere in canonicals hands | 18:07 |
| clarkb | kevko: I would try setting https://opendev.org/openstack/kolla-ansible/src/branch/master/ansible/module_utils/kolla_podman_worker.py#L240 to true. We have valid certs on our registries so that should rule out https vs http as the issue (or fix it) | 18:07 |
| SvenKieske | mjk: can you link me to your launchpad account? maybe something is off with that? | 18:07 |
| mjk | https://launchpad.net/~mjknox | 18:08 |
| SvenKieske | kevko: rereading the thread tomorrow I guess | 18:08 |
| clarkb | mjk: can you recount the steps you've taken and where it breaks? | 18:08 |
| mjk | yup, I hit sign in on review.opendev, it punts me to login.ubuntu, click yes login me in, get a 404 and this is the landing URL https://review.opendev.org/SignInFailure,SIGN_IN,Contact+site+administrator | 18:09 |
| fungi | that can indicate the account was previously retired, or is a duplicate of another existing account, i think | 18:10 |
| SvenKieske | weird, have you per chance any browser extensions that do interfere with redirects? I personally use ublock origin and firefox containers but have no issue with them | 18:10 |
| clarkb | thanks. I've found the error in the logs. You are logging in with a new launchpad account that has the same email address as an existing launchpad and gerrit account. Gerrit won't let multiple accounts have the same email address so it errors | 18:10 |
| clarkb | fungi: yup its exactly that | 18:10 |
| kevko | clarkb: fungi: do you know what ? :D i've commented the pull command with ssh primary "sudo podman pull something " ...and now it runs suspiciously long (which means that actually ONLY pull probably not working ..but your proxy is OK ) :D :D :D | 18:10 |
| SvenKieske | you guys are just awesome :) I would never have thought about that :D mjk: did you maybe register already in the past? | 18:11 |
| clarkb | mjk: the options here are for you to login using your old launchpad account and use the existing gerrit account. | 18:11 |
| clarkb | or we disable the old account and make it inactive and then let gerrit create the new account for you on login with the new launchpad credentials | 18:11 |
| fungi | i'm stepping away for a moment to pick up dinner, but shouldn't be gone long (probably back in 15 minutes tops) | 18:11 |
| mjk | hrm, I will have to figure out the old account | 18:12 |
| mjk | I will pipe up if I can't figure it out | 18:12 |
| clarkb | I guess technically there is a third otpion. Use a different email address and let it create a new account for you | 18:13 |
| mjk | will use that as plan B if needed :) | 18:13 |
| clarkb | give me a few and I'll page back in the admin steps to inspect the old account in gerrit | 18:16 |
| clarkb | I think I have to promote my user then use the rest api because ssh doesnt' cut it for this | 18:16 |
| clarkb | that has given me a little bit of extra info. First this email address was the only one ever used. It wasn't a secondary email. Next the account is from 2016. And finally the openid associated with the old account gives me a 404 when I fetch it indicating the openid isn't part of ubuntu one anymore. I don't know why this is the case | 18:21 |
| SvenKieske | okay guys, thanks for helping out :) I guess I'm out for today. o/ | 18:21 |
| clarkb | what we can do is remove the primary email address, and launchpad ids with that email address from the old account then disable the account entirely. Then you can login with the email address and get a new account. This assumes you cannot recover the old ubuntu/launchpad openid | 18:22 |
| clarkb | or the third options I mentioned use another email address | 18:22 |
| clarkb | fwiw openstack-discuss has had a couple of messages since we upgraded mm3 | 18:36 |
| clarkb | maybe even 4? | 18:37 |
| fungi | okay, back | 18:38 |
| fungi | and yeah, foundation@openinfra had a message too | 18:39 |
| *** travisholton2 is now known as travisholton | 19:36 | |
| opendevreview | Julia Kreger proposed opendev/glean master: Fix glean docs build https://review.opendev.org/c/opendev/glean/+/899985 | 20:25 |
| opendevreview | Julia Kreger proposed opendev/glean master: Add support for tinycore linux https://review.opendev.org/c/opendev/glean/+/899219 | 20:26 |
| opendevreview | Julia Kreger proposed opendev/glean master: Fix glean docs build https://review.opendev.org/c/opendev/glean/+/899985 | 20:29 |
| clarkb | still no failuers from the letsencrypt job. Maybe the issue we had is similar to the kolla one on the openstack list. Some bug hidden inside of ansible and upgrading got us beyond it | 21:15 |
| kevko | clarkb fungi hmm, it looks like podman pull really not working (second thing is that python module is hiding real error of course) , but podman pull failing https://paste.openstack.org/show/bXulicWuWXDOHhoYNhfs/ | 21:42 |
| clarkb | kevko: https://mirror.dfw.rax.opendev.org:4447/v2/ works from here | 21:43 |
| opendevreview | Julia Kreger proposed opendev/glean master: Add support for tinycore linux https://review.opendev.org/c/opendev/glean/+/899219 | 21:43 |
| clarkb | so the service is generally up. Could be a networking problem I suppose. Is podman not willing to route over the internal network maybe? | 21:43 |
| kevko | clarkb: hmm, i've commented tests and write to file podman config and also raw podman pull https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_20b/899901/8/check/kolla-ansible-debian-podman/20bf125/primary/logs/ansible/deploy | 21:44 |
| fungi | kevko: mirror-int won't be reachable externally if that's how you tested | 21:44 |
| fungi | but mirror (without the -int) should still be testable over the internet | 21:44 |
| kevko | fungi no, it's the same ENV I would say | 21:44 |
| fungi | Get \"https://mirror-int.dfw.rax.opendev.org:4447/v2/\": dial tcp 10.209.161.66:4447: i/o timeout" | 21:45 |
| clarkb | I am able to get that url from nb01 which is in the same cloud as the mirror | 21:46 |
| fungi | where were you testing it from? did that run in one of our ci nodes in rax-dfw or somewhere else? | 21:46 |
| kevko | fungi: i was wondering if it can be a proxy problem ? https://paste.openstack.org/show/bsAG2v2LsHO0TsDNiFIK/ | 21:46 |
| clarkb | its from this job run https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_20b/899901/8/check/kolla-ansible-debian-podman/20bf125/ | 21:46 |
| fungi | ah, okay | 21:46 |
| clarkb | kevko: it would be nice if you linked to the zuul pages. makes linking to log lines possible | 21:46 |
| clarkb | also gives a lot more information about the change in question | 21:47 |
| kevko | https://review.opendev.org/c/openstack/kolla-ansible/+/899901 | 21:47 |
| kevko | clarkb: fungi: but check patch1 for a real change ... latest patchset is just my debug for now :) | 21:47 |
| kevko | it's nothing with a change I would say | 21:47 |
| fungi | if it was a proxy problem, i wonder why pip installing things wasn't affected | 21:48 |
| fungi | anyway, if we have the client address for this one, i can check the apache mod_proxy logs on the mirror server there again | 21:48 |
| clarkb | oh wait | 21:49 |
| clarkb | you hardcoded the mirror address you cannot do that | 21:49 |
| kevko | I don't know, sometimes I have a felling that podman is little little different as docker for example | 21:49 |
| clarkb | that job ran in ovh so ya it cannot reach mirror-int.dfw.rax | 21:49 |
| clarkb | so your test using podman is invalid | 21:49 |
| fungi | those mirror-int addresses will only be available from job nodes inside the same cloud providers | 21:49 |
| fungi | so only nodes booted in rax-dfw will be able to connect to mirror-int.dfw.rax.opendev.org | 21:50 |
| fungi | each cloud provider has its own mirror hostnames/urls that job nodes in that provider use | 21:51 |
| kevko | fungi: well, i copied that address from python error log ... | 21:51 |
| fungi | right, but then when you ran the job again it ran in a different cloud provider/region | 21:51 |
| kevko | ah okay, understand | 21:51 |
| kevko | cool, let me fix me test then | 21:51 |
| clarkb | kevko: zuul_site_mirror_fqdn this ansible var sets the fqdn. So you should be able to do something like https://{{ zuul_site_mirror_fqdn }}:4447/ as the mirror location | 21:55 |
| kevko | clarkb: thank you | 21:56 |
| clarkb | kevko: did you try setting verifyTls to true though? | 21:58 |
| clarkb | verify tls is the default when running the podman command so it used https (which we saw in that last log file). But the way the ansible module is written it defaults to false there and it was false when it failed so maybe it tried to do http instead? | 21:59 |
| kevko | clarkb: no, didn't tried yet .. | 22:02 |
| opendevreview | Merged opendev/glean master: Fix glean docs build https://review.opendev.org/c/opendev/glean/+/899985 | 22:21 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!