| apevec | There are some POST_FAILURE https://zuul.opendev.org/t/openstack/status#887503 for example, so no logs, can infra root see more what is failing? | 08:49 |
|---|---|---|
| apevec | https://zuul.opendev.org/t/openstack/build/f807d05737a144e48be47f3ebbf2bd26 is openstack-tox-py38 POST_FAILURE | 08:55 |
| *** dviroel_ is now known as dviroel | 11:47 | |
| fungi | apevec: thanks, tkajinam reported something similar in #openstack-infra | 12:50 |
| fungi | the lack of uploaded logs suggests we've got a misbehaving swift endpoint in one of our donor clouds. i'll see if i can track it down from local logs on the executors | 12:52 |
| fungi | 2023-07-05 05:44:03,177 DEBUG zuul.AnsibleJob.output: [e: b0ba181cf6ee4fa18459ebac491d5e21] [build: f807d05737a144e48be47f3ebbf2bd26] Ansible output: b'TASK [upload-logs-swift : Upload logs to swift] ********************************' | 12:59 |
| fungi | 2023-07-05 05:44:19,707 DEBUG zuul.AnsibleJob.output: [e: b0ba181cf6ee4fa18459ebac491d5e21] [build: f807d05737a144e48be47f3ebbf2bd26] Ansible output: b'fatal: [localhost]: FAILED! => {"censored": "the output has been hidden due to the fact that \'no_log: true\' was specified for this result", "changed": false}' | 12:59 |
| fungi | so it does appear to have failed uploading logs into swift, though not sure where or why yet, still digging | 13:00 |
| fungi | TASK [Select random swift provider for logs upload] chose {"_swift_provider_name": "ovh_gra"} | 13:03 |
| fungi | "Following the main incident with the Keystone API, some object storage products (PCS, Swift) are in degraded service." https://public-cloud.status-ovhcloud.com/incidents/kr48h5jhtk39 | 13:05 |
| fungi | https://public-cloud.status-ovhcloud.com/incidents/1shkj36zsphs seems to be the parent incident | 13:06 |
| fungi | looks like the swift impact probably started around 03:00 utc according to their info, might already be working again? | 13:09 |
| fungi | impact was listed as being to both the regions we use (bhs and gra) | 13:09 |
| fungi | https://zuul.opendev.org/t/openstack/build/1874efbb83db4c938323f8dade5cabe1 uploaded successfully to storage.gra.cloud.ovh.net a few minutes ago | 13:11 |
| fungi | apevec: tkajinam: i'm inclined to think this situation cleared up already, but if you see new examples which occur after, let's say 13:00 utc, let me know | 13:13 |
| apevec | ack, thanks fungi++ | 13:13 |
| opendevreview | Rodolfo Alonso proposed openstack/project-config master: Migrate "os-ken" project to Launchpad https://review.opendev.org/c/openstack/project-config/+/887726 | 16:39 |
| opendevreview | Merged openstack/project-config master: Migrate "os-ken" project to Launchpad https://review.opendev.org/c/openstack/project-config/+/887726 | 17:00 |
| *** dviroel_ is now known as dviroel | 19:48 | |
| opendevreview | Jeremy Stanley proposed opendev/system-config master: Update gitea to 1.19.4 https://review.opendev.org/c/opendev/system-config/+/887734 | 19:57 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!