| *** akrpan-pure has quit IRC | 00:07 | |
| *** tosky has quit IRC | 00:16 | |
| *** DSpider has quit IRC | 01:53 | |
| *** sshnaidm|rover has quit IRC | 01:59 | |
| *** sshnaidm has joined #opendev | 02:00 | |
| openstackgerrit | Matthew Thode proposed opendev/glean master: add hosts entries and ssh keys only once https://review.opendev.org/c/opendev/glean/+/768785 | 02:15 |
|---|---|---|
| *** ysandeep|away is now known as ysandeep|ruck | 03:41 | |
| *** whoami-rajat__ has joined #opendev | 04:36 | |
| prometheanfire | man mock feels so arcane | 04:55 |
| *** ykarel has joined #opendev | 05:12 | |
| prometheanfire | is there a reason we don't use magicmock? | 05:46 |
| prometheanfire | iirc I submitted a patch a while ago that was rejected because of it | 05:46 |
| prometheanfire | mock's open does not use iter | 05:46 |
| openstackgerrit | Matthew Thode proposed opendev/glean master: add hosts entries and ssh keys only once https://review.opendev.org/c/opendev/glean/+/768785 | 06:01 |
| prometheanfire | well, I'll let someone tell me not to use it (and suggest something themselves) | 06:01 |
| *** ykarel_ has joined #opendev | 06:21 | |
| *** ykarel has quit IRC | 06:23 | |
| *** ykarel_ is now known as ykarel | 07:36 | |
| *** slaweq has joined #opendev | 08:05 | |
| *** slaweq has quit IRC | 08:40 | |
| *** ykarel is now known as ykarel|lunch | 09:08 | |
| *** DSpider has joined #opendev | 09:15 | |
| *** zbr has quit IRC | 10:02 | |
| *** zbr has joined #opendev | 10:02 | |
| *** ozzzo has quit IRC | 10:03 | |
| *** ssbarnea has joined #opendev | 10:06 | |
| *** zbr has quit IRC | 10:07 | |
| *** zbr has joined #opendev | 10:08 | |
| *** zbr has quit IRC | 10:08 | |
| *** zbr has joined #opendev | 10:10 | |
| *** brinzhang0 has quit IRC | 10:29 | |
| *** tosky has joined #opendev | 10:50 | |
| *** ykarel|lunch is now known as ykarel | 11:13 | |
| *** ssbarnea has quit IRC | 11:43 | |
| *** ysandeep|ruck is now known as ysandeep|away | 12:35 | |
| *** ozzzo has joined #opendev | 14:36 | |
| openstackgerrit | Matthew Thode proposed opendev/glean master: add hosts entries and ssh keys only once https://review.opendev.org/c/opendev/glean/+/768785 | 15:06 |
| *** whoami-rajat__ has quit IRC | 15:45 | |
| fungi | seeing an unusual level of post_failure results with no manifest uploaded | 16:35 |
| fungi | i'll see if i can get something from the executor logs | 16:35 |
| prometheanfire | thanks | 16:41 |
| * prometheanfire smashes the recheck button | 16:41 | |
| fungi | connection timeouts hitting https://auth.cloud.ovh.net/ | 16:42 |
| fungi | yeah, i can't reach that | 16:43 |
| fungi | connection timeouts from anywhere i've tried, so the problem doesn't seem to be localized | 16:46 |
| prometheanfire | here too | 16:48 |
| fungi | wow, my traceroute to it from the east coast of north america hops through sydney au | 16:50 |
| prometheanfire | heh, someone advertise something they shouldn't? | 16:52 |
| fungi | mebbe | 16:53 |
| fungi | and yeah, i can't connect to ovh with openstackclient | 16:53 |
| prometheanfire | ya, I went through au as well, from dfw to iad to lax to syd | 16:53 |
| prometheanfire | did end up at the target I think | 16:54 |
| prometheanfire | 16 ip247.ip-51-68-65.eu (51.68.65.247) 214.068 ms 214.480 ms 214.411 ms | 16:54 |
| fungi | more or less same here... dc to lax to syd | 16:54 |
| fungi | yup, the ip address in dns for it responds to icmp echo | 16:54 |
| prometheanfire | heh, neat | 16:54 |
| fungi | and to udp traceroute probes | 16:54 |
| fungi | just not on 443/tcp | 16:54 |
| fungi | i'm waffling on emergency removing all our ovh regions from log uploads temporarily to at least keep builds from failing | 16:55 |
| prometheanfire | asked in #ovh | 16:56 |
| prometheanfire | not sure if we have someone specific to reach out to | 16:56 |
| *** hamalq has joined #opendev | 17:00 | |
| fungi | earliest occurrence i find in our executor logs is 16:10 utc, so it's been down almost an hour at this point | 17:01 |
| fungi | i can only seem to find public status/maintenance details for ovh cloud usa. i'll see if logging into the dashboard gives me any more useful info | 17:02 |
| fungi | http://travaux.ovh.net/ | 17:04 |
| fungi | syd<>lax backbone incident in progress since 17:36 (i guess that's cest?) | 17:05 |
| fungi | our overall ci system utilization is very low at the moment, so i'm tempted to just leave it and see if they resolve the problem soon | 17:06 |
| fungi | the incident details indicate " We have currently a saturation on this link , we are cheking to offload some traffic" | 17:08 |
| prometheanfire | some stuff here | 17:10 |
| prometheanfire | https://twitter.com/OVH_Status | 17:10 |
| fungi | yep, okay, so same stuff | 17:11 |
| fungi | i guess i'll give them a little longer before i force in an emergency base-jobs change to stop uploading there | 17:11 |
| prometheanfire | ack | 17:12 |
| fungi | if this were a typical thursday i would have just done it | 17:14 |
| fungi | ooh, i just got a response out of the api | 17:21 |
| fungi | it's working fairly reliably now | 17:22 |
| fungi | i'll give it a few more minutes of stability before i get my hopes up too much | 17:22 |
| prometheanfire | wise | 17:24 |
| fungi | but yeah, traceroute's still going through their sydney peer | 17:24 |
| fungi | still seems to be working | 17:47 |
| fungi | 17:05:22 was the last post_failure for that | 17:51 |
| fungi | surveying all the executor logs | 17:51 |
| fungi | only 23 builds were impacted, i think | 17:54 |
| fungi | but i'll status notice it just in case | 17:54 |
| *** ykarel has quit IRC | 17:56 | |
| *** hamalq has quit IRC | 18:04 | |
| *** hamalq has joined #opendev | 18:16 | |
| *** auristor has quit IRC | 18:19 | |
| *** hamalq has quit IRC | 18:20 | |
| *** auristor has joined #opendev | 18:20 | |
| fungi | looks like they closed the incident out at 19:05 cest, so ~40 minutes ago | 18:43 |
| fungi | #status notice An OVH network outage caused Zuul to report POST_FAILURE results with no summary/logs for some builds completing between 16:10 and 17:06 UTC; these should be safe to recheck now | 18:43 |
| openstackstatus | fungi: sending notice | 18:43 |
| -openstackstatus- NOTICE: An OVH network outage caused Zuul to report POST_FAILURE results with no summary/logs for some builds completing between 16:10 and 17:06 UTC; these should be safe to recheck now | 18:43 | |
| openstackstatus | fungi: finished sending notice | 18:46 |
| *** auristor has quit IRC | 19:48 | |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!