| *** brinzhang has joined #opendev | 00:21 | |
| *** tosky has quit IRC | 00:23 | |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: gerrit: set ownership on ~gerrit2/.ssh directory https://review.opendev.org/c/opendev/system-config/+/764395 | 02:14 |
|---|---|---|
| openstackgerrit | Ian Wienand proposed opendev/system-config master: gerrit: fix db/ mount for gate testing https://review.opendev.org/c/opendev/system-config/+/764396 | 02:14 |
| *** raukadah is now known as chandankumar | 04:08 | |
| *** brinzhang has quit IRC | 05:22 | |
| *** brinzhang has joined #opendev | 05:22 | |
| *** brinzhang has quit IRC | 05:24 | |
| *** brinzhang has joined #opendev | 05:24 | |
| *** brinzhang has quit IRC | 05:47 | |
| *** brinzhang has joined #opendev | 05:54 | |
| *** ykarel|away has joined #opendev | 06:33 | |
| *** ykarel|away is now known as ykarel | 06:33 | |
| *** ysandeep|away is now known as ysandeep | 06:36 | |
| *** lpetrut has joined #opendev | 07:04 | |
| *** slaweq has joined #opendev | 07:27 | |
| *** marios has joined #opendev | 07:38 | |
| *** eolivare has joined #opendev | 07:40 | |
| *** ykarel has quit IRC | 07:45 | |
| *** hashar has joined #opendev | 07:46 | |
| *** sboyron has joined #opendev | 07:51 | |
| *** ykarel has joined #opendev | 08:06 | |
| *** andrewbonney has joined #opendev | 08:08 | |
| *** rpittau|afk is now known as rpittau | 08:17 | |
| openstackgerrit | Merged openstack/diskimage-builder master: Install git-core instead of git on RH systems https://review.opendev.org/c/openstack/diskimage-builder/+/764369 | 08:28 |
| *** DSpider has joined #opendev | 08:29 | |
| *** mgoddard has joined #opendev | 08:50 | |
| *** fressi has joined #opendev | 08:56 | |
| *** fressi has quit IRC | 09:35 | |
| *** mgoddard has quit IRC | 10:02 | |
| *** dtantsur|afk is now known as dtantsur | 10:04 | |
| *** mgoddard has joined #opendev | 10:05 | |
| dtantsur | seeing timeouts from https://tarballs.opendev.org/ is it known? | 10:07 |
| dtantsur | not constantly, sometimes I can get through eventually | 10:08 |
| *** nautik has quit IRC | 10:27 | |
| *** fressi has joined #opendev | 10:30 | |
| *** fressi has quit IRC | 10:36 | |
| *** dulek has left #opendev | 10:36 | |
| *** fressi has joined #opendev | 10:38 | |
| *** brinzhang has quit IRC | 10:41 | |
| *** fressi has quit IRC | 10:53 | |
| *** fressi has joined #opendev | 10:56 | |
| *** tosky has joined #opendev | 11:57 | |
| *** fbo is now known as fbo|lunch | 12:01 | |
| *** fbo|lunch is now known as fbo | 12:55 | |
| openstackgerrit | Sorin Sbârnea proposed openstack/project-config master: add review-priority for tripleo-ci https://review.opendev.org/c/openstack/project-config/+/715069 | 13:34 |
| openstackgerrit | Albin Vass proposed zuul/zuul-jobs master: ensure-nimble: ensure curl is present https://review.opendev.org/c/zuul/zuul-jobs/+/764465 | 13:51 |
| fungi | dtantsur: i'll check the server, might be having intermittent packet loss to the afs backend or something | 13:53 |
| fungi | the kernel hasn't complained about busy/inaccessible volumes | 13:54 |
| fungi | load average spiked for a bit around 06:00-07:30z according to cacti | 13:56 |
| fungi | on the frontend server | 13:56 |
| *** mgoddard has quit IRC | 13:56 | |
| dtantsur | fungi: http://paste.openstack.org/show/800503/ | 13:58 |
| fungi | backend infrastructure for afs also looks generally healthy and not busy | 13:59 |
| fungi | dtantsur: could be you're running into cache misses on the frontend, backend communication for afs takes a while especially the larger the files get | 14:00 |
| fungi | oh, maybe i'm misreading your paste | 14:01 |
| fungi | what is ipa-cafile? | 14:01 |
| dtantsur | tried again, got 27 seconds | 14:02 |
| dtantsur | that's an artefact of my shell, ignore it | 14:02 |
| * dtantsur should probably move the current git branch to the left like all normal people | 14:02 | |
| fungi | oh, i see | 14:02 |
| fungi | yeah it's certainly acting slow to respond | 14:02 |
| dtantsur | traceroute looks sane, so not sure if it's my problem | 14:03 |
| fungi | command-line operations on the server to browse the webroot return instantaneously, so i don't think it's communication between the frontend and afs | 14:04 |
| priteau | Hello. Is releases.openstack.org having issue? Requests take a long time to complete or time out | 14:08 |
| fungi | priteau: yes, i'm looking into it. something's making our static sites frontend really slow to respond | 14:09 |
| fungi | i wonder if it's getting crawled really aggressively again | 14:12 |
| *** Tengu has quit IRC | 14:14 | |
| priteau | Black Friday shoppers looking for good deals on openstack tarballs? | 14:15 |
| fungi | tar usually is black, so maybe | 14:19 |
| fungi | i do see rather a lot of ip addresses hitting the legacy git redirects served from git.openstack.org | 14:20 |
| fungi | uses the same server | 14:20 |
| openstackgerrit | Albin Vass proposed zuul/zuul-jobs master: ensure-nimble: ensure depdencies are present https://review.opendev.org/c/zuul/zuul-jobs/+/764465 | 14:21 |
| fungi | lots of traffic from the semrush and ahrefs crawlers | 14:22 |
| * dtantsur likely has a lot of git.o.o references in his .git/config | 14:22 | |
| fungi | bu also a lot of git clients hitting it from various networks, yeah | 14:23 |
| fungi | docs.o.o has a bunch of traffic too though | 14:25 |
| fungi | petalbot is crawling it right now | 14:25 |
| fungi | and a fair amount of crawler traffic from bing and yandex on docs.o.o right now | 14:27 |
| fungi | 20187 requests today from an ip address in china unicom, looks like a crawler with a ua trying to look like a browser | 14:30 |
| fungi | also possible it's a nat address in front of lots of clients, i suppose. but only two user agents each with roughly half of those requests. one claims to be ie 11 on windows and the other headless chrome 77 on linux | 14:36 |
| *** rpittau is now known as rpittau|afk | 14:51 | |
| *** mgoddard has joined #opendev | 14:51 | |
| *** lpetrut has quit IRC | 15:02 | |
| *** chandankumar is now known as raukadah | 15:08 | |
| *** ykarel is now known as ykarel|away | 15:17 | |
| zbr | fungi: https://review.opendev.org/c/openstack/project-config/+/715069 please. | 15:25 |
| fungi | #status log restarted apache2 on static.opendev.org in order to troubleshoot very long response times | 15:31 |
| openstackstatus | fungi: finished logging | 15:31 |
| fungi | that seems to have cleared it up for some reason | 15:32 |
| fungi | the only theory i have to go on at the moment is that it's related to this one error in the main apache error.log: | 15:33 |
| fungi | [Fri Nov 27 07:06:00.135611 2020] [core:notice] [pid 9189:tid 140234591202240] AH00052: child pid 12448 exit signal Bus error (7) | 15:33 |
| avass | that doesn't sound like a fun error to investigate | 15:34 |
| fungi | when i stopped the service just now it complained in error.log about a bunch of processes not exiting in a timely fashion | 15:34 |
| fungi | [Fri Nov 27 15:30:03.034928 2020] [core:warn] [pid 9189:tid 140234591202240] AH00045: child process 18411 still did not exit, sending a SIGTERM | 15:35 |
| fungi | and so on | 15:35 |
| fungi | and then... | 15:36 |
| fungi | [Fri Nov 27 15:30:09.043384 2020] [core:error] [pid 9189:tid 140234591202240] AH00046: child process 18411 still did not exit, sending a SIGKILL | 15:36 |
| fungi | so i'm going with something happened, maybe a little over 8 hours ago, which got at least some apache worker processes "stuck" somehow but not in such a way as to add any useful errors in the log | 15:37 |
| fungi | potentially related to something which killed one of them with a bus error | 15:37 |
| openstackgerrit | Merged openstack/project-config master: add review-priority for tripleo-ci https://review.opendev.org/c/openstack/project-config/+/715069 | 15:38 |
| *** eolivare has quit IRC | 15:48 | |
| openstackgerrit | Ghanshyam Mann proposed openstack/project-config master: Notify placement changes on nova channel https://review.opendev.org/c/openstack/project-config/+/764469 | 15:52 |
| *** mgoddard has quit IRC | 15:56 | |
| openstackgerrit | Sorin Sbârnea proposed opendev/gerritbot master: Update container to use Python 3.8 builder https://review.opendev.org/c/opendev/gerritbot/+/764104 | 16:14 |
| *** marios is now known as marios|out | 16:46 | |
| *** hashar has quit IRC | 16:47 | |
| openstackgerrit | Albin Vass proposed zuul/zuul-jobs master: DNM: trigger upload_logs_s3 https://review.opendev.org/c/zuul/zuul-jobs/+/764476 | 16:53 |
| openstackgerrit | Albin Vass proposed zuul/zuul-jobs master: DNM: trigger upload_logs_s3 https://review.opendev.org/c/zuul/zuul-jobs/+/764476 | 16:56 |
| openstackgerrit | Albin Vass proposed zuul/zuul-jobs master: upload-logs-s3: fix test-jobs files: attribute https://review.opendev.org/c/zuul/zuul-jobs/+/764476 | 16:57 |
| openstackgerrit | Albin Vass proposed zuul/zuul-jobs master: DNM: trigger upload_logs_s3 https://review.opendev.org/c/zuul/zuul-jobs/+/764478 | 16:59 |
| *** tosky has quit IRC | 17:04 | |
| *** marios|out has quit IRC | 17:14 | |
| openstackgerrit | Ghanshyam Mann proposed openstack/project-config master: Notify placement changes on nova channel https://review.opendev.org/c/openstack/project-config/+/764469 | 17:14 |
| openstackgerrit | Albin Vass proposed zuul/zuul-jobs master: GZIPCompressedStream default read length to -1 https://review.opendev.org/c/zuul/zuul-jobs/+/764478 | 17:19 |
| openstackgerrit | Albin Vass proposed zuul/zuul-jobs master: GZIPCompressedStream default read length to -1 https://review.opendev.org/c/zuul/zuul-jobs/+/764478 | 17:20 |
| openstackgerrit | Albin Vass proposed zuul/zuul-jobs master: Add configuration to make logs public https://review.opendev.org/c/zuul/zuul-jobs/+/764483 | 17:45 |
| *** ykarel|away has quit IRC | 17:57 | |
| *** dtantsur is now known as dtantsur|afk | 18:05 | |
| openstackgerrit | Albin Vass proposed zuul/zuul-jobs master: Add configuration to make logs public https://review.opendev.org/c/zuul/zuul-jobs/+/764483 | 18:20 |
| *** tosky has joined #opendev | 19:03 | |
| *** ralonsoh has quit IRC | 19:23 | |
| *** whoami-rajat__ has quit IRC | 19:31 | |
| *** mugsie has quit IRC | 19:59 | |
| *** mugsie has joined #opendev | 20:01 | |
| *** slaweq has quit IRC | 20:26 | |
| *** andrewbonney has quit IRC | 20:38 | |
| *** Dmitrii-Sh has quit IRC | 20:52 | |
| *** Dmitrii-Sh has joined #opendev | 21:04 | |
| *** DSpider has quit IRC | 21:10 | |
| *** cgoncalves has quit IRC | 21:21 | |
| *** fressi has quit IRC | 21:22 | |
| *** osmanlicilegi has quit IRC | 21:22 | |
| *** raukadah has quit IRC | 21:24 | |
| *** avass has quit IRC | 21:25 | |
| *** sboyron has quit IRC | 21:26 | |
| *** avass has joined #opendev | 21:27 | |
| *** corvus has quit IRC | 21:29 | |
| *** corvus has joined #opendev | 21:30 | |
| *** fressi has joined #opendev | 21:30 | |
| *** fressi has quit IRC | 22:20 | |
| *** Tengu has joined #opendev | 23:06 | |
| *** tosky has quit IRC | 23:42 | |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!