Friday, 2020-11-27

*** brinzhang has joined #opendev00:21
*** tosky has quit IRC00:23
openstackgerritIan Wienand proposed opendev/system-config master: gerrit: set ownership on ~gerrit2/.ssh directory  https://review.opendev.org/c/opendev/system-config/+/76439502:14
openstackgerritIan Wienand proposed opendev/system-config master: gerrit: fix db/ mount for gate testing  https://review.opendev.org/c/opendev/system-config/+/76439602:14
*** raukadah is now known as chandankumar04:08
*** brinzhang has quit IRC05:22
*** brinzhang has joined #opendev05:22
*** brinzhang has quit IRC05:24
*** brinzhang has joined #opendev05:24
*** brinzhang has quit IRC05:47
*** brinzhang has joined #opendev05:54
*** ykarel|away has joined #opendev06:33
*** ykarel|away is now known as ykarel06:33
*** ysandeep|away is now known as ysandeep06:36
*** lpetrut has joined #opendev07:04
*** slaweq has joined #opendev07:27
*** marios has joined #opendev07:38
*** eolivare has joined #opendev07:40
*** ykarel has quit IRC07:45
*** hashar has joined #opendev07:46
*** sboyron has joined #opendev07:51
*** ykarel has joined #opendev08:06
*** andrewbonney has joined #opendev08:08
*** rpittau|afk is now known as rpittau08:17
openstackgerritMerged openstack/diskimage-builder master: Install git-core instead of git on RH systems  https://review.opendev.org/c/openstack/diskimage-builder/+/76436908:28
*** DSpider has joined #opendev08:29
*** mgoddard has joined #opendev08:50
*** fressi has joined #opendev08:56
*** fressi has quit IRC09:35
*** mgoddard has quit IRC10:02
*** dtantsur|afk is now known as dtantsur10:04
*** mgoddard has joined #opendev10:05
dtantsurseeing timeouts from https://tarballs.opendev.org/ is it known?10:07
dtantsurnot constantly, sometimes I can get through eventually10:08
*** nautik has quit IRC10:27
*** fressi has joined #opendev10:30
*** fressi has quit IRC10:36
*** dulek has left #opendev10:36
*** fressi has joined #opendev10:38
*** brinzhang has quit IRC10:41
*** fressi has quit IRC10:53
*** fressi has joined #opendev10:56
*** tosky has joined #opendev11:57
*** fbo is now known as fbo|lunch12:01
*** fbo|lunch is now known as fbo12:55
openstackgerritSorin Sb├órnea proposed openstack/project-config master: add review-priority for tripleo-ci  https://review.opendev.org/c/openstack/project-config/+/71506913:34
openstackgerritAlbin Vass proposed zuul/zuul-jobs master: ensure-nimble: ensure curl is present  https://review.opendev.org/c/zuul/zuul-jobs/+/76446513:51
fungidtantsur: i'll check the server, might be having intermittent packet loss to the afs backend or something13:53
fungithe kernel hasn't complained about busy/inaccessible volumes13:54
fungiload average spiked for a bit around 06:00-07:30z according to cacti13:56
fungion the frontend server13:56
*** mgoddard has quit IRC13:56
dtantsurfungi: http://paste.openstack.org/show/800503/13:58
fungibackend infrastructure for afs also looks generally healthy and not busy13:59
fungidtantsur: could be you're running into cache misses on the frontend, backend communication for afs takes a while especially the larger the files get14:00
fungioh, maybe i'm misreading your paste14:01
fungiwhat is ipa-cafile?14:01
dtantsurtried again, got 27 seconds14:02
dtantsurthat's an artefact of my shell, ignore it14:02
* dtantsur should probably move the current git branch to the left like all normal people14:02
fungioh, i see14:02
fungiyeah it's certainly acting slow to respond14:02
dtantsurtraceroute looks sane, so not sure if it's my problem14:03
fungicommand-line operations on the server to browse the webroot return instantaneously, so i don't think it's communication between the frontend and afs14:04
priteauHello. Is releases.openstack.org having issue? Requests take a long time to complete or time out14:08
fungipriteau: yes, i'm looking into it. something's making our static sites frontend really slow to respond14:09
fungii wonder if it's getting crawled really aggressively again14:12
*** Tengu has quit IRC14:14
priteauBlack Friday shoppers looking for good deals on openstack tarballs?14:15
fungitar usually is black, so maybe14:19
fungii do see rather a lot of ip addresses hitting the legacy git redirects served from git.openstack.org14:20
fungiuses the same server14:20
openstackgerritAlbin Vass proposed zuul/zuul-jobs master: ensure-nimble: ensure depdencies are present  https://review.opendev.org/c/zuul/zuul-jobs/+/76446514:21
fungilots of traffic from the semrush and ahrefs crawlers14:22
* dtantsur likely has a lot of git.o.o references in his .git/config14:22
fungibu also a lot of git clients hitting it from various networks, yeah14:23
fungidocs.o.o has a bunch of traffic too though14:25
fungipetalbot is crawling it right now14:25
fungiand a fair amount of crawler traffic from bing and yandex on docs.o.o right now14:27
fungi20187 requests today from an ip address in china unicom, looks like a crawler with a ua trying to look like a browser14:30
fungialso possible it's a nat address in front of lots of clients, i suppose. but only two user agents each with roughly half of those requests. one claims to be ie 11 on windows and the other headless chrome 77 on linux14:36
*** rpittau is now known as rpittau|afk14:51
*** mgoddard has joined #opendev14:51
*** lpetrut has quit IRC15:02
*** chandankumar is now known as raukadah15:08
*** ykarel is now known as ykarel|away15:17
zbrfungi: https://review.opendev.org/c/openstack/project-config/+/715069 please.15:25
fungi#status log restarted apache2 on static.opendev.org in order to troubleshoot very long response times15:31
openstackstatusfungi: finished logging15:31
fungithat seems to have cleared it up for some reason15:32
fungithe only theory i have to go on at the moment is that it's related to this one error in the main apache error.log:15:33
fungi[Fri Nov 27 07:06:00.135611 2020] [core:notice] [pid 9189:tid 140234591202240] AH00052: child pid 12448 exit signal Bus error (7)15:33
avassthat doesn't sound like a fun error to investigate15:34
fungiwhen i stopped the service just now it complained in error.log about a bunch of processes not exiting in a timely fashion15:34
fungi[Fri Nov 27 15:30:03.034928 2020] [core:warn] [pid 9189:tid 140234591202240] AH00045: child process 18411 still did not exit, sending a SIGTERM15:35
fungiand so on15:35
fungiand then...15:36
fungi[Fri Nov 27 15:30:09.043384 2020] [core:error] [pid 9189:tid 140234591202240] AH00046: child process 18411 still did not exit, sending a SIGKILL15:36
fungiso i'm going with something happened, maybe a little over 8 hours ago, which got at least some apache worker processes "stuck" somehow but not in such a way as to add any useful errors in the log15:37
fungipotentially related to something which killed one of them with a bus error15:37
openstackgerritMerged openstack/project-config master: add review-priority for tripleo-ci  https://review.opendev.org/c/openstack/project-config/+/71506915:38
*** eolivare has quit IRC15:48
openstackgerritGhanshyam Mann proposed openstack/project-config master: Notify placement changes on nova channel  https://review.opendev.org/c/openstack/project-config/+/76446915:52
*** mgoddard has quit IRC15:56
openstackgerritSorin Sb├órnea proposed opendev/gerritbot master: Update container to use Python 3.8 builder  https://review.opendev.org/c/opendev/gerritbot/+/76410416:14
*** marios is now known as marios|out16:46
*** hashar has quit IRC16:47
openstackgerritAlbin Vass proposed zuul/zuul-jobs master: DNM: trigger upload_logs_s3  https://review.opendev.org/c/zuul/zuul-jobs/+/76447616:53
openstackgerritAlbin Vass proposed zuul/zuul-jobs master: DNM: trigger upload_logs_s3  https://review.opendev.org/c/zuul/zuul-jobs/+/76447616:56
openstackgerritAlbin Vass proposed zuul/zuul-jobs master: upload-logs-s3: fix test-jobs files: attribute  https://review.opendev.org/c/zuul/zuul-jobs/+/76447616:57
openstackgerritAlbin Vass proposed zuul/zuul-jobs master: DNM: trigger upload_logs_s3  https://review.opendev.org/c/zuul/zuul-jobs/+/76447816:59
*** tosky has quit IRC17:04
*** marios|out has quit IRC17:14
openstackgerritGhanshyam Mann proposed openstack/project-config master: Notify placement changes on nova channel  https://review.opendev.org/c/openstack/project-config/+/76446917:14
openstackgerritAlbin Vass proposed zuul/zuul-jobs master: GZIPCompressedStream default read length to -1  https://review.opendev.org/c/zuul/zuul-jobs/+/76447817:19
openstackgerritAlbin Vass proposed zuul/zuul-jobs master: GZIPCompressedStream default read length to -1  https://review.opendev.org/c/zuul/zuul-jobs/+/76447817:20
openstackgerritAlbin Vass proposed zuul/zuul-jobs master: Add configuration to make logs public  https://review.opendev.org/c/zuul/zuul-jobs/+/76448317:45
*** ykarel|away has quit IRC17:57
*** dtantsur is now known as dtantsur|afk18:05
openstackgerritAlbin Vass proposed zuul/zuul-jobs master: Add configuration to make logs public  https://review.opendev.org/c/zuul/zuul-jobs/+/76448318:20
*** tosky has joined #opendev19:03
*** ralonsoh has quit IRC19:23
*** whoami-rajat__ has quit IRC19:31
*** mugsie has quit IRC19:59
*** mugsie has joined #opendev20:01
*** slaweq has quit IRC20:26
*** andrewbonney has quit IRC20:38
*** Dmitrii-Sh has quit IRC20:52
*** Dmitrii-Sh has joined #opendev21:04
*** DSpider has quit IRC21:10
*** cgoncalves has quit IRC21:21
*** fressi has quit IRC21:22
*** osmanlicilegi has quit IRC21:22
*** raukadah has quit IRC21:24
*** avass has quit IRC21:25
*** sboyron has quit IRC21:26
*** avass has joined #opendev21:27
*** corvus has quit IRC21:29
*** corvus has joined #opendev21:30
*** fressi has joined #opendev21:30
*** fressi has quit IRC22:20
*** Tengu has joined #opendev23:06
*** tosky has quit IRC23:42

Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!