| *** DSpider has quit IRC | 00:09 | |
| ianw | yeah, 04 was just brought up to not conflict with the openstack.org versions | 00:16 |
|---|---|---|
| ianw | back in ancient history; when we all lived care-free outside and with clean air; i.e. a few months ago | 00:17 |
| fungi | in the beforetime, the long-longago | 00:19 |
| ianw | fungi: speaking of, i'm not letting my ipv6 issues go without a fight. NAT works for me, but libvirt still has issues https://www.redhat.com/archives/libvir-list/2020-September/msg00528.html | 00:21 |
| fungi | wow, yeah | 00:24 |
| fungi | ipv6 nat... /me shudders | 00:24 |
| fungi | but i guess if it saves you from doing dhcp6-pd from your router through your workstation/laptop to the vm... | 00:25 |
| ianw | fungi: if you're in a reviewing mode, i'd like to get rid of https://review.opendev.org/#/q/status:open+topic:s-c-ansible-devel-testing | 00:32 |
| *** mlavalle has quit IRC | 00:47 | |
| openstackgerrit | Arun S A G proposed opendev/gerritlib master: Add support for TCP keep alive in gerritlib https://review.opendev.org/750849 | 00:56 |
| openstackgerrit | Merged opendev/system-config master: tox: drop test-requirements.txt https://review.opendev.org/747328 | 01:18 |
| openstackgerrit | Merged opendev/system-config master: Collect tox logs https://review.opendev.org/747329 | 01:18 |
| ianw | thanks! | 01:21 |
| ianw | gentoo trying to build. there's also a request up for f32 which i can work on soon | 01:21 |
| ianw | 2020-09-10 01:47:37.345 | + locale-gen -q | 01:50 |
| ianw | 2020-09-10 01:47:38.924 | * C.UTF-8: [error] cannot write output files to `C.UTF-8': File exists | 01:50 |
| ianw | 2020-09-10 01:47:40.158 | * Adding locales to archive ... [ !! ] | 01:50 |
| ianw | gentoo fails here now | 01:51 |
| ianw | prometheanfire: ^ | 01:51 |
| fungi | ianw: should 747597 still be getting a POST_FAILURE on system-config-run-base-ansible-devel or is that stale? | 01:52 |
| ianw | fungi: umm, i think that's probably stale from before we had the projects in project-config | 01:54 |
| ianw | although we'll need those ara changes in for it to work | 01:54 |
| fungi | yeah, i figured we were still waiting on that | 01:54 |
| ianw | dmsimard: maybe when you get a sec you can look over the 2.10 fixes for stable @ https://review.opendev.org/#/c/747337/ | 01:55 |
| fungi | but once dmsimard has time to decide there, hopefully we'll have that job back | 01:55 |
| ianw | yeah, and we can speculatively test a lot of the community module changes too with it, which is pretty cool | 01:55 |
| fungi | agreed, that'll be slick | 01:56 |
| prometheanfire | ianw: hi | 02:07 |
| prometheanfire | huh, that's new | 02:07 |
| prometheanfire | maybe we should be running with `-q -k -u` | 02:08 |
| prometheanfire | ianw: that's running in an infra element? If so, it can probably just be removed, since the base dib gentoo stuff runs it for us | 02:36 |
| ianw | that's in ... /tmp/in_target.d/pre-install.d/02-gentoo-00-set-profile | 02:37 |
| prometheanfire | or if it's the gentoo element that's failed, 02-gentoo-00-set-profile, then maybe that can be removed | 02:37 |
| prometheanfire | it's odd | 02:38 |
| prometheanfire | let me build a basic image and see if I can reproduce | 02:38 |
| ianw | https://nb01.opendev.org/gentoo-17-0-systemd-0000143982.log | 02:38 |
| ianw | nb01.openstack.org now points to some other host, we should remove that | 02:39 |
| prometheanfire | systemd profile running with these elements 'gentoo simple-init growroot vm openssh-server block-device-efi enable-serial-console' | 02:40 |
| prometheanfire | hmm | 02:42 |
| prometheanfire | 2020-09-10 02:40:47.395 | dib-run-parts 02-gentoo-00-set-profile completed | 02:42 |
| ianw | #status log removed old nb01/2.openstack.org dns entries | 02:46 |
| openstackstatus | ianw: finished logging | 02:46 |
| prometheanfire | ianw: I'm not sure why that happened, only thing I can think of is if locale-gen is running earlier | 03:23 |
| prometheanfire | but I don't see it in the log | 03:23 |
| openstackgerrit | Ian Wienand proposed openstack/project-config master: Add Fedora 32 builds https://review.opendev.org/750642 | 03:45 |
| openstackgerrit | Merged openstack/project-config master: Add Fedora 32 builds https://review.opendev.org/750642 | 04:06 |
| *** ykarel|away has joined #opendev | 04:22 | |
| *** ykarel|away is now known as ykarel | 04:23 | |
| *** ykarel has quit IRC | 04:30 | |
| *** bhagyashris|rove is now known as bhagyashri|rover | 04:32 | |
| *** ykarel has joined #opendev | 04:37 | |
| *** ykarel_ has joined #opendev | 04:51 | |
| *** ykarel has quit IRC | 04:54 | |
| *** user_19173783170 has joined #opendev | 04:55 | |
| user_19173783170 | hello | 04:56 |
| *** fressi has joined #opendev | 04:56 | |
| *** cloudnull has quit IRC | 05:08 | |
| *** ykarel__ has joined #opendev | 05:12 | |
| *** ykarel_ has quit IRC | 05:14 | |
| *** ysandeep|away is now known as ysandeep | 05:18 | |
| *** ykarel__ is now known as ykarel | 05:22 | |
| *** user_19173783170 has quit IRC | 05:31 | |
| *** ykarel_ has joined #opendev | 06:01 | |
| *** fressi has quit IRC | 06:01 | |
| *** ykarel has quit IRC | 06:03 | |
| *** cloudnull has joined #opendev | 06:05 | |
| *** ykarel_ is now known as ykarel | 06:15 | |
| *** qchris has quit IRC | 06:21 | |
| *** qchris has joined #opendev | 06:33 | |
| AJaeger | infra-root, please review https://review.opendev.org/#/c/597402/ to retire pynotedb. | 06:38 |
| AJaeger | gmann, clarkb , we're down to 52 errors - the stable branches for midonet need the x/networking-l2gw changes as well... | 06:39 |
| AJaeger | ianw: thanks! Could you review https://review.opendev.org/#/c/750635/3 as well? Then it's removed completely... | 06:45 |
| ianw | lgtm, thanks | 06:46 |
| AJaeger | thanks | 06:50 |
| *** hashar has joined #opendev | 06:50 | |
| openstackgerrit | Jan Kubovy proposed zuul/zuul-jobs master: WIP: Temporary disable sudo revocation https://review.opendev.org/750881 | 06:53 |
| openstackgerrit | Merged openstack/project-config master: pynotedb: remove project from infrastructure systems https://review.opendev.org/750635 | 06:58 |
| openstackgerrit | Jan Kubovy proposed zuul/zuul-jobs master: WIP: Temporary disable sudo revocation for tox https://review.opendev.org/750881 | 07:12 |
| AJaeger | gmann, clarkb: There were other repo retirements besides networking-l2gw as well - but that repo has the most fallout. I've send patches for some other repos already. | 07:13 |
| *** priteau has joined #opendev | 07:16 | |
| *** fressi has joined #opendev | 07:24 | |
| *** tosky has joined #opendev | 07:32 | |
| *** priteau has quit IRC | 07:47 | |
| *** andrewbonney has joined #opendev | 07:47 | |
| *** priteau has joined #opendev | 07:53 | |
| *** moppy has quit IRC | 08:01 | |
| *** moppy has joined #opendev | 08:02 | |
| openstackgerrit | Sorin Sbarnea (zbr) proposed opendev/elastic-recheck master: Enable configuration via environment variables https://review.opendev.org/746890 | 08:05 |
| openstackgerrit | Sorin Sbarnea (zbr) proposed opendev/elastic-recheck master: Refreshed bindep https://review.opendev.org/750904 | 08:27 |
| openstackgerrit | Sorin Sbarnea (zbr) proposed opendev/elastic-recheck master: Use pytest for queries https://review.opendev.org/750445 | 08:29 |
| *** jaicaa has quit IRC | 08:29 | |
| *** jaicaa has joined #opendev | 08:31 | |
| *** DSpider has joined #opendev | 08:31 | |
| *** ysandeep is now known as ysandeep|lunch | 08:34 | |
| *** ykarel_ has joined #opendev | 08:35 | |
| *** ykarel has quit IRC | 08:36 | |
| *** ykarel_ is now known as ykarel|lunch | 08:40 | |
| *** ykarel|lunch has quit IRC | 09:12 | |
| *** dtantsur|afk is now known as dtantsur | 09:14 | |
| *** ykarel has joined #opendev | 09:19 | |
| *** fressi has quit IRC | 09:21 | |
| *** ysandeep|lunch is now known as ysandeep | 09:35 | |
| *** hashar has quit IRC | 09:58 | |
| *** hashar has joined #opendev | 09:58 | |
| *** fressi has joined #opendev | 10:01 | |
| openstackgerrit | Merged opendev/elastic-recheck master: Refreshed bindep https://review.opendev.org/750904 | 10:04 |
| *** fressi has quit IRC | 10:13 | |
| openstackgerrit | Sorin Sbarnea (zbr) proposed opendev/elastic-recheck master: Use pytest for queries https://review.opendev.org/750445 | 10:24 |
| openstackgerrit | Sorin Sbarnea (zbr) proposed opendev/elastic-recheck master: Enable configuration via environment variables https://review.opendev.org/746890 | 10:24 |
| *** fressi has joined #opendev | 10:25 | |
| *** fressi has quit IRC | 10:44 | |
| openstackgerrit | Sorin Sbarnea (zbr) proposed opendev/elastic-recheck master: Create elastic-recheck container image https://review.opendev.org/750958 | 10:54 |
| *** dmellado has quit IRC | 11:18 | |
| openstackgerrit | Sorin Sbarnea (zbr) proposed opendev/elastic-recheck master: Create elastic-recheck container image https://review.opendev.org/750958 | 11:19 |
| *** dmellado has joined #opendev | 11:19 | |
| openstackgerrit | Sorin Sbarnea (zbr) proposed opendev/elastic-recheck master: Create elastic-recheck container image https://review.opendev.org/750958 | 11:32 |
| *** dmellado has quit IRC | 11:33 | |
| openstackgerrit | Sorin Sbarnea (zbr) proposed opendev/elastic-recheck master: Create elastic-recheck container image https://review.opendev.org/750958 | 11:34 |
| *** dmellado has joined #opendev | 11:34 | |
| openstackgerrit | Merged opendev/elastic-recheck master: Use pytest for queries https://review.opendev.org/750445 | 11:38 |
| openstackgerrit | Sorin Sbarnea (zbr) proposed opendev/elastic-recheck master: pylint fixes https://review.opendev.org/750963 | 11:42 |
| openstackgerrit | Sorin Sbarnea (zbr) proposed opendev/elastic-recheck master: pylint: fixed imports https://review.opendev.org/750968 | 12:04 |
| *** dmellado has quit IRC | 12:10 | |
| *** dmellado has joined #opendev | 12:11 | |
| openstackgerrit | Merged opendev/elastic-recheck master: pylint fixes https://review.opendev.org/750963 | 12:14 |
| *** hashar has quit IRC | 12:31 | |
| *** hashar has joined #opendev | 12:37 | |
| *** stephenfin has joined #opendev | 12:37 | |
| *** hashar has quit IRC | 12:40 | |
| openstackgerrit | Sorin Sbarnea (zbr) proposed opendev/elastic-recheck master: pylint: fixed imports https://review.opendev.org/750968 | 12:47 |
| mnaser | infra-root: https://review.opendev.org/#/c/750519/ does this failure happen because someone needs to create an afs vol for the ceph debs? | 12:48 |
| *** ykarel_ has joined #opendev | 12:48 | |
| *** ykarel has quit IRC | 12:51 | |
| fungi | mnaser: judging from vos listvol output, i expect so: http://paste.openstack.org/show/797716/ | 13:05 |
| fungi | https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_b8d/750519/1/check/system-config-run-mirror-x86/b8d8998/bridge.openstack.org/ara-report/result/f9f3445c-f5c0-41e7-a30f-0012aaa9f081/ | 13:10 |
| fungi | "src file does not exist, use \"force=yes\" if you really want to create the link: /afs/openstack.org/mirror/ceph-deb-octopus" | 13:10 |
| mnaser | fungi: yeah, that's my thought too. could i get that volume? :-) | 13:11 |
| fungi | it'll probably be 30 minutes, i need to finish up a couple things and switch computers | 13:11 |
| fungi | but syre | 13:11 |
| fungi | sure | 13:11 |
| *** priteau has quit IRC | 13:11 | |
| mnaser | fungi: unless i can push up a system-config patch, im not sure if that's managed manually or not | 13:12 |
| mnaser | i can gladly help with that too | 13:12 |
| dmsimard | fungi, ianw: thanks for the ping, I +W'd the ara fix | 13:12 |
| fungi | mnaser: it's not orchestrated unfortunately, i need to auth to kerberos and then run the commands documented here: afs.html | 13:13 |
| fungi | er, here: https://docs.opendev.org/opendev/system-config/latest/afs.html#mirrors | 13:14 |
| mnaser | fungi: ah gotcha, okay, i'll hold off. thank you :) | 13:14 |
| *** priteau has joined #opendev | 13:15 | |
| openstackgerrit | Sean McGinnis proposed openstack/project-config master: Remove noop jobs for python-adjutantclient https://review.opendev.org/750990 | 13:25 |
| *** ykarel_ is now known as ykarel | 13:25 | |
| *** Goneri has joined #opendev | 13:26 | |
| *** hashar has joined #opendev | 13:48 | |
| openstackgerrit | Sorin Sbarnea (zbr) proposed opendev/elastic-recheck master: pylint: fixed logging-not-lazy https://review.opendev.org/750997 | 13:57 |
| openstackgerrit | Merged opendev/elastic-recheck master: pylint: fixed imports https://review.opendev.org/750968 | 14:11 |
| *** ykarel is now known as ykarel|away | 14:18 | |
| openstackgerrit | Sorin Sbarnea (zbr) proposed opendev/elastic-recheck master: pylint: 6 more https://review.opendev.org/751001 | 14:29 |
| openstackgerrit | Andreas Jaeger proposed openstack/project-config master: Retire devstack-plugin-pika project https://review.opendev.org/748712 | 14:40 |
| *** ykarel|away has quit IRC | 14:45 | |
| openstackgerrit | Sorin Sbarnea (zbr) proposed opendev/elastic-recheck master: Create elastic-recheck container image https://review.opendev.org/750958 | 15:01 |
| *** avass has joined #opendev | 15:09 | |
| openstackgerrit | Sorin Sbarnea (zbr) proposed opendev/elastic-recheck master: pytlint: 4 more https://review.opendev.org/751021 | 15:12 |
| openstackgerrit | Sorin Sbarnea (zbr) proposed opendev/elastic-recheck master: pylint: 4 more https://review.opendev.org/751021 | 15:16 |
| *** hashar has quit IRC | 15:17 | |
| openstackgerrit | Merged opendev/elastic-recheck master: pylint: fixed logging-not-lazy https://review.opendev.org/750997 | 15:20 |
| openstackgerrit | Merged opendev/elastic-recheck master: pylint: 6 more https://review.opendev.org/751001 | 15:25 |
| *** mlavalle has joined #opendev | 16:02 | |
| *** dtantsur is now known as dtantsur|afk | 16:22 | |
| *** ysandeep is now known as ysandeep|away | 16:25 | |
| openstackgerrit | Oleksandr Kozachenko proposed openstack/project-config master: Add some more monasca projects in vexxhost tenant https://review.opendev.org/751046 | 16:31 |
| openstackgerrit | Oleksandr Kozachenko proposed openstack/project-config master: Add some more monasca projects in vexxhost tenant https://review.opendev.org/751046 | 16:32 |
| *** dmellado_ has joined #opendev | 16:33 | |
| *** dmellado has quit IRC | 16:34 | |
| *** dmellado_ is now known as dmellado | 16:34 | |
| fungi | okay... what am i doing wrong? from a shell on mirror.ca-ymq-1.vexxhost.opendev.org i can `ls -l /afs/openstack.org/mirror/ceph-deb-octopus/` and see a dummy file i created in there, but https://mirror.ca-ymq-1.vexxhost.opendev.org/ceph-deb-octopus/ returns 404 even after fully restarting apache | 16:36 |
| fungi | must not have made it to one of the replicas yet for some reason? | 16:38 |
| fungi | https://static.opendev.org/mirror/ceph-deb-octopus/ is working fine | 16:38 |
| *** dmellado has quit IRC | 16:39 | |
| fungi | oh! duh | 16:39 |
| *** dmellado has joined #opendev | 16:39 | |
| fungi | apache looks in /var/www/mirror and we create symlinks there, so it's not going to appear on mirror servers until 750519 merges | 16:40 |
| clarkb | ya the symlinks expose things | 16:40 |
| fungi | #status log added new mirror.deb-octopus volume mounted at /afs/.openstack.org/mirror/ceph-deb-octopus with replicas and set acls consistent with other reprepro mirrors | 16:41 |
| openstackstatus | fungi: finished logging | 16:41 |
| Open10K8S | Hi Team. Please check this PS https://review.opendev.org/751046 . | 16:42 |
| Open10K8S | Thank you | 16:42 |
| *** dmellado has quit IRC | 16:49 | |
| clarkb | infra-root I'm around now and can keep an eye on https://review.opendev.org/#/c/749853/3 if it is approval worthy. Once that lands my next step will be to delete the old server and its volume | 16:51 |
| *** dmellado has joined #opendev | 16:55 | |
| *** dmellado has quit IRC | 17:12 | |
| openstackgerrit | Merged openstack/project-config master: Add some more monasca projects in vexxhost tenant https://review.opendev.org/751046 | 17:12 |
| *** dmellado has joined #opendev | 17:13 | |
| clarkb | fungi: frickler have time for https://review.opendev.org/#/c/749777/ and https://review.opendev.org/#/c/749766/ ? Thats setuptools clean up from the earlier situation with the 50 release | 17:18 |
| fungi | clarkb: on a related topic, i've replaced the cinder volumes for nb01 and nb02... should i do the same for nb04 or are we settled on deleting the server? | 17:25 |
| clarkb | I'm good with deleting it based on available disk space | 17:26 |
| clarkb | and i think ianw said it should be fine too? | 17:26 |
| clarkb | I'll write up a change to pause it now I guess? | 17:26 |
| fungi | that would be awesome. i'll skip it for now and move its volume to the delete list on the etherpad as a reminder | 17:26 |
| clarkb | sounds good, thanks | 17:27 |
| *** dmellado has quit IRC | 17:27 | |
| fungi | #status log cinder volumes for nb01 and nb02 have been replaced and cleaned up | 17:27 |
| openstackstatus | fungi: finished logging | 17:27 |
| openstackgerrit | Clark Boylan proposed openstack/project-config master: Pause image builds on nb04.opendev.org https://review.opendev.org/751066 | 17:31 |
| clarkb | I'm going to stop the builder on nb03.openstack.org now just to be sure there is no fallout from that (don't expect any) | 17:32 |
| clarkb | #status log Stopped nodepool-builder on nb03.openstack.org in preparation for its deletion | 17:33 |
| openstackstatus | clarkb: finished logging | 17:33 |
| mnaser | fungi: thank you for the addition | 17:33 |
| openstackgerrit | Mohammed Naser proposed opendev/system-config master: Add ceph octopus mirrors https://review.opendev.org/750519 | 17:34 |
| *** dmellado has joined #opendev | 17:46 | |
| fungi | #status log cinder volume for wiki-dev has been replaced and cleaned up | 17:58 |
| openstackstatus | fungi: finished logging | 17:58 |
| fungi | now we're just down to the volumes for nb04 (soon to be deleted) and wiki.o.o (which will need deeper surgery to solve) | 17:58 |
| fungi | plus a bunch of unused volumes stuck in error_deleting state now, but once i've got the rest taken care of i'll dump that list into the open rax ticket about the maintenance letting them know those are unused | 17:59 |
| mnaser | zuul.opendev.org seems sluggy-ish | 18:06 |
| mnaser | but i cant nail down a specific reason | 18:06 |
| mnaser | it feels like dns because it responds quickly after a long time | 18:07 |
| fungi | specific views? | 18:07 |
| fungi | ahh | 18:07 |
| clarkb | we did just upgrade the scheduler and the web server (and that included a db update) | 18:07 |
| fungi | zuul-web has an entire core pegged, fwiw | 18:08 |
| fungi | accordnig to top | 18:08 |
| mnaser | `time curl https://zuul.opendev.org/api/tenant/openstack/config-errors` took 6s | 18:08 |
| mnaser | `time curl https://zuul.opendev.org/api/tenant/openstack/build/7e8db9764bc248668a823648913f7403` took 4.66s here so it seems like something is off | 18:08 |
| fungi | yeah, top continues to report a zuul-web process consuming 100-103% cpu | 18:09 |
| fungi | the system itself is still mostly idle because it has 8 processors | 18:10 |
| fungi | but i think zuul-web is single-threaded and so probably starved for cycles | 18:10 |
| clarkb | memory pressure is fine too (thats a common cause of slowness with zuul if we have a leak) | 18:10 |
| fungi | hrm, though no the zuul-web cpu utilization does seem to be getting evenly spread across all the cpus | 18:11 |
| fungi | so it's not that it's eating all of a cpu | 18:11 |
| clarkb | we did change the prioritization of gearman tasks iirc | 18:11 |
| *** gmann is now known as gmann_afk | 18:11 | |
| clarkb | could be something like that maybe? | 18:11 |
| clarkb | fwiw I agree my browser seems to exhibit similar | 18:12 |
| *** andrewbonney has quit IRC | 18:13 | |
| openstackgerrit | Merged openstack/project-config master: Pause image builds on nb04.opendev.org https://review.opendev.org/751066 | 18:14 |
| clarkb | looking in the zuul web debug log the gearman tasks don't seem to take very long. about 100ms or so each | 18:14 |
| fungi | cpu usage also doesn't look particularly higher than it had been before the restart: http://cacti.openstack.org/cacti/graph.php?action=view&local_graph_id=64789&rra_id=all | 18:14 |
| clarkb | though the openstack tenant requests take longer (its got a lot of stuff going on compared to the others I guess) | 18:15 |
| fungi | mnaser: when did you first notice it being slow? | 18:16 |
| clarkb | there are a lot of requests for /api/tenant/openstack/status | 18:16 |
| clarkb | which we should be caching for longer than the period between those requests I thought | 18:16 |
| mnaser | fungi: to be honest, it was a little choppy when i submitted the revision to the ceph patch which means | 18:16 |
| clarkb | I wonder if that is related | 18:16 |
| fungi | we do seem to have *just* reached full utilization of available ram for cache memory | 18:16 |
| mnaser | since at least 45 minutes | 18:16 |
| fungi | established tcp connections is up quite a bit compared to previous days, but that can easily be a symptom of it taking longer to fulfil requests: http://cacti.openstack.org/cacti/graph.php?action=view&local_graph_id=64791&rra_id=all | 18:18 |
| clarkb | the status response is ~1MB whcih is well below our max cache size of 8MB for memory and 10MB for disk (I think we're using disk?) | 18:19 |
| clarkb | we aren't logging that we have cache hits like the mirrors do | 18:20 |
| fungi | also the traffic levels on eth0 don't look appreciably higher than usual for a weekday | 18:20 |
| clarkb | so I wonder if that is at least part of the issue (not using the cache) | 18:20 |
| clarkb | the cache dir has content | 18:22 |
| clarkb | fungi: could it be the .* isn't treated like a regex in our CacheEnable url-string directives? | 18:26 |
| fungi | mmm | 18:26 |
| fungi | http://httpd.apache.org/docs/current/mod/mod_cache.html#cacheenable | 18:29 |
| fungi | i don't see anywhere in there indicating you can use a regex, and the examples are all paths, urls or empty strings | 18:29 |
| fungi | there are a couple of host wildcard examples | 18:30 |
| clarkb | I wonder if that is at least part of the issue here (we aren't caching and taking load off of zuul as expected) | 18:30 |
| fungi | but they're fancy syntax, neither traditional regex nor glob | 18:30 |
| fungi | looks like we can put them into a <location> block though | 18:30 |
| clarkb | fungi: maybe we should drop the .* in /static/ and see if that changes anything | 18:30 |
| clarkb | since its supposed to cache at or below the url-string supplied so the .* in /static/ is not helpful here | 18:31 |
| fungi | or locationmatch i guess in this case | 18:31 |
| clarkb | and then ya if that makes things happy we can use the location stuff | 18:32 |
| fungi | clarkb: yeah worth a try just making it /static/ | 18:32 |
| fungi | "The CacheEnable directive instructs mod_cache to cache urls at or below url-string." | 18:33 |
| fungi | emphasis on "or below" | 18:33 |
| fungi | so the trailing .* is at best redundant | 18:33 |
| openstackgerrit | Merged opendev/system-config master: Remove nodepool builder puppetry and nb03.openstack.org https://review.opendev.org/749853 | 18:33 |
| clarkb | oh ya we don't use .* on the openstack specific vhost | 18:34 |
| clarkb | let me look at the logs for that vhost | 18:34 |
| fungi | clarkb: but reading between the lines, it looks like if we need to get specific for /api/tenant/.*/status we should probably put that particular cacheenable in a <locationmatch> container | 18:35 |
| clarkb | looking at logs for the openstack vhost we combine the logs into a single log file: zuul-ssl-access.log and I see no cache hits there at all :/ | 18:36 |
| fungi | do we have mod_cache set to log its info? | 18:37 |
| clarkb | what is odd is we seem to have cache content in the cache storage location (but its all hashed paths and I have no idea how to parse that to see what we are caching) | 18:37 |
| clarkb | oh maybe that is what we are missing. Let me compare to the mirrors | 18:37 |
| clarkb | ya the mirrors use a custome log format | 18:38 |
| clarkb | I'll add that to zuul with the .* stop in /status/ | 18:38 |
| fungi | also corvus might be interested in this topic if he's available | 18:38 |
| fungi | i guess https://opendev.org/opendev/system-config/src/branch/master/playbooks/roles/zuul-web/templates/zuul.vhost.j2 is where we'll need to patch it when we work out the details | 18:42 |
| fungi | :q | 18:42 |
| fungi | heh, this is not vi | 18:42 |
| openstackgerrit | Clark Boylan proposed opendev/system-config master: Improve logging and cache config for zuul web proxy https://review.opendev.org/751078 | 18:43 |
| clarkb | fungi: ^ yup | 18:43 |
| clarkb | I think we get better logs then work from there | 18:43 |
| clarkb | I'm going to figure out lunch while that does its thing | 18:46 |
| mnaser | infra-root: https://review.opendev.org/#/c/750519/ is ready for a +2 so we can get deb octopus mirroring :) | 18:56 |
| mnaser | should be pretty straight forward | 18:56 |
| fungi | so if i `openstack volume attach ...` and the volume state changes to "attaching" for a few seconds and then back to "available" again, where's the best place (as a non-administrator of that cloud) to look for clues as to why it didn't attach? | 18:59 |
| fungi | given i've already tried dmesg as an obvious first guess, and come up with nothing | 19:00 |
| mnaser | fungi: not much you can do, that seems like a cloud-level issue | 19:01 |
| *** hashar has joined #opendev | 19:01 | |
| mnaser | once it goes into attaching, that means nova and cinder have started talking to start attaching the port | 19:01 |
| mnaser | s/port/volume/ | 19:02 |
| fungi | so there's no synchronous attach call which will spit out an actual error or somethingf | 19:02 |
| mnaser | not off the top of my head, it is possible that the instance action log might log something | 19:02 |
| mnaser | nova instance-action-list <instance> | 19:02 |
| fungi | it does log the attach volume calls at least | 19:03 |
| fungi | with corresponding request ids | 19:04 |
| fungi | | attach_volume | req-a7ec807c-101a-4f27-b630-ae65547dd4a3 | - | 2020-09-10T18:56:24.000000 | | 19:04 |
| fungi | the message column just contains "-" though | 19:04 |
| fungi | oh well, i'll do this the old fashioned way and copy the volume contents to the rootfs, twiddle them into production, detach the old volume, *then* try to attach yet another volume | 19:06 |
| fungi | maybe xen is particular about allowing a very limited number of devices for this old node | 19:07 |
| fungi | i know it doesn't allow more than 16 block devices to be attached at once, but this instance is nowhere near that limit | 19:08 |
| fungi | so am strongly suspecting it's related to being booted from a snapshot of a prior instance booted from an ancient "legacy cloud" flavor | 19:09 |
| fungi | maybe that setup only allowed a single volume attachment or something similarly insane | 19:09 |
| zer0c00l | fungi: After adding keep alive, the bot seems to be holding up for 16+ hours. Perhaps wait and see over the weekend? | 19:11 |
| zer0c00l | https://review.opendev.org/#/c/750849/ | 19:11 |
| fungi | zer0c00l: yeah, that sort of thing can definitely be unpredictable, but a very good sign so far! | 19:13 |
| fungi | long ago i worked for an employer whose firewalls had a default tcp idle policy of 120 seconds. had to set ssh keepalive on by default for openssh on my workstation since otherwise my sessions would hang as soon as they hit a lull | 19:15 |
| fungi | it was very frustrating | 19:15 |
| *** priteau has quit IRC | 19:16 | |
| zer0c00l | It is frustrating! I have seen it miss events and my co-workers complain that it didn't pick up their upstream contributions. | 19:18 |
| openstackgerrit | Merged opendev/system-config master: Improve logging and cache config for zuul web proxy https://review.opendev.org/751078 | 19:51 |
| openstackgerrit | Merged openstack/project-config master: Retire devstack-plugin-pika project https://review.opendev.org/748712 | 20:05 |
| clarkb | fungi: zuul's new logging seems to show what we assumed | 20:07 |
| clarkb | fungi: there are cache messages for the openstack vhost but not the opendev vhost | 20:08 |
| clarkb | fungi: were you working on a locationmatch yet? | 20:08 |
| openstackgerrit | Merged openstack/project-config master: Add openstack/osops https://review.opendev.org/749834 | 20:18 |
| openstackgerrit | Merged openstack/project-config master: Set neutron-lib stable ACLs https://review.opendev.org/750643 | 20:18 |
| openstackgerrit | Clark Boylan proposed opendev/system-config master: Use LocationMatch to cache zuul api statuses https://review.opendev.org/751099 | 20:18 |
| clarkb | fungi: ^ something like that I think? | 20:18 |
| clarkb | then once thats landed we can reevaluate if zuul is still slow | 20:18 |
| clarkb | (but at least then we should be caching what we expect to be caching taking any unnecessary load off of zuul itself) | 20:18 |
| openstackgerrit | Adam Coldrick proposed opendev/storyboard master: Optimise the Story browsing query https://review.opendev.org/742046 | 20:25 |
| fungi | clarkb: oh, sorry, i stepped away to cook dinner, back now and will review asap | 20:28 |
| openstackgerrit | Adam Coldrick proposed opendev/storyboard master: Optimise the Story browsing query https://review.opendev.org/742046 | 20:28 |
| clarkb | no worries I was catching up after lunch and figured that was our next step based on what I saw in the logs | 20:29 |
| clarkb | you might want to check the logs too to confirm (when dinner is sorted) | 20:29 |
| *** hashar has quit IRC | 20:33 | |
| *** openstackgerrit has quit IRC | 20:36 | |
| fungi | clarkb: yeah, logs look right now | 21:10 |
| fungi | except that there's little point in setting combined-cache on the http vhost as it's merely a redirect | 21:10 |
| fungi | but it hurts nothing | 21:11 |
| fungi | and i approved the locationmatch change, will keep an eye out | 21:11 |
| clarkb | thanks | 21:11 |
| mnaser | clarkb: do you have a sec to review https://review.opendev.org/#/c/750519/ | 21:19 |
| clarkb | ya sorry still diggin garound in zuul web logs. I'll look at that now | 21:23 |
| clarkb | fwiw I'm not sure the cache lenght is long enough to help much based on the openstack vhost caching | 21:24 |
| clarkb | but that vhost may also get a lot less traffic so we'll see when the other change merges | 21:24 |
| clarkb | mnaser: its missing a thing. Left a comment inline | 21:29 |
| *** openstackgerrit has joined #opendev | 21:31 | |
| openstackgerrit | Mohammed Naser proposed opendev/system-config master: Add ceph octopus mirrors https://review.opendev.org/750519 | 21:31 |
| mnaser | clarkb: ah yes, that's an obvious one. fixed, thank you | 21:31 |
| *** rosmaita has joined #opendev | 21:37 | |
| openstackgerrit | Merged opendev/system-config master: Use LocationMatch to cache zuul api statuses https://review.opendev.org/751099 | 21:47 |
| *** Goneri has quit IRC | 21:49 | |
| fungi | clarkb: ^ that's in place | 22:13 |
| clarkb | hrm still not logging that it is trying to cache statuses | 22:16 |
| clarkb | it does seem to with the openstack vhost though | 22:18 |
| fungi | yeah, confirmed, maybe it didn't read the config? | 22:19 |
| fungi | though it is listing cache info for /static/... | 22:28 |
| clarkb | maybe I got the regex wrong or didn't location match properly | 22:28 |
| clarkb | should we try dropping the ^ maybe? | 22:28 |
| fungi | so maybe LocationMatch "^/api/tenant/.*/status" doesn't match what we expect, yeah | 22:29 |
| fungi | i'll take a closer look at the locationmatch directive docs | 22:29 |
| clarkb | thanks | 22:30 |
| fungi | http://httpd.apache.org/docs/current/mod/core.html#locationmatch | 22:30 |
| fungi | that looks like what we have | 22:30 |
| openstackgerrit | Merged opendev/system-config master: Add ceph octopus mirrors https://review.opendev.org/750519 | 22:32 |
| clarkb | hrm ya that even includes the ^ root | 22:41 |
| *** tosky has quit IRC | 22:42 | |
| fungi | i'm stumped | 22:46 |
| fungi | could we have put a location directive somewhere they get ignored? | 22:47 |
| *** gmann_afk is now known as gmann | 22:49 | |
| clarkb | maybe it needs to be before the rewrite rule? | 22:49 |
| clarkb | I know rewrite rules themselves are order dependent | 22:49 |
| clarkb | not sure how that interacts with other directives | 22:49 |
| clarkb | oh hey I didn't realize https://review.opendev.org/#/c/749853/3 did end up merging | 22:52 |
| clarkb | ianw: if you're around now any last objections to deleting nb03.openstack.org and its cinder volume? | 22:52 |
| ianw | clarkb: hey, yep, looking at some zuul nodejs stuff so that's why i'm quiet :) no objections to cleaning it up, new server seems to be working great | 22:53 |
| clarkb | great, I'll procede with that now as I'm stumped by the zuul web caching thing | 22:53 |
| clarkb | ianw: one thing I noticed was I used a different network for the new server than the old server. The old server is on the os-control-network-ipv4 network and new one is on os-control-network (which has ipv6 too) | 22:57 |
| clarkb | ianw: does that change anything? | 22:57 |
| clarkb | I guess if ipv6 is an issue we do the netplan hack like we did in vexxhost or rebuild on the -ipv4 network | 22:58 |
| * clarkb proceeds with deletion | 22:58 | |
| ianw | clarkb: my memory is hazy ... i think that maybe we got floating ip's assigned to the other one | 22:58 |
| ianw | kevinz might remember | 22:58 |
| clarkb | hrm we have a floating ip on the current one | 22:59 |
| clarkb | so I expect its fine | 22:59 |
| fungi | hopefully the reason was purely historical and no longer relevant | 23:00 |
| clarkb | #status log Deleted nb03.openstack.org. It has been replaced by nb03.opendev.org. | 23:00 |
| openstackstatus | clarkb: finished logging | 23:01 |
| clarkb | ianw: have the nlXY records in openstack.org dns been cleaned up yet? | 23:01 |
| clarkb | if not I can clean those up when I do nb03 | 23:01 |
| ianw | clarkb: umm, i don't think so. i did find the old nbxx.openstack.org addresses were redirecting to some exciting new unrelated site when i accidetnally typod that yesterday | 23:01 |
| *** mlavalle has quit IRC | 23:02 | |
| openstackgerrit | Jeremy Stanley proposed opendev/engagement master: Initial commit https://review.opendev.org/729293 | 23:03 |
| fungi | everybody needs a little excitement these days, i suppose | 23:04 |
| clarkb | er nlXY are still openstack | 23:05 |
| clarkb | its nb0X that are not | 23:05 |
| clarkb | (I haven't deleted the wrong records yet thankfully) | 23:05 |
| clarkb | and it looks like the nb records have been cleaned | 23:06 |
| clarkb | so I think we're good now | 23:07 |
| clarkb | ianw: ^ fyi | 23:07 |
| ianw | right, i thought so, but then i thought i missed something :) | 23:08 |
| ianw | we did an inplace replacement of the launchers | 23:08 |
| clarkb | ianw: also we've paused nb04 so that its images can rotate out | 23:10 |
| clarkb | then probably next week we can delete it next | 23:10 |
| fungi | clarkb: so back to caching, maybe we need a locationmatch on http://127.0.0.1:9000/api/tenant/.*/status | 23:13 |
| ianw | ++ | 23:13 |
| clarkb | fungi: or just drop the ^ and it should match a longer prefix that way? | 23:13 |
| fungi | yeah, that probably wouldn't hurt | 23:14 |
| clarkb | whats weird is the openstack one seems to work | 23:14 |
| clarkb | but that isn't use a location match because it isn't a regex so maybe order matters for locationmatches | 23:14 |
| clarkb | fungi: maybe we want to hand edit the file and reload really quick to check? | 23:14 |
| fungi | i'll try it now, yeah | 23:15 |
| fungi | that doesn't seem to have made any difference, even after a reload | 23:16 |
| fungi | replacing the .* with a specific tenant hasn't either | 23:18 |
| clarkb | fungi: maybe switch back to CacheEnable for the openstack tenant and see if that works? | 23:18 |
| clarkb | if that doesn't work its possible the status isn't cacheable for some reason in that context (but is when using the openstack vhost) | 23:18 |
| fungi | clarkb: tried it and still not getting cached, so i'm starting to think it's something about the response that's making it uncacheable (but then how is it cached through the other vhost?) | 23:25 |
| clarkb | weird | 23:27 |
| openstackgerrit | Adam Coldrick proposed opendev/storyboard master: Optimise the Story browsing query https://review.opendev.org/742046 | 23:34 |
| openstackgerrit | Adam Coldrick proposed opendev/storyboard master: Unpin webob and hacking to fix compatibility issues https://review.opendev.org/751117 | 23:34 |
| *** DSpider has quit IRC | 23:44 | |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!