| *** jamesmcarthur has joined #openstack-infra | 00:12 | |
| *** rlandy|bbl is now known as rlandy | 00:32 | |
| *** Goneri has quit IRC | 00:39 | |
| *** hamalq has quit IRC | 00:41 | |
| *** SpamapS has quit IRC | 00:44 | |
| *** ryohayakawa has joined #openstack-infra | 00:45 | |
| *** portdirect has quit IRC | 00:51 | |
| *** gmann has quit IRC | 00:51 | |
| *** portdirect has joined #openstack-infra | 00:51 | |
| *** gmann has joined #openstack-infra | 00:51 | |
| *** seongsoocho has quit IRC | 00:52 | |
| *** seongsoocho has joined #openstack-infra | 00:54 | |
| *** SpamapS has joined #openstack-infra | 00:58 | |
| *** rfolco has quit IRC | 00:58 | |
| *** ociuhandu has joined #openstack-infra | 01:01 | |
| *** jamesmcarthur has quit IRC | 01:04 | |
| *** ociuhandu has quit IRC | 01:05 | |
| *** tetsuro has joined #openstack-infra | 01:13 | |
| *** ricolin has joined #openstack-infra | 01:21 | |
| *** jamesmcarthur has joined #openstack-infra | 01:37 | |
| *** jamesmcarthur has quit IRC | 01:48 | |
| *** rlandy has quit IRC | 01:50 | |
| *** ramishra has joined #openstack-infra | 01:53 | |
| *** d34dh0r53 has joined #openstack-infra | 01:55 | |
| *** dchen is now known as dchen|away | 02:04 | |
| *** dchen|away is now known as dchen | 02:27 | |
| *** ktsuyuzaki is now known as kota_ | 02:32 | |
| *** tetsuro has quit IRC | 02:45 | |
| *** jamesmcarthur has joined #openstack-infra | 02:51 | |
| *** rcernin has quit IRC | 02:59 | |
| *** hongbin has joined #openstack-infra | 03:00 | |
| *** d34dh0r53 has quit IRC | 03:02 | |
| *** rcernin has joined #openstack-infra | 03:03 | |
| *** SpamapS has quit IRC | 03:06 | |
| *** SpamapS has joined #openstack-infra | 03:09 | |
| *** jamesmcarthur has quit IRC | 03:15 | |
| *** SpamapS has quit IRC | 03:15 | |
| *** armax has quit IRC | 03:33 | |
| *** dchen is now known as dchen|away | 03:42 | |
| *** tetsuro has joined #openstack-infra | 03:51 | |
| *** tetsuro has quit IRC | 04:01 | |
| *** tetsuro has joined #openstack-infra | 04:02 | |
| *** tetsuro has quit IRC | 04:06 | |
| *** gyee has quit IRC | 04:10 | |
| *** hongbin has quit IRC | 04:20 | |
| *** vishalmanchanda has joined #openstack-infra | 04:22 | |
| *** ykarel has joined #openstack-infra | 04:28 | |
| *** evrardjp has quit IRC | 04:33 | |
| *** evrardjp has joined #openstack-infra | 04:33 | |
| *** ysandeep|away is now known as ysandeep|rover | 04:38 | |
| *** dchen|away is now known as dchen | 04:43 | |
| *** Lucas_Gray has quit IRC | 04:49 | |
| *** dchen is now known as dchen|away | 04:57 | |
| *** ociuhandu has joined #openstack-infra | 05:03 | |
| *** SpamapS has joined #openstack-infra | 05:06 | |
| *** ociuhandu has quit IRC | 05:07 | |
| *** mtreinish has quit IRC | 05:08 | |
| *** ykarel has quit IRC | 05:16 | |
| *** udesale has joined #openstack-infra | 05:27 | |
| *** ykarel has joined #openstack-infra | 05:31 | |
| *** ykarel is now known as ykarel|mtg | 05:33 | |
| *** ykarel_ has joined #openstack-infra | 05:36 | |
| *** ykarel|mtg has quit IRC | 05:39 | |
| *** dchen|away is now known as dchen | 05:47 | |
| *** lmiccini has joined #openstack-infra | 05:48 | |
| *** mtreinish has joined #openstack-infra | 05:54 | |
| *** lxkong has quit IRC | 05:55 | |
| *** bhagyashris|away is now known as bhagyashris | 06:01 | |
| *** marios has joined #openstack-infra | 06:03 | |
| *** lpetrut has joined #openstack-infra | 06:04 | |
| *** tetsuro has joined #openstack-infra | 06:06 | |
| *** ykarel_ is now known as ykarel | 06:08 | |
| *** tetsuro has quit IRC | 06:11 | |
| *** kevinz has joined #openstack-infra | 06:28 | |
| *** eolivare has joined #openstack-infra | 06:43 | |
| *** slaweq has joined #openstack-infra | 06:48 | |
| *** yolanda has quit IRC | 06:52 | |
| *** yolanda has joined #openstack-infra | 06:52 | |
| *** nightmare_unreal has joined #openstack-infra | 06:56 | |
| *** jcapitao has joined #openstack-infra | 07:12 | |
| *** dklyle has quit IRC | 07:19 | |
| *** dchen is now known as dchen|away | 07:25 | |
| *** andrewbonney has joined #openstack-infra | 07:29 | |
| *** rcernin has quit IRC | 07:32 | |
| *** tosky has joined #openstack-infra | 07:42 | |
| *** ociuhandu has joined #openstack-infra | 07:52 | |
| *** ociuhandu has quit IRC | 07:52 | |
| *** ociuhandu has joined #openstack-infra | 07:53 | |
| *** ralonsoh has joined #openstack-infra | 07:53 | |
| *** jpena|off is now known as jpena | 07:54 | |
| *** dchen|away is now known as dchen | 07:55 | |
| *** lucasagomes has joined #openstack-infra | 08:06 | |
| *** bhagyashris is now known as bhagyashris|lunc | 08:10 | |
| *** ysandeep|rover is now known as ysandeep|lunch | 08:11 | |
| *** xek has joined #openstack-infra | 08:13 | |
| *** hashar has joined #openstack-infra | 08:25 | |
| *** pkopec has joined #openstack-infra | 08:27 | |
| *** dchen is now known as dchen|away | 08:33 | |
| *** derekh has joined #openstack-infra | 08:41 | |
| *** rcernin has joined #openstack-infra | 08:48 | |
| *** priteau has joined #openstack-infra | 08:49 | |
| *** ociuhandu_ has joined #openstack-infra | 08:53 | |
| *** rcernin has quit IRC | 08:54 | |
| *** ociuhandu has quit IRC | 08:56 | |
| *** tetsuro has joined #openstack-infra | 08:58 | |
| *** ysandeep|lunch is now known as ysandeep | 09:05 | |
| *** ricolin has quit IRC | 09:07 | |
| *** ysandeep is now known as ysandeep|rover | 09:11 | |
| *** dtantsur|afk is now known as dtantsur | 09:12 | |
| *** Lucas_Gray has joined #openstack-infra | 09:14 | |
| *** ociuhandu_ has quit IRC | 09:25 | |
| *** ociuhandu has joined #openstack-infra | 09:31 | |
| *** ociuhandu has quit IRC | 09:31 | |
| *** ociuhandu has joined #openstack-infra | 09:31 | |
| *** tetsuro has quit IRC | 09:32 | |
| *** tetsuro has joined #openstack-infra | 09:40 | |
| *** bhagyashris|lunc is now known as bhagyashris | 09:41 | |
| *** tetsuro has quit IRC | 09:48 | |
| *** tetsuro has joined #openstack-infra | 09:51 | |
| *** tetsuro has quit IRC | 09:53 | |
| *** udesale_ has joined #openstack-infra | 09:58 | |
| *** udesale has quit IRC | 10:01 | |
| *** ramishra has quit IRC | 10:15 | |
| *** rcernin has joined #openstack-infra | 10:19 | |
| *** rcernin has quit IRC | 10:43 | |
| *** xek has quit IRC | 10:49 | |
| *** ramishra has joined #openstack-infra | 10:58 | |
| *** dchen|away is now known as dchen | 11:02 | |
| *** jcapitao is now known as jcapitao_lunch | 11:04 | |
| *** bhagyashris is now known as bhagyashris|away | 11:06 | |
| *** lucasagomes has quit IRC | 11:11 | |
| *** lucasagomes has joined #openstack-infra | 11:12 | |
| *** xek has joined #openstack-infra | 11:24 | |
| *** jpena is now known as jpena|lunch | 11:28 | |
| *** bhagyashris|away is now known as bhagyashris | 11:36 | |
| *** lbragstad has quit IRC | 11:45 | |
| *** ricolin has joined #openstack-infra | 11:47 | |
| *** dchen is now known as dchen|away | 11:50 | |
| *** dchen|away is now known as dchen | 11:50 | |
| *** priteau has quit IRC | 11:55 | |
| *** priteau has joined #openstack-infra | 11:56 | |
| *** jcapitao_lunch is now known as jcapitao | 11:58 | |
| *** evrardjp has quit IRC | 12:00 | |
| *** tkajinam has quit IRC | 12:02 | |
| *** rfolco has joined #openstack-infra | 12:02 | |
| *** hashar has quit IRC | 12:05 | |
| *** rlandy has joined #openstack-infra | 12:10 | |
| *** jamesmcarthur has joined #openstack-infra | 12:16 | |
| *** ysandeep|rover is now known as ysandeep|brb | 12:26 | |
| *** ryohayakawa has quit IRC | 12:26 | |
| *** dchen is now known as dchen|away | 12:29 | |
| *** ralonsoh has quit IRC | 12:34 | |
| *** ralonsoh has joined #openstack-infra | 12:34 | |
| *** jpena|lunch is now known as jpena | 12:34 | |
| *** ysandeep|brb is now known as ysandeep|rover | 12:37 | |
| *** xek_ has joined #openstack-infra | 12:48 | |
| *** lbragstad has joined #openstack-infra | 12:50 | |
| *** dchen|away is now known as dchen | 12:50 | |
| *** xek has quit IRC | 12:51 | |
| *** piotrowskim has joined #openstack-infra | 12:56 | |
| *** ociuhandu has quit IRC | 12:57 | |
| *** ociuhandu has joined #openstack-infra | 12:57 | |
| *** rcernin has joined #openstack-infra | 12:59 | |
| *** rcernin has quit IRC | 13:03 | |
| *** dchen is now known as dchen|away | 13:05 | |
| *** dchen|away is now known as dchen | 13:09 | |
| *** dciabrin has quit IRC | 13:10 | |
| *** dciabrin has joined #openstack-infra | 13:10 | |
| *** Goneri has joined #openstack-infra | 13:13 | |
| *** jamesmcarthur has quit IRC | 13:18 | |
| *** zxiiro has joined #openstack-infra | 13:19 | |
| *** Lucas_Gray has quit IRC | 13:22 | |
| *** ramishra has quit IRC | 13:27 | |
| *** Lucas_Gray has joined #openstack-infra | 13:29 | |
| *** dchen is now known as dchen|away | 13:30 | |
| *** dchen|away is now known as dchen | 13:31 | |
| *** ramishra has joined #openstack-infra | 13:39 | |
| *** dchen is now known as dchen|away | 13:41 | |
| *** d34dh0r53 has joined #openstack-infra | 13:41 | |
| *** d34dh0r53 has joined #openstack-infra | 13:42 | |
| *** dchen|away is now known as dchen | 13:47 | |
| *** dciabrin has quit IRC | 13:48 | |
| *** dciabrin has joined #openstack-infra | 13:49 | |
| *** ramishra has quit IRC | 13:59 | |
| *** ramishra has joined #openstack-infra | 14:00 | |
| openstackgerrit | Monty Taylor proposed zuul/zuul-jobs master: Add a job for publishing a site to netlify https://review.opendev.org/739047 | 14:04 |
|---|---|---|
| *** ysandeep|rover is now known as ysandeep|away | 14:05 | |
| *** rlandy is now known as rlandy|training | 14:07 | |
| openstackgerrit | Monty Taylor proposed zuul/zuul-jobs master: Add a job for publishing a site to netlify https://review.opendev.org/739047 | 14:09 |
| *** ykarel is now known as ykarel|away | 14:28 | |
| *** hashar has joined #openstack-infra | 14:28 | |
| *** jamesmcarthur has joined #openstack-infra | 14:30 | |
| *** dchen is now known as dchen|away | 14:33 | |
| *** auristor has quit IRC | 14:34 | |
| *** lpetrut has quit IRC | 14:40 | |
| *** auristor has joined #openstack-infra | 14:41 | |
| *** dklyle has joined #openstack-infra | 14:41 | |
| *** armax has joined #openstack-infra | 14:47 | |
| *** auristor has quit IRC | 15:01 | |
| *** jamesmcarthur has quit IRC | 15:02 | |
| *** auristor has joined #openstack-infra | 15:07 | |
| *** ykarel|away has quit IRC | 15:13 | |
| *** auristor has quit IRC | 15:30 | |
| *** udesale_ has quit IRC | 15:40 | |
| *** auristor has joined #openstack-infra | 15:40 | |
| *** marios has quit IRC | 15:58 | |
| *** jcapitao has quit IRC | 16:04 | |
| *** xek_ has quit IRC | 16:06 | |
| *** priteau has quit IRC | 16:08 | |
| *** lmiccini has quit IRC | 16:09 | |
| clarkb | logged in docker hub users are not subject to rate limiting | 16:12 |
| clarkb | I expect that authenticating with docker hub and using our mirrors would work but that may affect our ability to cache things? | 16:13 |
| clarkb | the other aspect to consider is keeping those credentials either safe or sufficiently limited in scope that their exposure in random jobs is a non issue | 16:14 |
| clarkb | weshay|ruck: ^ that may be worth investigating | 16:14 |
| *** lucasagomes has quit IRC | 16:18 | |
| weshay|ruck | sec sorry | 16:26 |
| *** Lucas_Gray has quit IRC | 16:26 | |
| fungi | clarkb: yeah, i noticed that as well but wasn't sure how to effectively auth the proxied requests | 16:29 |
| *** dtantsur is now known as dtantsur|afk | 16:32 | |
| clarkb | fungi: I think the protocol requests a token and then supplies that in the requests | 16:34 |
| clarkb | it may be that you need to auth to dockerhub first then use the token with the mirror? | 16:34 |
| *** rlandy|training is now known as rlandy | 16:35 | |
| clarkb | mostly just trying to look at the various options, I haven't tested anything | 16:39 |
| weshay|ruck | clarkb, hrm.. interesting.. zbr FYI ^ | 16:43 |
| *** ociuhandu has quit IRC | 16:45 | |
| *** hashar has quit IRC | 16:46 | |
| *** hamalq has joined #openstack-infra | 16:46 | |
| *** derekh has quit IRC | 17:00 | |
| *** ykarel|away has joined #openstack-infra | 17:03 | |
| *** andrewbonney has quit IRC | 17:04 | |
| *** jpena is now known as jpena|off | 17:16 | |
| *** nightmare_unreal has quit IRC | 17:33 | |
| *** ykarel|away has quit IRC | 17:41 | |
| *** redrobot has quit IRC | 17:41 | |
| *** ralonsoh has quit IRC | 17:49 | |
| *** ociuhandu has joined #openstack-infra | 17:54 | |
| *** ociuhandu has quit IRC | 17:58 | |
| zbr | based on https://docs.docker.com/registry/recipes/mirror/ -- i believe that we need a dual setup, a pull-only mirror and a push-registry for pushes, one that does not use the mirror. | 18:02 |
| zbr | decoupling pulls from pulls, and we can configure the mirror to use auth, in order to avoid being rate limited | 18:04 |
| clarkb | zbr: we do pushes directly already which is basically the same thing? | 18:04 |
| clarkb | but also we don't run a proper registry like that because there is no way to prune | 18:05 |
| fungi | yeah, the first half of that suggestion is already done | 18:18 |
| fungi | we're only talking about read operations here, write operations are completely separate | 18:19 |
| fungi | if we did authenticate the proxy we'd certainly use a different account than the various accounts used for pushing images to various dockerhub namespaces | 18:19 |
| clarkb | well and I'm suggested we authenticate the client and not change the proxy | 18:20 |
| clarkb | but yes different accounts for sure | 18:20 |
| fungi | in that case i would be surprised if we could actually cache the downloads | 18:20 |
| clarkb | fungi: we mitm so it should work I think | 18:20 |
| clarkb | and we already tell apache to cache even though it shouldn't under its "what is cacheable" rules | 18:21 |
| fungi | i guess it depends on whether the request for the file incorporates credentials, and whether that makes it unique in apache mod_proxy's eyes | 18:22 |
| clarkb | I think we'll be ok because it uses headers | 18:24 |
| clarkb | and we tell the server that its ok to cache the data anyway | 18:25 |
| clarkb | ah we ignore the query string parameters | 18:25 |
| clarkb | not necessarily headers | 18:26 |
| *** jamesden_ has joined #openstack-infra | 18:27 | |
| clarkb | reading mod_cache docs I think we may be ok because the supplied header is provided by the client and the response either gives you the data or you get a 401 unauthorized | 18:28 |
| clarkb | but we can also ignore the authentication header with an apache config in case that ends up being used to identify the resources | 18:30 |
| *** jamesdenton has quit IRC | 18:30 | |
| fungi | worth a try, i guess. so projets would encode a dockerhub credential as a zuul secret? | 18:32 |
| clarkb | I think so | 18:33 |
| clarkb | actually this should work becuse iirc you have to do anonymous auth with docker hub already | 18:34 |
| clarkb | all we'd be changing is going from a token belonging to anonymous to an actual user | 18:34 |
| clarkb | its possible that the cache may still be scoped by auth token I suppose but I don't think its a consistent token for anonymous either | 18:37 |
| clarkb | possible that is already an underlying bug in the system if so | 18:37 |
| *** eolivare has quit IRC | 18:37 | |
| fungi | just worried that we're probably hitting this to a great extent because of limited cache space and aggressive expiring leading to frequent cache misses | 18:44 |
| clarkb | that can be too, but I'm not sure we have much ability to increase disk size. At in some of the clouds we're limited iirc | 18:44 |
| clarkb | we can possibly make bigger caches in clouds where more cinder quota is available | 18:44 |
| fungi | well, also htcacheclean speed is limited by i/o bandwidth, so if we make the caches too big we can't prune them as fast as we fill them and the run away from us filling the disk and generating new and even more fun errors | 18:45 |
| clarkb | good point | 18:45 |
| clarkb | oh it will also be proportional to the max-servers count in the clouds since that roughly maps to jobs/time | 18:49 |
| fungi | yep, we hit this issue the most in our largest regions, no surprise there | 18:50 |
| *** xek_ has joined #openstack-infra | 18:51 | |
| fungi | if we had infinite storage and infinitesimal iowait then it would be roughly even across regions, but ultimately there's pressure from new cache additions forcing earlier content out before it expires | 18:51 |
| fungi | and that pressure increases in more active/higher quota regions | 18:52 |
| clarkb | as a sanity check on the dfw mirror the current log file for docker v2 api says 25939 cache hits and 6883 misses | 18:54 |
| clarkb | looking at the log though it seems like the python-requests requests for docker things may not be caching for some reason | 18:57 |
| clarkb | if docker client is used then it will say cache hit vs cache miss etc | 18:57 |
| fungi | python-requests is probably zuul-registry? | 18:58 |
| fungi | that could be worth looking into | 18:58 |
| clarkb | fungi: no its tripleo | 18:58 |
| fungi | or skopeo? | 18:58 |
| fungi | ahh | 18:58 |
| clarkb | they've written their own client | 18:58 |
| clarkb | and this is why I know about the authentication thing | 18:59 |
| clarkb | it had trouble with that, but if I had to make a hunch here part of the problem is they are bypassing the cache entirely looking at the logs | 18:59 |
| clarkb | I don't understand why though | 18:59 |
| fungi | so basically none of tripleo's requests are actually being served from the cache? | 18:59 |
| fungi | that could explain this recently becoming an issue | 18:59 |
| clarkb | at least on a quick glance any from their python client seem to lack cache hit or miss indications | 18:59 |
| clarkb | fungi: yup that is what I'm thinking now | 18:59 |
| clarkb | but I need to context switch to the meeting | 18:59 |
| fungi | but now, infra meeting | 19:00 |
| *** hashar has joined #openstack-infra | 19:17 | |
| *** tosky has quit IRC | 19:25 | |
| *** hashar has quit IRC | 19:50 | |
| openstackgerrit | Ian Wienand proposed openstack/project-config master: Add pyca/project-config https://review.opendev.org/743143 | 19:54 |
| openstackgerrit | Ian Wienand proposed openstack/project-config master: Add pyca 3rd-party CI tenant https://review.opendev.org/743144 | 19:54 |
| *** mordred has quit IRC | 20:05 | |
| clarkb | weshay|ruck: EmilienM zbr ^ see scrollback but tldr is I think tripleo's python client may be doing something that prevents apache from caching things. And that could lead to more upstream requests and being rate limited. Whereas the docker client does not have these problems based on apache logs | 20:08 |
| EmilienM | mwhahaha: ^ fyi | 20:08 |
| clarkb | I don't know why this is happening and the logs between the two look very similar exept for the user agent strings | 20:09 |
| clarkb | that maybe points to headers? since they aren't all recorded | 20:09 |
| EmilienM | we use "podman_image" ansible module to pull / pre-fetch container images | 20:09 |
| EmilienM | but the tripleo container image prepare step is indeed in pure python on our side | 20:09 |
| clarkb | "python-requests/2.22.0" is the user agent when we don't cache | 20:10 |
| mwhahaha | fetching tags/metadata aren't cachable | 20:10 |
| mwhahaha | and have a short life token (5misn) | 20:10 |
| clarkb | thats getting things like /cloudflare/registry-v2/docker/registry/v2/blobs/sha256/99/9984f3dc6de930df9464afc6fed39d9be3c1331ed07f4cd2395ce2eddc0986fc/data | 20:10 |
| clarkb | which is not metadata but the actual layer info aiui | 20:10 |
| mwhahaha | it's likely having header issues then | 20:11 |
| mwhahaha | we don't specify any no-cache stuff | 20:11 |
| mwhahaha | let me go look in the code | 20:11 |
| fungi | we also try to ignore no-cache pragma and cache busting stuff where we can | 20:11 |
| mwhahaha | we've been doing this for years now tho | 20:12 |
| clarkb | mwhahaha: sure I'm just reporting what I see in the logs today | 20:12 |
| fungi | but maybe we can improve on that too | 20:12 |
| clarkb | "docker/19.03.12 go/go1.13.10 git-commit/48a66213fe kernel/4.15.0-112-generic os/linux arch/amd64 UpstreamClient(docker-sdk-python/4.2.2)" UA triggers caching just fine | 20:12 |
| clarkb | and is used by kolla | 20:12 |
| clarkb | but the tripleo urls seem to be hit by python-requests and do not trigger caching | 20:12 |
| mwhahaha | maybe we need to add allow caching to our code or something | 20:14 |
| mwhahaha | not certain what the defaults are with python-requests | 20:14 |
| weshay|ruck | 0/ | 20:18 |
| *** mordred has joined #openstack-infra | 20:18 | |
| clarkb | I'm trying to put what I'm seeing together into a paste | 20:18 |
| weshay|ruck | +21 | 20:20 |
| weshay|ruck | or just 1 | 20:20 |
| mwhahaha | we're not providing a Cache-Control header in our requests, trying to see what ges sent in the docker client | 20:20 |
| clarkb | http://paste.openstack.org/show/796392/ I think that shows that the manifest data isn't cached as expected, but then the docker client does manage to use cached data for the layer blobs | 20:26 |
| mwhahaha | so those two things are doing different operations | 20:28 |
| openstackgerrit | Ian Wienand proposed openstack/project-config master: Add pyca 3rd-party CI tenant https://review.opendev.org/743144 | 20:28 |
| mwhahaha | the layer should be cached, but the other bits are different actions | 20:28 |
| clarkb | I'm not sure I understand. Each section capture multiple operations | 20:28 |
| clarkb | I wanted to compare the ops between clients | 20:28 |
| mwhahaha | right but they aren't doing the same thing we do | 20:28 |
| mwhahaha | so we need to get a tags list, kolla doesn't | 20:29 |
| clarkb | the last GET in both sequence is basically the same | 20:29 |
| mwhahaha | kolla uses specific tags | 20:29 |
| clarkb | I would expect that last GET to be cacheable in both cases | 20:29 |
| clarkb | that is the bit that isn't working how I expect | 20:29 |
| mwhahaha | how many times is that layer cached | 20:29 |
| mwhahaha | er fetched | 20:29 |
| clarkb | mwhahaha: why does that matter? we should get a cache missed if it is the first request but we don't get that either | 20:29 |
| mwhahaha | size? | 20:30 |
| mwhahaha | do have a min size for cache? | 20:30 |
| *** pkopec has quit IRC | 20:30 | |
| clarkb | the size is minimal for both cases | 20:30 |
| clarkb | the kolla layer is less than a kb, the tripleo layer about 23 kb | 20:30 |
| clarkb | we have a max size but its much larger than that | 20:30 |
| mwhahaha | that seems wrong for layer | 20:31 |
| clarkb | I think its not necessarily a complete layer | 20:31 |
| mwhahaha | so their request is a partial request using range | 20:31 |
| clarkb | docker chunks them up into those blobs | 20:31 |
| mwhahaha | the reason why i asked how many times is c308ad9b6a688a9bb05caccf8055b3e3910e27a51305a1ee19434d5bd3485847 fetched is if that's the first time, it won't be a hit | 20:32 |
| clarkb | mwhahaha: I know that but in that case apache will still log it doesn't have it in the cache | 20:32 |
| mwhahaha | right but that's not a client side ask | 20:33 |
| mwhahaha | the only way tripleoclient could break that is if we set cache-control to not allow caching but we don't | 20:33 |
| clarkb | something about that client is causing apache to not even try the cache | 20:33 |
| clarkb | thats not true | 20:33 |
| clarkb | apache has rules for what is cacheable and not | 20:33 |
| mwhahaha | ... | 20:34 |
| clarkb | $IP - - [2020-07-28 20:32:46.765] "GET /cloudflare/registry-v2/docker/registry/v2/blobs/sha256/27/27f31552aaaf7d0b8034b3ef6df2425ff0a54e75904366d490054e84722eaef4/data?verify=1595971366-6IJLMBtHINoelVCBx73kEyldqgE%3D HTTP/1.1" 200 806941 cache miss: attempting entity save | 20:34 |
| clarkb | "http://mirror.dfw.rax.opendev.org:8082/v2/kolla/centos-source-masakari-monitors/blobs/sha256:27f31552aaaf7d0b8034b3ef6df2425ff0a54e75904366d490054e84722eaef4" "docker/19.03.12 go/go1.13.10 git-commit/48a66213fe kernel/4.18.0-193.6.3.el8_2.x86_64 os/linux arch/amd64 UpstreamClient(docker-sdk-python/4.2.2)" | 20:34 |
| clarkb | that is what a miss should look like, but in the tripleo case it doesn't even try | 20:34 |
| mwhahaha | the determination of caching is not a clientside ask in that case | 20:34 |
| fungi | running an aggressive cache in production in the past, i recall having to custom compile squid to disable some of the cache-busting signals it obeyed | 20:34 |
| mwhahaha | unless you can provide the exact headers/request item that's causing your config to not cache, it's not a client problem | 20:34 |
| *** Lucas_Gray has joined #openstack-infra | 20:34 | |
| clarkb | yes, I understand. But what is cacheable is in part determined by the clients request | 20:35 |
| mwhahaha | no it's not | 20:35 |
| mwhahaha | it only is if we are specifyign cache related headers | 20:35 |
| mwhahaha | and we're not | 20:35 |
| mwhahaha | that's what i'm saying | 20:35 |
| mwhahaha | unless python-requests disables it by default which i do not believe it does | 20:35 |
| mwhahaha | so as far as our request, we're not specifying any cache control related headers in the client request | 20:36 |
| mwhahaha | my thought is maybe we need to be and that's the problem but i don't see it being set in docker client either | 20:36 |
| mwhahaha | unless go's http request automatically enables it | 20:36 |
| clarkb | what I'm trying to say is the the cache control headers are not the only things that apache considers | 20:36 |
| clarkb | I'm trying to find docs on that now | 20:36 |
| mwhahaha | etag/cache-control/etc | 20:37 |
| clarkb | for example we specifically ignore query strings in the apache config for docker because by default apache would treat all urls with a query string as uncachable | 20:37 |
| mwhahaha | yea i know there are multiple but we're not specifying any | 20:37 |
| mwhahaha | so unless you know something about python-requests that i don't, it's unclear to me how this is a client issue | 20:37 |
| clarkb | ok I'm just trying to help figure this out | 20:38 |
| clarkb | I don't know for sure it is a client issue but I do know the official client works fine | 20:38 |
| mwhahaha | If the response contains an "Authorization:" header, it must also contain an "s-maxage", "must-revalidate" or "public" option in the "Cache-Control:" header, or it won't be cached. | 20:38 |
| clarkb | your special client does not | 20:38 |
| mwhahaha | that's likely that we must specify cache-control | 20:38 |
| mwhahaha | because docker requests have Authorization | 20:38 |
| clarkb | I'm not trying to blame anyone or any software, I'm just following what the logs tell me | 20:38 |
| clarkb | the logs tell me that tripleo's python client behaves differently and that likely causes some of the request limiting we see | 20:39 |
| clarkb | (because without caching we'll make more requests) | 20:39 |
| mwhahaha | let me throw in a Cache-Control: public in and see if that addresses it | 20:39 |
| fungi | also our dockerclient read requests mostly aren't authenticated either, eight | 20:40 |
| fungi | s/eight/right/ ? | 20:40 |
| mwhahaha | all docker.io requests require auth. they might be doing something upfront | 20:40 |
| mwhahaha | well maybe not all, most | 20:40 |
| mwhahaha | so metadata and layer fetching does | 20:40 |
| mwhahaha | because cloudflare says nope if your token expires | 20:40 |
| clarkb | yes you have to "authenticate" as an anonymous user even if fetching anonymously | 20:40 |
| *** vishalmanchanda has quit IRC | 20:41 | |
| clarkb | this is also why I think things would work if we authenticated as a propre user (to avoid the rate limits entirely) | 20:43 |
| clarkb | but maybe it is sufficient to use morecached data | 20:43 |
| mwhahaha | https://review.opendev.org/#/c/743629/ | 20:44 |
| mwhahaha | that might address it | 20:44 |
| mwhahaha | though that might break something else | 20:45 |
| *** rlandy is now known as rlandy|brb | 20:45 | |
| mwhahaha | but i'll play with it | 20:45 |
| clarkb | I think you may only want to set it on the sha256 addressed entities | 20:45 |
| clarkb | because those will never change | 20:45 |
| clarkb | (and if they do we have a sha256 collision and bigger problems) | 20:45 |
| mwhahaha | right maybe just the layer request | 20:46 |
| *** evrardjp has joined #openstack-infra | 20:55 | |
| *** gyee has joined #openstack-infra | 21:02 | |
| *** rlandy|brb is now known as rlandy | 21:02 | |
| *** lxkong has joined #openstack-infra | 21:03 | |
| *** rlandy has quit IRC | 21:28 | |
| *** smarcet has joined #openstack-infra | 21:29 | |
| *** auristor has quit IRC | 21:32 | |
| *** slaweq has quit IRC | 21:34 | |
| erbarr | is there a way to pass 3 flat networks in devstack? | 21:41 |
| *** redrobot has joined #openstack-infra | 21:43 | |
| *** smarcet has left #openstack-infra | 21:43 | |
| *** auristor has joined #openstack-infra | 21:47 | |
| fungi | erbarr: you might have more luck asking in #openstack-qa where the devstack maintainers hang out | 21:48 |
| erbarr | fungi: cool, thanks! | 21:49 |
| *** xek_ has quit IRC | 22:03 | |
| gouthamr | hello o/ we've a gerrit issue that with a patch in manila that i need help with... | 22:05 |
| gouthamr | https://review.opendev.org/742289/ is a backport; and it was accidentally proposed twice (https://review.opendev.org/#/c/742290/1) - I abandoned the latter change and even tried modifying the Change-ID, but zuul is still unable to vote on it - it runs the jobs fine: https://zuul.opendev.org/t/openstack/buildsets?change=742289 | 22:05 |
| *** auristor has quit IRC | 22:06 | |
| *** d34dh0r53 has quit IRC | 22:09 | |
| *** rcernin has joined #openstack-infra | 22:09 | |
| fungi | i'm surprised gerrit allowed the same change-id to be pushed twice for the same project+branch | 22:12 |
| fungi | i didn't even know that was possible | 22:12 |
| gouthamr | fungi: an unfortunate, unhandled race condition perhaps | 22:15 |
| clarkb | I'm guessing you'll need a new change id | 22:16 |
| *** auristor has joined #openstack-infra | 22:18 | |
| gouthamr | clarkb: ack, i can do that and detach the cherry-pick from the parent | 22:19 |
| *** auristor has quit IRC | 22:20 | |
| *** rcernin has quit IRC | 22:22 | |
| gouthamr | i created a fresh cherry-pick: https://review.opendev.org/743640; i'll report back in case zuul has issues again | 22:24 |
| fungi | thanks, it seems like something i wouldn't have expected gerrit to allow (normally it would either amend the same change with a new patch set or reject the push) | 22:32 |
| fungi | if they were somehow pushed at the same moment then, yeah, maybe there's a race | 22:32 |
| gouthamr | yes, they were pushed within seconds of each other, from my IRC logs: | 22:36 |
| gouthamr | [2020-07-21T14:38:31-0700] <openstackgerrit> Tom Barron proposed openstack/manila stable/train: Update LVM volume extend https://review.opendev.org/742289 | 22:36 |
| gouthamr | [2020-07-21T14:38:32-0700] <openstackgerrit> Goutham Pacha Ravi proposed openstack/manila stable/train: Update LVM volume extend https://review.opendev.org/742290 | 22:36 |
| *** rcernin has joined #openstack-infra | 22:37 | |
| gouthamr | 1 second to be precise | 22:37 |
| openstackgerrit | Merged openstack/project-config master: Add pyca/project-config https://review.opendev.org/743143 | 22:46 |
| *** rcernin has quit IRC | 22:50 | |
| *** rcernin has joined #openstack-infra | 22:50 | |
| *** tkajinam has joined #openstack-infra | 22:52 | |
| *** dviroel has joined #openstack-infra | 22:53 | |
| clarkb | mwhahaha: I'm looking at the mirror and grepping for logs related to this job against your change https://zuul.opendev.org/t/openstack/stream/61aa12e9474840c8969fc8426541fa41?logfile=console.log I don't see the caching happening. So maybe there is more going on. Also another thing I notice is that the same blobs seem to be fetched multiple times by the same host under the same job, that may be | 23:05 |
| clarkb | something we can optimize too (iirc the docker client would find it locally and not refetch) | 23:05 |
| clarkb | might need to tcpdump -A a docker pull vs what your client is doing? | 23:06 |
| mwhahaha | Depends this isn't the same process as a docker pull. Additionally we should have some locking to prevent layers from being fetched excessively | 23:08 |
| *** zer0c00l has joined #openstack-infra | 23:10 | |
| mwhahaha | If you have layer blobs Isa we can probably coordinate them in the logs | 23:10 |
| clarkb | mwhahaha: 6af36aed3fd9edb878221b7f08c5bcf0b4375fe4f33d179ffaacded52c476fce a blob for centos-binary-memcached appears to have been fetched 5 times by one of the hosts running that job | 23:10 |
| clarkb | I was grepping for 'cache' which matched on the memcached portion of the url | 23:10 |
| clarkb | there may be others (or not) not sure yet | 23:10 |
| clarkb | I'm trying to actually look at cases for multiple fetches now | 23:12 |
| zer0c00l | This has been pending for a while https://review.opendev.org/#/c/720832/ | 23:15 |
| zer0c00l | Any chance i can get this reviewed? | 23:15 |
| zer0c00l | should be an easy one. | 23:15 |
| clarkb | mwhahaha: http://paste.openstack.org/show/796396/ | 23:18 |
| openstackgerrit | Merged openstack/project-config master: Add pyca 3rd-party CI tenant https://review.opendev.org/743144 | 23:19 |
| clarkb | zer0c00l: see my comment I believe a different change fixed it | 23:21 |
| *** piotrowskim has quit IRC | 23:22 | |
| *** dchen|away is now known as dchen | 23:34 | |
| *** hamalq has quit IRC | 23:48 | |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!