| *** tosky has quit IRC | 00:00 | |
| ianw | dmsimard: re our discussion; there's a few ara changes for stable in https://review.opendev.org/#/q/topic:s-c-ansible-devel-testing to bring stable/0.x support, it gets the jobs green and works with our system-config gate tests of -devel branch (https://review.opendev.org/716449) | 00:23 |
|---|---|---|
| ianw | infra-root: https://review.opendev.org/#/q/topic:s-c-ansible-devel-testing is a stack of changes to get the -devel job working again, related to ^ | 00:24 |
| *** DSpider has quit IRC | 00:27 | |
| kevinz | ianw: Hi morning! Sorry I was on PTO last Thursday and Friday | 02:11 |
| kevinz | ianw: Any problem with Linaro Cloud? | 02:11 |
| ianw | kevinz: hey, no worries! i was having trouble starting a new mirror host in the control plane tenant, didn't seem to be getting ipv4 | 02:11 |
| kevinz | o, let me check now | 02:11 |
| ianw | i want to replace the mirror with a focal node, we seem to still have the issues of the bionic node shutting down randomly | 02:11 |
| kevinz | ianw: sure, let me check the problem | 02:12 |
| ianw | i can start a node if you like, lmn | 02:12 |
| kevinz | OK, you can start a node. And I will check if the IPV4 pool is full | 02:13 |
| ianw | ok, it was getting an address, just no response | 02:13 |
| ianw | 139.178.85.140 is what it's been given | 02:14 |
| dmsimard | ianw: hey, I'll have a look when I have a chance | 02:16 |
| kevinz | ianw: you mean that it can be assigned a IPv4 ,but could not pingable from outside right? | 02:16 |
| dmsimard | oh right, 0.x has ansible as a dependency | 02:21 |
| dmsimard | for that config | 02:21 |
| ianw | kevinz: yeah, and it's not connecting out either | 02:36 |
| ianw | kevinz: ipv6 works | 02:36 |
| ianw | dmsimard: yeah, it's three hacks really :) are we the main users of the 0.x branch? | 02:38 |
| dmsimard | i replied on the patches, thanks for that | 02:41 |
| dmsimard | hard to keep track who is using it still, maybe osa/kolla/tripleo too, would need to check | 02:42 |
| dmsimard | their playbooks work with 1.x though: https://api.trunk.demo.recordsansible.org/?path=openstack-ansible | 02:44 |
| dmsimard | hard requirement on py3 hurt some adoption | 02:45 |
| dmsimard | maybe the new cli in 1.x will be enough to warrant upgrading :p | 02:46 |
| ianw | dmsimard: sorry if i'm out of the loop; it's mostly the static generation -- is there a path for that? | 02:48 |
| dmsimard | static generation is in 1.x but somewhat limited: no search or pagination | 02:49 |
| ianw | ahh, ok, well that's enough. i can look at upgrading then | 02:49 |
| dmsimard | this is static content: https://93b12f56c043f4c1608b-13b744a8e90ddbccef6c4bc9edcf0403.ssl.cf2.rackcdn.com/747538/1/check/ara-basic-ansible-devel/dde9313/server/static/ | 02:50 |
| dmsimard | ah looks like the links are broken :/ | 02:50 |
| ianw | dmsimard: do you mean put in a check if running with ansible 2.10? or just try to import it an catch the exception? | 02:51 |
| dmsimard | I haven't had to look at those in a bit, not sure if it's legit broken or an issue with swift | 02:51 |
| dmsimard | hmmm maybe either could work | 02:52 |
| dmsimard | I'd just rather not touch what is already working for previous versions | 02:52 |
| dmsimard | can add a if for 2.10 or try/catch for probably a similar result | 02:52 |
| dmsimard | getting late for me, see you later | 02:56 |
| ianw | dmsimard: NP, thanks for looking :) i updated 747337 to check for importerror | 02:57 |
| kevinz | ianw: could you re-triagger the creation focal node? | 03:41 |
| ianw | kevinz: sure! | 03:41 |
| kevinz | ianw: this time it worked | 03:43 |
| ianw | kevinz: ok, looking good :) | 03:43 |
| ianw | it's just about to test ipv6 | 03:43 |
| ianw | yep, working ... cool. | 03:44 |
| kevinz | I think the reason should be one L3-agent could not update the iptables.. | 03:44 |
| ianw | kevinz: i ended up creating a few because of system-config errors, i wonder if that broke something | 03:51 |
| ianw | on friday i mean | 03:51 |
| *** tkajinam has quit IRC | 04:03 | |
| *** tkajinam has joined #opendev | 04:03 | |
| *** ysandeep|away is now known as ysandeep | 04:14 | |
| *** ykarel|away has joined #opendev | 04:42 | |
| *** ykarel|away is now known as ykarel | 04:42 | |
| openstackgerrit | Ian Wienand proposed opendev/zone-opendev.org master: Add linaro mirror02 (focal mirror) https://review.opendev.org/747615 | 04:45 |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: Add linaro Focal mirror https://review.opendev.org/747617 | 05:00 |
| *** raukadah is now known as chandankumar | 05:45 | |
| ianw | infra-root: ^ host is up, with a 200g volume spit for apache/openafs caches, so those should be good. not doesn't switch the CNAME till we've verified it | 05:46 |
| *** elod_off is now known as elod | 06:00 | |
| *** yoctozepto has quit IRC | 06:10 | |
| *** yoctozepto has joined #opendev | 06:13 | |
| *** mnasiadka has quit IRC | 06:16 | |
| *** mnasiadka has joined #opendev | 06:19 | |
| *** seongsoocho has quit IRC | 06:24 | |
| *** seongsoocho has joined #opendev | 06:25 | |
| *** dtantsur|afk is now known as dtantsur | 06:35 | |
| *** AJaeger has quit IRC | 06:36 | |
| *** jhesketh_ has quit IRC | 06:39 | |
| *** jhesketh has joined #opendev | 06:41 | |
| *** AJaeger has joined #opendev | 06:45 | |
| *** sshnaidm|afk is now known as sshnaidm | 07:03 | |
| *** fressi has joined #opendev | 07:10 | |
| *** andrewbonney has joined #opendev | 07:11 | |
| *** AJaeger has quit IRC | 07:17 | |
| *** hashar has joined #opendev | 07:20 | |
| *** tosky has joined #opendev | 07:20 | |
| openstackgerrit | Sorin Sbarnea (zbr) proposed opendev/system-config master: Improved ask read-only message https://review.opendev.org/747631 | 07:24 |
| *** bolg has joined #opendev | 07:44 | |
| *** DSpider has joined #opendev | 07:57 | |
| *** moppy has quit IRC | 08:01 | |
| *** moppy has joined #opendev | 08:01 | |
| *** ysandeep is now known as ysandeep|lunch | 08:23 | |
| openstackgerrit | Albin Vass proposed zuul/zuul-jobs master: DNM: Add unified synchronize-repos role that works with linux and windows https://review.opendev.org/740005 | 08:25 |
| openstackgerrit | Albin Vass proposed zuul/zuul-jobs master: Update synchronize-repos https://review.opendev.org/740110 | 08:25 |
| openstackgerrit | Albin Vass proposed zuul/zuul-jobs master: synchronize-repos: Remove unecessary git path modifications https://review.opendev.org/747640 | 08:25 |
| *** auristor has quit IRC | 08:53 | |
| *** auristor has joined #opendev | 08:54 | |
| *** hashar has quit IRC | 09:12 | |
| *** ysandeep|lunch is now known as ysandeep | 09:12 | |
| openstackgerrit | Sorin Sbarnea (zbr) proposed opendev/gerritlib master: Fixed POLLIN event check https://review.opendev.org/729966 | 09:21 |
| *** priteau has joined #opendev | 09:24 | |
| openstackgerrit | Sorin Sbarnea (zbr) proposed zuul/zuul-jobs master: Enable linting of test-playbooks https://review.opendev.org/731471 | 09:25 |
| *** AJaeger has joined #opendev | 09:32 | |
| *** fressi has quit IRC | 10:07 | |
| *** fressi has joined #opendev | 10:10 | |
| *** fressi has quit IRC | 10:44 | |
| *** fressi has joined #opendev | 10:50 | |
| *** fressi has quit IRC | 11:35 | |
| *** fressi has joined #opendev | 11:36 | |
| *** ysandeep is now known as ysandeep|brb | 11:40 | |
| openstackgerrit | Sorin Sbarnea (zbr) proposed opendev/elastic-recheck master: Enable configuration via environment variables https://review.opendev.org/746890 | 11:54 |
| *** ysandeep|brb is now known as ysandeep | 11:54 | |
| openstackgerrit | Sorin Sbarnea (zbr) proposed opendev/elastic-recheck master: Enable configuration via environment variables https://review.opendev.org/746890 | 12:10 |
| zbr | clarkb: let me know when we can resume on e-r | 12:11 |
| *** hashar has joined #opendev | 12:49 | |
| *** owalsh has quit IRC | 13:15 | |
| *** owalsh has joined #opendev | 13:16 | |
| *** ykarel is now known as ykarel|away | 13:30 | |
| openstackgerrit | Lon Hohberger proposed openstack/diskimage-builder master: rhel-common: Provide method to select module streams https://review.opendev.org/747732 | 13:53 |
| openstackgerrit | Lon Hohberger proposed openstack/diskimage-builder master: rhel-common: Provide method to select module streams https://review.opendev.org/747732 | 13:56 |
| *** icey has quit IRC | 14:01 | |
| *** icey has joined #opendev | 14:08 | |
| *** ykarel|away has quit IRC | 14:25 | |
| *** ysandeep is now known as ysandeep|dinner | 14:38 | |
| *** chandankumar is now known as raukadah | 14:52 | |
| *** mlavalle has joined #opendev | 14:55 | |
| clarkb | zbr: I'm hoping today, but need to catch up on reviews and emails first. I figured I'd approve the puppet change then check that the install flipped over to python3 | 14:56 |
| *** qchris has quit IRC | 14:57 | |
| *** tkajinam_ has joined #opendev | 14:59 | |
| *** seongsoocho_ has joined #opendev | 15:01 | |
| *** dviroel_ has joined #opendev | 15:01 | |
| *** cloudnull7 has joined #opendev | 15:02 | |
| *** seongsoocho has quit IRC | 15:03 | |
| *** tkajinam has quit IRC | 15:03 | |
| *** dviroel has quit IRC | 15:03 | |
| *** cloudnull has quit IRC | 15:03 | |
| *** seongsoocho_ is now known as seongsoocho | 15:03 | |
| *** dviroel_ is now known as dviroel | 15:03 | |
| *** cloudnull7 is now known as cloudnull | 15:03 | |
| *** mlavalle has quit IRC | 15:03 | |
| *** mlavalle has joined #opendev | 15:03 | |
| *** zbr has quit IRC | 15:04 | |
| *** zbr has joined #opendev | 15:05 | |
| *** ildikov has quit IRC | 15:06 | |
| *** ildikov has joined #opendev | 15:08 | |
| *** qchris has joined #opendev | 15:10 | |
| clarkb | if anyone groks pandoc betterthan me, there is a thread about gitea's rst rendering on openstack-discuss. It drops out when hitting the zuul job role directives whereas github's rendering seems to just ignore the error and render them "raw" | 15:20 |
| clarkb | Our gitea is configured to use pandoc for the rendering and I wonder if there is an option we can set to have it ignore those unknown directives rather than giving up | 15:21 |
| openstackgerrit | Matthew Thode proposed openstack/diskimage-builder master: update various gentoo bits https://review.opendev.org/747502 | 15:21 |
| *** lseki has quit IRC | 15:22 | |
| *** donnyd has quit IRC | 15:22 | |
| *** hillpd has quit IRC | 15:22 | |
| *** priteau has quit IRC | 15:22 | |
| *** auristor has quit IRC | 15:22 | |
| *** andrewbonney has quit IRC | 15:22 | |
| *** mwhahaha has quit IRC | 15:22 | |
| *** mnaser has quit IRC | 15:22 | |
| *** ysandeep|dinner is now known as ysandeep | 15:27 | |
| *** mwhahaha has joined #opendev | 15:31 | |
| *** weshay|ruck has quit IRC | 15:31 | |
| *** weshay has joined #opendev | 15:32 | |
| *** lseki has joined #opendev | 15:33 | |
| *** donnyd has joined #opendev | 15:33 | |
| *** hillpd has joined #opendev | 15:33 | |
| *** priteau has joined #opendev | 15:34 | |
| *** auristor has joined #opendev | 15:34 | |
| *** andrewbonney has joined #opendev | 15:34 | |
| *** mnaser has joined #opendev | 15:34 | |
| *** fressi has quit IRC | 15:35 | |
| clarkb | rst2html.py does similar, but includes the error in the output document | 15:57 |
| *** weshay has quit IRC | 16:05 | |
| *** weshay has joined #opendev | 16:05 | |
| *** ysandeep is now known as ysandeep|away | 16:16 | |
| *** iurygregory has quit IRC | 16:18 | |
| *** dtantsur is now known as dtantsur|afk | 16:19 | |
| clarkb | looking at review-test we seem to have put review-site (and /home/gerrit2) on the root device. I think this may have been done to make snapshotting easy. Unfortuntely I think that won't lead to very accurate testing since production git repos are on a cinder volume and that is where notedb will be hosted | 16:24 |
| clarkb | are we able to snapshot a cinder volume? I think we can and that may be a better option for us to test similar "devices" | 16:24 |
| fungi | i'm totally on vacation, but couldn't you just attach and mount a cinder volume there? | 16:24 |
| clarkb | (though we're back to cinder v1 api on rax iirc) | 16:24 |
| clarkb | fungi: we can but what I'm not sure about is our snapshotting ability | 16:25 |
| fungi | ahh | 16:25 |
| clarkb | I also need to poke around the srever more and get a better feel for its general state | 16:25 |
| clarkb | a lot of the data seems to be there but gerrit isn't running | 16:25 |
| fungi | rsync data from rootfs to a cinder volume and then (bind)mount it into place before each test? that way you can just discard the content on it | 16:26 |
| clarkb | ya that may be the simple thing /me was pulling up cinder docs to check if snapshots there are viable | 16:26 |
| clarkb | cinder v2 and v3 do snapshotting | 16:27 |
| clarkb | v1 docs aren't even published (anymore) | 16:27 |
| *** iurygregory has joined #opendev | 16:27 | |
| clarkb | but also I'm thinking it might be easier to not try and keep this test server in full sync with the prod server | 16:27 |
| clarkb | we can build a point in time from $nowish, then that should be a reasonable enough approximation | 16:28 |
| clarkb | because the syncing adds another layer of complexity on top of the snapshotting | 16:28 |
| clarkb | if we snapshot then drift due to syncing wiht prod how do we reconcile that with a snapshot restore, etc | 16:28 |
| clarkb | fungi: while you arne't here, you did confirm that v1 volume api worked with rax right? | 16:30 |
| clarkb | did a change get pushed to encode that in our clouds.yaml? | 16:30 |
| *** hashar has quit IRC | 16:48 | |
| *** tosky has quit IRC | 16:49 | |
| fungi | i confirmed it worked in my homedir to remove the v2 overrides | 16:59 |
| *** andrewbonney has quit IRC | 17:08 | |
| *** cmurphy_afk is now known as cmurphy | 17:14 | |
| clarkb | anyone else want to review https://review.opendev.org/#/c/729336/11 to convert e-r to python3 on our deployment? I'm able to watch that go in (and revert/fix/etc if necessary) | 17:20 |
| openstackgerrit | Sorin Sbarnea (zbr) proposed zuul/zuul-jobs master: bindep: Add missing virtualenv and fixed repo install https://review.opendev.org/693637 | 17:29 |
| openstackgerrit | Sorin Sbarnea (zbr) proposed zuul/zuul-jobs master: bindep: Fixed runtime warnings https://review.opendev.org/747781 | 17:33 |
| openstackgerrit | Sorin Sbarnea (zbr) proposed zuul/zuul-jobs master: bindep: install packages one by one https://review.opendev.org/693637 | 17:37 |
| openstackgerrit | Sorin Sbarnea (zbr) proposed openstack/diskimage-builder master: Validate virtualenv and pip https://review.opendev.org/707104 | 17:54 |
| openstackgerrit | Merged zuul/zuul-jobs master: ara-report: add option for artifact prefix https://review.opendev.org/747100 | 19:05 |
| openstackgerrit | Merged zuul/zuul-jobs master: tox: include command output in log/error https://review.opendev.org/747325 | 19:05 |
| *** priteau has quit IRC | 19:31 | |
| fungi | infra-root: rackspace opened a ticket to let us know there was a problem with the host for zm07 and that it's been rebooted | 19:39 |
| clarkb | the server is up and zuul-merger is running on it | 19:40 |
| fungi | good deal | 19:40 |
| openstackgerrit | Merged opendev/puppet-elastic_recheck master: Use py3 with elastic-recheck https://review.opendev.org/729336 | 19:49 |
| openstackgerrit | Brian Rosmaita proposed opendev/system-config master: Turn off rendering of RST files by default https://review.opendev.org/747796 | 20:48 |
| clarkb | ok e-r hasn't updated its install because the install resource is subscribed to the git repo and is subscribe only | 20:49 |
| clarkb | zbr: ^ that means if you land an e-r change the next hourly puppet run should do it | 20:49 |
| openstackgerrit | Merged opendev/infra-specs master: Central Authentication Service https://review.opendev.org/731838 | 21:05 |
| clarkb | ianw: when your day starts I wanted to ask about my dib package list parsing change | 21:36 |
| clarkb | (mostly if there are any concerns wit hthat approach) | 21:36 |
| ianw | clarkb: sorry, yeah will review | 22:15 |
| ianw | i don't think so | 22:15 |
| clarkb | ianw: I reviewed your ansible-devel stack and left a couple comments on one or two of them | 22:18 |
| prometheanfire | ianw: is debian-minimal considered 'maintained' (as opposed to ironic agent) I ask because it sems to not install what's needed to add keys | 22:28 |
| prometheanfire | 2020-08-24 22:28:09.535 | dib-run-parts Running /tmp/dib_build.trgNuKIW/hooks/root.d/75-debian-minimal-baseinstall | 22:28 |
| prometheanfire | 2020-08-24 22:28:09.624 | E: gnupg, gnupg2 and gnupg1 do not seem to be installed, but one of them is required for this operation | 22:28 |
| ianw | clarkb: thanks, will go over it | 22:28 |
| ianw | promethenaifre: hrm, well we build it in the gate, so ... yeah? | 22:29 |
| ianw | what release are you seeing that for? | 22:29 |
| prometheanfire | buster | 22:29 |
| clarkb | we do install gnupg2 in infra-package-needs | 22:29 |
| prometheanfire | I think the install is done too late | 22:30 |
| clarkb | I think debootstrap dropped it from its default list | 22:30 |
| clarkb | so we just add it on after? | 22:30 |
| prometheanfire | ah | 22:30 |
| prometheanfire | https://github.com/openstack/diskimage-builder/blob/master/diskimage_builder/elements/debian-minimal/root.d/75-debian-minimal-baseinstall adds keys at the start | 22:30 |
| prometheanfire | installs gpg at the end | 22:30 |
| prometheanfire | some fun ordering issues there | 22:30 |
| * prometheanfire would just like a way to add it to debootstrap | 22:31 | |
| prometheanfire | DIB_DEBOOTSTRAP_EXTRA_ARGS probably? | 22:31 |
| clarkb | /opt/dib_tmp/dib_build.l7lqZiFA/hooks/root.d/75-debian-minimal-baseinstall:main:78 : sudo chroot /opt/dib_tmp/dib_build.l7lqZiFA/mnt /usr/bin/apt-get install -y systemd-sysv busybox sudo gnupg2 python3 <- that is when we install it according to our logs | 22:32 |
| clarkb | ah we use our mirrors in our builds | 22:33 |
| clarkb | and we disable verification with those. I bet that explains how it got through | 22:33 |
| prometheanfire | my issue is an interaction between adding a external repo with an apt key also needed | 22:34 |
| prometheanfire | trying with export DIB_DEBOOTSTRAP_EXTRA_ARGS='--include=gpg' | 22:35 |
| clarkb | prometheanfire: debian installs gpgv in debootstrap | 22:35 |
| clarkb | which is a minimal gpg version used only to verify signatures | 22:35 |
| clarkb | I think we can just do debootstrap then install gpg after? | 22:35 |
| clarkb | where is it failing for you? (since I think this is what we do) | 22:36 |
| prometheanfire | clarkb: part of 75-debian-minimal-baseinstall is to add the keys | 22:36 |
| prometheanfire | https://github.com/openstack/diskimage-builder/blob/master/diskimage_builder/elements/debian-minimal/root.d/75-debian-minimal-baseinstall#L28-L30 | 22:36 |
| prometheanfire | not just verification | 22:36 |
| clarkb | we don't see mto run that (probably because we don't have gpg keys to verify with the debootstrap step) | 22:38 |
| prometheanfire | ya | 22:38 |
| clarkb | now that said, I think you can just copy the files in place with modern apt | 22:38 |
| clarkb | so rather than run apt-key add - we can copy the contents instead? | 22:39 |
| prometheanfire | adding ya, probably better to change how that's done, iirc both stretch and buster can do it | 22:39 |
| clarkb | that may also be why gpg proper was dropped as a dep | 22:39 |
| prometheanfire | I'll play with it later tonight and make a review (unless someone beats me to it) | 22:40 |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: tox: drop test-requirements.txt https://review.opendev.org/747328 | 22:43 |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: Collect tox logs https://review.opendev.org/747329 | 22:43 |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: Fix ansible-devel job for Ansible 2.10 changes https://review.opendev.org/716449 | 22:43 |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: Ansible devel testing: install ansible-collections from checkout https://review.opendev.org/747597 | 22:43 |
| ianw | ahh, that would make sense, yes we probably do use our mirrors everywhere and thus avoid gpg errors | 22:44 |
| ianw | clarkb: you ok with https://review.opendev.org/747615 to add the linaro mirror02 dns? sorry should have been a deps on, need it for LE | 22:48 |
| clarkb | ianw: yup +2. That reminds me did the launch node script get updated to exclude dsa keys properly | 22:52 |
| * clarkb tries to find that change again | 22:52 | |
| clarkb | it did https://review.opendev.org/#/c/744821/4 if anyone else wants to review that one | 22:53 |
| ianw | clarkb: yeah, it did. i tried it out, and then realised it needs one other thing too ... | 22:53 |
| ianw | because we were re-using ip addresses, the no verify key didn't work, also need to set the hosts file to null | 22:54 |
| ianw | then i wonder though, what wins between sshfp records and known_hosts? | 22:54 |
| clarkb | oh interesting | 22:54 |
| ianw | are we aware of the botocore install errors | 22:54 |
| clarkb | I'm not aware of botocore issues (in what context?) | 22:55 |
| ianw | sorry, zuul-jobs is where i'm seeing it. wasn't sure if was a global breakage ... investigating | 22:55 |
| ianw | 1.14.48 of boto3 released about 4 hours ago | 22:56 |
| ianw | "ERROR: No matching distribution found for botocore<1.18.0,>=1.17.48 (from boto3->-r /home/zuul/src/opendev.org/zuul/zuul-jobs/test-requirements.txt (line 24))" | 22:59 |
| ianw | https://236fa39aba5b694d3ffa-c578cacb39dd1edf606b634ec77d1998.ssl.cf1.rackcdn.com/746423/11/gate/tox-py35/7885730/job-output.txt | 22:59 |
| *** mlavalle has quit IRC | 23:00 | |
| clarkb | https://pypi.org/project/botocore/1.17.48/ seems to be the latest version | 23:00 |
| clarkb | but it isn't at https://mirror.mtl01.inap.opendev.org/pypi/simple/botocore/ | 23:00 |
| clarkb | https://pypi.org/simple/botocore/ does have it | 23:01 |
| clarkb | stale cdn nodes again? | 23:01 |
| clarkb | https://mirror.dfw.rax.opendev.org/pypi/simple/botocore/ has it so ya I think this must be the CDN acting up | 23:01 |
| ianw | :/ | 23:02 |
| ianw | do i remember correctly fungi manually clearing caches? | 23:03 |
| clarkb | fungi did a bunch of wget/curls to see if the problem was persistent last week iir | 23:03 |
| clarkb | we only cache those indexes for 10 minutes | 23:03 |
| clarkb | I think the hope was to identify a specific backend that was bad but I don't think that happened | 23:04 |
| openstackgerrit | Ian Wienand proposed zuul/zuul-jobs master: Add ensure-rust role https://review.opendev.org/746423 | 23:05 |
| ianw | let's see if ^ goes better ... | 23:06 |
| openstackgerrit | Merged opendev/zone-opendev.org master: Add linaro mirror02 (focal mirror) https://review.opendev.org/747615 | 23:11 |
| *** elod has quit IRC | 23:18 | |
| *** elod has joined #opendev | 23:20 | |
| *** iurygregory has quit IRC | 23:21 | |
| openstackgerrit | Merged zuul/zuul-jobs master: Add ensure-rust role https://review.opendev.org/746423 | 23:34 |
| ianw | for reference, the MTL cache is definitely getting a different result for the pypi index page for botocore | 23:36 |
| ianw | "Update - We're currently investigating performance issues with our URL and Surrogate Key purging services. Purge All and all other services are unaffected. " ... dunno what that means | 23:38 |
| clarkb | https://www.docker.com/blog/scaling-docker-to-serve-millions-more-developers-network-egress/ | 23:39 |
| clarkb | we're likely going to need to rethink docker image pulls | 23:39 |
| ianw | yeah i saw that ... we should be able to heavily proxy it though? | 23:40 |
| clarkb | maybe, all requests to docker hub are authenticated | 23:40 |
| clarkb | and apache doesn't cache any of those authenticated requests (even though it ca) | 23:40 |
| clarkb | that likely means we need to look at a different caching tool, something which will cache authenticated requests if they are marked public content | 23:40 |
| clarkb | also interesting in there they rate limited blob requests not manifests requests, what we current cache ar ethe blob requests so would hopefully be in a better spot but they are changing it to rate limit manifests (arg!) | 23:41 |
| clarkb | "Stay tuned in the coming weeks for a blog post about configuring CI and production systems in light of these changes." | 23:43 |
| clarkb | I guess we wait for their thoughts on CI in particular and go from there | 23:43 |
| clarkb | there will also be open source plans which we may want to sign up for opendev and zuul | 23:43 |
| clarkb | corvus: ^ fyi | 23:43 |
| corvus | i guess they are reconciling their accounts payable with their business model | 23:43 |
| clarkb | I think my only real gripe with the changes are the move from blob to manifest based limits | 23:44 |
| clarkb | we cache the blobs which are the expensive bits already but will be penalized for trying to be good citiizens upfront | 23:44 |
| clarkb | fwiw cache-control: public is the header specifier that indicates you can cache a request that was requested with an authorization header | 23:45 |
| clarkb | apache seems to ignore that. I assume there exists some cache out there that doesn't | 23:45 |
| clarkb | (docker hub does seem to set that header properly too) | 23:46 |
| clarkb | also we can just drop the caching entirely and as long as IPs don't get recycled too often probably be fine | 23:47 |
| * clarkb hopes the post for CI has some better ideas | 23:47 | |
| corvus | clarkb: it is a distributed ci system ;) | 23:47 |
| ianw | just logging for reference but here are the mtl headers giving bad botocore results ATM on mtl01.inap http://paste.openstack.org/show/797099/ | 23:47 |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!