| opendevreview | Merged openstack/diskimage-builder master: update gentoo source suffix (where it finds the file to download) https://review.opendev.org/c/openstack/diskimage-builder/+/818678 | 00:41 |
|---|---|---|
| ianw | the bootloader is wrong | 01:00 |
| ianw | i don't really understand how this passed testing | 01:00 |
| opendevreview | Ian Wienand proposed openstack/diskimage-builder master: [wip] fix f35 bootloader setup https://review.opendev.org/c/openstack/diskimage-builder/+/818708 | 04:24 |
| opendevreview | Ian Wienand proposed openstack/diskimage-builder master: [wip] fix f35 bootloader setup https://review.opendev.org/c/openstack/diskimage-builder/+/818708 | 04:41 |
| opendevreview | Ian Wienand proposed openstack/diskimage-builder master: bootloader: specify distros for grubby install https://review.opendev.org/c/openstack/diskimage-builder/+/818710 | 05:16 |
| *** ysandeep|out is now known as ysandeep | 05:42 | |
| ianw | well i'm stumpted. https://zuul.opendev.org/t/openstack/build/aaf6fa41e8f94c98887e76915e17b515 shows that the gate tests are setting "root=LABEL=cloudimg-fs" (and, ergo, why f35 boots in the gate) | 07:36 |
| ianw | the build logs https://nb01.opendev.org/fedora-35-0000000008.log and https://dd25c8fff5148bff9a3d-e4098ecd70541ccfec6de5edaad12e3e.ssl.cf5.rackcdn.com/818708/2/check/dib-nodepool-functional-openstack-fedora-35-containerfile-src/aaf6fa4/nodepool/builds/test-image-0000000001.log look the same | 07:37 |
| ianw | but the gate builds have images with "root=UUID=<wrong-uuid>" | 07:38 |
| ianw | i think that https://review.opendev.org/c/openstack/diskimage-builder/+/818710 is generally correct, in making the image use grubby to setup the bootloader | 07:39 |
| ianw | but it's unsatisfying to not have an exact root-cause for the production failure | 07:39 |
| opendevreview | Ian Wienand proposed openstack/diskimage-builder master: bootloader: specify distros for grubby install https://review.opendev.org/c/openstack/diskimage-builder/+/818710 | 07:43 |
| *** ykarel is now known as ykarel|lunch | 07:53 | |
| opendevreview | Albin Vass proposed zuul/zuul-jobs master: Make prepare-workspace-git root configurable https://review.opendev.org/c/zuul/zuul-jobs/+/818716 | 08:02 |
| *** ysandeep is now known as ysandeep|lunch | 08:18 | |
| opendevreview | wangxiyuan proposed openstack/project-config master: Add openEuler 20.03 LTS SP2 node https://review.opendev.org/c/openstack/project-config/+/818723 | 09:10 |
| *** ysandeep|lunch is now known as ysandeep | 09:24 | |
| *** ildikov_ is now known as ildikov | 09:30 | |
| *** simondodsley_ is now known as simondodsley | 09:31 | |
| *** TheJulia_ is now known as TheJulia | 09:32 | |
| *** ykarel|lunch is now known as ykarel | 09:53 | |
| *** odyssey4me is now known as Guest6506 | 10:03 | |
| *** rlandy is now known as rlandy|ruck | 11:16 | |
| *** ajya[m] is now known as ajya | 12:52 | |
| fungi | rackspace says logstash-worker12 had to be rebooted roughlt 17.5 hours ago due to a host problem | 13:53 |
| fungi | looks like its four jenkins-log-worker started successfully at boot though | 13:54 |
| *** ykarel is now known as ykarel|away | 14:29 | |
| sshnaidm | fungi, clarkb hey, just for clarification - we don't have Ansible Galaxy mirror or cache somewhere, right? | 14:46 |
| fungi | sshnaidm: i don't know of one... does galaxy have its own site which serves roles? | 14:53 |
| sshnaidm | https://galaxy.ansible.com | 14:57 |
| sshnaidm | it's the main one | 14:57 |
| fungi | what protocol is used to fetch roles from it? | 15:01 |
| fungi | is it just normal http gets? | 15:01 |
| fungi | looks like the url for the ansible-galaxy cli is overridable, so in theory we could point it to a caching proxy: https://docs.ansible.com/ansible/latest/galaxy/user_guide.html?extIdCarryOver=true&sc_cid=701f2000001OH6uAAG#configuring-the-ansible-galaxy-client | 15:07 |
| fungi | the bigger question is whether galaxy.ansible.com is just a frontend/indexing service, or actually hosts the data for which we'd need to proxy requests | 15:08 |
| fungi | for example, we cache pypi.org but it really only serves indices, all the packages are served from files.pythonhosted.org instead | 15:09 |
| fungi | so we have to cache that as well | 15:09 |
| sshnaidm | I think galaxy.ansible.com actually hosts this content | 15:10 |
| sshnaidm | and it's just HTTPS protocol | 15:10 |
| fungi | i know at one point it only served as a pointer to things in github, but i guess they improved the design | 15:11 |
| sshnaidm | fungi, the link looks like: https://galaxy.ansible.com/download/community-general-4.0.2.tar.gz | 15:11 |
| fungi | oh, cool, so roles are bundled into compressed tarballs, that does probably make this simpler | 15:12 |
| sshnaidm | yeah, it's possible to clone from git repo, from tarball, from local directory, from ansible galaxy server | 15:13 |
| opendevreview | Jeremy Stanley proposed opendev/system-config master: Cache Ansible Galaxy on CI mirror servers https://review.opendev.org/c/opendev/system-config/+/818787 | 15:23 |
| fungi | sshnaidm: ^ does that make sense? | 15:23 |
| *** ysandeep is now known as ysandeep|out | 15:26 | |
| sshnaidm | fungi, I think it totally does.. I'd like to test | 15:27 |
| fungi | well, there's a test included which will at least make sure it's proxying the main page, but yeah we'll probably have to do some more extensive testing to confirm downloads really end up going through the proxy when told to | 15:28 |
| fungi | sshnaidm: bad news, i'll wip it... | 15:29 |
| fungi | Connecting to ansible-galaxy.s3.amazonaws.com (ansible-galaxy.s3.amazonaws.com)|52.216.84.40|:443... connected. | 15:29 |
| sshnaidm | fungi, yeah, just noticed | 15:29 |
| fungi | so the tarballs aren't served from galaxy.ansible.com, they're served from amazon sq | 15:29 |
| sshnaidm | it's hosted on aws | 15:29 |
| fungi | er, s3 | 15:29 |
| fungi | yeah, i'll see if i can adjust the config to cover that | 15:30 |
| sshnaidm | it looks like https://ansible-galaxy.s3.amazonaws.com/artifact/bc/fe717cf4cb44..... | 15:30 |
| sshnaidm | so to cache "https://ansible-galaxy.s3.amazonaws.com/artifact/" ? | 15:30 |
| opendevreview | Jeremy Stanley proposed opendev/system-config master: Cache Ansible Galaxy on CI mirror servers https://review.opendev.org/c/opendev/system-config/+/818787 | 15:40 |
| fungi | that seems to be consistent with how we handle similar situations for other proxies | 15:40 |
| fungi | we basically shadow a "ansible-galaxy.s3.amazonaws.com" directory at galaxy.ansible.com and then proxy it to s3, and rewrite the remote url in responses so that it looks like a directory on the proxy site instead | 15:41 |
| sshnaidm | yeah | 15:42 |
| sshnaidm | looks good | 15:42 |
| sshnaidm | rlandy|ruck, marios fyi ^ | 15:42 |
| fungi | so as long as https://galaxy.ansible.com/ansible-galaxy.s3.amazonaws.com/ is never actually used for anything, that shouldn't cause problems | 15:42 |
| fungi | and that was we can easily expand it if they use more than one storage backend (like the quay proxy does) | 15:43 |
| fungi | er, that way we can | 15:43 |
| sshnaidm | nice | 15:44 |
| rlandy|ruck | fungi: thanks for the help here | 15:44 |
| opendevreview | Arx Cruz proposed opendev/elastic-recheck rdo: Fix ER bot to report back to gerrit with bug/error report https://review.opendev.org/c/opendev/elastic-recheck/+/805638 | 15:45 |
| fungi | i'm trying to think up a good way to test the recursion stays on the proxy base url | 15:45 |
| opendevreview | Jeremy Stanley proposed opendev/system-config master: Cache Ansible Galaxy on CI mirror servers https://review.opendev.org/c/opendev/system-config/+/818787 | 15:53 |
| fungi | i think that ought to confirm it | 15:53 |
| Clark[m] | fungi: is there a reason you don't host it on 443 like pypi? | 16:16 |
| Clark[m] | Also fwiw we've used the zuul connection cache to cache git repos for things like roles in the past. Not sure we want to keep doing that particularly with GitHub impacting zuul restarts due to throttling | 16:17 |
| marios | thanks sshnaidm fungi | 16:18 |
| fungi | Clark[m]: oh, good point, i guess we can just tell ansible to use a namespaced path | 16:19 |
| fungi | i didn't consider it | 16:19 |
| fungi | i'll revise | 16:19 |
| fungi | Clark[m]: as for using git, apparently ansible galaxy is all compressed tarballs now not just pointers to git repos? | 16:19 |
| Clark[m] | It does both I think. You can consume the roles from a git url if you choose or via the Galaxy tarball | 16:20 |
| sshnaidm | Clark[m], that's right, but currently galaxy provides tarballs from aws s3 | 16:21 |
| sshnaidm | collections can be installed both from git, local dir, tarball, galaxy.. | 16:22 |
| sshnaidm | maybe even worth to have a zuul role that downloads configured collections from galaxy before the job? | 16:23 |
| sshnaidm | and then in job you can install them from local dir, for example | 16:23 |
| Clark[m] | Yes that's basically what we already do. You add it to zuul's project list then require the repo in your job | 16:24 |
| opendevreview | Jeremy Stanley proposed opendev/system-config master: Cache Ansible Galaxy on CI mirror servers https://review.opendev.org/c/opendev/system-config/+/818787 | 16:34 |
| opendevreview | Jeremy Stanley proposed opendev/system-config master: Cache Ansible Galaxy on CI mirror servers https://review.opendev.org/c/opendev/system-config/+/818787 | 16:36 |
| *** migarcia is now known as migarcia|away | 16:37 | |
| *** migarcia|away is now known as migarcia | 16:37 | |
| sshnaidm | fungi, but it won't work when galaxy server will be down, right? | 16:41 |
| fungi | sshnaidm: it may work briefly if we have it cached | 16:41 |
| fungi | but as soon as caches expire it will return failures, sure | 16:41 |
| sshnaidm | fungi, but someone need to redirect to aws..? | 16:42 |
| fungi | the redirects are cached | 16:42 |
| sshnaidm | oh, that's great | 16:42 |
| fungi | but also they don't redirect to aws, they simply redirect back to the proxy because the s3 urls are rewritten | 16:42 |
| fungi | that's the ProxyPassReverse "/galaxy-s3/" "https://ansible-galaxy.s3.amazonaws.com/" | 16:43 |
| sshnaidm | I see, awesome | 16:43 |
| fungi | it replaces "https://ansible-galaxy.s3.amazonaws.com/" in the responses with "/galaxy-s3/" | 16:43 |
| fungi | so it will be https://mirror.whatever.opendev.org/galaxy-s3/artifact/... | 16:44 |
| sshnaidm | great, probably worth to have a big cache time :) | 16:44 |
| *** sshnaidm is now known as sshnaidm|afk | 16:45 | |
| fungi | but like clarkb pointed out, if those roles/collections area available from git repositories then setting them as required-projects for your job will be even more stable since zuul keeps a local cache of git repos and pushes them onto the job node | 16:45 |
| fungi | and then you can also depends-on pull requests for those roles, in theory | 16:46 |
| fungi | anyway, assuming the test in included in 818787 is sane, it should prove the download urls are being properly forced through the proxy | 16:49 |
| jrosser | i'm unclear if we're talking about collections developed on gerrit/opendev or more generally collections whose source is in git(hub)..... | 16:49 |
| jrosser | in practive you'll always need both | 16:49 |
| fungi | jrosser: yes | 16:49 |
| fungi | our zuul has a connection to github and can use repositories there | 16:49 |
| jrosser | for context, OSA moved very early to ansible-base + collections and almost straight away we migrated to install from github rather than the published ones on galaxy | 16:50 |
| jrosser | as the API/whatever was way too unreliable for use in CI | 16:50 |
| jrosser | if there is a way to optimise that and use local copies it would be interesting | 16:50 |
| fungi | if the galaxy api is super unstable, then yes the proxy will only hide some of it | 16:50 |
| fungi | jrosser: looks like we already consume a few collections that way... https://opendev.org/openstack/project-config/src/branch/master/zuul/main.yaml#L1449-L1452 | 16:54 |
| jrosser | oh interesting, that covers a bunch of it actually https://opendev.org/openstack/openstack-ansible/src/branch/master/ansible-collection-requirements.yml | 16:55 |
| fungi | if you look a few lines past there, we also have repos for some roles hosted in github in namespaces like infrawatch and stackhpc | 16:55 |
| fungi | so yes, if you add those to a job's required-projects list then you can find them checked out on disk in the job | 16:56 |
| fungi | if you're truly testing the integration of those roles and collections then that's probably a better solution. on the other hand if you're testing automation which fetches things from galaxy then the caching proxy is probably better | 16:58 |
| *** marios is now known as marios|out | 16:58 | |
| opendevreview | daniel.pawlik proposed openstack/project-config master: Add OpenStack CI Log Processing project in infra docs https://review.opendev.org/c/openstack/project-config/+/818811 | 17:11 |
| *** sshnaidm|afk is now known as sshnaidm | 17:16 | |
| sshnaidm | fungi, can we add additional collections to https://opendev.org/openstack/project-config/src/branch/master/zuul/main.yaml#L1449-L1452 ? So that we can clone them as "required-projects" | 17:16 |
| fungi | sshnaidm: sure, that's what the feature's for | 17:17 |
| opendevreview | Jeremy Stanley proposed opendev/system-config master: Cache Ansible Galaxy on CI mirror servers https://review.opendev.org/c/opendev/system-config/+/818787 | 17:19 |
| sshnaidm | fungi, but it will be always latest code, right? | 17:21 |
| sshnaidm | I think we'd better stick to some known good version | 17:21 |
| Clark[m] | It will be the entire git repo so you can checkout any ref or tag you like | 17:22 |
| Clark[m] | By default it will be up to date | 17:22 |
| sshnaidm | ack | 17:23 |
| fungi | yes, what clarkb said | 17:25 |
| fungi | it can be a specific branch or a specific tag or a specific commit, anything that repo has you can check out explicitly on the job node | 17:26 |
| fungi | it'll be as if you had just cloned the repo at the start of the build | 17:26 |
| fungi | clarkb: were you suggesting i put it on 80/443 or on 8080/4443? it looks like you suggested the former, but i get the impression that's just for backward compatibility with when we used to host an actual pypi mirror and the latter is preferred? | 17:32 |
| sshnaidm | it's interesting: when you install a collection from repo, but it has dependency on other collection - the deps will be downloaded from galaxy. You need to clone all deps and explicitly install them before or together with the desired collection.. | 17:37 |
| opendevreview | Shnaidman Sagi (Sergey) proposed openstack/project-config master: Add additional collections to projects https://review.opendev.org/c/openstack/project-config/+/818813 | 17:42 |
| sshnaidm | fungi, Clark[m] ^ | 17:42 |
| Clark[m] | fungi: I think we split out to 4443 when we were worried it would break stuff on 443 to add proxying. But now we know it's fine so can prefer the default port | 18:08 |
| Clark[m] | Then docker and some other things just don't work on a subroot so have to be at / | 18:08 |
| fungi | okay, in that case 818787 is probably ready | 18:09 |
| fungi | though i have test failures to sort out still | 18:09 |
| fungi | so i guess not entirely ready | 18:09 |
| *** lucasagomes_ is now known as lucasagomes | 18:22 | |
| opendevreview | Jeremy Stanley proposed opendev/system-config master: Cache Ansible Galaxy on CI mirror servers https://review.opendev.org/c/opendev/system-config/+/818787 | 18:50 |
| sshnaidm | Clark[m], can we +w https://review.opendev.org/c/openstack/project-config/+/818813 ? | 19:16 |
| *** sshnaidm is now known as sshnaidm|afk | 19:26 | |
| Clark[m] | sshnaidm: I'm not at a computer with credentials to vote today but looks like it is just lacking approval if mordred or fungi want to +A it | 19:35 |
| fungi | config-core: ^ | 19:36 |
| fungi | i can single-core approve it if needed though | 19:36 |
| mordred | I +2d | 19:37 |
| fungi | thanks, i'll approve | 19:37 |
| opendevreview | Merged openstack/project-config master: Add additional collections to projects https://review.opendev.org/c/openstack/project-config/+/818813 | 19:44 |
| opendevreview | Jeremy Stanley proposed opendev/system-config master: Create a new lists.openinfra.dev mailing list site https://review.opendev.org/c/opendev/system-config/+/818826 | 19:48 |
| opendevreview | Ian Wienand proposed openstack/diskimage-builder master: bootloader: specify distros for grubby install https://review.opendev.org/c/openstack/diskimage-builder/+/818710 | 20:06 |
| ianw | fungi: if you have a sec https://review.opendev.org/c/openstack/project-config/+/818278 will help move my github branch of dstat_graph into an opendev project | 20:06 |
| fungi | oh, yep! | 20:07 |
| opendevreview | Merged openstack/project-config master: Add dstat_graph project https://review.opendev.org/c/openstack/project-config/+/818278 | 20:20 |
| *** ysandeep|out is now known as ysandeep | 23:38 | |
| *** rlandy|ruck is now known as rlandy|out | 23:44 | |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!