| *** DSpider has quit IRC | 00:23 | |
| clarkb | sr | 00:30 |
|---|---|---|
| clarkb | derp | 00:30 |
| ianw | you know we don't actually have centos-7 arm64 nodes. that's going to be a bummer for the new wheel jobs that will fail on that | 00:35 |
| ianw | i do not know why i didn't think of that before now | 00:35 |
| openstackgerrit | Merged openstack/diskimage-builder master: Add dependency on yamllint https://review.opendev.org/731245 | 00:38 |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: Prepare to drop Python 2 support https://review.opendev.org/731378 | 00:47 |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: Prepare to drop Python 2 support https://review.opendev.org/731378 | 00:51 |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: Drop support for python2 https://review.opendev.org/728889 | 00:51 |
| *** Meiyan has joined #opendev | 00:56 | |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: Prepare to drop Python 2 support https://review.opendev.org/731378 | 01:04 |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: Drop support for python2 https://review.opendev.org/728889 | 01:04 |
| fungi | ianw: d'oh, right... hrw were you planning to try to add centos-7-arm64 nodes? | 01:10 |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: Prepare to drop Python 2 support https://review.opendev.org/731378 | 01:10 |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: Drop support for python2 https://review.opendev.org/728889 | 01:10 |
| ianw | it hasn't been gate tested | 01:10 |
| openstackgerrit | Ian Wienand proposed openstack/project-config master: wheel-build: remove centos-7-arm jobs https://review.opendev.org/731380 | 01:14 |
| ianw | fungi: ^ | 01:14 |
| ianw | we should also run a requirements check | 01:15 |
| ianw | a check in requirements i should say | 01:15 |
| *** ysandeep|away is now known as ysandeep | 01:16 | |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: Prepare to drop Python 2 support https://review.opendev.org/731378 | 01:28 |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: Drop support for python2 https://review.opendev.org/728889 | 01:28 |
| *** sshnaidm|afk is now known as sshnaidm|off | 01:34 | |
| ianw | error: Curl error (28): Timeout was reached for http://mirror.regionone.linaro-us.opendev.org/centos/8/AppStream/aarch64/os/repodata/repomd.xml [Operation too slow. Less than 1000 bytes/sec transferred the last 30 seconds] | 01:37 |
| *** mlavalle has quit IRC | 01:37 | |
| ianw | that's probably not a great start for the wheel build jobs | 01:37 |
| *** cloudnull has quit IRC | 01:43 | |
| *** cloudnull has joined #opendev | 01:51 | |
| openstackgerrit | Ian Wienand proposed openstack/project-config master: build-wheels: ensure virtualenv https://review.opendev.org/731387 | 02:01 |
| openstackgerrit | Merged openstack/diskimage-builder master: use stage3 instead of stage4 for gentoo builds https://review.opendev.org/717177 | 02:02 |
| openstackgerrit | Merged openstack/diskimage-builder master: Pre-install xz package in opensuse chroot https://review.opendev.org/730992 | 02:06 |
| ianw | so at this point, i'm not aware of any current platform issues with dib. so this should be a nice place to drop py2 support | 02:07 |
| *** redrobot has quit IRC | 02:22 | |
| openstackgerrit | Ian Wienand proposed openstack/project-config master: build-wheels: ensure virtualenv https://review.opendev.org/731387 | 02:39 |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: Prepare to drop Python 2 support https://review.opendev.org/731378 | 02:45 |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: Drop support for python2 https://review.opendev.org/728889 | 02:45 |
| openstackgerrit | Ian Wienand proposed openstack/project-config master: build-wheels: ensure virtualenv https://review.opendev.org/731387 | 02:54 |
| openstackgerrit | Merged openstack/project-config master: wheel-build: remove centos-7-arm jobs https://review.opendev.org/731380 | 02:57 |
| *** Eighth_Doctor is now known as Conan_Kudo | 03:04 | |
| *** Conan_Kudo is now known as Eighth_Doctor | 03:04 | |
| openstackgerrit | Ian Wienand proposed openstack/project-config master: Move wheel build role/job outside project-config https://review.opendev.org/731401 | 03:32 |
| openstackgerrit | Ian Wienand proposed openstack/project-config master: Move wheel build role/job outside project-config https://review.opendev.org/731401 | 03:49 |
| openstackgerrit | Ian Wienand proposed openstack/project-config master: Move wheel build role/job outside project-config https://review.opendev.org/731401 | 03:50 |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: Prepare to drop Python 2 support https://review.opendev.org/731378 | 04:04 |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: Drop support for python2 https://review.opendev.org/728889 | 04:04 |
| openstackgerrit | Andreas Jaeger proposed zuul/zuul-jobs master: WIP: Sync with configure-mirrors https://review.opendev.org/731293 | 04:17 |
| *** ykarel|away is now known as ykarel | 04:23 | |
| *** sgw has quit IRC | 05:06 | |
| openstackgerrit | Andreas Jaeger proposed zuul/zuul-jobs master: Add configure-os-mirrors role https://review.opendev.org/677578 | 05:18 |
| *** dpawlik has joined #opendev | 06:07 | |
| openstackgerrit | Andreas Jaeger proposed zuul/zuul-jobs master: Fix typo: Deprecrated https://review.opendev.org/731439 | 06:10 |
| openstackgerrit | Ian Wienand proposed openstack/project-config master: Move wheel build role/job outside project-config https://review.opendev.org/731401 | 06:16 |
| AJaeger | ianw: regarding the wheel build jobs: these are openstack specific, so no sense to find an opendev repo - do you agree? | 06:26 |
| ianw | hrw/fungi/AJaeger: I *think* i've got everything in order for https://review.opendev.org/#/q/topic:arm64-wheel+(status:open+OR+status:merged) | 06:26 |
| ianw | AJaeger: as in move it to opendev-base-jobs? | 06:27 |
| AJaeger | ianw: I now dug deeper - it iterates over opesntack/requirements, so the implementation is openstack specific. | 06:29 |
| AJaeger | so, openstack-zuul-jobs is probably fine. And yes, opendev/base-jobs was an alternative if we think we would generalize it and include further repos | 06:30 |
| ianw | yeah, i mean it *could* be even further abstracted into something more generic ... but not in this change :) | 06:30 |
| AJaeger | agreed, not in this change - and I see nothing in other projects; so leave it there. | 06:31 |
| ianw | ohhh, all the requirements branches will have to update the job names | 06:31 |
| ianw | i'm out of time for that now, but that only affects removing them from project config | 06:31 |
| AJaeger | left a note on the openstack-zuul-jobs change, you renamed jobs to "cache" but talk in description about "CI mirrors". Want to change that as well? | 06:32 |
| AJaeger | on the other hand the URL is "mirror..." | 06:32 |
| jrosser | i am looking at these dib-lint errors https://zuul.opendev.org/t/openstack/build/56dd0c43bb444c43b9384f3452031082/log/job-output.txt#610 | 06:33 |
| jrosser | is the right thing to do to add the '---' as the error suggests? | 06:33 |
| ianw | jrosser: oh, hrm, we included yamllint to detect duplicate keys, it should be skipping that check | 06:35 |
| ianw | jrosser: ohhh, damn, i guess that yamllint doesn't pickup the .yamllint file in the dib directory | 06:37 |
| jrosser | i just tried it locally here and there are four occurrences of the error | 06:37 |
| ianw | i guess you're running dib-lint from an install? | 06:38 |
| jrosser | i pip installed it yes, to try to understand what was happening | 06:38 |
| ianw | jrosser: yeah, it's not getting the .yamllint from the dib source tree that would make it ignore that | 06:39 |
| ianw | that's not the intent | 06:39 |
| ianw | the reason i did it was because it picked up duplicate keys better than what we had, which was just load the yaml files with pyyaml | 06:39 |
| jrosser | i figured i'd ask as this felt like a case of "new linter" and i just am unlucky with that patch | 06:40 |
| *** jaicaa has quit IRC | 06:42 | |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: Revert "dib-lint: use yamllint to parse YAML files" https://review.opendev.org/731441 | 06:45 |
| ianw | jrosser: ^ | 06:45 |
| *** jaicaa has joined #opendev | 06:45 | |
| jrosser | thankyou :) | 06:45 |
| ianw | jrosser: i will do a point release so that doesn't get lost in the last 2.x release | 06:47 |
| jrosser | sure, i'm keen to get that patch to the focal images landed fairly soon if thats possible | 06:50 |
| *** hashar has joined #opendev | 06:51 | |
| openstackgerrit | Merged openstack/diskimage-builder master: Revert "dib-lint: use yamllint to parse YAML files" https://review.opendev.org/731441 | 07:00 |
| ianw | 2.37.3 should fix it, a recheck in a bit | 07:07 |
| *** DSpider has joined #opendev | 07:10 | |
| openstackgerrit | Merged zuul/zuul-jobs master: Fix typo: Deprecrated https://review.opendev.org/731439 | 07:11 |
| *** tosky has joined #opendev | 07:44 | |
| openstackgerrit | Merged openstack/project-config master: Don't install python-dev on focal https://review.opendev.org/731237 | 07:54 |
| *** moppy has quit IRC | 08:01 | |
| *** moppy has joined #opendev | 08:01 | |
| *** slaweq_ has joined #opendev | 08:06 | |
| *** slaweq has quit IRC | 08:07 | |
| frickler | infra-root: I deleted the latest focal image in order to trigger a rebuild with ^^. I also noticed some oldish images built by nb04, should we clean these up manually? http://paste.openstack.org/show/794083/ | 08:11 |
| *** ykarel is now known as ykarel|lunch | 08:13 | |
| *** icarusfactor has quit IRC | 08:14 | |
| frickler | also, how would one run these commands on the new dockerized servers like nb01? we might want to add some docs about it to https://docs.openstack.org/infra/system-config/nodepool.html | 08:14 |
| *** roman_g has joined #opendev | 08:24 | |
| openstackgerrit | Sorin Sbarnea (zbr) proposed zuul/zuul-jobs master: Bump ansible-lint to 4.3.0 https://review.opendev.org/702679 | 08:25 |
| *** ravsingh has joined #opendev | 08:33 | |
| openstackgerrit | Andreas Jaeger proposed openstack/project-config master: Add root_marker_dir for releasenotes publish https://review.opendev.org/731462 | 08:34 |
| *** xiaolin has joined #opendev | 08:34 | |
| *** tkajinam has quit IRC | 08:37 | |
| *** dtantsur|afk is now known as dtantsur | 08:40 | |
| openstackgerrit | Sorin Sbarnea (zbr) proposed zuul/zuul-jobs master: Enable linting of test-playbooks https://review.opendev.org/731471 | 08:45 |
| *** ykarel|lunch is now known as ykarel | 08:52 | |
| openstackgerrit | Sorin Sbarnea (zbr) proposed zuul/zuul-jobs master: Bump ansible-lint to 4.3.0 https://review.opendev.org/702679 | 08:53 |
| *** rpittau|afk is now known as rpittau | 08:53 | |
| *** ysandeep is now known as ysandeep|lunch | 09:35 | |
| *** ravsingh has quit IRC | 09:39 | |
| openstackgerrit | Sorin Sbarnea (zbr) proposed zuul/zuul-jobs master: Bump ansible-lint to 4.3.0 https://review.opendev.org/702679 | 09:45 |
| *** ravsingh has joined #opendev | 09:52 | |
| *** Meiyan has quit IRC | 09:55 | |
| *** tosky_ has joined #opendev | 09:57 | |
| *** tosky is now known as Guest48633 | 09:57 | |
| *** tosky_ is now known as tosky | 09:57 | |
| openstackgerrit | Sorin Sbarnea (zbr) proposed zuul/zuul-jobs master: Bump ansible-lint to 4.3.0 https://review.opendev.org/702679 | 10:09 |
| *** slaweq_ is now known as slaweq | 10:15 | |
| *** rpittau is now known as rpittau|bbl | 10:18 | |
| *** ysandeep|lunch is now known as ysandeep | 10:18 | |
| *** sshnaidm|off has quit IRC | 10:55 | |
| openstackgerrit | Tristan Cacqueray proposed zuul/zuul-jobs master: shake-build: add shake build system job https://review.opendev.org/731365 | 11:10 |
| jrosser | focal image seems to be build without python2 now https://nb01.opendev.org/ubuntu-focal-0000009842.log, thats cool | 11:17 |
| openstackgerrit | Sorin Sbarnea (zbr) proposed opendev/elastic-recheck master: Replace pep8 jobs with linters https://review.opendev.org/731497 | 11:17 |
| *** rpittau|bbl is now known as rpittau | 12:05 | |
| openstackgerrit | Dmitriy Rabotyagov (noonedeadpunk) proposed openstack/project-config master: Replace vexxhost sjc1 with ca region https://review.opendev.org/731508 | 12:12 |
| *** dtantsur is now known as dtantsur|brb | 12:14 | |
| *** ravsingh has quit IRC | 12:38 | |
| openstackgerrit | Oleksandr Kozachenko proposed zuul/zuul-jobs master: Add the namespace in wait-for-pods role https://review.opendev.org/731292 | 12:44 |
| openstackgerrit | Oleksandr Kozachenko proposed zuul/zuul-jobs master: Add namespace in the collect-k8s-logs role https://review.opendev.org/731319 | 12:44 |
| openstackgerrit | Tristan Cacqueray proposed zuul/zuul-jobs master: shake-build: add shake build system job https://review.opendev.org/731365 | 12:48 |
| openstackgerrit | Sean McGinnis proposed openstack/project-config master: Drop legacy-tempest-dsvm-neutron-full-* jobs https://review.opendev.org/731518 | 12:52 |
| *** priteau has joined #opendev | 12:58 | |
| openstackgerrit | Oleksandr Kozachenko proposed zuul/zuul-jobs master: Add addons args in ensure-kubernetes role https://review.opendev.org/731523 | 13:06 |
| *** dtantsur|brb is now known as dtantsur | 13:09 | |
| openstackgerrit | Oleksandr Kozachenko proposed zuul/zuul-jobs master: Add the namespace in wait-for-pods role https://review.opendev.org/731292 | 13:09 |
| openstackgerrit | Oleksandr Kozachenko proposed zuul/zuul-jobs master: Add namespace in the collect-k8s-logs role https://review.opendev.org/731319 | 13:09 |
| openstackgerrit | Merged zuul/zuul-jobs master: shake-build: add shake build system job https://review.opendev.org/731365 | 13:28 |
| openstackgerrit | Nate Johnston proposed openstack/project-config master: Add no-op jobs for Congress preparatory to retirement https://review.opendev.org/731534 | 13:37 |
| openstackgerrit | Albin Vass proposed zuul/zuul-jobs master: upload-artifactory: fix variable misspelling https://review.opendev.org/731535 | 13:37 |
| openstackgerrit | Albin Vass proposed zuul/zuul-jobs master: upload-artifactory: fix variable misspelling https://review.opendev.org/731535 | 13:38 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Split inventory into multiple dirs and move hostvars https://review.opendev.org/730991 | 13:46 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Override bridge hostvars directly https://review.opendev.org/731258 | 13:46 |
| *** sgw has joined #opendev | 13:47 | |
| mordred | frickler: good point on docs ... we might also want to put in a helper script so that just running "nodepool" works as expected (we've done that in some other places) | 13:47 |
| mordred | frickler: until then, using docker exec to execute a command in the container is the easiest | 13:49 |
| mordred | for instance: | 13:49 |
| mordred | root@nb01:~# docker exec -it nodepool-builder-compose_nodepool-builder_1 nodepool list | 13:49 |
| mordred | and you can see the container name from docker ps | 13:49 |
| *** ykarel is now known as ykarel|away | 13:50 | |
| mordred | clarkb: https://review.opendev.org/#/c/731224/ and its two parents are ready for review and should be safe to land | 14:06 |
| mordred | clarkb: the children are also ready for review - I think the first failure is ephemeral and I rechecked | 14:07 |
| corvus | mordred, frickler: yeah, i've just been running commands on nl01 because of the "docker exec" thing, so a helper script would be great | 14:10 |
| mordred | corvus: I'll get a patch up for one | 14:11 |
| *** rpittau is now known as rpittau|brb | 14:18 | |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Install nodepool helper scripts on nodepool hosts https://review.opendev.org/731549 | 14:20 |
| mordred | corvus, frickler: ^^ | 14:20 |
| openstackgerrit | Albin Vass proposed zuul/zuul-jobs master: upload-artifactory: fix variable misspelling https://review.opendev.org/731535 | 14:26 |
| *** rpittau|brb is now known as rpittau | 14:29 | |
| *** ravsingh has joined #opendev | 14:38 | |
| *** ravsingh has quit IRC | 14:47 | |
| clarkb | mordred: we're ok with the move of exim in https://review.opendev.org/#/c/730937/8 ? | 14:55 |
| clarkb | mordred: iirc it was in the top dir roles/ dir to make it more reconsumable | 14:55 |
| clarkb | (I see how it makes file matchers simpler so agree with the intent just not sure if we expect that ot break anyone) | 14:55 |
| clarkb | mordred: I +2'd it but didn't approve in case we want to sort through ^ a bit more | 14:57 |
| mordred | clarkb: yeah - that was the intent - but I don't know that in practice that reconsumability ever occurred | 14:59 |
| clarkb | ya I've certainly never heard of someone doing it | 14:59 |
| clarkb | pabelanger with windmill installs for ansible might be the only case I can think of where it may have happened | 14:59 |
| mordred | and I think at this point given our general staffing size I don't thnk we should chase theoretical reconsumability to hard | 15:00 |
| mordred | clarkb: nod | 15:00 |
| clarkb | when we move it that will cause external ansible runs relying on that path to fail because the role won't be available right? | 15:01 |
| clarkb | basically it will fail "safely" | 15:01 |
| clarkb | I think given that would be the failure more we can move it and then if people show up asking questions we'll learn something | 15:01 |
| *** Guest70400 has joined #opendev | 15:03 | |
| clarkb | jvb01 seems to ahve the same snmp problems as meetpad. I wonder if all new ipv6 addrs in rax can't route back to cacti's existing addr | 15:05 |
| *** Guest70400 is now known as redrobot | 15:05 | |
| clarkb | I'll give it the same ipv4 address override in a bit (mostly worried about not having data during the ptg then we can maybe look at new cacti or somethign) | 15:05 |
| fungi | yeah, i think the issue is likely with the cacti server | 15:07 |
| clarkb | as a heads up it is going to get very warm here today so I'm going to try and get a bike ride in early. THen I've also got AC person out as well as $school stuff so my morning is going to be weird | 15:07 |
| *** avass has joined #opendev | 15:07 | |
| fungi | yeah, we're trying to get an estimate this week for a new heating and cooling system. ours is 12 years old and the latest repair bill is steep what with leaks in multiple places, so may be time to send it on to valhalla | 15:09 |
| fungi | if we can get them to actually show up, i'll likely get sucked into that for a while | 15:10 |
| mordred | fungi: have you considered just learning to enjoy being hot? | 15:10 |
| fungi | i don't mind it at all. but i got married | 15:12 |
| openstackgerrit | Merged opendev/system-config master: Clean up base playbook https://review.opendev.org/730985 | 15:25 |
| clarkb | anyone else want to weigh in on the exim role move in https://review.opendev.org/#/c/730937/8 ? if not I think we can approve it | 15:27 |
| clarkb | fungi: frickler ^ | 15:28 |
| openstackgerrit | Merged openstack/project-config master: Replace vexxhost sjc1 with ca region https://review.opendev.org/731508 | 15:28 |
| openstackgerrit | Albin Vass proposed zuul/zuul-jobs master: upload-artifactory: documentation fix https://review.opendev.org/731170 | 15:28 |
| clarkb | infra-root also I'll plan to approve https://review.opendev.org/#/c/730862/ once my bike ride is done as I should be able to monitor it from that point on | 15:30 |
| frickler | I'm getting a bunch of broken files when running download-logs.sh. reason is that base_url ends with a /, and here another / is added https://opendev.org/zuul/zuul-jobs/src/branch/master/roles/local-log-download/templates/download-logs.sh.j2#L69 which gives 404s at least from ovh.net | 15:38 |
| clarkb | I've added njohnston to https://review.opendev.org/#/admin/groups/232,members to facilitate the changes at https://review.opendev.org/#/c/728818/4 | 15:38 |
| openstackgerrit | Merged openstack/project-config master: Add no-op jobs for Congress preparatory to retirement https://review.opendev.org/731534 | 15:39 |
| clarkb | frickler: usually multiple /'s are collapsed together because unix filesystems will do that. But swift is approximating paths there as it isn't a file system and we probably hit the error due to that | 15:40 |
| clarkb | frickler: likely we'll need to fix the script | 15:40 |
| frickler | clarkb: yeah, just double-check that the url always ends in a /. eoding now, I can take a look tomorrow | 15:41 |
| openstackgerrit | Nate Johnston proposed openstack/project-config master: Retire the Congress project https://review.opendev.org/721742 | 15:49 |
| openstackgerrit | Sorin Sbarnea (zbr) proposed zuul/zuul-jobs master: Bump ansible-lint to 4.3.0 https://review.opendev.org/702679 | 15:50 |
| openstackgerrit | Merged zuul/zuul-jobs master: upload-artifactory: documentation fix https://review.opendev.org/731170 | 15:50 |
| AJaeger | clarkb, please tell njohnston to abandon all open reviews for cyborg as well | 15:51 |
| openstackgerrit | Nate Johnston proposed openstack/project-config master: Retire the Congress project https://review.opendev.org/721742 | 15:51 |
| clarkb | AJaeger: cyborg or congress? | 15:53 |
| AJaeger | clarkb: congress - argh ;( | 15:56 |
| openstackgerrit | Nate Johnston proposed openstack/project-config master: Retire the Congress project https://review.opendev.org/721742 | 16:02 |
| *** ysandeep is now known as ysandeep|afk | 16:02 | |
| *** bhagyashris|ruck is now known as bhagyashris | 16:05 | |
| *** hashar has quit IRC | 16:05 | |
| *** rpittau is now known as rpittau|afk | 16:08 | |
| openstackgerrit | Oleksandr Kozachenko proposed zuul/zuul-jobs master: Add the namespace in wait-for-pods role https://review.opendev.org/731292 | 16:13 |
| openstackgerrit | Oleksandr Kozachenko proposed zuul/zuul-jobs master: Add namespace in the collect-k8s-logs role https://review.opendev.org/731319 | 16:13 |
| *** priteau has quit IRC | 16:14 | |
| openstackgerrit | Oleksandr Kozachenko proposed zuul/zuul-jobs master: Add namespace in the collect-k8s-logs role https://review.opendev.org/731319 | 16:14 |
| *** priteau has joined #opendev | 16:15 | |
| openstackgerrit | Oleksandr Kozachenko proposed zuul/zuul-jobs master: Add addons args in ensure-kubernetes role https://review.opendev.org/731523 | 16:16 |
| *** dpawlik has quit IRC | 16:19 | |
| *** priteau has quit IRC | 16:19 | |
| *** roman_g has quit IRC | 16:21 | |
| *** mlavalle has joined #opendev | 16:29 | |
| mordred | clarkb, corvus: you're both smarter than I am - do you see why the service-mirror job is timing out in https://review.opendev.org/#/c/730999 and https://review.opendev.org/#/c/730991 - it failed with the timeout in a recheck too - so it seems to be a consistent timeout | 16:33 |
| clarkb | mordred: it has to build afs with dkms iirc and that is slow | 16:33 |
| clarkb | thats part of the slowness I remember, not sure if that is enough to account for a timeout though (I thought we bumped the timeout to an hour too?) | 16:34 |
| clarkb | hrm no thats a half hour timeout | 16:34 |
| clarkb | I think for jobs that install the afs kernel module we need longer timeouts | 16:34 |
| mordred | clarkb: it seems to be timing out in the testinfra tests - oh, but I wonder if adding the iptables role to the playbook is enough to bump it over 30 | 16:40 |
| clarkb | mordred: ya it may timeout later but a huge chunk of time is eaten by the afs compile | 16:41 |
| clarkb | and ya its possible that adding more ansible to it is enough to bump it over the threshold there | 16:41 |
| clarkb | I'm going to pop out now for that bike ride. Back in a bit and I'll approve that https mirrors change then | 16:44 |
| openstackgerrit | Monty Taylor proposed opendev/puppet-openstack_infra_spec_helper master: Install hosts file into service location https://review.opendev.org/731583 | 16:47 |
| *** priteau has joined #opendev | 16:49 | |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Split inventory into multiple dirs and move hostvars https://review.opendev.org/730991 | 16:51 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Override bridge hostvars directly https://review.opendev.org/731258 | 16:51 |
| mordred | clarkb: I bumped the timeout to see if that helps - and also fixed up the places where our puppet tests were confused | 16:51 |
| openstackgerrit | Oleksandr Kozachenko proposed zuul/zuul-jobs master: Add the namespace in wait-for-pods role https://review.opendev.org/731292 | 16:52 |
| openstackgerrit | Oleksandr Kozachenko proposed zuul/zuul-jobs master: Add namespace in the collect-k8s-logs role https://review.opendev.org/731319 | 16:52 |
| openstackgerrit | Merged zuul/zuul-jobs master: Bump ansible-lint to 4.3.0 https://review.opendev.org/702679 | 16:55 |
| *** priteau has quit IRC | 16:55 | |
| *** ysandeep|afk is now known as ysandeep | 16:57 | |
| *** dtantsur is now known as dtantsur|afk | 17:00 | |
| * prometheanfire needs to watch the infra gentoo build now that the gentoo image is being built from stage3 (from upstream) | 17:05 | |
| *** ysandeep is now known as ysandeep|away | 17:13 | |
| AJaeger | config-core, congress is retired now - please finish with +2A on https://review.opendev.org/#/c/721742 | 17:16 |
| *** olaph has quit IRC | 17:20 | |
| openstackgerrit | Sorin Sbarnea (zbr) proposed zuul/zuul-jobs master: Enable linting of test-playbooks https://review.opendev.org/731471 | 17:27 |
| *** njohnston has joined #opendev | 17:28 | |
| openstackgerrit | Merged zuul/zuul-jobs master: Rename test install role to ensure- https://review.opendev.org/730720 | 17:47 |
| *** dtroyer has quit IRC | 17:48 | |
| *** cloudnull has quit IRC | 17:48 | |
| *** cloudnull has joined #opendev | 17:49 | |
| openstackgerrit | Merged openstack/project-config master: Retire the Congress project https://review.opendev.org/721742 | 18:04 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Run iptables in service playbooks instead of base https://review.opendev.org/730999 | 18:07 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Split inventory into multiple dirs and move hostvars https://review.opendev.org/730991 | 18:07 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Override bridge hostvars directly https://review.opendev.org/731258 | 18:07 |
| openstackgerrit | Sorin Sbarnea (zbr) proposed zuul/zuul-jobs master: test-playbooks: improved syntax https://review.opendev.org/731591 | 18:12 |
| *** diablo_rojo has joined #opendev | 18:16 | |
| diablo_rojo | clarkb, fungi mordred corvus and anyone else interested in meetpad :) | 18:17 |
| diablo_rojo | I pulled numbers based on teams that said they were interested in meetpad | 18:17 |
| diablo_rojo | and it looks like from 13 -17 UTC you could have as many as 190 users.. | 18:17 |
| diablo_rojo | Oh, actually 215ish | 18:17 |
| diablo_rojo | Across 10 rooms. | 18:18 |
| diablo_rojo | (shout out to wendallkaters for helping me pull the numbers) | 18:19 |
| *** wendallkaters_ has joined #opendev | 18:19 | |
| clarkb | diablo_rojo: thank you for pulling those. we may want to laumch 2 or 3 more jvbs then | 18:20 |
| diablo_rojo | Yeah | 18:20 |
| diablo_rojo | Friday the numbers go way down to under 100 | 18:21 |
| diablo_rojo | Its just Tuesday - Thursday you need to worry most about. | 18:22 |
| diablo_rojo | Monday is in the 130 range | 18:22 |
| clarkb | hrm maybe we add a third jvb I think that givesus rough coverage for monday if we fogure ~40 per jvb | 18:24 |
| clarkb | then based on that we can add more capacity for middle of the week | 18:25 |
| clarkb | or we can rollout all at once and it is easy enough to turn them off | 18:25 |
| *** wendallkaters_ is now known as wendallkaters | 18:32 | |
| diablo_rojo | clarkb, I think one before Monday would be good. | 18:33 |
| diablo_rojo | And then I guess you can see how that goes and decide if things are sketchy to add another one or two? | 18:33 |
| diablo_rojo | ..or three? | 18:33 |
| AJaeger | FYI, we have now config-errors in openstack tenant due to congress removal - murano and openstack-ansible repos still have required-proejcts. I pushed changes for master, will backport | 18:34 |
| AJaeger | mnaser: ^ | 18:38 |
| corvus | diablo_rojo, clarkb: yeah -- i think i'd almost just as soon go ahead and launch 2 or 3 more tomorrow so there's no delay if we need them. | 18:50 |
| clarkb | corvus: ya I think I'm coming around to that too | 19:00 |
| clarkb | fwiw any infra root can add new ones just pick the next digit up from jvb0X, boot it, then add it to the inventory should get it enrolled | 19:01 |
| clarkb | I'm going to improve the base job change to use https with our mirrors now | 19:01 |
| clarkb | s/improve/approve/ | 19:04 |
| clarkb | post bike ride typos | 19:04 |
| openstackgerrit | Merged opendev/base-jobs master: Use mirrors with ssl globally https://review.opendev.org/730862 | 19:08 |
| clarkb | I'll keep an eye out for any problems with and feel free to point out any oddities you may see, but its pretty well tested in zuul-jobs and with base-test so I think we're good | 19:10 |
| clarkb | lookign at a xenial job it is correctly using http not https | 19:16 |
| corvus | clarkb: "approve" is the opposite of "improve", right? :) | 19:19 |
| openstackgerrit | Sorin Sbarnea (zbr) proposed zuul/zuul-jobs master: Enable linting of test-playbooks https://review.opendev.org/731471 | 19:26 |
| *** priteau has joined #opendev | 19:38 | |
| *** priteau has quit IRC | 19:43 | |
| openstackgerrit | Sorin Sbarnea (zbr) proposed zuul/zuul-jobs master: test-playbooks: avoid warnings with shell/command https://review.opendev.org/731605 | 19:47 |
| clarkb | spot checks of new changes entering openstack's check queue show not obvious issues yet. I expect this is working as testing expected it to | 19:48 |
| openstackgerrit | Albin Vass proposed zuul/zuul-jobs master: Update docs for os specific task guidelines https://review.opendev.org/731606 | 19:49 |
| openstackgerrit | Albin Vass proposed zuul/zuul-jobs master: Update guideline docs for os specific tasks https://review.opendev.org/731606 | 19:50 |
| openstackgerrit | Albin Vass proposed zuul/zuul-jobs master: Update guideline docs for os specific tasks https://review.opendev.org/731606 | 19:52 |
| clarkb | I've also managed to find that pip is using https pypi mirror in some job logs so that aspect looks good too | 19:54 |
| clarkb | since nothing is on fire I'm going to do local reboots for patching | 19:54 |
| mordred | clarkb, corvus: WOOHOO! The whole stack to https://review.opendev.org/#/c/731258 is green | 19:55 |
| openstackgerrit | Albin Vass proposed zuul/zuul-jobs master: Update guideline docs for os specific tasks https://review.opendev.org/731606 | 19:55 |
| mordred | clarkb: and woot on ssl mirror | 19:55 |
| fungi | i managed to get a/v on my workstation functioning for meetpad in preparation for next week | 19:57 |
| fungi | yay! | 19:57 |
| fungi | for a ~$1 webcam, it's not half bad | 20:00 |
| openstackgerrit | Albin Vass proposed zuul/zuul-jobs master: Update guideline docs for os specific tasks https://review.opendev.org/731606 | 20:06 |
| openstackgerrit | Albin Vass proposed zuul/zuul-jobs master: Update guideline docs for os specific tasks https://review.opendev.org/731606 | 20:08 |
| *** dtroyer has joined #opendev | 20:09 | |
| *** priteau has joined #opendev | 20:21 | |
| *** priteau has quit IRC | 20:26 | |
| *** slaweq has quit IRC | 20:33 | |
| mordred | fungi: nice! | 20:49 |
| clarkb | fungi: I'm looking over our ptg etherpad and the authentication spec is on there (I added it a while back) | 21:04 |
| clarkb | fungi: do you think we might have a spec to review before the end of our PTG time? | 21:05 |
| clarkb | I've taken notes on who we might want around for various conversations, but if you are interested in something particular feel free to add your own annotations | 21:11 |
| clarkb | https://etherpad.opendev.org/p/opendev-virtual-ptg-june-2020 is the etherpad | 21:11 |
| mordred | oh authentication | 21:12 |
| mordred | such a cruel mistress | 21:12 |
| mordred | fungi, clarkb: also - https://review.opendev.org/#/q/topic:reorg-base is green and ready - and I think I'm confident that our testing is good enough that it should be good to go (I had to fix enough things that testing found that make me confident the right things are being exercised) | 21:14 |
| clarkb | mordred: I think you can approve https://review.opendev.org/#/c/730937/ since no one objected to the move earlier | 21:15 |
| mordred | clarkb: cool | 21:15 |
| mordred | clarkb, fungi: don't miss https://review.opendev.org/#/c/731583/ - which is in infra-spec-helper | 21:16 |
| mordred | clarkb: I really want those puppet jobs to go away - but still haven't gotten them replaced with system-config-run- jobs | 21:16 |
| clarkb | mordred: why does it matter that that file is in the service dir when we set the path to it with -i ? | 21:20 |
| fungi | clarkb: i can up the priority on refactoring all the text. i can have something to review either way, it's just a question of how many review comments i can avoid by cleaning it up better up front | 21:21 |
| clarkb | that is so the relative path to host vars works? | 21:21 |
| mordred | clarkb: yes | 21:26 |
| mordred | clarkb: (we _could_ set up an ansible.cfg that also points to our yamlgroup plugin and then sets inventory to the written hosts file as well as the groups.yaml - but that seemed like extra work in a system that should ultimately be going away anyway) | 21:28 |
| clarkb | mordred: comments on https://review.opendev.org/#/c/730999/7 | 21:30 |
| clarkb | mordred: thanks for confirming | 21:31 |
| openstackgerrit | Merged opendev/system-config master: Move base roles into a base subdir https://review.opendev.org/730937 | 21:36 |
| mordred | clarkb: I like the idea of renaming service-letsencrypt | 21:38 |
| openstackgerrit | Merged opendev/system-config master: Move rackspace repo management to the repos role https://review.opendev.org/731224 | 21:39 |
| *** mlavalle has quit IRC | 21:45 | |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Run iptables in service playbooks instead of base https://review.opendev.org/730999 | 21:48 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Split inventory into multiple dirs and move hostvars https://review.opendev.org/730991 | 21:48 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Override bridge hostvars directly https://review.opendev.org/731258 | 21:48 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Rename service-letsencrypt to just letsencrypt https://review.opendev.org/731617 | 21:48 |
| mordred | clarkb: added a rename patch there | 21:48 |
| *** mlavalle has joined #opendev | 21:50 | |
| clarkb | ianw: if you're around today you may want to review https://review.opendev.org/#/c/731617/1 to make sure we aren't missing anything obvious with that le playbook rename | 21:55 |
| clarkb | mordred: is it ok to be scared pulling up https://review.opendev.org/#/c/730991/12 ? :) | 21:59 |
| clarkb | I know its mostly mechanical but a lot of stuff moving around there | 21:59 |
| mordred | clarkb: it's totally fine to be scared :) | 22:09 |
| ianw | clarkb: looking :) | 22:14 |
| ianw | clarkb/mordred: before we start moving letsencrypt, could i get one more eye on https://review.opendev.org/#/c/728743/ to add the sslcheck generation so that's running? | 22:16 |
| ianw | note that's ontop of the testinfra change to pass the inventory as a fixture -- https://review.opendev.org/#/c/729418 | 22:16 |
| jrosser | repos are now https? i have an interesting conundrum now of how i install ca-certificates to overcome that | 22:17 |
| *** DSpider has quit IRC | 22:19 | |
| ianw | jrosser: hrm what are you starting from that doesn't have certs that cover letsencrypt by default? | 22:19 |
| jrosser | an lxc rootfs which it would appear does not have ca-certificates in it initially | 22:20 |
| clarkb | jrosser: yes I converted them today now that we have ssl available on all mirrors | 22:20 |
| jrosser | there is this sort of thing i can do Acquire::https::Verify-Peer "false"; and so on | 22:21 |
| ianw | jrosser: hrrmmm, i don't want to be unhelpful but it does sound like that is doomed to fail in many ways :) | 22:21 |
| clarkb | jrosser: that gives us an extra level of assurance there isn't anything funny going on because we can't use gpg to sign our apt repos. Its also important for pypi which has no other verification | 22:21 |
| clarkb | jrosser: ianw: with bwrap and other light weight container tools its common to bind mount the hosts certs into the container | 22:21 |
| clarkb | mordred: left some thoughts on that change. Mostly I got confused and then think I sorted it out and that resulted in a suggestion | 22:22 |
| clarkb | mordred: but you should double check that my understanding is actually correct | 22:22 |
| jrosser | hmm it's still also serving http, is that going to continue? | 22:23 |
| clarkb | jrosser: probably? at least I haven't thought about turning it off yet | 22:23 |
| clarkb | jrosser: can you not bind mount in the cert trusts from the host? | 22:23 |
| clarkb | (zuul has to do this very thing with bwrap) | 22:23 |
| jrosser | this has potentially broken every branch for OSA | 22:24 |
| jrosser | and i really don't want to go messing with the functional parts in stable branches for building container images | 22:24 |
| jrosser | so the easiest answer for me would be to adjust the mirror URL from https to http | 22:25 |
| clarkb | ya that will work | 22:25 |
| clarkb | I hadn't considered turning off http | 22:25 |
| jrosser | ok cool - i can work round it | 22:25 |
| clarkb | so we can probably leave that in place | 22:25 |
| clarkb | (as a side note if we created out own signing key for those repos I think the lxc containers would similarly break, so we can't address this problem that way either) | 22:27 |
| jrosser | if it came to it, looks like i could be a bit more surgical with Acquire::https::repo.domain.tld::Verify-Peer "false"; | 22:27 |
| clarkb | the underlying issue being that reprepro builds its own indexes and so we aren't mirroring what upstream has published and can't just piggy back off of their signing key | 22:28 |
| clarkb | with https we're verifying the connectivity between the hosts and not the package content directly | 22:28 |
| clarkb | with gpgp + http the package contents are validated but we'd have to add our own key to the trust chain for that so similar problem with slightly different tools | 22:29 |
| jrosser | yes, we use an aptly server here and have do use our own signing key for that | 22:29 |
| clarkb | jrosser: as a side note only bionic and newer are using https | 22:31 |
| clarkb | and buster and newer? I 'm less confident on the debian side | 22:31 |
| clarkb | but basically debian and ubuntu where apt is new enough to support https out of the box | 22:31 |
| clarkb | so not all of your stable branches will be affected potentially | 22:31 |
| jrosser | i don't think we have any xenial left, but having talked about it i think there's several things i can do which are not particularly invasive | 22:33 |
| clarkb | but ya we'll keep the http up | 22:33 |
| clarkb | that'll keep that option open | 22:33 |
| jrosser | btw the removing of python2 from focal has worked | 22:34 |
| clarkb | jrosser: we also confirmed that ansible with zuul should prefer auto not auto legacy | 22:34 |
| jrosser | the jobs get an awful lot further today than before | 22:34 |
| clarkb | so I think it is possible there is a bug in ansibles lookup tables or however it is determining what auto should be | 22:34 |
| clarkb | (because it really should prefer python3 at this point I think) | 22:34 |
| mordred | ianw: +2 on both from me - I don't believe your patches and mine will conflict - assuming it can figure out the change and the rename. left a comment on https://review.opendev.org/#/c/728743/ though | 22:45 |
| ianw | mordred: thanks, yeah agree no conflict | 22:47 |
| ianw | will add that matcher | 22:47 |
| ianw | "E: Failed to fetch https://mirror.regionone.linaro-us.opendev.org/ubuntu-ports/pool/main/l/lzo2/liblzo2-2_2.08-1.2_arm64.deb 403 Forbidden" ... that's a new one | 22:52 |
| ianw | https://zuul.opendev.org/t/openstack/build/8006451998854e7fb1220218a84eec64/logs | 22:52 |
| clarkb | ianw: did we somehow sync fileperms that prevent apache from serving it? | 22:53 |
| mordred | clarkb: I responded to yoru comments - good comments - I'll probably respin that with a rename to gate-hosts.yaml | 22:53 |
| clarkb | ianw: I've approved the test fixture change and its child was already approved so both things should end up going in soon | 22:54 |
| *** tkajinam has joined #opendev | 22:54 | |
| clarkb | ianw: mordred editing https://review.opendev.org/#/c/728743/19/playbooks/host_vars/gitea01.opendev.org.yaml will conflict with mordreds changes to the host vars I think | 22:54 |
| clarkb | at a git level I mean | 22:54 |
| ianw | clarkb: it's ... weird. i'm looking at the logs and there's 403 errors, but i get the file locally | 22:56 |
| ianw | maybe i spoke too soon, i got one file, another like https://mirror.regionone.linaro-us.opendev.org/ubuntu-ports/pool/universe/p/python-iniparse/python-iniparse_0.4-2.2_all.deb is just hanging | 22:57 |
| ianw | ahh, eventaully it came back 403 | 22:57 |
| ianw | [Thu May 28 22:51:39 2020] afs: Lost contact with file server 104.130.138.161 in cell openstack.org (code -1) (all multi-homed ip addresses down for the server) | 22:59 |
| ianw | [Thu May 28 22:51:57 2020] afs: file server 104.130.138.161 in cell openstack.org is back up (code 0) (multi-homed address; other same-host interfaces may still be down) | 22:59 |
| ianw | [Thu May 28 23:00:37 2020] afs: Lost contact with file server 104.130.138.161 in cell openstack.org (code -1) (all multi-homed ip addresses down for the server) | 22:59 |
| ianw | [Thu May 28 23:00:50 2020] afs: file server 104.130.138.161 in cell openstack.org is back up (code 105) (multi-homed address; other same-host interfaces may still be down | 22:59 |
| ianw | that's interesting that the kernel logs say it's 23:00 and the date on the system is Thu May 28 22:59:17 UTC 2020 | 23:00 |
| openstackgerrit | Oleksandr Kozachenko proposed zuul/zuul-jobs master: Add the namespace in wait-for-pods role https://review.opendev.org/731292 | 23:04 |
| openstackgerrit | Oleksandr Kozachenko proposed zuul/zuul-jobs master: Add namespace in the collect-k8s-logs role https://review.opendev.org/731319 | 23:04 |
| *** factor has joined #opendev | 23:06 | |
| openstackgerrit | Oleksandr Kozachenko proposed zuul/zuul-jobs master: Add addons args in ensure-kubernetes role https://review.opendev.org/731523 | 23:16 |
| openstackgerrit | Merged opendev/system-config master: testinfra: pass inventory and zuul data https://review.opendev.org/729418 | 23:18 |
| ianw | clarkb: ^ i'm thinking with that we could update the mirror jobs to query remotely. i dunno if that really buys us anything, but it is kind of cool | 23:20 |
| ianw | mirror testinfra jobs i mean | 23:20 |
| *** tosky has quit IRC | 23:20 | |
| clarkb | ianw: ya, I actually got them to, but in a more convoluted way | 23:21 |
| clarkb | I think we can update the tests to be more simple | 23:21 |
| clarkb | I'm about to transition to making fire and cooking food though | 23:21 |
| openstackgerrit | Oleksandr Kozachenko proposed zuul/zuul-jobs master: Add addons args in ensure-kubernetes role https://review.opendev.org/731523 | 23:24 |
| openstackgerrit | Merged opendev/system-config master: Generate ssl check list directly from letsencrypt variables https://review.opendev.org/728743 | 23:31 |
| fungi | clarkb: revisiting the ansible auto python detection, could it be older pre-auto-capable ansible versions in jobs? or maybe nested ansible is actually where folks are hitting it? | 23:44 |
| clarkb | fungi: our default is 2.8 (maybe 2.9 now) and that is new enough for auto proper aiui | 23:45 |
| clarkb | fungi: corvus confirmed with a inventory file from a job showing auto was set | 23:45 |
| fungi | so... nested ansible defaulting to legacy? | 23:47 |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: Prepare to drop Python 2 support https://review.opendev.org/731378 | 23:47 |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: Drop support for python2 https://review.opendev.org/728889 | 23:47 |
| clarkb | fungi: hrm was it nested ansible? | 23:47 |
| clarkb | that could explain it too I suppose | 23:47 |
| fungi | i honestly don't recall what the example was now | 23:48 |
| clarkb | it was an OSA job running against focal | 23:48 |
| clarkb | but I'm not sure if that was zuul level ansible or nested now that you've got me thinking about hte nested possibility | 23:48 |
| clarkb | jrosser: ^ was your ubuntu focal issue with zuul's ansible or a nested ansible run? | 23:48 |
| ianw | i was looking at the matching stuff, because suse isn't in there either | 23:51 |
| ianw | so falls back to python2 | 23:51 |
| clarkb | oh is it a lack of table entries that falls back to python2? | 23:53 |
| clarkb | maybe we can convince them to invert that now that python2 is dead (ish) | 23:53 |
| ianw | yeah i added a note on one of the changes when we were dropping pip-and-virtualenv from suse, and python2 setuptools wasn't there | 23:54 |
| ianw | https://review.opendev.org/#/c/726728/ was it | 23:57 |
| ianw | https://github.com/ansible/ansible/blob/devel/lib/ansible/config/base.yml#L1504 | 23:58 |
| ianw | i think https://github.com/ansible/ansible/blob/devel/lib/ansible/config/base.yml#L1521 is the bit it falls into | 23:59 |
| ianw | the order is /usr/bin/python then 3.7, 3.6, 3.5 ... so if py2 is installed, it's preferred | 23:59 |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!