| ianw | it looks like it's only in devel | 00:03 |
|---|---|---|
| ianw | no hang on, it is on 2.9 | 00:05 |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: Update to Ansible 2.9.8 https://review.opendev.org/726981 | 00:07 |
| ianw | hang on, we're too bleeding edge | 00:09 |
| ianw | mattclay committed 40 minutes ago ... that's the 2.9.8 release commit but it's not released, yet | 00:09 |
| ianw | https://github.com/ansible/ansible/commit/ed2a39d68e17e1b1d4c5670e1847cf448cf20f53 | 00:09 |
| ianw | it is quite awesome that we find this in the gate, and not in production however :) | 00:14 |
| *** mlavalle has quit IRC | 00:15 | |
| *** diablo_rojo has joined #opendev | 00:17 | |
| openstackgerrit | Merged opendev/system-config master: Move build-essential arm64 things to base https://review.opendev.org/726039 | 00:57 |
| openstackgerrit | Merged opendev/system-config master: service-bridge: skip osc/kubectl things for arm64 https://review.opendev.org/726040 | 00:57 |
| clarkb | ++ | 01:08 |
| *** tkajinam has quit IRC | 01:29 | |
| *** tkajinam has joined #opendev | 01:29 | |
| ianw | ... interesting ... bionic has an arm 16.04-hwe kernel, which we've been unconditionally installing | 01:30 |
| clarkb | I think we install hwe because xenial non hwe had bugs on arm? | 01:34 |
| clarkb | Im guessing bionic inherited that | 01:34 |
| *** ysandeep|away is now known as ysandeep | 01:36 | |
| ianw | yeah, unintentionally | 01:38 |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: [wip] focal ubuntu-minimal testing https://review.opendev.org/725752 | 01:43 |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: ubuntu-minimal : only install 16.04 HWE kernel on xenial https://review.opendev.org/726996 | 01:43 |
| cmurphy | ianw: on our train branches we're still seeing "virtualenv: command not found" on opensuse even with https://review.opendev.org/726526 e.g. https://zuul.opendev.org/t/openstack/build/f2ed9a3541d34f6598ef133b1c183bc0 | 01:46 |
| ianw | cmurphy: hrmm, ok let me look | 01:54 |
| ianw | i guess it's running under python2 | 02:01 |
| ianw | we can either run ensure-virtualenv or unconditionally use "python3 -m venv" | 02:01 |
| ianw | cmurphy: https://review.opendev.org/727000 Add ensure-virtualenv role .. should do it, will test | 02:07 |
| cmurphy | ianw: awesome, ty | 02:21 |
| ianw | well thank me if/when it works :) | 02:21 |
| ianw | [repo-update|http://mirror.dfw.rax.opendev.org/opensuse/update/leap/15.1/oss/] Valid metadata not found at specified URL | 03:27 |
| ianw | they all seemed to fail with something like that | 03:27 |
| ianw | https://static.opendev.org/mirror/logs/rsync-mirrors/opensuse.log seems like things are in sync | 03:29 |
| ianw | AJaeger: ^ perhaps something to look into? see results @ https://review.opendev.org/#/c/726628/ | 03:30 |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: ubuntu-minimal : only install 16.04 HWE kernel on xenial https://review.opendev.org/726996 | 04:07 |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: [wip] focal ubuntu-minimal testing https://review.opendev.org/725752 | 04:07 |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: package-installs: allow when filter to be a list https://review.opendev.org/727049 | 04:07 |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: ubuntu-minimal: fix HWE install for focal https://review.opendev.org/727050 | 04:07 |
| *** diablo_rojo has quit IRC | 04:25 | |
| *** ykarel|away is now known as ykarel | 04:41 | |
| ianw | mordred / clarkb: ^ so that stack should fix things up to build arm64 focal images | 04:43 |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: ubuntu-minimal: Add Ubuntu Focal test build https://review.opendev.org/725752 | 04:45 |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: ubuntu-minimal: Add Ubuntu Focal test build https://review.opendev.org/725752 | 04:46 |
| AJaeger | dirk, cmurphy, do you konw what mirror to use? See ianw's question above | 05:17 |
| cmurphy | is it/was it in the middle of updating? that repo seems to work for me now | 05:22 |
| AJaeger | cmurphy: looks like it. | 05:23 |
| AJaeger | "epodata/84548613659f4ab18f2cd48ee350fd4f99da47aa947597b61462f5243a0860f3-primary.xml.gz' not found on medium 'http://mirror.dfw.rax.opendev.org/opensuse/update/leap/15.1/oss/'" - that file exists now | 05:23 |
| ianw | cmurphy: our updates should be atomic ... but if the mirror was out of sync we would have an atomically out of date mirror :) | 05:23 |
| AJaeger | ianw: I think we can recheck and should be good again | 05:23 |
| ianw | thtat was @ 2020-05-12 02:20:52.897737 | 05:24 |
| ianw | + date --iso-8601=ns | 05:26 |
| ianw | 2020-05-12T02:21:17,796218602+00:00 | 05:26 |
| ianw | that is suspiciously close | 05:26 |
| ianw | (that's from https://static.opendev.org/mirror/logs/rsync-mirrors/opensuse.log) | 05:27 |
| openstackgerrit | Andreas Jaeger proposed zuul/zuul-jobs master: Add new non-npm specific javascript jobs https://review.opendev.org/726547 | 05:38 |
| openstackgerrit | Andreas Jaeger proposed zuul/zuul-jobs master: Extract ensure-javascript-build-tool role https://review.opendev.org/726900 | 05:38 |
| openstackgerrit | Andreas Jaeger proposed zuul/zuul-jobs master: Add new non-npm specific javascript jobs https://review.opendev.org/726547 | 05:53 |
| openstackgerrit | Andreas Jaeger proposed zuul/zuul-jobs master: Extract ensure-javascript-build-tool role https://review.opendev.org/726900 | 05:53 |
| *** dpawlik has joined #opendev | 06:05 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/project-config master: Normalize projects.yaml https://review.opendev.org/727062 | 06:10 |
| dirk | ianw: the problem is that zypper revaluates it's local cache only once per hour. If it changes in-between then it throws an ugly error | 06:11 |
| dirk | We could change the invocation to always revalidate, which should not be a performance problem as the mirror is local | 06:13 |
| *** tkajinam has quit IRC | 06:35 | |
| *** tkajinam has joined #opendev | 06:36 | |
| openstackgerrit | Andreas Jaeger proposed openstack/project-config master: Fix flake8 testing https://review.opendev.org/727070 | 06:41 |
| *** roman_g has quit IRC | 07:00 | |
| ianw | fungi: so not much luck with ovh mirrors; as of now we don't have the right flavors and as of now ansible 2.9.8 is not released either (which would be required for a focal node) so ... yeah it's pretty much where you left it :) | 07:08 |
| *** ralonsoh has joined #opendev | 07:33 | |
| *** tosky has joined #opendev | 07:35 | |
| *** lpetrut has joined #opendev | 07:52 | |
| *** dtantsur|afk is now known as dtantsur | 07:54 | |
| *** rpittau|afk is now known as rpittau | 07:58 | |
| *** ykarel is now known as ykarel|lunch | 08:22 | |
| *** slaweq has joined #opendev | 08:27 | |
| slaweq | frickler: hi, maybe we can continue here | 08:27 |
| slaweq | frickler: so I have patch https://review.opendev.org/727103 for tobiko | 08:28 |
| slaweq | frickler: and in fact the issue may happen in one of the jobs: tobiko-devstack-faults-centos-7 or tobiko-devstack-faults-ubuntu-bionic | 08:28 |
| frickler | slaweq: o.k., I'll set holds for both | 08:29 |
| *** ysandeep is now known as ysandeep|lunch | 08:31 | |
| slaweq | frickler: thx a lot | 08:32 |
| *** lpetrut_ has joined #opendev | 08:39 | |
| *** lpetrut has quit IRC | 08:42 | |
| *** ykarel|lunch is now known as ykarel | 08:51 | |
| *** sshnaidm|afk is now known as sshnaidm | 08:52 | |
| frickler | slaweq: welcome, please let me know your ssh key (once again ;) so I can give you access once the node(s) got held | 08:58 |
| slaweq | frickler: http://paste.openstack.org/show/793417/ | 09:00 |
| *** priteau has joined #opendev | 09:02 | |
| *** DSpider has joined #opendev | 09:23 | |
| *** ysandeep|lunch is now known as ysandeep | 09:25 | |
| *** jhesketh has joined #opendev | 09:31 | |
| ttx | Looks like the latest ptgbot changes were not picked up by the puppet-ptgbot module... Is there anything new that needs to be done for it to happen? | 09:31 |
| ttx | /cc fungi ^ | 09:31 |
| openstackgerrit | Albin Vass proposed zuul/zuul-jobs master: Ensure output dirs are empty https://review.opendev.org/727135 | 09:54 |
| openstackgerrit | Albin Vass proposed zuul/zuul-jobs master: Ensure output dirs are empty https://review.opendev.org/727135 | 10:03 |
| *** ysandeep is now known as ysandeep|brb | 10:09 | |
| slaweq | frickler: ping, one of the jobs failed: https://06d50ddbf1635bcfe515-db6919ac06726062cc29a59f6b4e7f0a.ssl.cf2.rackcdn.com/727103/1/check/tobiko-devstack-faults-centos-7/3949807/ | 10:10 |
| *** yuri has quit IRC | 10:15 | |
| openstackgerrit | Albin Vass proposed zuul/zuul-jobs master: Ensure output dirs are empty https://review.opendev.org/727135 | 10:15 |
| *** rpittau is now known as rpittau|bbl | 10:19 | |
| *** dpawlik has quit IRC | 10:21 | |
| *** dpawlik has joined #opendev | 10:21 | |
| frickler | slaweq: added you for root access on all three nodes, let me know if you need anything further. I'm still keeping the hold for the bionic job unless you tell me you won't need it | 11:00 |
| *** ysandeep|brb is now known as ysandeep | 11:11 | |
| openstackgerrit | Albin Vass proposed zuul/zuul-jobs master: Drop support for ansible 2.6 https://review.opendev.org/727158 | 11:13 |
| slaweq | frickler: thx a lot, I will ping You once I will finish | 11:46 |
| *** hashar has joined #opendev | 11:52 | |
| *** rpittau|bbl is now known as rpittau | 12:21 | |
| openstackgerrit | Andreas Jaeger proposed openstack/project-config master: Remove noop-jobs from oslo.tools https://review.opendev.org/727178 | 12:29 |
| *** zbr_ has joined #opendev | 12:35 | |
| *** zbr_ has quit IRC | 12:36 | |
| *** slittle1 has quit IRC | 12:38 | |
| *** elod_pto is now known as elod | 12:42 | |
| fungi | ttx: looking now | 13:00 |
| fungi | the bot was last restarted may 11 | 13:01 |
| fungi | which i guess was yesterday | 13:02 |
| fungi | yeah, 16:05 utc yesterday | 13:02 |
| fungi | current git checkout and installation of the software is cc11180 from april 1 | 13:04 |
| *** slittle1 has joined #opendev | 13:07 | |
| fungi | mordred: looking at /var/log/ansible/service-eavesdrop.yaml.log on bridge.o.o i see the install-ansible-roles task failing with "fatal: destination path '/etc/ansible/roles/puppet' already exists and is not an empty directory." | 13:09 |
| fungi | i have to assume that's not expected | 13:09 |
| slaweq | frickler: infra_root: hi, I'm done with debugging on the nodes 104.239.136.110, 104.239.141.232 and 104.239.136.111 | 13:11 |
| slaweq | You can clean them now | 13:12 |
| slaweq | thx a lot for help | 13:12 |
| mordred | fungi: that's the thing we fixed yesterday :( | 13:12 |
| mordred | fungi: oh - so yeah - last time that ran was before we fixed it | 13:14 |
| mordred | fungi: so we should just trigger a new run | 13:14 |
| frickler | slaweq: deleted those nodes, do you still want the bionic hold or should I drop that one, too? | 13:14 |
| ttx | fungi: there was code posted since then which it did not pick up... In theory the puppet run should trigger and deploy new files ? | 13:15 |
| ttx | Like 6 days ago acf550e1a2 pushed a new version of logo.png which the puppet run is supposed to deploy | 13:15 |
| fungi | mordred: aha, so i can reenqueue into deploy the last change which ran infra-prod-service-eavesdrop | 13:16 |
| fungi | ttx: yes, but how puppet gets run on eavesdrop has changed as of the past month to be more immediate when changes merge (in theory) | 13:16 |
| ttx | ah, that would certainly explain | 13:16 |
| fungi | and is no longer being run periodically, i guess | 13:16 |
| ttx | fungi: should I add a new job to ptgbot that would pick up changes immediately? | 13:17 |
| mordred | ttx: sadly that particular combo is a bit more problematic | 13:18 |
| fungi | ttx: the infra-prod-service-eavesdrop job should in theory take care of it, i think. i've reenqueued 725103,2 into the deploy pipeline, which is the most recent change to have triggered that job | 13:19 |
| ttx | ok let's see | 13:19 |
| fungi | and at least if it doesn't, then we can continue to troubleshoot from there | 13:19 |
| openstackgerrit | Merged zuul/zuul-jobs master: Add new non-npm specific javascript jobs https://review.opendev.org/726547 | 13:37 |
| openstackgerrit | Merged openstack/project-config master: Fix flake8 testing https://review.opendev.org/727070 | 13:39 |
| *** redrobot has quit IRC | 13:39 | |
| openstackgerrit | Merged zuul/zuul-jobs master: Fail and direct user to use ensure-* version of roles https://review.opendev.org/726448 | 13:42 |
| *** openstackstatus has quit IRC | 13:53 | |
| *** openstackstatus has joined #opendev | 13:53 | |
| *** ChanServ sets mode: +v openstackstatus | 13:53 | |
| slaweq | frickler: no, You can drop it, thx a lot | 13:59 |
| ttx | mordred, fungi: job reported success but the log file is full of errors, so not sure it worked | 14:01 |
| ttx | From the ptgbot perspective, it did work, so I'm happy enough | 14:01 |
| fungi | ttx: thanks, i'll dig deeper on the errors, but glad that much is working | 14:05 |
| openstackgerrit | Merged openstack/project-config master: Normalize projects.yaml https://review.opendev.org/727062 | 14:06 |
| *** pramchan has joined #opendev | 14:12 | |
| fungi | ttx: oh, if you mean this error it's expected: https://zuul.opendev.org/t/openstack/build/5883eb79d16641c39aaf6f7b698ac694/log/job-output.txt#409 | 14:18 |
| fungi | it's running on our ssh bastion and we don't run the console logger there, for added safety | 14:19 |
| fungi | we redirect and append the output to a local logfile on the bastion instead | 14:20 |
| mordred | one of these days we're going to finish fixing that console logging | 14:21 |
| *** ysandeep is now known as ysandeep|afk | 14:26 | |
| *** tkajinam has quit IRC | 14:37 | |
| *** ralonsoh has quit IRC | 14:40 | |
| *** hashar has quit IRC | 14:40 | |
| *** ralonsoh has joined #opendev | 14:40 | |
| *** ralonsoh has quit IRC | 14:44 | |
| clarkb | fungi: ttx I manually restarted ptgbot after it netsplit away | 14:48 |
| fungi | clarkb: yep, i remember seeing that | 14:49 |
| fungi | so that's why there was a restart yesterday | 14:49 |
| clarkb | yes | 14:50 |
| openstackgerrit | Merged openstack/project-config master: Increase tag-releases job timeout https://review.opendev.org/726525 | 15:04 |
| *** dtantsur is now known as dtantsur|afk | 15:08 | |
| *** ysandeep|afk is now known as ysandeep | 15:11 | |
| openstackgerrit | Merged openstack/project-config master: Set python-path for SUSE builds https://review.opendev.org/726728 | 15:23 |
| *** mlavalle has joined #opendev | 15:27 | |
| openstackgerrit | Albin Vass proposed zuul/zuul-jobs master: Add upload-artifactory role https://review.opendev.org/725678 | 15:31 |
| openstackgerrit | Monty Taylor proposed zuul/zuul-jobs master: Split browser and non-browser targetting js jobs https://review.opendev.org/727231 | 15:33 |
| *** roman_g has joined #opendev | 15:33 | |
| *** slittle1 has quit IRC | 15:35 | |
| *** Guest49293 has joined #opendev | 15:37 | |
| *** Guest49293 is now known as redrobot | 15:38 | |
| openstackgerrit | Monty Taylor proposed zuul/zuul-jobs master: Split browser and non-browser targetting js jobs https://review.opendev.org/727231 | 15:38 |
| openstackgerrit | Monty Taylor proposed zuul/zuul-jobs master: Add firefox to javascript browser testing depends https://review.opendev.org/727234 | 15:38 |
| openstackgerrit | Monty Taylor proposed zuul/zuul-jobs master: Split browser and non-browser targetting js test jobs https://review.opendev.org/727231 | 15:43 |
| openstackgerrit | Monty Taylor proposed zuul/zuul-jobs master: Add firefox to javascript browser testing depends https://review.opendev.org/727234 | 15:43 |
| openstackgerrit | Monty Taylor proposed zuul/zuul-jobs master: Split browser and non-browser targetting js test jobs https://review.opendev.org/727231 | 15:45 |
| openstackgerrit | Tristan Cacqueray proposed zuul/zuul-jobs master: fetch-sphinx-output: introduce zuul_use_fetch_output https://review.opendev.org/681905 | 15:47 |
| *** lpetrut_ has quit IRC | 15:51 | |
| clarkb | ttx: smcginnis: mordred has mentioned there was a change made to zuul to not reconfigure zuul on tag events. We expect many tag events in the near future so mordred was suggesting we might restart the zuul scheduler today before that. I believe zuul did the reconfigure on tag events during previous releases so I think chances are we'll be fine but thought I'd bring it up here in case we want to go ahead with | 15:55 |
| clarkb | a zuul restart | 15:55 |
| clarkb | the biggest risk is it will slow zuul down as it has to process all of those unnecessary configuration updates. | 15:55 |
| clarkb | we did just restart zuul on friday and it seems happy since then. I think the risk tehrefore is low that the restart will cause problems | 15:56 |
| clarkb | but restarting does slwo down the queues as we have to restart all the jobs | 15:56 |
| mordred | clarkb, ttx: https://review.opendev.org/#/c/726213/ - it has merged | 15:56 |
| mordred | yeah - it's a tradeoff to be sure | 15:56 |
| mordred | but if we hit a slow point, then perhaps a quick restart wouldn't be terrible | 15:57 |
| smcginnis | If the team thinks it's safe that it won't cause any issues when we process our final releases, then I'm fine with getting in a quick restart before then. | 15:59 |
| ttx | we'll be tagging a lot tomorrow morning for sure | 15:59 |
| smcginnis | ttx were just talking about timing. Targeting 9UTC to kick off the release. | 15:59 |
| ttx | I have a hard time assessing which path is safest | 16:00 |
| clarkb | I expect we'll notice zuul slow down as it reconfigures, but it'll chew through that backlog then be fine | 16:00 |
| smcginnis | That is the existing behavior for the past few series final releases, right? | 16:00 |
| clarkb | smcginnis: yes I believe so | 16:00 |
| clarkb | I think corvus recently said its about 6 seconds per dynamic reconfiguration | 16:01 |
| clarkb | if that number is correct then I think we can expect a delay of 6 seconds * number of tags (roughly) | 16:01 |
| smcginnis | There's > 85 repos being tagged. A few have multiple repos per deliverable file, so it's likely close to 100. | 16:03 |
| openstackgerrit | Tristan Cacqueray proposed zuul/zuul-jobs master: fetch-sphinx-tarball: introduce zuul_use_fetch_output https://review.opendev.org/681870 | 16:03 |
| clarkb | smcginnis: ok ~10 minutes is probably not that bad all considered | 16:03 |
| smcginnis | Inefficient, but not unreasonable. | 16:03 |
| ttx | nah it's fine, that patch takes a long time anyway | 16:03 |
| clarkb | mordred: ^ is that your understanding of how we will be affected? | 16:03 |
| mordred | clarkb: yeah - so with those numbers then it should probably be fine | 16:03 |
| mordred | and maybe we don't worry about it | 16:04 |
| mordred | it hasn't been enough of a problem for us to investigate before teh bmw folks hit the issue | 16:04 |
| corvus | ++ will cost more than it will save | 16:04 |
| smcginnis | I guess I would feel safer not introducing behavior changes right before doing the big release, but up to y'all if you think it's worth doing or not. | 16:05 |
| smcginnis | As long as we are good to go by 9UTC tomorrow morning. | 16:05 |
| clarkb | I'm hearing that we should leave it as is since we've lived through the behavior before and the expected impact, while noticeable, will be manageable | 16:07 |
| ttx | +1 | 16:07 |
| openstackgerrit | Albin Vass proposed zuul/zuul-jobs master: Add upload-artifactory role https://review.opendev.org/725678 | 16:09 |
| *** rpittau is now known as rpittau|afk | 16:10 | |
| openstackgerrit | Tristan Cacqueray proposed zuul/zuul-jobs master: fetch-sphinx-output: introduce zuul_use_fetch_output https://review.opendev.org/681905 | 16:15 |
| openstackgerrit | Tristan Cacqueray proposed zuul/zuul-jobs master: fetch-sphinx-tarball: introduce zuul_use_fetch_output https://review.opendev.org/681870 | 16:15 |
| *** pramchan has quit IRC | 16:22 | |
| openstackgerrit | Merged zuul/zuul-jobs master: Extract ensure-javascript-build-tool role https://review.opendev.org/726900 | 16:28 |
| openstackgerrit | Monty Taylor proposed zuul/zuul-jobs master: Split browser and non-browser targetting js test jobs https://review.opendev.org/727231 | 16:31 |
| tobiash | corvus, mordred: 6 seconds per dynamic reconfigure is quick enough so this probably won't hurt much. Further zuul anyway does deduplication so if you push more tags at once that will result in only one reconfig | 16:37 |
| mordred | tobiash: good point | 16:37 |
| openstackgerrit | Tristan Cacqueray proposed zuul/zuul-jobs master: fetch-sphinx-output: introduce zuul_use_fetch_output https://review.opendev.org/681905 | 16:38 |
| tobiash | we were hurt much more with 100-200s reconfig time and projects constantly pushing tags all day long | 16:39 |
| *** dpawlik has quit IRC | 16:40 | |
| mordred | tobiash: yikes! | 16:47 |
| openstackgerrit | Tristan Cacqueray proposed zuul/zuul-jobs master: fetch-sphinx-tarball: introduce zuul_use_fetch_output https://review.opendev.org/681870 | 16:54 |
| openstackgerrit | Tristan Cacqueray proposed zuul/zuul-jobs master: fetch-sphinx-tarball: add missing zuul_success default https://review.opendev.org/727272 | 16:54 |
| *** ykarel is now known as ykarel|away | 16:55 | |
| openstackgerrit | Merged zuul/zuul-jobs master: Split browser and non-browser targetting js test jobs https://review.opendev.org/727231 | 17:10 |
| *** priteau has quit IRC | 17:17 | |
| *** ysandeep is now known as ysandeep|away | 17:19 | |
| *** iurygregory has quit IRC | 17:27 | |
| openstackgerrit | Albin Vass proposed zuul/zuul-jobs master: WIP: Remove requiring tox_envlist https://review.opendev.org/726829 | 17:33 |
| openstackgerrit | Tristan Cacqueray proposed zuul/zuul-jobs master: fetch-sphinx-tarball: introduce zuul_use_fetch_output https://review.opendev.org/681870 | 17:43 |
| openstackgerrit | Albin Vass proposed zuul/zuul-jobs master: Add upload-artifactory role https://review.opendev.org/725678 | 18:00 |
| openstackgerrit | Albin Vass proposed zuul/zuul-jobs master: Add upload-artifactory role https://review.opendev.org/725678 | 18:02 |
| *** hashar has joined #opendev | 18:08 | |
| *** iurygregory has joined #opendev | 18:31 | |
| openstackgerrit | Monty Taylor proposed zuul/zuul-jobs master: Combine javascript deployment and deployment-tarball jobs https://review.opendev.org/727370 | 18:51 |
| openstackgerrit | Jeremy Stanley proposed opendev/zone-opendev.org master: Replace OVH CI mirrors https://review.opendev.org/727376 | 18:57 |
| openstackgerrit | Jeremy Stanley proposed opendev/system-config master: Replace OVH CI mirrors https://review.opendev.org/727388 | 19:12 |
| openstackgerrit | Jeremy Stanley proposed opendev/system-config master: Add OpenEdge CI mirror to Cacti config https://review.opendev.org/727389 | 19:12 |
| *** diablo_rojo has joined #opendev | 19:13 | |
| openstackgerrit | Monty Taylor proposed zuul/zuul-jobs master: Combine javascript deployment and deployment-tarball jobs https://review.opendev.org/727370 | 19:16 |
| openstackgerrit | Jeremy Stanley proposed openstack/project-config master: Add OVH CI mirrors to opendev.org map https://review.opendev.org/727392 | 19:20 |
| openstackgerrit | Jeremy Stanley proposed openstack/project-config master: Revert "Temporarily disable OVH" https://review.opendev.org/727393 | 19:20 |
| openstackgerrit | Merged opendev/zone-opendev.org master: Replace OVH CI mirrors https://review.opendev.org/727376 | 19:28 |
| AJaeger | infra-root, speaking about ovh: Is swiftupload ok again? See https://review.opendev.org/#/c/726028/ | 19:31 |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: Update to Ansible 2.9.8 https://review.opendev.org/726981 | 19:33 |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: Add focal testing for mirror nodes https://review.opendev.org/726970 | 19:33 |
| openstackgerrit | Tristan Cacqueray proposed zuul/zuul-jobs master: fetch-sphinx-tarball: introduce zuul_use_fetch_output https://review.opendev.org/681870 | 19:50 |
| openstackgerrit | Albin Vass proposed zuul/zuul-jobs master: Drop support for ansible 2.6 https://review.opendev.org/727158 | 19:51 |
| openstackgerrit | Albin Vass proposed zuul/zuul-jobs master: Drop support for ansible 2.7 https://review.opendev.org/727410 | 19:51 |
| openstackgerrit | Gonéri Le Bouder proposed zuul/zuul-jobs master: bindep: ensure virtualenv is present https://review.opendev.org/727413 | 19:55 |
| openstackgerrit | Tristan Cacqueray proposed zuul/zuul-jobs master: fetch-sphinx-tarball: introduce zuul_use_fetch_output https://review.opendev.org/681870 | 19:55 |
| *** slaweq has quit IRC | 19:57 | |
| ianw | AJaeger: that's a good point ... maybe we just put it back and see if it fails? | 19:58 |
| fungi | clarkb: since you reviewed the other topic:ovh-mirrors changes can you also take a look at 727388 (it's the only other outstanding piece before we can move forward on those) | 19:58 |
| mordred | fungi: if we wanted to not have compose start containers, I'd suggest we shift back to podman rather than using systemd to start the containers | 19:58 |
| mordred | because then we can run them as a non-priv user in the first place | 19:59 |
| AJaeger | ianw: we could... | 19:59 |
| fungi | mordred: got it. i know we'd talked about having systemd unit shims calling docker-compose so we don't have to remember where/what those configs are | 19:59 |
| mordred | but I don't know that we're to that point | 19:59 |
| clarkb | fungi: yes looking | 19:59 |
| clarkb | then I'm grabbing lunch | 19:59 |
| fungi | thanks! | 19:59 |
| * mordred thinks one of the nice things about docker is that he doesn't have to deal with systemd - but maybe he's in the minority there | 20:00 | |
| fungi | i've already got the cinder volumes carved up, formatted, and configured to mount at /var/cache/apache2 and /var/cache/openafs on the new mirrors, but can't actually mount them yet until ansible applies the mirror playbook | 20:01 |
| clarkb | fungi: I've approved https://review.opendev.org/#/c/727388/1 but I noticed a thing in it that may be a preexisting bug we should followup on | 20:01 |
| fungi | clarkb: good catch, i'll push a followup now | 20:02 |
| fungi | correcting any in that file | 20:02 |
| openstackgerrit | Jeremy Stanley proposed opendev/system-config master: Add missing HTTPS ports in ssldomains file https://review.opendev.org/727418 | 20:04 |
| fungi | clarkb: ^ thanks again | 20:04 |
| fungi | those seemed to be the only two occurrences anyway | 20:05 |
| fungi | anyway, time to cook dinner, i'll check back after and see if ansible playbooks are up to the point where i can mount those logical volumes and test | 20:06 |
| openstackgerrit | Albin Vass proposed zuul/zuul-jobs master: Don't require tox_envlist https://review.opendev.org/726829 | 20:13 |
| openstackgerrit | Albin Vass proposed zuul/zuul-jobs master: Don't require tox_envlist https://review.opendev.org/726829 | 20:14 |
| *** dpawlik has joined #opendev | 20:17 | |
| openstackgerrit | Merged openstack/project-config master: Add OVH CI mirrors to opendev.org map https://review.opendev.org/727392 | 20:17 |
| *** sshnaidm is now known as sshnaidm|afk | 20:18 | |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: package-installs: allow when filter to be a list https://review.opendev.org/727049 | 20:22 |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: ubuntu-minimal: fix HWE install for focal https://review.opendev.org/727050 | 20:22 |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: ubuntu-minimal : only install 16.04 HWE kernel on xenial https://review.opendev.org/726996 | 20:22 |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: ubuntu-minimal: Add Ubuntu Focal test build https://review.opendev.org/725752 | 20:22 |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: block device: update variable name https://review.opendev.org/727431 | 20:22 |
| ianw | clarkb: ^ if you could look over that stack at some point, it should give us arm64 focal builds | 20:25 |
| ianw | and also stop us using the 16.04 hwe kernel on 10.04 for arm64 | 20:26 |
| ianw | s/10.04/18.04/ even | 20:26 |
| ianw | i'll have to double-check on Xinliang's comments about the variants available on arm64 | 20:26 |
| ianw | https://www.flake8rules.com/rules/E741.html (I O l get confused for 1/0) seems to have come up a few times with i guess new flake8 ... not sure how i feel about it | 20:28 |
| ianw | i note the font they use on the title of that page makes them indistinguishable; but fonts I code with I don't recall ever having an issue | 20:29 |
| clarkb | ianw: I purposefully select a font that dinstuishes O/0, 1/l and so on. | 20:30 |
| clarkb | source code pro | 20:30 |
| corvus | i'm opposed to enabling e741 | 20:36 |
| corvus | (also, i thought we dealt with this like a year ago already) | 20:36 |
| corvus | mordred, clarkb: is https://review.opendev.org/727367 okay for us? not going to cause memory issues? | 20:36 |
| corvus | (i think the current state there is "dont use jemalloc fixes all problems on any python version" so i think it's okay, just double checking) | 20:37 |
| clarkb | corvus: yes that is my assumption | 20:37 |
| clarkb | I think it should be safe. we saw similar memory behavior with 3.8 and 3.7. Removing jemalloc fixed 3.8, yet to be confirmed for 3.7 bu expected to be the same situation | 20:38 |
| mordred | agree | 20:38 |
| mordred | we should, obviously, pay attention when we restart :) | 20:39 |
| *** yuri has joined #opendev | 20:53 | |
| *** hashar has quit IRC | 21:01 | |
| openstackgerrit | Tristan Cacqueray proposed zuul/zuul-jobs master: fetch-sphinx-tarball: introduce zuul_use_fetch_output https://review.opendev.org/681870 | 21:12 |
| clarkb | ianw: did some digging on kernel packages for arm and x86 and I think we only have to not-arch: arm64 the -kvm kernel package | 21:25 |
| mordred | ianw: I built a focal arm image - I'm guessing it was probably not very good given those things :) | 21:26 |
| openstackgerrit | James E. Blair proposed opendev/system-config master: Run Zuul, Nodepool, and Zookeeper as the "container" user https://review.opendev.org/726958 | 21:27 |
| mordred | ianw: fwiw: https://i.imgur.com/eG1yKbp.png | 21:30 |
| mordred | ianw: (that's what that commit message looks like in gerrit for me) | 21:30 |
| mnaser | minikube released a new version 21 hours ago and our k8s jobs have been failing since | 21:31 |
| mnaser | it seems to be related to the CNI to an extent, but the logs don't provide much help right now :( | 21:31 |
| clarkb | mordred: https://i.imgur.com/vNfosYN.png that is what it looks like to me | 21:31 |
| mnaser | is it possible to get an autohold to debug a bit more and 1) capture _why_ the CNI is not working (it's just giving timeouts) and 2) add appropriate logging to capture the thing that is borked | 21:31 |
| *** Spock_ncc1701 has joined #opendev | 21:32 | |
| mordred | clarkb: yeah - that's a much better font :) | 21:32 |
| mnaser | seeing things like "failed to read pod IP from plugin/docker: Couldn't find network status for kube-system/coredns-66bff467f8-lrrsh through plugin: invalid network status for" but it looks like things are borked | 21:33 |
| mordred | clarkb: in the console/vim there is no confusion for me - my main point mostly being that it's entirely likely that people amongst our contributor base may have browsers that render things confusingly | 21:33 |
| clarkb | mnaser: can you give us nifo on the jobs/changes to hold for? | 21:33 |
| clarkb | mordred: ya, I still think people should use better fonts :) | 21:33 |
| mnaser | clarkb: change # 727217 -- job: "openstack-operator:functional" | 21:34 |
| mordred | clarkb: I honestly have no idea how to improve that font situation | 21:34 |
| mnaser | that should be enough | 21:34 |
| mnaser | actually this will test a really interesting scenario | 21:34 |
| clarkb | mordred: if you were on linux I could tell you how | 21:34 |
| mnaser | autohold with a buildset running | 21:34 |
| mordred | clarkb: it's a web page being rendered by firefox | 21:34 |
| mnaser | buildset registry* | 21:34 |
| mordred | clarkb: yeah - just saying - I mean, I don't override fonts in websites | 21:34 |
| mnaser | i wonder if it'll kill the buildset registry with autohold or it'll keep it around | 21:35 |
| mordred | and I imagine most people don't override fonts in websites | 21:35 |
| clarkb | mordred: ya gerrit is just using the monospace font there | 21:35 |
| clarkb | mordred: so you should pick a better monospace font in firefox and that will fix it I think | 21:35 |
| mordred | or we could put in a css setting into our gerrit to pick a better monospace font by default - like perhaps hack | 21:35 |
| clarkb | firefox -> preferences -> general -> fonts and colors -> advanced | 21:35 |
| clarkb | mordred: ya does hack distinguish though? | 21:36 |
| mordred | very much so | 21:36 |
| clarkb | cool. I use source code pro because whatever font I had way back when for monospace didn't distinguish | 21:36 |
| clarkb | I guess if we set hack and it isn't also served or pointed to on the internet then the brwoser will fall back to default monospaced font? | 21:38 |
| clarkb | so we may needt to do a bit more to make it work as expected but that should still be doable | 21:38 |
| clarkb | mnaser: I've run the hold command. Though I've realized I didn't use the container so I Think it was using the old global install. I don't think that is a problem though | 21:40 |
| mnaser | yeah i think it'll probably be fine | 21:40 |
| clarkb | mordred: ^ that should've continued to work bceause gearman right? | 21:40 |
| mnaser | let me kick a recheck | 21:40 |
| fungi | mordred: clarkb: are hack and source code pro distributed under free licenses? | 21:41 |
| clarkb | fungi: source code pro is | 21:41 |
| clarkb | looks like hack is as well | 21:41 |
| fungi | i was using a classic c64 font in my terminal, but recently switched to excelsior | 21:41 |
| clarkb | https://github.com/source-foundry/Hack#license and https://github.com/adobe-fonts/source-code-pro/blob/release/LICENSE.md | 21:42 |
| corvus | i just want a reality check here -- are we really planning on customizing gerrit's font? and doing that before the upgrade to 2.16? | 21:42 |
| clarkb | corvus: no I don't think so | 21:42 |
| clarkb | corvus: I think we are saying it is possible to do that and a simple change might help a chunk of users if we can identify a widely installed monospaced font with distinguishable glyphs. Fixing it more broadly is more effort | 21:42 |
| clarkb | (and maybe gerrit itself should fix it) | 21:43 |
| corvus | ok. because if folks are just shooting the breeze about fonts, cool, i'll go back to work. | 21:43 |
| corvus | clarkb: maybe gerrit itself *has* fixed it :) | 21:43 |
| clarkb | corvus: good point | 21:43 |
| clarkb | curiously the "Hack Regular" I have installed on tumbleweed doesn't seem to match what they've got on their website | 21:45 |
| clarkb | oh because I'm a derp | 21:46 |
| corvus | we decided to ignore 741 in zuul 2.5 years ago. i still think that's a good decision. this is not a big problem, but to the extent it is a problem, it should not be solved by a linting rule. code editors and viewers should have good fonts. that includes gerrit, but i don't think we should take up the issue until after we're running 3.x (iff it's still an issue). | 21:46 |
| mordred | corvus: yeah - my main thing was that I don't think it's super helpful for us to say "people shouldn't be confused by this because they should have good fonts" when we are serving an application for viewing code that in fact is configured to show the code in fonts that are inappropriate. I agree that it shoudl be fixed - just that we should keep the snark towards people experiencing the problem to a minimum | 21:48 |
| mordred | whilst we are running a service that contributes currently to the issue | 21:48 |
| corvus | i read no snark | 21:48 |
| mordred | I 100% agree that the soiution on our side is "get gerrit upgraded" | 21:48 |
| clarkb | I think its also fair to say "use a better monospace font" is an option people have in the interim (and they'll need it for their code editor either way) | 21:48 |
| openstackgerrit | melanie witt proposed zuul/zuul-jobs master: DNM Try out running sphinx-build in parallel for releasenotes https://review.opendev.org/727473 | 21:49 |
| openstackgerrit | Jeremy Stanley proposed opendev/system-config master: Update docs.airshipit.org functional test https://review.opendev.org/727474 | 21:50 |
| fungi | a quick review of ^ would be appreciated to get system-config changes merging again | 21:50 |
| clarkb | fungi: maybe we should update that to be a simple 2XX return code check? | 21:51 |
| clarkb | +2 either way | 21:51 |
| *** panda has quit IRC | 21:51 | |
| fungi | yeah, i gather the intent of that test is to ensure we're serving the correct content from the correct vhosts | 21:52 |
| clarkb | thats a good point, but hard to ensure that remains fixed over time | 21:52 |
| fungi | so dropping back to an http 200 check would lose that property, if we consider it important | 21:52 |
| fungi | there may be other ways to improve the job though to retain that degree of robustness without relying on real-world changing content to do so | 21:53 |
| *** panda has joined #opendev | 21:53 | |
| *** roman_g has quit IRC | 21:53 | |
| fungi | any of the sites being tested in that job have a roughly equal chance of causing the same sort of regression over time | 21:54 |
| openstackgerrit | Guillaume Chauvel proposed zuul/zuul-jobs master: tox siblings installed packages: Add PEP 440 direct reference format https://review.opendev.org/727475 | 21:54 |
| *** roman_g has joined #opendev | 21:58 | |
| ianw | clarkb: thanks, i'm not sure what the semantics are though for kernels that don't exist with package-installs ... it's not really defined | 21:58 |
| ianw | corvus: i've just seen at least 2 changes fly by, and i think it's actually probably related to projects dropping py2 support and in the process also jumping flake8 versions maybe | 21:59 |
| clarkb | ya openstack is running into issues with it I think | 21:59 |
| ianw | i don't really care, just maybe something old becoming new again | 22:00 |
| mordred | ianw: flake8 made a release in teh last 24 hours | 22:00 |
| clarkb | there was a bunp to fix some other problem hwihc pulled in new rules | 22:00 |
| clarkb | and lots of projects don't pin that toolchain anymore I guess | 22:00 |
| mordred | my understanding is that openstack is working on having hacking pin it | 22:00 |
| clarkb | mordred: I thought hackign already did? | 22:00 |
| ianw | i thought we pinned and updated on releases | 22:00 |
| mordred | https://review.opendev.org/#/c/727133/ | 22:01 |
| clarkb | ianw: that was the old system and apparently it isn't used anymore everywhere | 22:01 |
| mordred | it was pinned to <4 | 22:01 |
| mordred | but that wasn't good enough :) | 22:01 |
| ianw | well it would not be the first instance of me being well behind the times! | 22:01 |
| clarkb | mordred: aha | 22:01 |
| mordred | the other new-but-breaking change is *wonderful* and has already caught actual bugs :) | 22:02 |
| ianw | mordred: did you include the vm element when building focal arm64 images? i'm not totally sure how they could have built as it would have tried to use a non-existant kernel | 22:02 |
| mordred | but - is also breaking/churn-inducing | 22:02 |
| clarkb | mordred: whats the other change? | 22:03 |
| mordred | clarkb: checks for unused format arguments | 22:03 |
| ianw | focal mirror testing passed with fresh ansible 2.9.8 -> https://review.opendev.org/#/c/726970/ | 22:03 |
| mordred | clarkb: which caused me to find a place in sdk where we were passing arguments meant for the surrounding method to to the format string which was silently ignoring them :) | 22:03 |
| mordred | ianw: oh - you know what? I think I didn't actually build a focal image | 22:04 |
| mordred | ianw: forgot to set DIB_RELEASE | 22:04 |
| mordred | ianw: how about I delete that image now :) | 22:04 |
| ianw | the static job failures seem to be related to asserts checking airship site ... assert 'Airship Treasuremap' in '\n\n<!DOCTYPE html> | 22:05 |
| fungi | ianw: yes, please review https://review.opendev.org/727474 | 22:06 |
| fungi | once that merges i can recheck the ovh mirrors change | 22:07 |
| ianw | ahh, very good. well then i think we've missed the boat with ovh, but post upgrading bridge to 2.9.8 (https://review.opendev.org/726981) i think we can try focal control plane hosts if we like | 22:08 |
| fungi | yeah, i'd rather not leave a substantial chunk of our ci resources offline any longer | 22:08 |
| fungi | but the experimentation was useful, sounds like | 22:08 |
| mordred | ianw: that has a sad panda | 22:09 |
| ianw | mordred: yeah it's the aforementioned static airship thing | 22:09 |
| mordred | oh - that's the thing | 22:09 |
| mordred | yeah | 22:09 |
| corvus | mordred: https://review.opendev.org/726472 is failing because cacti02 is not in the inventory in the gate job; so i think the approach generally works, it's just that we have one firewall rule that applies to every host, and it's referencing a host in the inventory which isn't present in most gate tests. | 22:09 |
| * mordred is now caught up | 22:09 | |
| mordred | corvus: yay! and also boo | 22:10 |
| corvus | mordred: we might solve this by adding the 'cacti' group as in https://review.opendev.org/726475 -- but there's the question of testing that... | 22:10 |
| mordred | yeah | 22:10 |
| mordred | corvus: we could add cacti02.openstack.org to the inventory in run-base | 22:12 |
| corvus | maybe we add one of the 'base' nodes to the cacti group in the gate test, so that 'bionic' is a cacti host and we add it to the firewall rules? or we add a cacti host (as you just suggested)...? | 22:12 |
| mordred | it would mean adding an iptables rule in test allowing a produciton serever ingress ... | 22:12 |
| mordred | or yeah - your thing | 22:12 |
| mordred | corvus: or - we could add the bridge host to the cacti group in run-bae | 22:13 |
| mordred | base | 22:13 |
| mordred | corvus: it's always there | 22:13 |
| corvus | mordred: oh were you suggesting we add the real cacti host to the test inventory? | 22:13 |
| mordred | yeah | 22:13 |
| corvus | mordred: i think that might cause us to run base roles on it? | 22:13 |
| mordred | corvus: oh - that's an excellent point - let's not do that | 22:13 |
| corvus | i mean, it's not dangerous; it'd fail in check. but it won't work. :) | 22:14 |
| mordred | yeah | 22:14 |
| mordred | what about adding bridge to the cacti group? | 22:14 |
| mordred | that way the roles will run on a server they don't need to run on - but it's not likely to be dangeous - and we'll have a known value we can check other hosts's iptables rules for | 22:14 |
| corvus | seems okay? maybe a little squirrely in terms of 'polluting' the test, but probably not a problem in practice? | 22:15 |
| corvus | here's another option: we leave the exesting dns_a filter in place for hosts, and pretty much just use it for this case, while we use the new hostvars group stuff for everything else | 22:15 |
| mordred | "does zk01 have a rule allowing cacti traffic from {{ bridge.openstack.org.public_v4 }}" | 22:15 |
| mordred | corvus: that's a great idea | 22:16 |
| mordred | corvus: because "open a port to cacti" is different than "please connect me to all of the zk servers whichever they happen to be" | 22:16 |
| corvus | mordred: well, i was planning on using the group thing for cacti too: "open ports for all the cacti servers" | 22:16 |
| corvus | but it's not a big cost to keep that as it is | 22:17 |
| corvus | we don't change that server often | 22:17 |
| mordred | yeah - it would be nice to do it all that way | 22:17 |
| mordred | but adding a cacti server to every change is maybe a bit much | 22:17 |
| corvus | well, how about i just do that for now, and we can revisit the cacti thing later if we want | 22:17 |
| mordred | ++ | 22:18 |
| openstackgerrit | James E. Blair proposed opendev/system-config master: Add iptables_extra_allowed_groups https://review.opendev.org/726475 | 22:20 |
| *** Spock_ncc1701 has quit IRC | 22:20 | |
| mordred | corvus: you are so much better at jinja filters than I am | 22:27 |
| openstackgerrit | Tristan Cacqueray proposed zuul/zuul-jobs master: fetch-sphinx-tarball: introduce zuul_use_fetch_output https://review.opendev.org/681870 | 22:32 |
| corvus | mordred: well, we'll see if that passes | 22:33 |
| *** roman_g has quit IRC | 22:33 | |
| clarkb | corvus: mordred one queston on that inline | 22:35 |
| mnaser | clarkb: if you're still around, the job has failed -- node #0016528215 -- 88afcc93-415e-4267-9834-23de32b00d33 at rax-iad, ip: 104.130.26.157 | 22:38 |
| mnaser | https://github.com/mnaser.keys is my keys :) | 22:38 |
| clarkb | mnaser: one moment please | 22:38 |
| clarkb | mnaser: root@104.130.26.157 should work for you now | 22:41 |
| mnaser | clarkb: im in, thank you | 22:42 |
| clarkb | we've got quite a few held nodes. might need to do some cleanup | 22:42 |
| clarkb | they are all fairly recent things though so maybe not | 22:43 |
| clarkb | looks like ianw has some pip and docker stuff in flight and corvus is debugging the zuul/nodepool jobs? | 22:43 |
| ianw | clarkb: i'll run through mine in a bit, i'm sure most can go | 22:44 |
| openstackgerrit | Monty Taylor proposed zuul/zuul-jobs master: Combine javascript deployment and deployment-tarball jobs https://review.opendev.org/727370 | 22:45 |
| openstackgerrit | Monty Taylor proposed zuul/zuul-jobs master: Combine javascript deployment and deployment-tarball jobs https://review.opendev.org/727370 | 22:46 |
| *** tkajinam has joined #opendev | 22:54 | |
| openstackgerrit | Merged opendev/system-config master: Update docs.airshipit.org functional test https://review.opendev.org/727474 | 23:14 |
| openstackgerrit | melanie witt proposed zuul/zuul-jobs master: DNM Try out running sphinx-build in parallel for releasenotes https://review.opendev.org/727473 | 23:14 |
| *** tosky has quit IRC | 23:14 | |
| clarkb | fungi: ^ I think you can recheck now | 23:14 |
| fungi | clarkb: indeed, perfect timing | 23:28 |
| *** mlavalle has quit IRC | 23:53 | |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!
