Tuesday, 2020-05-12

ianwit looks like it's only in devel00:03
ianwno hang on, it is on 2.900:05
openstackgerritIan Wienand proposed opendev/system-config master: Update to Ansible 2.9.8  https://review.opendev.org/72698100:07
ianwhang on, we're too bleeding edge00:09
ianwmattclay committed 40 minutes ago ... that's the 2.9.8 release commit but it's not released, yet00:09
ianwhttps://github.com/ansible/ansible/commit/ed2a39d68e17e1b1d4c5670e1847cf448cf20f5300:09
ianwit is quite awesome that we find this in the gate, and not in production however :)00:14
*** mlavalle has quit IRC00:15
*** diablo_rojo has joined #opendev00:17
openstackgerritMerged opendev/system-config master: Move build-essential arm64 things to base  https://review.opendev.org/72603900:57
openstackgerritMerged opendev/system-config master: service-bridge: skip osc/kubectl things for arm64  https://review.opendev.org/72604000:57
clarkb++01:08
*** tkajinam has quit IRC01:29
*** tkajinam has joined #opendev01:29
ianw... interesting ... bionic has an arm 16.04-hwe kernel, which we've been unconditionally installing01:30
clarkbI think we install hwe because xenial non hwe had bugs on arm?01:34
clarkbIm guessing bionic inherited that01:34
*** ysandeep|away is now known as ysandeep01:36
ianwyeah, unintentionally01:38
openstackgerritIan Wienand proposed openstack/diskimage-builder master: [wip] focal ubuntu-minimal testing  https://review.opendev.org/72575201:43
openstackgerritIan Wienand proposed openstack/diskimage-builder master: ubuntu-minimal : only install 16.04 HWE kernel on xenial  https://review.opendev.org/72699601:43
cmurphyianw: on our train branches we're still seeing "virtualenv: command not found" on opensuse even with https://review.opendev.org/726526 e.g. https://zuul.opendev.org/t/openstack/build/f2ed9a3541d34f6598ef133b1c183bc001:46
ianwcmurphy: hrmm, ok let me look01:54
ianwi guess it's running under python202:01
ianwwe can either run ensure-virtualenv or unconditionally use "python3 -m venv"02:01
ianwcmurphy: https://review.opendev.org/727000 Add ensure-virtualenv role .. should do it, will test02:07
cmurphyianw: awesome, ty02:21
ianwwell thank me if/when it works :)02:21
ianw[repo-update|http://mirror.dfw.rax.opendev.org/opensuse/update/leap/15.1/oss/] Valid metadata not found at specified URL03:27
ianwthey all seemed to fail with something like that03:27
ianwhttps://static.opendev.org/mirror/logs/rsync-mirrors/opensuse.log seems like things are in sync03:29
ianwAJaeger: ^ perhaps something to look into?  see results @ https://review.opendev.org/#/c/726628/03:30
openstackgerritIan Wienand proposed openstack/diskimage-builder master: ubuntu-minimal : only install 16.04 HWE kernel on xenial  https://review.opendev.org/72699604:07
openstackgerritIan Wienand proposed openstack/diskimage-builder master: [wip] focal ubuntu-minimal testing  https://review.opendev.org/72575204:07
openstackgerritIan Wienand proposed openstack/diskimage-builder master: package-installs: allow when filter to be a list  https://review.opendev.org/72704904:07
openstackgerritIan Wienand proposed openstack/diskimage-builder master: ubuntu-minimal: fix HWE install for focal  https://review.opendev.org/72705004:07
*** diablo_rojo has quit IRC04:25
*** ykarel|away is now known as ykarel04:41
ianwmordred / clarkb: ^ so that stack should fix things up to build arm64 focal images04:43
openstackgerritIan Wienand proposed openstack/diskimage-builder master: ubuntu-minimal: Add Ubuntu Focal test build  https://review.opendev.org/72575204:45
openstackgerritIan Wienand proposed openstack/diskimage-builder master: ubuntu-minimal: Add Ubuntu Focal test build  https://review.opendev.org/72575204:46
AJaegerdirk, cmurphy, do you konw what mirror to use? See ianw's question above05:17
cmurphyis it/was it in the middle of updating? that repo seems to work for me now05:22
AJaegercmurphy: looks like it.05:23
AJaeger"epodata/84548613659f4ab18f2cd48ee350fd4f99da47aa947597b61462f5243a0860f3-primary.xml.gz' not found on medium 'http://mirror.dfw.rax.opendev.org/opensuse/update/leap/15.1/oss/'" - that file exists now05:23
ianwcmurphy: our updates should be atomic ... but if the mirror was out of sync we would have an atomically out of date mirror :)05:23
AJaegerianw: I think we can recheck and should be good again05:23
ianwthtat was @ 2020-05-12 02:20:52.89773705:24
ianw+ date --iso-8601=ns05:26
ianw2020-05-12T02:21:17,796218602+00:0005:26
ianwthat is suspiciously close05:26
ianw(that's from https://static.opendev.org/mirror/logs/rsync-mirrors/opensuse.log)05:27
openstackgerritAndreas Jaeger proposed zuul/zuul-jobs master: Add new non-npm specific javascript jobs  https://review.opendev.org/72654705:38
openstackgerritAndreas Jaeger proposed zuul/zuul-jobs master: Extract ensure-javascript-build-tool role  https://review.opendev.org/72690005:38
openstackgerritAndreas Jaeger proposed zuul/zuul-jobs master: Add new non-npm specific javascript jobs  https://review.opendev.org/72654705:53
openstackgerritAndreas Jaeger proposed zuul/zuul-jobs master: Extract ensure-javascript-build-tool role  https://review.opendev.org/72690005:53
*** dpawlik has joined #opendev06:05
openstackgerritOpenStack Proposal Bot proposed openstack/project-config master: Normalize projects.yaml  https://review.opendev.org/72706206:10
dirkianw: the problem is that zypper revaluates it's local cache only once per hour. If it changes in-between then it throws an ugly error06:11
dirkWe could change the invocation to always revalidate, which should not be a performance problem as the mirror is local06:13
*** tkajinam has quit IRC06:35
*** tkajinam has joined #opendev06:36
openstackgerritAndreas Jaeger proposed openstack/project-config master: Fix flake8 testing  https://review.opendev.org/72707006:41
*** roman_g has quit IRC07:00
ianwfungi: so not much luck with ovh mirrors; as of now we don't have the right flavors and as of now ansible 2.9.8 is not released either (which would be required for a focal node) so ... yeah it's pretty much where you left it :)07:08
*** ralonsoh has joined #opendev07:33
*** tosky has joined #opendev07:35
*** lpetrut has joined #opendev07:52
*** dtantsur|afk is now known as dtantsur07:54
*** rpittau|afk is now known as rpittau07:58
*** ykarel is now known as ykarel|lunch08:22
*** slaweq has joined #opendev08:27
slaweqfrickler: hi, maybe we can continue here08:27
slaweqfrickler: so I have patch https://review.opendev.org/727103 for tobiko08:28
slaweqfrickler: and in fact the issue may happen in one of the jobs: tobiko-devstack-faults-centos-7 or tobiko-devstack-faults-ubuntu-bionic08:28
fricklerslaweq: o.k., I'll set holds for both08:29
*** ysandeep is now known as ysandeep|lunch08:31
slaweqfrickler: thx a lot08:32
*** lpetrut_ has joined #opendev08:39
*** lpetrut has quit IRC08:42
*** ykarel|lunch is now known as ykarel08:51
*** sshnaidm|afk is now known as sshnaidm08:52
fricklerslaweq: welcome, please let me know your ssh key (once again ;) so I can give you access once the node(s) got held08:58
slaweqfrickler: http://paste.openstack.org/show/793417/09:00
*** priteau has joined #opendev09:02
*** DSpider has joined #opendev09:23
*** ysandeep|lunch is now known as ysandeep09:25
*** jhesketh has joined #opendev09:31
ttxLooks like the latest ptgbot changes were not picked up by the puppet-ptgbot module... Is there anything new that needs to be done for it to happen?09:31
ttx /cc fungi ^09:31
openstackgerritAlbin Vass proposed zuul/zuul-jobs master: Ensure output dirs are empty  https://review.opendev.org/72713509:54
openstackgerritAlbin Vass proposed zuul/zuul-jobs master: Ensure output dirs are empty  https://review.opendev.org/72713510:03
*** ysandeep is now known as ysandeep|brb10:09
slaweqfrickler: ping, one of the jobs failed: https://06d50ddbf1635bcfe515-db6919ac06726062cc29a59f6b4e7f0a.ssl.cf2.rackcdn.com/727103/1/check/tobiko-devstack-faults-centos-7/3949807/10:10
*** yuri has quit IRC10:15
openstackgerritAlbin Vass proposed zuul/zuul-jobs master: Ensure output dirs are empty  https://review.opendev.org/72713510:15
*** rpittau is now known as rpittau|bbl10:19
*** dpawlik has quit IRC10:21
*** dpawlik has joined #opendev10:21
fricklerslaweq: added you for root access on all three nodes, let me know if you need anything further. I'm still keeping the hold for the bionic job unless you tell me you won't need it11:00
*** ysandeep|brb is now known as ysandeep11:11
openstackgerritAlbin Vass proposed zuul/zuul-jobs master: Drop support for ansible 2.6  https://review.opendev.org/72715811:13
slaweqfrickler: thx a lot, I will ping You once I will finish11:46
*** hashar has joined #opendev11:52
*** rpittau|bbl is now known as rpittau12:21
openstackgerritAndreas Jaeger proposed openstack/project-config master: Remove noop-jobs from oslo.tools  https://review.opendev.org/72717812:29
*** zbr_ has joined #opendev12:35
*** zbr_ has quit IRC12:36
*** slittle1 has quit IRC12:38
*** elod_pto is now known as elod12:42
fungittx: looking now13:00
fungithe bot was last restarted may 1113:01
fungiwhich i guess was yesterday13:02
fungiyeah, 16:05 utc yesterday13:02
fungicurrent git checkout and installation of the software is cc11180 from april 113:04
*** slittle1 has joined #opendev13:07
fungimordred: looking at /var/log/ansible/service-eavesdrop.yaml.log on bridge.o.o i see the install-ansible-roles task failing with "fatal: destination path '/etc/ansible/roles/puppet' already exists and is not an empty directory."13:09
fungii have to assume that's not expected13:09
slaweqfrickler: infra_root: hi, I'm done with debugging on the nodes 104.239.136.110, 104.239.141.232 and 104.239.136.11113:11
slaweqYou can clean them now13:12
slaweqthx a lot for help13:12
mordredfungi: that's the thing we fixed yesterday :(13:12
mordredfungi: oh - so yeah - last time that ran was before we fixed it13:14
mordredfungi: so we should just trigger a new run13:14
fricklerslaweq: deleted those nodes, do you still want the bionic hold or should I drop that one, too?13:14
ttxfungi: there was code posted since then which it did not pick up... In theory the puppet run should trigger and deploy new files ?13:15
ttxLike 6 days ago acf550e1a2 pushed a new version of logo.png which the puppet run is supposed to deploy13:15
fungimordred: aha, so i can reenqueue into deploy the last change which ran infra-prod-service-eavesdrop13:16
fungittx: yes, but how puppet gets run on eavesdrop has changed as of the past month to be more immediate when changes merge (in theory)13:16
ttxah, that would certainly explain13:16
fungiand is no longer being run periodically, i guess13:16
ttxfungi: should I add a new job to ptgbot that would pick up changes immediately?13:17
mordredttx: sadly that particular combo is a bit more problematic13:18
fungittx: the infra-prod-service-eavesdrop job should in theory take care of it, i think. i've reenqueued 725103,2 into the deploy pipeline, which is the most recent change to have triggered that job13:19
ttxok let's see13:19
fungiand at least if it doesn't, then we can continue to troubleshoot from there13:19
openstackgerritMerged zuul/zuul-jobs master: Add new non-npm specific javascript jobs  https://review.opendev.org/72654713:37
openstackgerritMerged openstack/project-config master: Fix flake8 testing  https://review.opendev.org/72707013:39
*** redrobot has quit IRC13:39
openstackgerritMerged zuul/zuul-jobs master: Fail and direct user to use ensure-* version of roles  https://review.opendev.org/72644813:42
*** openstackstatus has quit IRC13:53
*** openstackstatus has joined #opendev13:53
*** ChanServ sets mode: +v openstackstatus13:53
slaweqfrickler: no, You can drop it, thx a lot13:59
ttxmordred, fungi: job reported success but the log file is full of errors, so not sure it worked14:01
ttxFrom the ptgbot perspective, it did work, so I'm happy enough14:01
fungittx: thanks, i'll dig deeper on the errors, but glad that much is working14:05
openstackgerritMerged openstack/project-config master: Normalize projects.yaml  https://review.opendev.org/72706214:06
*** pramchan has joined #opendev14:12
fungittx: oh, if you mean this error it's expected: https://zuul.opendev.org/t/openstack/build/5883eb79d16641c39aaf6f7b698ac694/log/job-output.txt#40914:18
fungiit's running on our ssh bastion and we don't run the console logger there, for added safety14:19
fungiwe redirect and append the output to a local logfile on the bastion instead14:20
mordredone of these days we're going to finish fixing that console logging14:21
*** ysandeep is now known as ysandeep|afk14:26
*** tkajinam has quit IRC14:37
*** ralonsoh has quit IRC14:40
*** hashar has quit IRC14:40
*** ralonsoh has joined #opendev14:40
*** ralonsoh has quit IRC14:44
clarkbfungi: ttx I manually restarted ptgbot after it netsplit away14:48
fungiclarkb: yep, i remember seeing that14:49
fungiso that's why there was a restart yesterday14:49
clarkbyes14:50
openstackgerritMerged openstack/project-config master: Increase tag-releases job timeout  https://review.opendev.org/72652515:04
*** dtantsur is now known as dtantsur|afk15:08
*** ysandeep|afk is now known as ysandeep15:11
openstackgerritMerged openstack/project-config master: Set python-path for SUSE builds  https://review.opendev.org/72672815:23
*** mlavalle has joined #opendev15:27
openstackgerritAlbin Vass proposed zuul/zuul-jobs master: Add upload-artifactory role  https://review.opendev.org/72567815:31
openstackgerritMonty Taylor proposed zuul/zuul-jobs master: Split browser and non-browser targetting js jobs  https://review.opendev.org/72723115:33
*** roman_g has joined #opendev15:33
*** slittle1 has quit IRC15:35
*** Guest49293 has joined #opendev15:37
*** Guest49293 is now known as redrobot15:38
openstackgerritMonty Taylor proposed zuul/zuul-jobs master: Split browser and non-browser targetting js jobs  https://review.opendev.org/72723115:38
openstackgerritMonty Taylor proposed zuul/zuul-jobs master: Add firefox to javascript browser testing depends  https://review.opendev.org/72723415:38
openstackgerritMonty Taylor proposed zuul/zuul-jobs master: Split browser and non-browser targetting js test jobs  https://review.opendev.org/72723115:43
openstackgerritMonty Taylor proposed zuul/zuul-jobs master: Add firefox to javascript browser testing depends  https://review.opendev.org/72723415:43
openstackgerritMonty Taylor proposed zuul/zuul-jobs master: Split browser and non-browser targetting js test jobs  https://review.opendev.org/72723115:45
openstackgerritTristan Cacqueray proposed zuul/zuul-jobs master: fetch-sphinx-output: introduce zuul_use_fetch_output  https://review.opendev.org/68190515:47
*** lpetrut_ has quit IRC15:51
clarkbttx: smcginnis: mordred has mentioned there was a change made to zuul to not reconfigure zuul on tag events. We expect many tag events in the near future so mordred was suggesting we might restart the zuul scheduler today before that. I believe zuul did the reconfigure on tag events during previous releases so I think chances are we'll be fine but thought I'd bring it up here in case we want to go ahead with15:55
clarkba zuul restart15:55
clarkbthe biggest risk is it will slow zuul down as it has to process all of those unnecessary configuration updates.15:55
clarkbwe did just restart zuul on friday and it seems happy since then. I think the risk tehrefore is low that the restart will cause problems15:56
clarkbbut restarting does slwo down the queues as we have to restart all the jobs15:56
mordredclarkb, ttx: https://review.opendev.org/#/c/726213/ - it has merged15:56
mordredyeah - it's a tradeoff to be sure15:56
mordredbut if we hit a slow point, then perhaps a quick restart wouldn't be terrible15:57
smcginnisIf the team thinks it's safe that it won't cause any issues when we process our final releases, then I'm fine with getting in a quick restart before then.15:59
ttxwe'll be tagging a lot tomorrow morning for sure15:59
smcginnisttx were just talking about timing. Targeting 9UTC to kick off the release.15:59
ttxI have a hard time assessing which path is safest16:00
clarkbI expect we'll notice zuul slow down as it reconfigures, but it'll chew through that backlog then be fine16:00
smcginnisThat is the existing behavior for the past few series final releases, right?16:00
clarkbsmcginnis: yes I believe so16:00
clarkbI think corvus recently said its about 6 seconds per dynamic reconfiguration16:01
clarkbif that number is correct then I think we can expect a delay of 6 seconds * number of tags (roughly)16:01
smcginnisThere's > 85 repos being tagged. A few have multiple repos per deliverable file, so it's likely close to 100.16:03
openstackgerritTristan Cacqueray proposed zuul/zuul-jobs master: fetch-sphinx-tarball: introduce zuul_use_fetch_output  https://review.opendev.org/68187016:03
clarkbsmcginnis: ok ~10 minutes is probably not that bad all considered16:03
smcginnisInefficient, but not unreasonable.16:03
ttxnah it's fine, that patch takes a long time anyway16:03
clarkbmordred: ^ is that your understanding of how we will be affected?16:03
mordredclarkb: yeah - so with those numbers then it should probably be fine16:03
mordredand maybe we don't worry about it16:04
mordredit hasn't been enough of a problem for us to investigate before teh bmw folks hit the issue16:04
corvus++ will cost more than it will save16:04
smcginnisI guess I would feel safer not introducing behavior changes right before doing the big release, but up to y'all if you think it's worth doing or not.16:05
smcginnisAs long as we are good to go by 9UTC tomorrow morning.16:05
clarkbI'm hearing that we should leave it as is since we've lived through the behavior before and the expected impact, while noticeable, will be manageable16:07
ttx+116:07
openstackgerritAlbin Vass proposed zuul/zuul-jobs master: Add upload-artifactory role  https://review.opendev.org/72567816:09
*** rpittau is now known as rpittau|afk16:10
openstackgerritTristan Cacqueray proposed zuul/zuul-jobs master: fetch-sphinx-output: introduce zuul_use_fetch_output  https://review.opendev.org/68190516:15
openstackgerritTristan Cacqueray proposed zuul/zuul-jobs master: fetch-sphinx-tarball: introduce zuul_use_fetch_output  https://review.opendev.org/68187016:15
*** pramchan has quit IRC16:22
openstackgerritMerged zuul/zuul-jobs master: Extract ensure-javascript-build-tool role  https://review.opendev.org/72690016:28
openstackgerritMonty Taylor proposed zuul/zuul-jobs master: Split browser and non-browser targetting js test jobs  https://review.opendev.org/72723116:31
tobiashcorvus, mordred: 6 seconds per dynamic reconfigure is quick enough so this probably won't hurt much. Further zuul anyway does deduplication so if you push more tags at once that will result in only one reconfig16:37
mordredtobiash: good point16:37
openstackgerritTristan Cacqueray proposed zuul/zuul-jobs master: fetch-sphinx-output: introduce zuul_use_fetch_output  https://review.opendev.org/68190516:38
tobiashwe were hurt much more with 100-200s reconfig time and projects constantly pushing tags all day long16:39
*** dpawlik has quit IRC16:40
mordredtobiash: yikes!16:47
openstackgerritTristan Cacqueray proposed zuul/zuul-jobs master: fetch-sphinx-tarball: introduce zuul_use_fetch_output  https://review.opendev.org/68187016:54
openstackgerritTristan Cacqueray proposed zuul/zuul-jobs master: fetch-sphinx-tarball: add missing zuul_success default  https://review.opendev.org/72727216:54
*** ykarel is now known as ykarel|away16:55
openstackgerritMerged zuul/zuul-jobs master: Split browser and non-browser targetting js test jobs  https://review.opendev.org/72723117:10
*** priteau has quit IRC17:17
*** ysandeep is now known as ysandeep|away17:19
*** iurygregory has quit IRC17:27
openstackgerritAlbin Vass proposed zuul/zuul-jobs master: WIP: Remove requiring tox_envlist  https://review.opendev.org/72682917:33
openstackgerritTristan Cacqueray proposed zuul/zuul-jobs master: fetch-sphinx-tarball: introduce zuul_use_fetch_output  https://review.opendev.org/68187017:43
openstackgerritAlbin Vass proposed zuul/zuul-jobs master: Add upload-artifactory role  https://review.opendev.org/72567818:00
openstackgerritAlbin Vass proposed zuul/zuul-jobs master: Add upload-artifactory role  https://review.opendev.org/72567818:02
*** hashar has joined #opendev18:08
*** iurygregory has joined #opendev18:31
openstackgerritMonty Taylor proposed zuul/zuul-jobs master: Combine javascript deployment and deployment-tarball jobs  https://review.opendev.org/72737018:51
openstackgerritJeremy Stanley proposed opendev/zone-opendev.org master: Replace OVH CI mirrors  https://review.opendev.org/72737618:57
openstackgerritJeremy Stanley proposed opendev/system-config master: Replace OVH CI mirrors  https://review.opendev.org/72738819:12
openstackgerritJeremy Stanley proposed opendev/system-config master: Add OpenEdge CI mirror to Cacti config  https://review.opendev.org/72738919:12
*** diablo_rojo has joined #opendev19:13
openstackgerritMonty Taylor proposed zuul/zuul-jobs master: Combine javascript deployment and deployment-tarball jobs  https://review.opendev.org/72737019:16
openstackgerritJeremy Stanley proposed openstack/project-config master: Add OVH CI mirrors to opendev.org map  https://review.opendev.org/72739219:20
openstackgerritJeremy Stanley proposed openstack/project-config master: Revert "Temporarily disable OVH"  https://review.opendev.org/72739319:20
openstackgerritMerged opendev/zone-opendev.org master: Replace OVH CI mirrors  https://review.opendev.org/72737619:28
AJaegerinfra-root, speaking about ovh: Is swiftupload ok again? See https://review.opendev.org/#/c/726028/19:31
openstackgerritIan Wienand proposed opendev/system-config master: Update to Ansible 2.9.8  https://review.opendev.org/72698119:33
openstackgerritIan Wienand proposed opendev/system-config master: Add focal testing for mirror nodes  https://review.opendev.org/72697019:33
openstackgerritTristan Cacqueray proposed zuul/zuul-jobs master: fetch-sphinx-tarball: introduce zuul_use_fetch_output  https://review.opendev.org/68187019:50
openstackgerritAlbin Vass proposed zuul/zuul-jobs master: Drop support for ansible 2.6  https://review.opendev.org/72715819:51
openstackgerritAlbin Vass proposed zuul/zuul-jobs master: Drop support for ansible 2.7  https://review.opendev.org/72741019:51
openstackgerritGonĂ©ri Le Bouder proposed zuul/zuul-jobs master: bindep: ensure virtualenv is present  https://review.opendev.org/72741319:55
openstackgerritTristan Cacqueray proposed zuul/zuul-jobs master: fetch-sphinx-tarball: introduce zuul_use_fetch_output  https://review.opendev.org/68187019:55
*** slaweq has quit IRC19:57
ianwAJaeger: that's a good point ... maybe we just put it back and see if it fails?19:58
fungiclarkb: since you reviewed the other topic:ovh-mirrors changes can you also take a look at 727388 (it's the only other outstanding piece before we can move forward on those)19:58
mordredfungi: if we wanted to not have compose start containers, I'd suggest we shift back to podman rather than using systemd to start the containers19:58
mordredbecause then we can run them as a non-priv user in the first place19:59
AJaegerianw: we could...19:59
fungimordred: got it. i know we'd talked about having systemd unit shims calling docker-compose so we don't have to remember where/what those configs are19:59
mordredbut I don't know that we're to that point19:59
clarkbfungi: yes looking19:59
clarkbthen I'm grabbing lunch19:59
fungithanks!19:59
* mordred thinks one of the nice things about docker is that he doesn't have to deal with systemd - but maybe he's in the minority there20:00
fungii've already got the cinder volumes carved up, formatted, and configured to mount at /var/cache/apache2 and /var/cache/openafs on the new mirrors, but can't actually mount them yet until ansible applies the mirror playbook20:01
clarkbfungi: I've approved https://review.opendev.org/#/c/727388/1 but I noticed a thing in it that may be a preexisting bug we should followup on20:01
fungiclarkb: good catch, i'll push a followup now20:02
fungicorrecting any in that file20:02
openstackgerritJeremy Stanley proposed opendev/system-config master: Add missing HTTPS ports in ssldomains file  https://review.opendev.org/72741820:04
fungiclarkb: ^ thanks again20:04
fungithose seemed to be the only two occurrences anyway20:05
fungianyway, time to cook dinner, i'll check back after and see if ansible playbooks are up to the point where i can mount those logical volumes and test20:06
openstackgerritAlbin Vass proposed zuul/zuul-jobs master: Don't require tox_envlist  https://review.opendev.org/72682920:13
openstackgerritAlbin Vass proposed zuul/zuul-jobs master: Don't require tox_envlist  https://review.opendev.org/72682920:14
*** dpawlik has joined #opendev20:17
openstackgerritMerged openstack/project-config master: Add OVH CI mirrors to opendev.org map  https://review.opendev.org/72739220:17
*** sshnaidm is now known as sshnaidm|afk20:18
openstackgerritIan Wienand proposed openstack/diskimage-builder master: package-installs: allow when filter to be a list  https://review.opendev.org/72704920:22
openstackgerritIan Wienand proposed openstack/diskimage-builder master: ubuntu-minimal: fix HWE install for focal  https://review.opendev.org/72705020:22
openstackgerritIan Wienand proposed openstack/diskimage-builder master: ubuntu-minimal : only install 16.04 HWE kernel on xenial  https://review.opendev.org/72699620:22
openstackgerritIan Wienand proposed openstack/diskimage-builder master: ubuntu-minimal: Add Ubuntu Focal test build  https://review.opendev.org/72575220:22
openstackgerritIan Wienand proposed openstack/diskimage-builder master: block device: update variable name  https://review.opendev.org/72743120:22
ianwclarkb: ^ if you could look over that stack at some point, it should give us arm64 focal builds20:25
ianwand also stop us using the 16.04 hwe kernel on 10.04 for arm6420:26
ianws/10.04/18.04/ even20:26
ianwi'll have to double-check on Xinliang's comments about the variants available on arm6420:26
ianwhttps://www.flake8rules.com/rules/E741.html (I O l get confused for 1/0) seems to have come up a few times with i guess new flake8 ... not sure how i feel about it20:28
ianwi note the font they use on the title of that page makes them indistinguishable; but fonts I code with I don't recall ever having an issue20:29
clarkbianw: I purposefully select a font that dinstuishes O/0, 1/l and so on.20:30
clarkbsource code pro20:30
corvusi'm opposed to enabling e74120:36
corvus(also, i thought we dealt with this like a year ago already)20:36
corvusmordred, clarkb: is https://review.opendev.org/727367 okay for us?  not going to cause memory issues?20:36
corvus(i think the current state there is "dont use jemalloc fixes all problems on any python version" so i think it's okay, just double checking)20:37
clarkbcorvus: yes that is my assumption20:37
clarkbI think it should be safe. we saw similar memory behavior with 3.8 and 3.7. Removing jemalloc fixed 3.8, yet to be confirmed for 3.7 bu expected to be the same situation20:38
mordredagree20:38
mordredwe should, obviously, pay attention when we restart :)20:39
*** yuri has joined #opendev20:53
*** hashar has quit IRC21:01
openstackgerritTristan Cacqueray proposed zuul/zuul-jobs master: fetch-sphinx-tarball: introduce zuul_use_fetch_output  https://review.opendev.org/68187021:12
clarkbianw: did some digging on kernel packages for arm and x86 and I think we only have to not-arch: arm64 the -kvm kernel package21:25
mordredianw: I built a focal arm image - I'm guessing it was probably not very good given those things :)21:26
openstackgerritJames E. Blair proposed opendev/system-config master: Run Zuul, Nodepool, and Zookeeper as the "container" user  https://review.opendev.org/72695821:27
mordredianw: fwiw: https://i.imgur.com/eG1yKbp.png21:30
mordredianw: (that's what that commit message looks like in gerrit for me)21:30
mnaserminikube released a new version 21 hours ago and our k8s jobs have been failing since21:31
mnaserit seems to be related to the CNI to an extent, but the logs don't provide much help right now :(21:31
clarkbmordred: https://i.imgur.com/vNfosYN.png that is what it looks like to me21:31
mnaseris it possible to get an autohold to debug a bit more and 1) capture _why_ the CNI is not working (it's just giving timeouts) and 2) add appropriate logging to capture the thing that is borked21:31
*** Spock_ncc1701 has joined #opendev21:32
mordredclarkb: yeah - that's a much better font :)21:32
mnaserseeing things like "failed to read pod IP from plugin/docker: Couldn't find network status for kube-system/coredns-66bff467f8-lrrsh through plugin: invalid network status for" but it looks like things are borked21:33
mordredclarkb: in the console/vim there is no confusion for me - my main point mostly being that it's entirely likely that people amongst our contributor base may have browsers that render things confusingly21:33
clarkbmnaser: can you give us nifo on the jobs/changes to hold for?21:33
clarkbmordred: ya, I still think people should use better fonts :)21:33
mnaserclarkb: change # 727217 -- job: "openstack-operator:functional"21:34
mordredclarkb: I honestly have no idea how to improve that font situation21:34
mnaserthat should be enough21:34
mnaseractually this will test a really interesting scenario21:34
clarkbmordred: if you were on linux I could tell you how21:34
mnaserautohold with a buildset running21:34
mordredclarkb: it's a web page being rendered by firefox21:34
mnaserbuildset registry*21:34
mordredclarkb: yeah - just saying - I mean, I don't override fonts in websites21:34
mnaseri wonder if it'll kill the buildset registry with autohold or it'll keep it around21:35
mordredand I imagine most people don't override fonts in websites21:35
clarkbmordred: ya gerrit is just using the monospace font there21:35
clarkbmordred: so you should pick a better monospace font in firefox and that will fix it I think21:35
mordredor we could put in a css setting into our gerrit to pick a better monospace font by default - like perhaps hack21:35
clarkbfirefox -> preferences -> general -> fonts and colors -> advanced21:35
clarkbmordred: ya does hack distinguish though?21:36
mordredvery much so21:36
clarkbcool. I use source code pro because whatever font I had way back when for monospace didn't distinguish21:36
clarkbI guess if we set hack and it isn't also served or pointed to on the internet then the brwoser will fall back to default monospaced font?21:38
clarkbso we may needt to do a bit more to make it work as expected but that should still be doable21:38
clarkbmnaser: I've run the hold command. Though I've realized I didn't use the container so I Think it was using the old global install. I don't think that is a problem though21:40
mnaseryeah i think it'll probably be fine21:40
clarkbmordred: ^ that should've continued to work bceause gearman right?21:40
mnaserlet me kick a recheck21:40
fungimordred: clarkb: are hack and source code pro distributed under free licenses?21:41
clarkbfungi: source code pro is21:41
clarkblooks like hack is as well21:41
fungii was using a classic c64 font in my terminal, but recently switched to excelsior21:41
clarkbhttps://github.com/source-foundry/Hack#license and https://github.com/adobe-fonts/source-code-pro/blob/release/LICENSE.md21:42
corvusi just want a reality check here -- are we really planning on customizing gerrit's font?  and doing that before the upgrade to 2.16?21:42
clarkbcorvus: no I don't think so21:42
clarkbcorvus: I think we are saying it is possible to do that and a simple change might help a chunk of users if we can identify a widely installed monospaced font with distinguishable glyphs. Fixing it more broadly is more effort21:42
clarkb(and maybe gerrit itself should fix it)21:43
corvusok.  because if folks are just shooting the breeze about fonts, cool, i'll go back to work.21:43
corvusclarkb: maybe gerrit itself *has* fixed it :)21:43
clarkbcorvus: good point21:43
clarkbcuriously the "Hack Regular" I have installed on tumbleweed doesn't seem to match what they've got on their website21:45
clarkboh because I'm a derp21:46
corvuswe decided to ignore 741 in zuul 2.5 years ago.  i still think that's a good decision.  this is not a big problem, but to the extent it is a problem, it should not be solved by a linting rule.  code editors and viewers should have good fonts.  that includes gerrit, but i don't think we should take up the issue until after we're running 3.x (iff it's still an issue).21:46
mordredcorvus: yeah - my main thing was that I don't think it's super helpful for us to say "people shouldn't be confused by this because they should have good fonts" when we are serving an application for viewing code that in fact is configured to show the code in fonts that are inappropriate. I agree that it shoudl be fixed - just that we should keep the snark towards people experiencing the problem to a minimum21:48
mordredwhilst we are running a service that contributes currently to the issue21:48
corvusi read no snark21:48
mordredI 100% agree that the soiution on our side is "get gerrit upgraded"21:48
clarkbI think its also fair to say "use a better monospace font" is an option people have in the interim (and they'll need it for their code editor either way)21:48
openstackgerritmelanie witt proposed zuul/zuul-jobs master: DNM Try out running sphinx-build in parallel for releasenotes  https://review.opendev.org/72747321:49
openstackgerritJeremy Stanley proposed opendev/system-config master: Update docs.airshipit.org functional test  https://review.opendev.org/72747421:50
fungia quick review of ^ would be appreciated to get system-config changes merging again21:50
clarkbfungi: maybe we should update that to be a simple 2XX return code check?21:51
clarkb+2 either way21:51
*** panda has quit IRC21:51
fungiyeah, i gather the intent of that test is to ensure we're serving the correct content from the correct vhosts21:52
clarkbthats a good point, but hard to ensure that remains fixed over time21:52
fungiso dropping back to an http 200 check would lose that property, if we consider it important21:52
fungithere may be other ways to improve the job though to retain that degree of robustness without relying on real-world changing content to do so21:53
*** panda has joined #opendev21:53
*** roman_g has quit IRC21:53
fungiany of the sites being tested in that job have a roughly equal chance of causing the same sort of regression over time21:54
openstackgerritGuillaume Chauvel proposed zuul/zuul-jobs master: tox siblings installed packages: Add PEP 440 direct reference format  https://review.opendev.org/72747521:54
*** roman_g has joined #opendev21:58
ianwclarkb: thanks, i'm not sure what the semantics are though for kernels that don't exist with package-installs ... it's not really defined21:58
ianwcorvus: i've just seen at least 2 changes fly by, and i think it's actually probably related to projects dropping py2 support and in the process also jumping flake8 versions maybe21:59
clarkbya openstack is running into issues with it I think21:59
ianwi don't really care, just maybe something old becoming new again22:00
mordredianw: flake8 made a release in teh last 24 hours22:00
clarkbthere was a bunp to fix some other problem hwihc pulled in new rules22:00
clarkband lots of projects don't pin that toolchain anymore I guess22:00
mordredmy understanding is that openstack is working on having hacking pin it22:00
clarkbmordred: I thought hackign already did?22:00
ianwi thought we pinned and updated on releases22:00
mordredhttps://review.opendev.org/#/c/727133/22:01
clarkbianw: that was the old system and apparently it isn't used anymore everywhere22:01
mordredit was pinned to <422:01
mordredbut that wasn't good enough :)22:01
ianwwell it would not be the first instance of me being well behind the times!22:01
clarkbmordred: aha22:01
mordredthe other new-but-breaking change is *wonderful* and has already caught actual bugs :)22:02
ianwmordred: did you include the vm element when building focal arm64 images?  i'm not totally sure how they could have built as it would have tried to use a non-existant kernel22:02
mordredbut - is also breaking/churn-inducing22:02
clarkbmordred: whats the other change?22:03
mordredclarkb: checks for unused format arguments22:03
ianwfocal mirror testing passed with fresh ansible 2.9.8 -> https://review.opendev.org/#/c/726970/22:03
mordredclarkb: which caused me to find a place in sdk where we were passing arguments meant for the surrounding method to to the format string which was silently ignoring them :)22:03
mordredianw: oh - you know what? I think I didn't actually build a focal image22:04
mordredianw: forgot to set DIB_RELEASE22:04
mordredianw: how about I delete that image now :)22:04
ianwthe static job failures seem to be related to asserts checking airship site ... assert 'Airship Treasuremap' in '\n\n<!DOCTYPE html>22:05
fungiianw: yes, please review https://review.opendev.org/72747422:06
fungionce that merges i can recheck the ovh mirrors change22:07
ianwahh, very good.  well then i think we've missed the boat with ovh, but post upgrading bridge to 2.9.8 (https://review.opendev.org/726981) i think we can try focal control plane hosts if we like22:08
fungiyeah, i'd rather not leave a substantial chunk of our ci resources offline any longer22:08
fungibut the experimentation was useful, sounds like22:08
mordredianw: that has a sad panda22:09
ianwmordred: yeah it's the aforementioned static airship thing22:09
mordredoh - that's the thing22:09
mordredyeah22:09
corvusmordred: https://review.opendev.org/726472 is failing because cacti02 is not in the inventory in the gate job; so i think the approach generally works, it's just that we have one firewall rule that applies to every host, and it's referencing a host in the inventory which isn't present in most gate tests.22:09
* mordred is now caught up22:09
mordredcorvus: yay! and also boo22:10
corvusmordred: we might solve this by adding the 'cacti' group as in https://review.opendev.org/726475  --  but there's the question of testing that...22:10
mordredyeah22:10
mordredcorvus: we could add cacti02.openstack.org to the inventory in run-base22:12
corvusmaybe we add one of the 'base' nodes to the cacti group in the gate test, so that 'bionic' is a cacti host and we add it to the firewall rules?  or we add a cacti host (as you just suggested)...?22:12
mordredit would mean adding an iptables rule in test allowing a produciton serever ingress ...22:12
mordredor yeah - your thing22:12
mordredcorvus: or - we could add the bridge host to the cacti group in run-bae22:13
mordredbase22:13
mordredcorvus: it's always there22:13
corvusmordred: oh were you suggesting we add the real cacti host to the test inventory?22:13
mordredyeah22:13
corvusmordred: i think that might cause us to run base roles on it?22:13
mordredcorvus: oh - that's an excellent point - let's not do that22:13
corvusi mean, it's not dangerous; it'd fail in check.  but it won't work.  :)22:14
mordredyeah22:14
mordredwhat about adding bridge to the cacti group?22:14
mordredthat way the roles will run on a server they don't need to run on - but it's not likely to be dangeous - and we'll have a known value we can check other hosts's iptables rules for22:14
corvusseems okay?  maybe a little squirrely in terms of 'polluting' the test, but probably not a problem in practice?22:15
corvushere's another option: we leave the exesting dns_a filter in place for hosts, and pretty much just use it for this case, while we use the new hostvars group stuff for everything else22:15
mordred"does zk01 have a rule allowing cacti traffic from {{ bridge.openstack.org.public_v4 }}"22:15
mordredcorvus: that's a great idea22:16
mordredcorvus: because "open a port to cacti" is different than "please connect me to all of the zk servers whichever they happen to be"22:16
corvusmordred: well, i was planning on using the group thing for cacti too: "open ports for all the cacti servers"22:16
corvusbut it's not a big cost to keep that as it is22:17
corvuswe don't change that server often22:17
mordredyeah - it would be nice to do it all that way22:17
mordredbut adding a cacti server to every change is maybe a bit much22:17
corvuswell, how about i just do that for now, and we can revisit the cacti thing later if we want22:17
mordred++22:18
openstackgerritJames E. Blair proposed opendev/system-config master: Add iptables_extra_allowed_groups  https://review.opendev.org/72647522:20
*** Spock_ncc1701 has quit IRC22:20
mordredcorvus: you are so much better at jinja filters than I am22:27
openstackgerritTristan Cacqueray proposed zuul/zuul-jobs master: fetch-sphinx-tarball: introduce zuul_use_fetch_output  https://review.opendev.org/68187022:32
corvusmordred: well, we'll see if that passes22:33
*** roman_g has quit IRC22:33
clarkbcorvus: mordred one queston on that inline22:35
mnaserclarkb: if you're still around, the job has failed -- node #0016528215 -- 88afcc93-415e-4267-9834-23de32b00d33 at rax-iad, ip: 104.130.26.15722:38
mnaserhttps://github.com/mnaser.keys is my keys :)22:38
clarkbmnaser: one moment please22:38
clarkbmnaser: root@104.130.26.157 should work for you now22:41
mnaserclarkb: im in, thank you22:42
clarkbwe've got quite a few held nodes. might need to do some cleanup22:42
clarkbthey are all fairly recent things though so maybe not22:43
clarkblooks like ianw has some pip and docker stuff in flight and corvus is debugging the zuul/nodepool jobs?22:43
ianwclarkb: i'll run through mine in a bit, i'm sure most can go22:44
openstackgerritMonty Taylor proposed zuul/zuul-jobs master: Combine javascript deployment and deployment-tarball jobs  https://review.opendev.org/72737022:45
openstackgerritMonty Taylor proposed zuul/zuul-jobs master: Combine javascript deployment and deployment-tarball jobs  https://review.opendev.org/72737022:46
*** tkajinam has joined #opendev22:54
openstackgerritMerged opendev/system-config master: Update docs.airshipit.org functional test  https://review.opendev.org/72747423:14
openstackgerritmelanie witt proposed zuul/zuul-jobs master: DNM Try out running sphinx-build in parallel for releasenotes  https://review.opendev.org/72747323:14
*** tosky has quit IRC23:14
clarkbfungi: ^ I think you can recheck now23:14
fungiclarkb: indeed, perfect timing23:28
*** mlavalle has quit IRC23:53

Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!