Monday, 2021-05-03

*** hamalq has joined #opendev01:49
*** hamalq has quit IRC01:54
*** whoami-rajat has quit IRC02:49
openstackgerritSteve Baker proposed openstack/diskimage-builder master: Search for block-device-${ARCH}.yaml before default  https://review.opendev.org/c/openstack/diskimage-builder/+/78912703:36
openstackgerritIan Wienand proposed opendev/system-config master: [dnm] testing using gr-date-formatter in zuul plugin  https://review.opendev.org/c/opendev/system-config/+/78888203:39
ianwthat was quite an adventure for a one-ish line fix.03:47
*** hamalq has joined #opendev03:50
*** hamalq has quit IRC03:55
*** vishalmanchanda has joined #opendev04:54
*** ykarel has joined #opendev05:24
*** ysandeep|away is now known as ysandeep05:33
openstackgerritIan Wienand proposed opendev/system-config master: gerrit: enable plugin reload in dev mode  https://review.opendev.org/c/opendev/system-config/+/78913005:43
*** hamalq has joined #opendev05:51
*** hamalq has quit IRC05:56
*** sboyron has joined #opendev06:12
*** hamalq has joined #opendev06:27
*** hamalq has quit IRC06:31
*** hamalq has joined #opendev06:31
openstackgerritIan Wienand proposed openstack/project-config master: bullseye: set python-path  https://review.opendev.org/c/openstack/project-config/+/78913306:33
*** gnuoy has joined #opendev06:44
openstackgerritMerged openstack/project-config master: bullseye: set python-path  https://review.opendev.org/c/openstack/project-config/+/78913307:00
*** fressi has quit IRC07:01
*** hashar has joined #opendev07:05
*** fressi has joined #opendev07:07
*** dtantsur|afk is now known as dtantsur07:10
*** rpittau|afk is now known as rpittau07:19
*** avass has joined #opendev07:25
*** tosky has joined #opendev07:50
*** whoami-rajat has joined #opendev07:56
*** sshnaidm|off is now known as sshnaidm08:12
*** fressi has quit IRC08:33
*** hamalq has quit IRC08:46
*** fressi has joined #opendev08:47
*** ricolin has joined #opendev08:54
*** cgoncalves has quit IRC08:58
*** cgoncalves has joined #opendev09:00
*** fressi has quit IRC09:09
*** fressi has joined #opendev09:11
*** cgoncalves has quit IRC09:14
*** ysandeep is now known as ysandeep|lunch09:15
*** cgoncalves has joined #opendev09:16
*** fressi has quit IRC09:25
openstackgerritMerged opendev/base-jobs master: Switch the base-test job to use ubuntu-focal nodes  https://review.opendev.org/c/opendev/base-jobs/+/78909709:25
*** fressi has joined #opendev09:27
*** hamalq has joined #opendev09:36
*** hamalq has quit IRC09:40
*** ysandeep|lunch is now known as ysandeep09:56
*** dtantsur is now known as dtantsur|brb10:13
openstackgerritMerged openstack/diskimage-builder master: Update the ironic jobs  https://review.opendev.org/c/openstack/diskimage-builder/+/78897210:13
*** ralonsoh_ has joined #opendev10:18
*** ralonsoh has quit IRC10:20
*** dmellado has quit IRC10:21
*** dmellado has joined #opendev10:22
*** fressi has quit IRC10:35
zigoHi there!11:10
zigoLooks like we do have a progress on the bullseye front, but it's not yet finished.11:11
*** ykarel is now known as ykarel|lunch11:11
zigohttps://3df9d8624791e146e7ca-5b369dc32177e920b38f658289f2b193.ssl.cf2.rackcdn.com/786772/3/check/puppet-openstack-integration-6-scenario001-tempest-debian-bullseye/69ac958/job-output.txt <--- 404 there ...11:11
zigoLooking more closely, it looks like this:11:11
zigohttps://mirror.kna1.airship-citycloud.opendev.org/debian-security/dists/bullseye-security/main/source/11:11
zigois missing the InRelease file.11:11
zigoOr rather, this one: https://mirror.kna1.airship-citycloud.opendev.org/debian-security/dists/bullseye-security/11:12
zigoNo InRelease, no Release.gpg ...11:12
zigoSo it appears unsigned.11:12
zigoianw: Do you know how to fix that?11:13
openstackgerritHitesh Kumar proposed openstack/diskimage-builder master: Migrate from testr to stestr  https://review.opendev.org/c/openstack/diskimage-builder/+/78924611:17
*** fressi has joined #opendev11:18
*** dtantsur|brb is now known as dtantsur11:29
*** ysandeep is now known as ysandeep|afk11:31
*** hamalq has joined #opendev11:36
*** hamalq has quit IRC11:41
*** ykarel|lunch is now known as ykarel11:53
fricklerzigo: that's bullseye vs. bullseye-security, see https://review.opendev.org/c/zuul/zuul-jobs/+/78909012:15
*** hamalq has joined #opendev12:17
*** ysandeep|afk is now known as ysandeep12:19
*** hamalq has quit IRC12:22
zigoRight.12:28
*** stand has quit IRC13:08
*** fressi has quit IRC13:13
*** artom has joined #opendev13:24
*** d34dh0r53 has joined #opendev13:30
openstackgerritThierry Carrez proposed openstack/project-config master: Add skyline project  https://review.opendev.org/c/openstack/project-config/+/78928413:39
*** fressi has joined #opendev13:44
*** fressi has quit IRC13:46
*** fressi has joined #opendev13:47
*** auristor has quit IRC13:59
*** auristor has joined #opendev14:00
*** ysandeep is now known as ysandeep|afk14:05
fungizigo: i have hopes my current recheck on 789090 will work now, or it will at least tell us what else is still wrong14:08
zigofungi: Ok.14:08
fungiapparently ansible's python autodiscovery didn't consider that a distro might include python 2.7 but not package any modules for it14:08
fungiso you have to explicitly tell ansible to use python3 on bullseye14:09
fungiotherwise it will try to install python-apt for its package management module and then choke on that not existing14:10
zigoI'd be using ftpsync rather than reprepro to do debian mirroring ...14:10
zigoThat's much much nicer.14:10
fungiwell, we don't currently have a problem with the mirroring14:10
zigoHow come it's not downloading the indices signatures ?14:11
fungijust turning up more places where we assume a consistent pattern for the security suite name, when we have to special-case versions earlier than bullseye14:11
fungiwe don't sign our indices, these mirrors are only intended for ci jobs, we don't want anyone trying to rely on them for production systems14:12
zigoLook-up here: https://mirror.kna1.airship-citycloud.opendev.org/debian-security/dists/bullseye-security/ <--- No InRelease or Release.gpg file ...14:12
fungiyep, intentional14:12
zigo:/14:12
zigoSo, resigning the archive with reprepro ?14:12
fungiif we did sign them, we'd need to add the signing key trust into our images14:12
fungireprepro is not configured to sign the indices currently14:13
zigoWell, you don't need to sign them, that's why you should be using ftpsync and keep using the key from debian-archive-keyring ...14:13
fungiexcept we're not running official mirrors, so we have to mirror from a second-order mirror, and they're all notoriously broken us in the past14:14
zigoAh... :/14:14
fungitoo many of the official mirrors don't actually mirror safely, and end up with indices which refer to packages they haven't pulled yet14:14
zigoI believe that any mirror in the official list is doing things correctly with ftpsync.14:15
zigoThe mirror people (like Julien Christau) do some basic checks to make sure of that before adding a mirror to the official list.14:15
fungii suppose it's also possible that ftpsync breaks. when we tried using second order mirrors in the past we'd regularly have jobs failing due indices referring to packages which weren't there14:16
fungithey'd usually "fix themselves" on the next update, but having your jobs break for hours until the mirror gets back into a coherent state or you manually switch which mirror you point to is painful14:17
fungireprepro guarantees our indices always refer to the package versions it retrieved14:18
*** hamalq has joined #opendev14:18
zigoOk, makes sense.14:18
fungiso the indices and package files can't get out of sync14:18
zigoThough ethz never failed on me ... :)14:18
zigo(I'm using ethz in Zurrick as a push for my mirror.infomaniak.com)14:19
fungiwe discussed an alternative though, we could use ftpsync and then run a consistency check on the result before we vos release it in afs14:19
fungibut having signed package indices does still increase the risk that someone (maybe accidentally) bakes a ci mirror url into production configs14:19
zigoI see.14:20
*** hamalq has quit IRC14:22
openstackgerritJeremy Stanley proposed zuul/zuul-jobs master: Add new Debian security mirror suite pattern  https://review.opendev.org/c/zuul/zuul-jobs/+/78909014:35
*** fressi has quit IRC14:43
*** ysandeep|afk is now known as ysandeep14:50
*** zbr has quit IRC14:51
*** zbr has joined #opendev14:52
*** avass has quit IRC15:04
*** avass has joined #opendev15:05
*** dhellmann has joined #opendev15:21
*** ykarel is now known as ykarel|away15:24
*** lbragstad_ is now known as lbragstad15:51
*** ysandeep is now known as ysandeep|away15:51
*** hamalq has joined #opendev16:00
fungia fresh debian-bullseye images is building as of a few seconds ago, with 789133 now in place16:03
clarkbfungi: I've approved a couple of the acl cleanups you pushed up that had a +2 already16:04
fungithanks!16:04
fungimainly trying to straighten out all our acl spaghetti16:04
clarkbI'm leaving +2s on the ones that don't already have +2s (though amotoki seems to have +1'd so I think you can probably approve those if you are ready?)16:04
fungionce they're all merged, the only exceptions will be non-opendev/ repos using the opendev/retired acl16:05
clarkbfungi: and we still need to decide if we watn to land the openstack/retired update16:05
fungii'm fine merging it, though we'll want to keep an eye on the deploy job16:05
fungii think it's reasonable to maintain consistency there, even if the things currently in meta-config are unlikely to be used for retired repos, we might add things to meta-config in the future which might be16:06
clarkbya maybe get your cleanups in first since I think those are more important (I +2'd four and approved two) then followup landing the retired.config update16:06
clarkbfungi: ++16:06
clarkbfungi: and ya I think if you want to approve the other 4 of yours that should be fine16:07
fungiand i need to start putting together our global gerrit config cleanup now that the non-retired openstack repos have explicit inherited release manager permissions16:08
fungiearly in the openstack release cycle like this is great timing, since it gives us plenty of time to iron out unanticipated problems16:08
clarkb++16:09
*** hamalq has quit IRC16:09
*** hamalq has joined #opendev16:14
openstackgerritMerged openstack/project-config master: Split x/monasca-vagrant out to its own Gerrit ACL  https://review.opendev.org/c/openstack/project-config/+/78905516:14
openstackgerritMerged openstack/project-config master: Split x/monitorstack out to its own Gerrit ACL  https://review.opendev.org/c/openstack/project-config/+/78905616:14
*** hamalq_ has joined #opendev16:16
clarkbfungi: ianw: also looks like the centos8-stream aarch64 glibc package may have updated?16:19
*** hamalq has quit IRC16:20
fungioh! nice16:21
clarkbhttps://mirror.iad.rax.opendev.org/centos/8-stream/BaseOS/aarch64/os/Packages/glibc-2.28-158.el8.aarch64.rpm exists anyway16:22
fungiexcellent. hrw: yoctozepto: ^ i think you were the ones waiting for that to be back to working again?16:23
fungiit's possible not everything that needed to be rebuilt with the fixed autotools has migrated yet, but worth testing again16:24
*** ykarel|away has quit IRC16:24
*** dtantsur is now known as dtantsur|afk16:32
*** rpittau is now known as rpittau|afk16:32
*** hashar is now known as hasharSkating16:33
*** hasharSkating is now known as hashar16:35
openstackgerritJeremy Stanley proposed opendev/system-config master: Clean up Gerrit global config documentation  https://review.opendev.org/c/opendev/system-config/+/78938316:37
*** tkajinam has quit IRC16:48
fungiclarkb: 789061 was also related acl cleanup16:52
fungii ran some scripted analysis over the projects.yaml comparing to the acls file tree and, surprisingly, that was the only orphaned acl file16:53
hrwfungi: great!16:54
fungii think it was held over from the puppet-openstack -> openstack-puppet -> puppet-openstack shuffle some years ago16:54
fungis/some/many/16:55
fungiseems like it happened around when i first started with the project in 201216:55
fungior was even already underway when i started16:55
hrwfungi: https://review.opendev.org/c/openstack/kolla/+/789272 enables that CI back16:56
hrwwill see16:56
openstackgerritMerged openstack/project-config master: Split vexxhost/base-jobs out to its own Gerrit ACL  https://review.opendev.org/c/openstack/project-config/+/78905316:57
clarkbfungi: approved16:58
openstackgerritMerged openstack/project-config master: Split Charms project ACLs between namespaces  https://review.opendev.org/c/openstack/project-config/+/78905417:00
openstackgerritMerged openstack/project-config master: Split Puppet project ACLs between namespaces  https://review.opendev.org/c/openstack/project-config/+/78905717:02
openstackgerritMerged openstack/project-config master: Split Xstatic project ACLs between namespaces  https://review.opendev.org/c/openstack/project-config/+/78905817:02
openstackgerritMerged openstack/project-config master: Remove unused Gerrit ACL for openstack-puppet  https://review.opendev.org/c/openstack/project-config/+/78906117:03
*** dhellmann_ has joined #opendev17:14
*** dhellmann has quit IRC17:15
*** dhellmann_ is now known as dhellmann17:16
corvuszuul (project) is still seeing a bunch of jobs fail due to yarn installation; is that happening anywhere else?  should we start trying to track that and identify if it's certain mirrors?17:32
clarkbI've not heard it happening elsewhere, but also I think openstack (which does the bulk of the job runs) doesn't do much js.17:33
clarkbcorvus: thinking out loud here: is yarn something that we could put in a "venv" on the test nodes somehow?17:44
*** hashar has quit IRC17:44
clarkbif so we might be able to include it in our test images17:44
fungiokay, looks like updated debian-bullseye images have uploaded to pretty much all our providers. rechecking the zuul-jobs fix now17:47
*** avass has quit IRC17:58
corvusclarkb: oh sorry i was sloppy in my description; it's yarn downloading packages from mirrors17:58
corvusclarkb: https://a45a99275ba2b41c83f4-16f1fcebf18a5553a04944eeab950404.ssl.cf1.rackcdn.com/789245/2/gate/zuul-tox-py38/bdf6611/job-output.txt17:58
corvus(sorry for direct log link; build is in progress)17:58
corvus2021-05-03 17:28:07.898575 | ubuntu-bionic | verbose 168.271561022 Error: https://mirror.regionone.limestone.opendev.org:4443/registry.npmjs/@patternfly/react-tokens/-/react-tokens-4.9.12.tgz: ESOCKETTIMEDOUT17:59
openstackgerritDaniel Blixt proposed zuul/zuul-jobs master: Make build-sshkey handling windows compatible  https://review.opendev.org/c/zuul/zuul-jobs/+/78066218:01
clarkbgot it, so this is yarn to our mirror and that is failing18:03
clarkbpossible that the backend is failing too (though that is to be determined)18:03
corvusyeah, if this is news to folks, i'll try to start keeping track of it18:10
clarkbyup that is new to me18:12
fungii saw it mentioned i think, but didn't know it was recurring18:13
corvushttp://paste.openstack.org/show/804933/18:15
corvuslooks like it's not limited to one mirror, but is only one of 2 specific packages?18:16
corvus(could be related to size, if those are larger files?)18:16
clarkbcorvus: and direct access exhibits the same problem looks like18:16
corvusthat's from logstash:  message:"error An unexpected error occurred"18:17
corvusyeah, i wonder what didn't use a mirror?18:17
clarkbthat makes me suspect it isn't our mirrors causing the problem, tehy are likely just bubbling it up when they hit it too (since direct access does the same thing)18:17
corvusthe two direct-access ones ran in ovh-bhs118:18
corvusbut we have later incidents using a mirror18:18
corvusfrom ovh-bhs118:18
corvusoh there's a third direct one too18:19
clarkbreact-tokens is only 663kb ish18:19
clarkbseems unlikely to be a size thing given ^18:20
corvusyeah18:20
corvusmaybe we just need to throw in a retry?18:20
clarkbcorvus: pip does retry up to 5 times out of the box (by default we don't do that)18:20
clarkbhaving other tools like yarn try that is not crazy18:20
clarkb*by default. We don't configure that special.18:21
corvusoh i know why we didn't use the mirror; we're missing a passenv in one of the tox envs18:21
openstackgerritJeremy Stanley proposed zuul/zuul-jobs master: Add new Debian security mirror suite pattern  https://review.opendev.org/c/zuul/zuul-jobs/+/78940318:31
openstackgerritJeremy Stanley proposed zuul/zuul-jobs master: Test Debian Buster and Bullseye  https://review.opendev.org/c/zuul/zuul-jobs/+/78940418:31
openstackgerritJeremy Stanley proposed zuul/zuul-jobs master: Add new Debian security mirror suite pattern  https://review.opendev.org/c/zuul/zuul-jobs/+/78909018:40
openstackgerritJeremy Stanley proposed zuul/zuul-jobs master: Test Debian Buster and Bullseye  https://review.opendev.org/c/zuul/zuul-jobs/+/78940418:40
*** vishalmanchanda has quit IRC19:13
clarkbI'm getting the infra meeting agenda put together now. If there is anything else you'd like to see on there please add it soon19:17
clarkb(I got TheJulia's questions about removing +r from our irc channels on there)19:17
fungii added the ubuntu-focal nodeset for base over the weekend19:18
fungialso there's the updated openstack artifact signing key changes, which includes a change to update our docs from rsa to ecc19:19
fungii didn't put that on the agenda but can if you think it's worth bringing up19:19
clarkbfungi: I'll add an item for the key19:19
clarkbI think it is worth mentioning. Let me know if you'd prefer to add it in19:20
fungiyeah, mainly looking for extra eyeballs on the rsa to ecc switch (rationale and implementation)19:20
fungii'll tack it on th ewiki real quick19:20
fungidone. also added the gerrit global config bit19:22
clarkbthanks!19:24
fungiprobably no need to put debian bullseye nodes on there unless people are in need of progress updates19:24
clarkbya I already pulled it off19:24
fungicool19:24
clarkbI agree, I think good progress is being made there and no real headaches, just fixing things19:25
fungias soon as we whack a mole, another pops up19:25
*** dmellado has quit IRC19:53
*** cgoncalves has quit IRC19:53
*** owalsh has quit IRC19:53
*** jaicaa has quit IRC19:54
*** lbragstad has quit IRC19:54
*** fdegir has quit IRC19:54
*** priteau has quit IRC19:54
*** calcmandan has quit IRC19:54
*** elod has quit IRC19:54
*** akahat|ruck has quit IRC19:54
*** icey has quit IRC19:54
*** dmellado has joined #opendev19:55
*** cgoncalves has joined #opendev19:55
*** owalsh has joined #opendev19:55
*** jaicaa has joined #opendev19:55
*** lbragstad has joined #opendev19:55
*** fdegir has joined #opendev19:55
*** priteau has joined #opendev19:55
*** calcmandan has joined #opendev19:55
*** elod has joined #opendev19:55
*** akahat|ruck has joined #opendev19:55
*** icey has joined #opendev19:55
*** dhellmann_ has joined #opendev20:11
*** dhellmann has quit IRC20:12
*** dhellmann_ is now known as dhellmann20:13
*** gothicserpent has quit IRC20:16
*** gothicserpent has joined #opendev20:22
*** slaweq_ has joined #opendev20:35
*** sboyron has quit IRC20:55
*** slaweq_ has quit IRC20:56
openstackgerritMerged zuul/zuul-jobs master: Add new Debian security mirror suite pattern  https://review.opendev.org/c/zuul/zuul-jobs/+/78909021:13
ianwcould i get a eye on https://review.opendev.org/c/opendev/system-config/+/788857 which adds stats prefixes, mostly to see if i mis-typed anything.  that should start separating our stats for api operations again21:40
clarkbianw: the sdk got a release and we built new nodepool iamges too? (I guess it won't hurt to prep this before hand but need that other stuff to happen too?)21:41
ianwyeah it should be ignored, but also yes the support is in the latest released sdk and we should have rebuilt images for dib changes very recently21:42
clarkbianw: one thing I noted inline21:43
ianwclarkb: ahh, good catch; to match the fields in the existing grafana that should be inap, you're right21:47
openstackgerritIan Wienand proposed opendev/system-config master: nodepool clouds: adds stats prefix  https://review.opendev.org/c/opendev/system-config/+/78885721:48
*** whoami-rajat has quit IRC22:03
*** ralonsoh_ has quit IRC22:09
*** snapdeal has joined #opendev22:14
*** snapdeal has quit IRC22:27
*** zimmerry has quit IRC22:28
clarkbianw: left a note about a small thing on https://review.opendev.org/c/opendev/system-config/+/78913022:35
ianwyeah, copy paste :)22:35
ianwevery time i try to do a minor thing to this plugin, it takes several hours to figure out how to build and deploy it, and about 10 minutes to do the change :)22:36
openstackgerritIan Wienand proposed opendev/system-config master: gerrit: enable plugin reload in dev mode  https://review.opendev.org/c/opendev/system-config/+/78913022:38
johnsomhttps://www.irccloud.com/pastebin/lPSr8QBQ/22:38
johnsomI have a zuul mystery I could use a second set of eyes on.22:38
johnsomWhy didn't the release notes check job run on this patch?22:38
clarkbjohnsom: the first thing I notice is that the results are from january22:42
clarkband a recent recheck doesn't seem to have posted anything22:42
johnsomHmm, good point it's old.22:42
clarkbIf I had to guess the january jobs ran when that file matcher iddn't exist22:43
clarkbwould be interesting to see if a recheck produces what you expect to see22:43
johnsomJust started one.22:43
johnsomNope, not on the list22:43
clarkbok it does not so we continue looking22:43
clarkbjohnsom: https://opendev.org/openstack/designate-tempest-plugin/src/branch/master/.zuul.yaml that is the config that is being used not the one in your paste22:45
clarkbthere are no release notes jobs there22:45
johnsomDOh!22:45
johnsomI crossed repos and didn't notice. That would do it.22:45
johnsomI need to get that added there. Thanks. Sigh22:46
clarkbno worries, happy to help22:46
*** tosky has quit IRC22:50
clarkbianw: it would be interesting to know if https://review.opendev.org/c/opendev/system-config/+/789130 makes that startup exception that we get from the plugin manager go away on the test setup23:08
ianwclarkb: as in in -> https://zuul.opendev.org/t/openstack/build/72c14cf4870f427387f3bdd8b96f73c7/log/review01.openstack.org/logs/error_log ?23:13

Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!