| mordred | yeah - there's a split of raised capabilities - it gets to behave like a config project for the purposes of config, but not for the purposes of execution | 00:00 |
|---|---|---|
| corvus | clarkb: it's basically revisiting every "if project.source_context.trusted" in the code and turning it from a binary into a ternary :) | 00:00 |
| clarkb | ah | 00:00 |
| corvus | probably by making it some sort of generic capabilities system, so when we go from 3->4 it's not as difficult as it will be to go from 2->3 :) | 00:01 |
| clarkb | fwiw I'm ok with mordred's intermediate solution | 00:01 |
| fungi | looks like we've had another spate of oom events on lists.o.o | 00:01 |
| corvus | i am as well | 00:01 |
| mnaser | i'm ok with the expectation that it won't be anyone other than opendev | 00:01 |
| fungi | 11 python processes killed since sunday | 00:01 |
| fungi | i'm going to restart all the queue managers on lists.o.o | 00:01 |
| corvus | mordred's solution should be forward-compatible with a future tenant-config-project feature | 00:01 |
| corvus | fungi: thanks | 00:01 |
| mnaser | you'll have a canary repo ready for that :p | 00:02 |
| clarkb | fungi: fwiw I wasn't really able to track it back to any specific message processing the last time I tried | 00:02 |
| mnaser | i'll throw up a change which modifies the acls and i guess discussion can happen there, ill summarize the idea in the commit in case someone goes back and wonders why | 00:02 |
| clarkb | but its still my hunch that one of the mailman pipelines is being unhappy about particular messages (and possibly that is an intentional dos) | 00:02 |
| corvus | mnaser: ++ | 00:03 |
| clarkb | mnaser: ++ | 00:03 |
| fungi | #status log restarted queue managers for all 5 mailman sites on lists.o.o following a spate of oom conditions | 00:04 |
| openstackstatus | fungi: finished logging | 00:04 |
| fungi | one thing i noticed with our etherpad puppetry... the content of /etc/apt/sources.list.d/nodesource.list doesn't seem to get updated automatically unless you blow away that file | 00:09 |
| fungi | i guess the nodejs puppet module takes care of that, and it seems to be something we're reconsuming from puppetforge, does that sound right? | 00:10 |
| clarkb | fungi: yes that sounds right from memory | 00:10 |
| openstackgerrit | Mohammed Naser proposed openstack/project-config master: vexxhost: move base-jobs to config-project https://review.opendev.org/716459 | 00:11 |
| mnaser | clarkb, corvus, mordred: ^ did my best to summarize, moved that repo to opendev/project-config acls and added it as a config project with only jobs/secret/nodesets | 00:11 |
| fungi | Mar 31 23:45:11 etherpad-dev01 puppet-user[28496]: (/Stage[main]/Nodejs::Repo::Nodesource::Apt/Apt::Source[nodesource]/Apt::Setting[list-nodesource]/File[/etc/apt/sources.list.d/nodesource.list]/ensure) | 00:11 |
| fungi | okay, so not fixing that behavior unless we fork it (or maybe there's a newer version on the forge) | 00:11 |
| fungi | i guess we can just live with that until the containerized version is done | 00:12 |
| fungi | https://etherpad-dev.openstack.org/ is back up and running again | 00:14 |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: [wip] fix ansible-devel job to actually run devel https://review.opendev.org/716449 | 00:14 |
| clarkb | fungi: and its only necessary during the infrequent cases of updating major nodejs versions right? | 00:14 |
| fungi | i guess | 00:15 |
| ianw | fungi: ReferenceError: io is not defined in https://etherpad-dev.openstack.org/javascripts/lib/ep_etherpad-lite/static/js/pad.js?callback=require.define at line 3' | 00:15 |
| fungi | new problem: | 00:15 |
| ianw | fungi: i am supposed to report that to my webmaster | 00:15 |
| fungi | ianw: beat me to it | 00:15 |
| clarkb | my guess is that is related to the npm install of etherpad itself | 00:15 |
| clarkb | (that should pull in all the deps and build js things appropriately) | 00:15 |
| clarkb | perhaps we need to retrigger that with the up to date nodejs | 00:15 |
| fungi | maybe it needs to rerun with new node in place, yeah | 00:15 |
| clarkb | (assuming nodejs 6 failed to build that properly in the past) | 00:16 |
| fungi | i did have to manually apt install nodejs to get the package to upgrade too | 00:16 |
| fungi | i'll wind the etherpad git repo back by one commit and see what puppet does on the next pulse | 00:16 |
| clarkb | ++ | 00:16 |
| fungi | did a `git reset --hard HEAD^1` in it | 00:17 |
| fungi | now behind origin/devel by 1 commit and can be fast-forwarded | 00:18 |
| *** dangtrinhnt has joined #opendev | 00:21 | |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: [wip] fix ansible-devel job to actually run devel https://review.opendev.org/716449 | 00:25 |
| *** dangtrinhnt has quit IRC | 00:37 | |
| mnaser | ok -- i'm running out of options, i'm trying to write a new job that runs a popular golang linter (golangci-lint): https://review.opendev.org/#/c/716452/ | 00:42 |
| mnaser | i've literally repeated the same exact playbooks locally and it runs just fine, it gives: [Errno 8] Exec format error: 'golangci-lint' -- in ci/infra nodes | 00:42 |
| mnaser | i tried running it with verbose but that hasn't really yielded to much more success, and it runs fine in a vm too so i'm pretty confused. could i get a hold on https://review.opendev.org/#/c/716453/ for golangci-lint | 00:43 |
| fungi | installed on a different platform maybe? | 00:43 |
| mnaser | fungi: i mean, i tried in an ubuntu docker container and tha tworked, and i downloaded the amd64 version (double checked) | 00:44 |
| *** dangtrinhnt has joined #opendev | 00:44 | |
| mnaser | and i literally did an ANSIBLE_ROLES_PATH=./roles ansible-playbook -i localhost, playbooks/golangci-lint/pre.yaml and then run.yaml and it worked fine | 00:44 |
| mnaser | hmm, maybe it's because i have go in this machine and its not in upstream | 00:45 |
| fungi | sudo zuul autohold --tenant vexxhost --project vexxhost/libvirtd_exporter --job golangci-lint-verbose --change 716453 --reason "mnaser investigating obscure go architecture error unable to reproduce elsewhere" --count 1 | 00:46 |
| fungi | does that look right? | 00:46 |
| mnaser | the reasoning is perfect | 00:46 |
| mnaser | :) | 00:46 |
| fungi | okay, the trap is set | 00:46 |
| fungi | lmk once it fails and i can add your ssh key | 00:46 |
| mnaser | fungi: ok, rechecked, i'll try to figure out why it happened | 00:48 |
| mnaser | fungi: failed! | 00:50 |
| fungi | you have ipv6 connectivity, right? this is a v6-only node | 00:52 |
| fungi | if not, we can release it and try again | 00:52 |
| fungi | also where do i find a copy of your ssh key(s)? | 00:53 |
| mnaser | fungi: i can find my way through a vm on our cloud | 00:53 |
| mnaser | fungi: https://github.com/mnaser.keys | 00:53 |
| mnaser | my local isp isn't wonderful when it comes for ipv6 :( | 00:53 |
| mnaser | office has ipv6, but not there | 00:53 |
| fungi | ssh root@2607:ff68:100:54:f816:3eff:fe14:5dbf | 00:54 |
| fungi | i only just recently worked out how to get prefix delegation via dhcp6 from my current residential broadband provider, and it's still a bit fiddly | 00:54 |
| fungi | but functional enough for me to drop my old tunnel to hurricane electric at least | 00:55 |
| mnaser | fungi: what's neat is their cell phone network is running on ipv6 which is cool, but not residental | 00:56 |
| clarkb | mnaser: thats how I ipv6 since home hasnone | 00:57 |
| clarkb | I bounce through cloud vm with it | 00:57 |
| clarkb | I'm hoping the ziply fiber aquisition results in ipv6 eventually though | 00:57 |
| *** dangtrinhnt has quit IRC | 00:58 | |
| mnaser | clarkb: hopefully so | 00:58 |
| mnaser | golangci-lint returns -bash: /usr/local/bin/golangci-lint: cannot execute binary file: Exec format error | 00:58 |
| mnaser | file says: /usr/local/bin/golangci-lint: Mach-O 64-bit x86_64 executable | 00:59 |
| * mnaser hmms | 00:59 | |
| fungi | so you're trying to run a binary built for macos? | 00:59 |
| fungi | Mach-O Mach-O man, i want to be a Mach-O man! (apologies to the village people) | 01:00 |
| mnaser | crap | 01:00 |
| mnaser | this is awkward | 01:00 |
| fungi | i'm not sure if linux has a mach compatibility shim lkm, if you really need it to run macos binaries on linux | 01:02 |
| openstackgerrit | Mohammed Naser proposed zuul/zuul-jobs master: golangci-lint: add job https://review.opendev.org/716452 | 01:02 |
| mnaser | fungi: sorry for that, i think we can rm that vm :( | 01:03 |
| fungi | i think macos/darwin's microkernel has elf compat but not the other way around | 01:03 |
| fungi | sure, will do | 01:03 |
| mnaser | yeah, you can run linux stuff but not macos, i assume this tool is "ultra optimized" | 01:03 |
| fungi | the error you were getting makes a lot more sense now, in retrospect | 01:04 |
| fungi | and i've released that node back into the aether from which it spawned | 01:04 |
| mnaser | fungi: cool, yay, now i have another type of error, but one that contains output! | 01:09 |
| fungi | lucky you! | 01:09 |
| fungi | puppet updated the etherpad deployment, but i'm still getting the same javascript error from it as before | 01:09 |
| fungi | unlucky me | 01:10 |
| fungi | however i think my evening is coming to a close so i'll take a fresh look at it in the morning if nothing new is on fire | 01:10 |
| fungi | since it's just the dev server | 01:10 |
| openstackgerrit | Mohammed Naser proposed zuul/zuul-jobs master: golangci-lint: add job https://review.opendev.org/716452 | 01:11 |
| johnsom | Hi there, I just got a "msg": "No viable v4 or v6 route found to opendev.org. The build node is assumed to be invalid.", | 01:12 |
| johnsom | https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_564/714004/6/check/neutron-ovn-provider-v2-scenario/564d705/job-output.txt | 01:12 |
| johnsom | Just an FYI, not blocking anything | 01:13 |
| mnaser | johnsom: uh that's very strange | 01:14 |
| mnaser | opendev.org: Temporary failure in name resolution | 01:15 |
| mnaser | oh an even more interesting | 01:15 |
| mnaser | fetch-output reported that the remote id changed for the machine | 01:16 |
| fungi | so possible there's a rogue vm nova has lost track of in that cloud getting into an arp fight with a job node | 01:16 |
| johnsom | Yeah, the remote ID thing is not unusual. I have seen that pop up every once in a while | 01:17 |
| *** dangtrinhnt has joined #opendev | 01:17 | |
| fungi | we see it from time to time across a seemingly random selection of sorts of jobs | 01:17 |
| johnsom | Yeah, just some zombie nova instance. We have special code in octavia to deal with them. | 01:18 |
| fungi | happens more often in some providers than others | 01:18 |
| mnaser | maybe good to report it to the provider | 01:18 |
| fungi | we try to when we can correlate them | 01:18 |
| openstackgerrit | Mohammed Naser proposed zuul/zuul-jobs master: golangci-lint: add job https://review.opendev.org/716452 | 01:23 |
| *** xavinux has joined #opendev | 01:33 | |
| openstackgerrit | Mohammed Naser proposed zuul/zuul-jobs master: golangci-lint: add job https://review.opendev.org/716452 | 01:34 |
| *** dangtrinhnt has quit IRC | 01:37 | |
| *** dangtrinhnt_ has joined #opendev | 01:37 | |
| *** xavinux has quit IRC | 01:42 | |
| *** dangtrinhnt_ has quit IRC | 01:45 | |
| *** dangtrinhnt has joined #opendev | 01:47 | |
| *** dangtrinhnt has quit IRC | 01:52 | |
| openstackgerrit | Mohammed Naser proposed zuul/zuul-jobs master: golangci-lint: add job https://review.opendev.org/716452 | 01:59 |
| *** dangtrinhnt has joined #opendev | 02:01 | |
| openstackgerrit | Mohammed Naser proposed zuul/zuul-jobs master: golangci-lint: add job https://review.opendev.org/716452 | 02:10 |
| *** ysandeep|away is now known as ysandeep|rover | 02:31 | |
| openstackgerrit | Mohammed Naser proposed zuul/zuul-jobs master: golangci-lint: add job https://review.opendev.org/716452 | 02:33 |
| openstackgerrit | Mohammed Naser proposed zuul/zuul-jobs master: golangci-lint: add job https://review.opendev.org/716452 | 02:33 |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: [wip] fix ansible-devel job to actually run devel https://review.opendev.org/716449 | 02:37 |
| ianw | i feel like something might have broken system-config : Line 1444: | 02:38 |
| ianw | Unknown project openstack/project-config | 02:38 |
| ianw | infra-prod-manage-projects | 02:40 |
| ianw | huh ... though it looks like the tests are still running | 02:43 |
| ianw | corvus / mnaser / mordred : ^ i feel like this is the intersection of all of you :) | 02:43 |
| ianw | https://review.opendev.org/#/c/716449/ is the review zuul is commenting on | 02:43 |
| openstackgerrit | Mohammed Naser proposed zuul/zuul-jobs master: golangci-lint: add job https://review.opendev.org/716452 | 02:47 |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: [wip] fix ansible-devel job to actually run devel https://review.opendev.org/716449 | 02:48 |
| mnaser | ianw: oh what did i do again poop | 02:48 |
| mnaser | ianw: wait, i wonder if thats the vexxhost tenant commenting on openstack/project-config ? | 02:50 |
| ianw | hrm, that might be it, and why the opendev side still runs | 02:51 |
| mnaser | ianw: yeah, we are loading opendev/project-config indeed | 02:51 |
| mnaser | i think i remember maybe pushing a patch to add a note to what tenant that was sending that | 02:52 |
| openstackgerrit | Mohammed Naser proposed openstack/project-config master: vexxhost: move base-jobs to config-project https://review.opendev.org/716459 | 02:53 |
| mnaser | ianw: i revised my patch and added opendev/project-config to it | 02:53 |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: [wip] fix ansible-devel job to actually run devel https://review.opendev.org/716449 | 02:57 |
| openstackgerrit | Mohammed Naser proposed zuul/zuul-jobs master: golangci-lint: add job https://review.opendev.org/716452 | 02:58 |
| openstackgerrit | Mohammed Naser proposed zuul/zuul-jobs master: golangci-lint: add job https://review.opendev.org/716452 | 03:05 |
| openstackgerrit | Mohammed Naser proposed zuul/zuul-jobs master: golangci-lint: add job https://review.opendev.org/716452 | 03:12 |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: [dnm] testing namespace collection usage https://review.opendev.org/716474 | 03:13 |
| *** dangtrinhnt has quit IRC | 03:25 | |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: [dnm] testing namespace collection usage https://review.opendev.org/716474 | 03:28 |
| *** bolg has quit IRC | 03:34 | |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: [dnm] testing namespace collection usage https://review.opendev.org/716474 | 03:43 |
| *** dangtrinhnt has joined #opendev | 04:00 | |
| *** ykarel|away is now known as ykarel | 04:10 | |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: Fix ansible-devel job for Ansible 2.10 changes https://review.opendev.org/716449 | 04:26 |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: [dnm] testing namespace collection usage https://review.opendev.org/716474 | 04:26 |
| ianw | mordred: i say we merge 716449, then we can even help out with testing the automatic namespace routing stuff | 04:28 |
| *** sgw has quit IRC | 05:07 | |
| *** sgw has joined #opendev | 05:24 | |
| *** bolg has joined #opendev | 05:39 | |
| openstackgerrit | Merged opendev/system-config master: Use ansible debug callback plugin https://review.opendev.org/716433 | 05:50 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/project-config master: Normalize projects.yaml https://review.opendev.org/716159 | 06:03 |
| *** DSpider has joined #opendev | 06:44 | |
| *** tobiash has quit IRC | 06:51 | |
| *** jhesketh has quit IRC | 06:52 | |
| *** jhesketh has joined #opendev | 06:53 | |
| *** tobiash has joined #opendev | 06:53 | |
| *** ysandeep|rover is now known as ysandeep|brb | 07:03 | |
| *** dangtrinhnt has quit IRC | 07:11 | |
| *** dangtrinhnt_ has joined #opendev | 07:12 | |
| openstackgerrit | Albin Vass proposed zuul/zuul-jobs master: Adds roles to install and run hashicorp packer https://review.opendev.org/709292 | 07:22 |
| *** tosky has joined #opendev | 07:32 | |
| *** rpittau|afk is now known as rpittau | 07:37 | |
| *** ysandeep|brb is now known as ysandeep | 07:52 | |
| *** ralonsoh has joined #opendev | 07:55 | |
| *** ysandeep is now known as ysandeep|rover | 07:57 | |
| openstackgerrit | Albin Vass proposed zuul/zuul-jobs master: Adds roles to install and run hashicorp packer https://review.opendev.org/709292 | 08:16 |
| openstackgerrit | Albin Vass proposed zuul/zuul-jobs master: Adds roles to install and run hashicorp packer https://review.opendev.org/709292 | 08:19 |
| *** ykarel is now known as ykarel|lunch | 08:30 | |
| *** dangtrinhnt has joined #opendev | 08:44 | |
| *** dangtrinhnt_ has quit IRC | 08:48 | |
| openstackgerrit | Albin Vass proposed zuul/zuul-jobs master: Adds variable to toggle whether to revoke sudo https://review.opendev.org/706248 | 09:05 |
| mnasiadka | Morning | 09:07 |
| mnasiadka | it seems CentOS 8 Extras repo is a bit stale on the mirrors, comparing http://mirror.centos.org/centos/8-stream/extras/x86_64/os/Packages/ to http://mirror.dfw.rax.opendev.org/centos/8/extras/x86_64/os/Packages/ | 09:07 |
| *** mrunge has joined #opendev | 09:16 | |
| *** dangtrinhnt has quit IRC | 09:16 | |
| *** dangtrinhnt_ has joined #opendev | 09:17 | |
| openstackgerrit | Sorin Sbarnea proposed openstack/diskimage-builder master: Validate virtualenv and pip https://review.opendev.org/707104 | 09:30 |
| *** osmanlicilegi has quit IRC | 09:49 | |
| *** osmanlicilegi has joined #opendev | 09:52 | |
| mrunge | hi there, who can refresh a cache for centos-extras in zuul? We have a patch failing, because of http://mirror.dfw.rax.opendev.org/centos/8/extras/x86_64/os/Packages/ outdated | 09:54 |
| *** ykarel|lunch is now known as ykarel | 09:59 | |
| *** rpittau is now known as rpittau|bbl | 10:16 | |
| *** dangtrinhnt_ has quit IRC | 10:26 | |
| *** ysandeep|rover is now known as ysandeep|afk | 10:56 | |
| *** njohnston has quit IRC | 11:20 | |
| frickler | infra-root: ^^ can't look myself currently | 11:20 |
| *** ysandeep|afk is now known as ysandeep|rover | 11:22 | |
| *** njohnston_ has joined #opendev | 11:23 | |
| *** dangtrinhnt has joined #opendev | 11:30 | |
| openstackgerrit | Tobias Henkel proposed zuul/zuul-jobs master: WIP: Try to fix unicode issue when parsing tox https://review.opendev.org/716560 | 11:53 |
| *** dangtrinhnt has quit IRC | 11:55 | |
| *** dangtrinhnt_ has joined #opendev | 11:56 | |
| openstackgerrit | Tobias Henkel proposed zuul/zuul-jobs master: Ignore errors when parsing tox output https://review.opendev.org/716561 | 11:59 |
| openstackgerrit | Albin Vass proposed zuul/zuul-jobs master: Adds roles to install and run hashicorp packer https://review.opendev.org/709292 | 12:02 |
| *** rpittau|bbl is now known as rpittau | 12:07 | |
| openstackgerrit | Albin Vass proposed zuul/zuul-jobs master: Adds roles to install and run hashicorp packer https://review.opendev.org/709292 | 12:11 |
| mrunge | any infra-root able to refresh http://mirror.dfw.rax.opendev.org/centos/8/extras/x86_64/os/Packages/ ? | 12:13 |
| openstackgerrit | Tobias Henkel proposed zuul/zuul-jobs master: Ignore errors when parsing tox output https://review.opendev.org/716561 | 12:28 |
| openstackgerrit | Tobias Henkel proposed zuul/zuul-jobs master: Ignore errors when parsing tox output https://review.opendev.org/716561 | 12:35 |
| *** roman_g has joined #opendev | 12:42 | |
| openstackgerrit | Sorin Sbarnea proposed zuul/zuul-jobs master: Add support for RedHat platforms on install-podman https://review.opendev.org/716578 | 12:47 |
| openstackgerrit | Mohammed Naser proposed zuul/zuul-jobs master: tox_parse_output: add no_log to tox_output https://review.opendev.org/716579 | 12:54 |
| openstackgerrit | Merged zuul/zuul-jobs master: Ignore errors when parsing tox output https://review.opendev.org/716561 | 12:55 |
| *** hashar has joined #opendev | 13:05 | |
| openstackgerrit | Merged zuul/zuul-jobs master: tox_parse_output: add no_log to tox_output https://review.opendev.org/716579 | 13:18 |
| *** dangtrinhnt_ has quit IRC | 13:20 | |
| *** dangtrinhnt has joined #opendev | 13:21 | |
| *** dangtrinhnt has quit IRC | 13:23 | |
| *** dangtrinhnt_ has joined #opendev | 13:23 | |
| *** dangtrinhnt_ has quit IRC | 13:24 | |
| *** roman_g has quit IRC | 13:34 | |
| *** ykarel is now known as ykarel|afk | 13:43 | |
| fungi | mnasiadka: mrunge: frickler: we currently mirror centos 8 via rsync from https://mirror.hackingand.coffee/centos/8/extras/x86_64/os/Packages/ | 13:47 |
| fungi | it looks like we're current with what's being served there | 13:47 |
| fungi | if there's a better rsync mirror we should be using, please propose an update | 13:47 |
| fungi | https://opendev.org/opendev/system-config/src/branch/master/playbooks/roles/mirror-update/files/centos-mirror-update#L34 | 13:47 |
| AJaeger | mordred: is everything imported from https://review.opendev.org/#/c/716159/ ? Then I'll merge | 13:48 |
| mordred | AJaeger: yes - I have manually imported the two repos that were missed from the upstream | 13:49 |
| mordred | we're assuming there was a hiccup from when we were running things manually | 13:49 |
| AJaeger | great, thanks mordred | 13:49 |
| mrunge | fungi, can we mirror from http://mirror.centos.org/centos/8/extras/x86_64/os/Packages/ ? | 13:50 |
| fungi | mrunge: is there a public rsync mirror for that? | 13:51 |
| mrunge | I'd bet | 13:51 |
| fungi | if you can test whether you can rsync from there, then please propose an update to the line i linked above | 13:51 |
| mnasiadka | fungi: the mirror being used is not really on the official mirror list - https://www.centos.org/download/mirrors/ | 13:52 |
| mrunge | fungi, where would I propose a change? | 13:52 |
| fungi | mrunge: the same system where you're proposing the changes which are being tested with packages from our mirror network, review.opendev.org | 13:53 |
| mrunge | fungi, I get that, but where is this in zuul or so, which repo has the info to mirror from? | 13:54 |
| fungi | mrunge: https://opendev.org/opendev/system-config/src/branch/master/playbooks/roles/mirror-update/files/centos-mirror-update#L34 | 13:54 |
| mnasiadka | mrunge: it's in this file: https://opendev.org/opendev/system-config/src/branch/master/playbooks/roles/mirror-update/files/centos-mirror-update - will you propose a change? | 13:54 |
| fungi | like i linked just above | 13:54 |
| mnasiadka | oops, fungi was faster ;) | 13:55 |
| mrunge | currently in a call. Will have a look later | 13:55 |
| mrunge | or who ever is faster than me | 13:55 |
| mrunge | thank you mnasiadka and fungi | 13:55 |
| fungi | cool, doesn't sound urgent but happy to review once someone has had a chance to test out and confirm they can rsync from somewhere else | 13:55 |
| openstackgerrit | Monty Taylor proposed zuul/zuul-jobs master: Strip ansi codes from pep8 message https://review.opendev.org/716598 | 13:56 |
| fungi | mrunge: mnasiadka: for reference, that mirror was chosen by ianw back in september, rationale is in the commit message for https://review.opendev.org/684437 | 13:57 |
| openstackgerrit | Albin Vass proposed zuul/zuul-jobs master: Adds roles to install and run hashicorp packer https://review.opendev.org/709292 | 13:57 |
| openstackgerrit | Monty Taylor proposed zuul/zuul-jobs master: Strip ansi codes from pep8 message https://review.opendev.org/716598 | 14:01 |
| openstackgerrit | Monty Taylor proposed zuul/zuul-jobs master: Add flag for toggling inline comments for linters https://review.opendev.org/716599 | 14:01 |
| openstackgerrit | Albin Vass proposed zuul/zuul-jobs master: Adds roles to install and run hashicorp packer https://review.opendev.org/709292 | 14:01 |
| openstackgerrit | Merged openstack/project-config master: Normalize projects.yaml https://review.opendev.org/716159 | 14:01 |
| *** ykarel|afk is now known as ykarel | 14:02 | |
| openstackgerrit | Monty Taylor proposed zuul/zuul-jobs master: Add tox_envlist to the inline comment https://review.opendev.org/716601 | 14:16 |
| openstackgerrit | Michal Nasiadka proposed opendev/system-config master: Change CentOS 8 upstream mirror https://review.opendev.org/716602 | 14:19 |
| mnasiadka | mrunge: ^^ | 14:19 |
| mrunge | you beat me, mnasiadka | 14:19 |
| mnaser | mordred: might having a look at https://review.opendev.org/716459 ? clarkb and corvus previously +2'd but we lost them because i added project-config to the list of projects that we don't load in our tenant b/c we were reporting to that project inside gerrit | 14:19 |
| mnasiadka | mrunge: oh well, next time you'll be faster ;) | 14:21 |
| mrunge | doesn't really matter, thank you for looking into that mnasiadka :) | 14:22 |
| mnasiadka | mrunge: it hurts kolla (not counting other projects), so it's in my interest ;) | 14:22 |
| openstackgerrit | Merged zuul/zuul-jobs master: Add flag for toggling inline comments for linters https://review.opendev.org/716599 | 14:23 |
| openstackgerrit | Merged openstack/project-config master: vexxhost: move base-jobs to config-project https://review.opendev.org/716459 | 14:31 |
| openstackgerrit | Mohammed Naser proposed zuul/zuul-jobs master: golangci-lint: add job https://review.opendev.org/716452 | 14:33 |
| openstackgerrit | Albin Vass proposed zuul/zuul-jobs master: Do not end host if correct go version is installed https://review.opendev.org/716607 | 14:34 |
| openstackgerrit | Monty Taylor proposed zuul/zuul-jobs master: Add tox_envlist to the inline comment https://review.opendev.org/716601 | 14:38 |
| openstackgerrit | Mohammed Naser proposed zuul/zuul-jobs master: golangci-lint: add job https://review.opendev.org/716452 | 14:42 |
| mordred | AJaeger, fungi : WOOHOO! https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_a37/716159/2/promote/infra-prod-manage-projects/a37619b/manage-projects.yaml.log | 14:45 |
| mordred | that's from the latest manage-projects run triggered by zuul for the cleanup patch | 14:45 |
| mordred | https://review.opendev.org/#/c/716159/ | 14:45 |
| mordred | mnaser: ^^ you can now check output of manage-projects without an infra-root | 14:46 |
| mnaser | mordred: !!! that's awesome | 14:48 |
| AJaeger | mordred: cool \o/ | 14:48 |
| *** ysandeep|rover is now known as ysandeep|away | 14:50 | |
| fungi | mnaser: mrunge: i've approved the mirror change, the next scheduled update pulse will be in a couple hours at 16:43z so we should hopefully see it take effect around then. i'm tailing the log from our cronjob so i can see when it does in case there are problems | 14:54 |
| fungi | er, mnasiadka ^ (sorry mnaser! tab-fail on my part) | 14:54 |
| mrunge | thank you fungi | 14:54 |
| mrunge | much appreciated | 14:54 |
| *** hashar has quit IRC | 14:55 | |
| fungi | thanks for spotting the stale mirror and working on updates. if this one doesn't do the trick for some reason we can try the other one you mentioned in your review comment | 14:55 |
| mrunge | sure. I'll keep an eye on that. | 14:56 |
| mnaser | infra-root: i think i broke zuul. | 14:57 |
| mnaser | once my change with depends-on for golangci-lint finished | 14:57 |
| mnaser | it got stuck for a little bit and didn't reportt | 14:57 |
| mnaser | and now all the pipelines are gone and the change didn't report: https://zuul.opendev.org/t/vexxhost/status | 14:57 |
| mnaser | waaaiat | 14:57 |
| mnaser | crap | 14:57 |
| mnaser | we merged the change that made the pipelines disappear. | 14:57 |
| mnaser | i didnt include load pipelines in that change | 14:58 |
| fungi | mnaser: thankfully this doesn't seem to have impacted any other tenants | 14:58 |
| mnaser | yes, this is because it was a vexxhost only tenant change | 14:58 |
| mnaser | so we just don't have a pipeline config right now | 14:58 |
| fungi | happy to fast-approve the fix when you push it | 14:58 |
| mnaser | ok, let me revise the change to load pipeline as well | 14:58 |
| *** lpetrut has joined #opendev | 14:59 | |
| mnasiadka | fungi: thanks! | 14:59 |
| openstackgerrit | Mohammed Naser proposed openstack/project-config master: vexxhost: load base-jobs & project-config pipelines https://review.opendev.org/716620 | 15:03 |
| *** njohnston_ is now known as njohnston | 15:03 | |
| mnaser | fungi: here's another approach | 15:03 |
| openstackgerrit | Monty Taylor proposed zuul/zuul-jobs master: Add tox_envlist to the inline comment https://review.opendev.org/716601 | 15:05 |
| openstackgerrit | Monty Taylor proposed zuul/zuul-jobs master: WIP Add testing for inline tox comments https://review.opendev.org/716623 | 15:05 |
| *** lpetrut has quit IRC | 15:06 | |
| mordred | corvus: don't know if you saw - but we had a successful manage-projects run with the output reported back to zuul: https://review.opendev.org/#/c/716159/ -> https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_a37/716159/2/promote/infra-prod-manage-projects/a37619b/manage-projects.yaml.log | 15:09 |
| corvus | mordred: that looks great :) | 15:11 |
| mordred | corvus: I'm quite pleased. which means I think we're actually ready for https://review.opendev.org/#/c/715957/ now! :) | 15:13 |
| openstackgerrit | Monty Taylor proposed zuul/zuul-jobs master: WIP Add testing for inline tox comments https://review.opendev.org/716623 | 15:16 |
| openstackgerrit | Merged opendev/system-config master: Change CentOS 8 upstream mirror https://review.opendev.org/716602 | 15:20 |
| fungi | revisiting yesterday's etherpad-dev error, love it when i paste an error into a web search and turn up one of our own conversations from 5 years ago: http://eavesdrop.openstack.org/irclogs/%23openstack-infra/%23openstack-infra.2015-09-02.log.html#t2015-09-02T15:34:31 | 15:20 |
| fungi | i guess it's possible we need to change up how we're proxying newer etherpad | 15:20 |
| mordred | fungi: awesome :) | 15:21 |
| mordred | fungi: did you see I got started on ansible/docker for etherpad yesterday? | 15:21 |
| fungi | i did! | 15:21 |
| *** hashar has joined #opendev | 15:21 | |
| *** ykarel is now known as ykarel|away | 15:22 | |
| mordred | woot. luckily I hadn't gotten to doing apache yet - so good timing if we need to change how we proxy :) | 15:23 |
| fungi | web console in ff tells me this: The resource from “https://etherpad-dev.openstack.org/socket.io/socket.io.js” was blocked due to MIME type (“text/html”) mismatch (X-Content-Type-Options: nosniff). | 15:25 |
| clarkb | reviewing that change is top of my list once I actually get started. Today is like tcp, slow start | 15:25 |
| openstackgerrit | Sorin Sbarnea proposed zuul/zuul-jobs master: Add support for RedHat platforms on install-podman https://review.opendev.org/716578 | 15:26 |
| fungi | clarkb: open a large receive window | 15:27 |
| fungi | from a shell on etherpad-dev, `http://localhost:9001/socket.io/socket.io.js` gives me a 404 | 15:29 |
| fungi | i'm assuming it should not | 15:29 |
| fungi | er, passing it to wget i mean | 15:29 |
| fungi | on etherpad.o.o it works, so i'm guessing that's a clue | 15:30 |
| clarkb | fungi: did it get rebuilt with newer nodejs after the git repo edit? | 15:31 |
| clarkb | maybe double check build process hasnt changed tok | 15:31 |
| clarkb | *too | 15:31 |
| fungi | aha, found puppet errors | 15:32 |
| fungi | Apr 1 00:13:51 etherpad-dev01 nodejs[32156]: #033[31m[2020-04-01 00:13:51.434] [ERROR] console - #033[39mFailed to load '/opt/etherpad-lite/etherpad-lite/node_modules/ep_etherpad-lite/node/hooks/express/socketio:expressCreateServer' for 'ep_etherpad-lite/socketio/hooks/expressCreateServer': Error: Cannot find module 'nodeify' | 15:32 |
| fungi | lots of "Cannot find module 'nodeify'" for a variety of different libraries | 15:33 |
| openstackgerrit | Ivan Kolodyazhny proposed openstack/project-config master: Add jobs for xstatic-graphlib https://review.opendev.org/716630 | 15:34 |
| fungi | i suppose it's looking for https://www.npmjs.com/package/nodeify | 15:35 |
| fungi | src/package-lock.json does declare it's a dependency at least | 15:36 |
| fungi | looks like that was added by c499a08 which first appears in the 1.8.0 tag (their latest release) | 15:39 |
| clarkb | fungi: their readme says to run bin/run.sh which is what we do iirc | 15:39 |
| clarkb | fungi: https://github.com/ether/etherpad-lite/issues/3252 maybe we need to run bin/installDeps.sh? | 15:42 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: WIP Dockerize etherpad https://review.opendev.org/716442 | 15:42 |
| fungi | i'll give it a try manually here and see what happens | 15:43 |
| clarkb | puppet does supposedly run that | 15:43 |
| mordred | clarkb: fwiw - in that ^^ I started from their upstream image, so once we're on it we should be able to stop caring about how to build etherpad ;) | 15:43 |
| clarkb | oh! except that we have a creates on the puppet | 15:43 |
| clarkb | so it will only run if the dir it creates isn't there | 15:43 |
| clarkb | so ya I bet tahts it, need to run that | 15:44 |
| clarkb | mordred: are we basically going to get that -1 on all system-config jobs until we move it into the opendev tenant? | 15:50 |
| mordred | yup | 15:50 |
| openstackgerrit | Albin Vass proposed zuul/zuul-jobs master: Rename install- roles to ensure- for consistency https://review.opendev.org/716636 | 15:50 |
| mordred | clarkb: maybe we should exclude project from the zuul config entry for it | 15:51 |
| clarkb | ++ if that makes the noise go away | 15:51 |
| mordred | oh - we already do | 15:51 |
| openstackgerrit | Albin Vass proposed zuul/zuul-jobs master: Rename install- roles to ensure- for consistency https://review.opendev.org/716636 | 15:52 |
| fungi | clarkb: and for some reason it complains "Error: EACCES: permission denied, mkdir '/opt/etherpad-lite/etherpad-lite/src/node_modules/wd/build'" even when run as root | 15:53 |
| clarkb | mordred: left note on the conatiner for etherpad change | 15:53 |
| fungi | looks like it blows away the node_modules dir completely and then chokes when trying to create a subdir under it | 15:54 |
| mordred | clarkb: yes - I was thinking a similar thing | 15:54 |
| clarkb | fungi: if you look in the script it seems to do a bunch of stuff with an || rm node_modules | 15:56 |
| clarkb | fungi: implying something is failing in that bunch of stuff? | 15:56 |
| fungi | ahh, i think it may also have wanted this run as eplite | 15:56 |
| fungi | okay, that seems to have worked better | 15:56 |
| fungi | chowned the tree to eplite, then used `sudo -H -u eplite bin/installDeps.sh` | 15:57 |
| openstackgerrit | Merged openstack/project-config master: vexxhost: load base-jobs & project-config pipelines https://review.opendev.org/716620 | 15:57 |
| clarkb | fungi: looks like puppet runs it as that user | 15:57 |
| fungi | yep | 15:57 |
| fungi | added 1003 packages from 1155 contributors and audited 13804 packages in 24.216s | 15:57 |
| clarkb | remember when people thought we had a lot of dependencies in openstack? | 15:57 |
| fungi | if only we'd written it all in javascript | 15:58 |
| fungi | just think of the bragging rights! | 15:58 |
| fungi | okay, https://etherpad-dev.openstack.org/ seems to be up and running | 15:59 |
| clarkb | https://etherpad-dev.openstack.org/p/clarkb-test2 loads for me with expected content too | 16:00 |
| clarkb | clarkb-test still fails to load | 16:00 |
| clarkb | (also expected) | 16:00 |
| corvus | that looks very much like the current etherpad | 16:00 |
| corvus | i thought it was sposed to be way different? | 16:00 |
| clarkb | corvus: the etherpad-lite readme gif indicates things look the same still | 16:00 |
| clarkb | not sure if that is just stale image though | 16:00 |
| openstackgerrit | Albin Vass proposed zuul/zuul-jobs master: WIP: Rename install- roles to ensure- for consistency https://review.opendev.org/716636 | 16:01 |
| fungi | yeah, i thought they had changed up the default theme | 16:02 |
| openstackgerrit | Albin Vass proposed zuul/zuul-jobs master: WIP: Rename install- roles to ensure- for consistency https://review.opendev.org/716636 | 16:03 |
| clarkb | https://github.com/ether/etherpad-lite/issues/3441#issuecomment-562896447 | 16:04 |
| clarkb | it is a settings.json thing, that is a file we manage with puppet. Possible we are undoing the "default" by supplying the file directly? | 16:04 |
| clarkb | yup we don't set skinName | 16:05 |
| clarkb | and that results in the empty value old school theme | 16:06 |
| clarkb | (according to settings.json.template) | 16:06 |
| fungi | there's a message about that when starting up too | 16:06 |
| mordred | clarkb: we have support for openid in the puppet etherpad - but we're not using that right? | 16:06 |
| fungi | or maybe it was in the puppet stdout | 16:06 |
| clarkb | mordred: correct, I ended up using it elsewhere but not in the openstack/opendev deployment | 16:07 |
| mordred | k | 16:07 |
| clarkb | mordred: I think it can be safely ignored for dockering | 16:07 |
| fungi | yeah, puppet | 16:07 |
| fungi | No "skinName" parameter found. Please check out settings.json.template and update your settings.json. Falling back to the default "no-skin". | 16:08 |
| clarkb | so they didn't really change the default | 16:08 |
| clarkb | they provide a default example config with a different value | 16:08 |
| clarkb | service default remains unchanged | 16:09 |
| fungi | seems that way | 16:09 |
| fungi | i can flip it in the config real quick if folks want to see | 16:09 |
| clarkb | fungi: if nothing else it will help confirm we're running the new thing | 16:10 |
| fungi | new theme temporarily in place until puppet undoes it | 16:13 |
| openstackgerrit | Albin Vass proposed zuul/zuul-jobs master: WIP: Rename install- roles to ensure- for consistency https://review.opendev.org/716636 | 16:14 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: WIP Dockerize etherpad https://review.opendev.org/716442 | 16:14 |
| clarkb | the LOADING.. banner doesn't ever seem to go away even after content has loaded | 16:14 |
| clarkb | though maybe it is trying to load a top banner that isn't coming in? | 16:14 |
| mordred | clarkb: ok - now with database and apache | 16:14 |
| fungi | clarkb: maybe | 16:14 |
| clarkb | my immediate reaction is I really don't like this :) | 16:14 |
| clarkb | its quite a bit less dense | 16:14 |
| fungi | also as i noted in that gh issue, it seems they've "optimized" it for portrait orientation screens (phones and tablets?) | 16:15 |
| clarkb | fungi: I think that may just be to limit width of text since people have an easier time reading shorter lines | 16:15 |
| *** rpittau is now known as rpittau|afk | 16:15 | |
| openstackgerrit | Monty Taylor proposed zuul/zuul-jobs master: Add tox_envlist to the inline comment https://review.opendev.org/716601 | 16:16 |
| clarkb | for drafting actual documents this is probably better, but for brainstorm scratch pad I think the original theme is likely better | 16:16 |
| clarkb | gives you much more room to work with | 16:16 |
| openstackgerrit | Albin Vass proposed zuul/zuul-jobs master: WIP: Rename install- roles to ensure- for consistency https://review.opendev.org/716636 | 16:17 |
| fungi | i suspect it's not all loading, as evidenced by the perpetual "loading" message you mentioned | 16:18 |
| openstackgerrit | Albin Vass proposed zuul/zuul-jobs master: WIP: Rename install- roles to ensure- for consistency https://review.opendev.org/716636 | 16:19 |
| fungi | probably more urls we need to whitelist in our proxy config | 16:19 |
| fungi | i'll check it out with a js debugging console shortly | 16:19 |
| clarkb | fungi: probably files under the theme/ dir or something | 16:20 |
| clarkb | src/static/skins is the source repo path | 16:20 |
| openstackgerrit | sebastian marcet proposed opendev/system-config master: Openstack Id production deploy v3.0.9 https://review.opendev.org/716653 | 16:22 |
| openstackgerrit | Monty Taylor proposed zuul/zuul-jobs master: WIP Add testing for inline tox comments https://review.opendev.org/716623 | 16:32 |
| openstackgerrit | Monty Taylor proposed zuul/zuul-jobs master: Require a / in a file path https://review.opendev.org/716655 | 16:32 |
| openstackgerrit | Monty Taylor proposed zuul/zuul-jobs master: Use os.path.exists https://review.opendev.org/716657 | 16:36 |
| openstackgerrit | Merged zuul/zuul-jobs master: Strip ansi codes from pep8 message https://review.opendev.org/716598 | 16:37 |
| openstackgerrit | Monty Taylor proposed zuul/zuul-jobs master: Check that a file exists for inline comments https://review.opendev.org/716655 | 16:38 |
| openstackgerrit | Monty Taylor proposed zuul/zuul-jobs master: WIP Add testing for inline tox comments https://review.opendev.org/716623 | 16:38 |
| clarkb | re using etherpad-dev with jitsi, the ssl cert on that server is currently self signed. I wonder how that will interact with jitsis nested loading. Maybe you have to navigate to the self signed location first, accept the cert, then load jitsi? | 16:38 |
| * clarkb makes change and we can test that | 16:38 | |
| openstackgerrit | Clark Boylan proposed opendev/system-config master: Switch meetpad to etherpad-dev https://review.opendev.org/716660 | 16:40 |
| openstackgerrit | Monty Taylor proposed zuul/zuul-jobs master: Check that a file exists for inline comments https://review.opendev.org/716655 | 16:40 |
| fungi | clarkb: the other thing i've been wondering about is whether having the meetpad and etherpad dns names in different domains is crossing a line from a browser tracking paranoia standpoint | 16:42 |
| clarkb | fungi: ya that was another thought that came up | 16:43 |
| clarkb | we can probably test that with etherpad-dev too fi we're adding ssl verification exceptions anyway | 16:43 |
| clarkb | (add a etherpad-dev.opendev.org cname and tell brwoser to add exception for that verification error too) | 16:43 |
| openstackgerrit | Monty Taylor proposed zuul/zuul-jobs master: Check that a file exists for inline comments https://review.opendev.org/716655 | 16:43 |
| openstackgerrit | Merged zuul/zuul-jobs master: Add tox_envlist to the inline comment https://review.opendev.org/716601 | 16:44 |
| openstackgerrit | Albin Vass proposed zuul/zuul-jobs master: Rename install-docker to ensure-docker for consistency https://review.opendev.org/716663 | 16:45 |
| mordred | clarkb: yah - and if we get happy with the docker patch above, we should likely take the opportunity to spin up etherpad01.opendev.org, roll out new code on it, LE it, and make etherpad.openstack.org a redirect like we do with gerrit | 16:46 |
| clarkb | mordred: ++ | 16:47 |
| mordred | clarkb: in fact, why don't I make that patch for etherpad.opendev and not for etherpad.openstack - that way we can land it and spin up a new server and see how it goes | 16:47 |
| mordred | in parallel to the other stuff | 16:47 |
| openstackgerrit | Clark Boylan proposed opendev/zone-opendev.org master: Add an etherpad-dev CNAME to openstack.org https://review.opendev.org/716665 | 16:49 |
| clarkb | mordred: we can just do ^ too | 16:49 |
| clarkb | (don't need a new server yet) | 16:49 |
| clarkb | oh in parallel ++ | 16:50 |
| fungi | wfm | 16:50 |
| mordred | yeah | 16:51 |
| mordred | like - might as well roll out docker on a clean host in this case | 16:51 |
| openstackgerrit | Albin Vass proposed zuul/zuul-jobs master: Rename install-kubernetes to ensure-kubernetes for consistency https://review.opendev.org/716667 | 16:53 |
| fungi | the great dockering seems like a fine fulcrum for switching over to using etherpad.opendev.org as the official service name anyway | 16:53 |
| fungi | (after -dev of course) | 16:54 |
| clarkb | fungi: we may not need -dev anymore with the system-config-run-etherpad job | 16:55 |
| clarkb | thats one of the great things about proper end to end testing there. We can treat it as the dev platform | 16:55 |
| fungi | yep | 16:56 |
| mordred | ++ | 16:56 |
| openstackgerrit | Mohammed Naser proposed zuul/zuul-jobs master: go-jobs: improve testing https://review.opendev.org/716668 | 16:56 |
| openstackgerrit | Monty Taylor proposed zuul/zuul-jobs master: Check that a file exists for inline comments https://review.opendev.org/716655 | 16:58 |
| openstackgerrit | Merged opendev/zone-opendev.org master: Add an etherpad-dev CNAME to openstack.org https://review.opendev.org/716665 | 17:02 |
| openstackgerrit | Albin Vass proposed zuul/zuul-jobs master: Rename install-kubernetes to ensure-kubernetes for consistency https://review.opendev.org/716667 | 17:06 |
| openstackgerrit | Albin Vass proposed zuul/zuul-jobs master: Rename install-openshift to ensure-openshift for consistency https://review.opendev.org/716675 | 17:06 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Make a new dockerized etherpad.opendev.org https://review.opendev.org/716442 | 17:09 |
| fungi | mnasiadka: mrunge: our centos 8 mirror seems to have updated as of a little while ago, please see if things are still working for you | 17:11 |
| fungi | (or are back to working, i guess) | 17:12 |
| mnaser | hmm | 17:12 |
| mnaser | Queue lengths: 2173 events, 0 management events, 11 results. | 17:12 |
| mnaser | that seems unusual, no? | 17:13 |
| mnaser | oh, it's zero now | 17:13 |
| fungi | they tend to back up during reconfigure events | 17:13 |
| clarkb | and no not very unusual | 17:13 |
| clarkb | our zuul is busy | 17:13 |
| fungi | seems like even a global pandemic doesn't slow down our workload | 17:14 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Make a new dockerized etherpad.opendev.org https://review.opendev.org/716442 | 17:16 |
| openstackgerrit | Albin Vass proposed zuul/zuul-jobs master: Rename install-docker to ensure-docker for consistency https://review.opendev.org/716663 | 17:19 |
| mnaser | oh yay | 17:19 |
| mnaser | i have my pipelines back. | 17:19 |
| openstackgerrit | Merged opendev/system-config master: Openstack Id production deploy v3.0.9 https://review.opendev.org/716653 | 17:20 |
| mordred | mnaser: \o/ | 17:23 |
| fungi | everyone loves their pipelines | 17:23 |
| openstackgerrit | Tobias Henkel proposed zuul/zuul-jobs master: Generalize parse tox output https://review.opendev.org/716263 | 17:29 |
| openstackgerrit | Tobias Henkel proposed zuul/zuul-jobs master: Strip source dir from file comments https://review.opendev.org/716264 | 17:29 |
| mordred | fungi: in soviet russia, pipeline loves everyone | 17:29 |
| openstackgerrit | Albin Vass proposed zuul/zuul-jobs master: Rename install-docker to ensure-docker for consistency https://review.opendev.org/716663 | 17:31 |
| openstackgerrit | Albin Vass proposed zuul/zuul-jobs master: Rename install-kubernetes to ensure-kubernetes for consistency https://review.opendev.org/716667 | 17:31 |
| openstackgerrit | Albin Vass proposed zuul/zuul-jobs master: Rename install-openshift to ensure-openshift for consistency https://review.opendev.org/716675 | 17:31 |
| openstackgerrit | Albin Vass proposed zuul/zuul-jobs master: Rename install-podman to ensure-podman for consistency https://review.opendev.org/716682 | 17:31 |
| openstackgerrit | Albin Vass proposed zuul/zuul-jobs master: Do not end host if correct go version is installed https://review.opendev.org/716607 | 17:34 |
| openstackgerrit | Albin Vass proposed zuul/zuul-jobs master: Adds roles to install and run hashicorp packer https://review.opendev.org/709292 | 17:36 |
| openstackgerrit | Merged zuul/zuul-jobs master: Check that a file exists for inline comments https://review.opendev.org/716655 | 17:38 |
| openstackgerrit | Mohammed Naser proposed zuul/zuul-jobs master: golangci-lint: add job https://review.opendev.org/716452 | 17:39 |
| openstackgerrit | Albin Vass proposed zuul/zuul-jobs master: Rename install-devstack to ensure-devstack for consistency https://review.opendev.org/716685 | 17:41 |
| mordred | clarkb: so - I'm happy with how manage-projects is running - I'd like to land the base/bridge one - whatcha thing? | 17:42 |
| mordred | think? | 17:42 |
| mordred | https://review.opendev.org/#/c/715957/ (it's already got 2x+2 - just checking in before I pull the trigger) | 17:42 |
| clarkb | mordred: ya I think we can do that next. Do we need to set allowed projects for those jobs too? | 17:43 |
| clarkb | we shouldn't need the semaphore because only system-config is running them in periodic and promote | 17:44 |
| fungi | testing the colibris skin on etherpad-dev with the js debugging console open, the only error it's reporting is "Error: setAuthorInfo: author (undefined) is not a string" | 17:44 |
| clarkb | but maybe we explicitly restrict it to system-config for now to avoid it growing outside? | 17:44 |
| mordred | clarkb: yeah - I thnik we don't need to | 17:44 |
| openstackgerrit | Albin Vass proposed zuul/zuul-jobs master: Rename install-javascript-packages to ensure-javascript-packages for consistency https://review.opendev.org/716687 | 17:44 |
| clarkb | fungi: look in the network debugger to see if its failing to load files | 17:44 |
| fungi | ahh | 17:44 |
| clarkb | they should show up as 404 or similar | 17:44 |
| mordred | clarkb: well - it won't work from other repos | 17:45 |
| mordred | clarkb: becuase it depends on the per-project ssh key | 17:45 |
| clarkb | mordred: it will work from project-cofig | 17:45 |
| clarkb | (we have that key on bridge too iirc) | 17:45 |
| mordred | yeah - but only from project-config | 17:45 |
| mordred | yeah | 17:45 |
| mordred | clarkb: if I add allowed-projects to the infra-prod-apply base job - that should let us just override it on child jobs yes? | 17:46 |
| openstackgerrit | Tobias Henkel proposed zuul/zuul-jobs master: Generalize parse tox output https://review.opendev.org/716263 | 17:46 |
| openstackgerrit | Tobias Henkel proposed zuul/zuul-jobs master: Strip source dir from file comments https://review.opendev.org/716264 | 17:46 |
| mordred | clarkb: actually - it's pointless | 17:47 |
| mordred | clarkb: allowed-projects is ignored by config projects :) | 17:47 |
| clarkb | oh heh | 17:47 |
| clarkb | ok | 17:47 |
| mordred | k. pulling the trigger | 17:48 |
| mordred | assuming we're happy with this one, I'll make the patches to do the same with everything else :) | 17:48 |
| fungi | clarkb: no error codes on any of the requests, though it did eventually stick a js error in the pad itself. trying to recreate now | 17:48 |
| openstackgerrit | Albin Vass proposed zuul/zuul-jobs master: Rename install-go to ensure-go for consistency https://review.opendev.org/716689 | 17:48 |
| openstackgerrit | Albin Vass proposed zuul/zuul-jobs master: Rename install-javascript-packages to ensure-javascript-packages for consistency https://review.opendev.org/716687 | 17:51 |
| openstackgerrit | Albin Vass proposed zuul/zuul-jobs master: Rename install-nodejs to ensure-nodejs for consistency https://review.opendev.org/716692 | 17:54 |
| openstackgerrit | Albin Vass proposed zuul/zuul-jobs master: Rename install-yarn to ensure-yarn for consistency https://review.opendev.org/716693 | 17:58 |
| openstackgerrit | Albin Vass proposed zuul/zuul-jobs master: Rename install-pdk-dependencies to ensure-pdk-dependencies for consistency https://review.opendev.org/716695 | 18:01 |
| openstackgerrit | Albin Vass proposed zuul/zuul-jobs master: Rename install-if-python to ensure-if-python for consistency https://review.opendev.org/716698 | 18:04 |
| openstackgerrit | Merged opendev/system-config master: Run manage-projects/base/bridge on system-config changes https://review.opendev.org/715957 | 18:11 |
| openstackgerrit | Albin Vass proposed zuul/zuul-jobs master: Rename install-docker to ensure-docker for consistency https://review.opendev.org/716663 | 18:14 |
| openstackgerrit | Albin Vass proposed zuul/zuul-jobs master: Rename install-kubernetes to ensure-kubernetes for consistency https://review.opendev.org/716667 | 18:15 |
| openstackgerrit | Albin Vass proposed zuul/zuul-jobs master: Rename install-openshift to ensure-openshift for consistency https://review.opendev.org/716675 | 18:16 |
| fungi | clarkb: so, interesting to note, if i start a new pad with the colibris theme set, it all finishes loading fine. might have just been something about the state of the clarkb-test2 pad? | 18:17 |
| fungi | https://etherpad-dev.openstack.org/p/NHKtmbuTD5MbrkqFHIp_ | 18:18 |
| clarkb | fungi: possibly | 18:18 |
| clarkb | fungi: maybe because I have things cached in the browser too | 18:18 |
| openstackgerrit | Albin Vass proposed zuul/zuul-jobs master: Rename install-devstack to ensure-devstack for consistency https://review.opendev.org/716685 | 18:19 |
| openstackgerrit | Albin Vass proposed zuul/zuul-jobs master: Rename install-javascript-packages to ensure-javascript-packages for consistency https://review.opendev.org/716687 | 18:20 |
| openstackgerrit | Albin Vass proposed zuul/zuul-jobs master: Rename install-nodejs to ensure-nodejs for consistency https://review.opendev.org/716692 | 18:20 |
| openstackgerrit | Albin Vass proposed zuul/zuul-jobs master: Rename install-yarn to ensure-yarn for consistency https://review.opendev.org/716693 | 18:20 |
| openstackgerrit | Albin Vass proposed zuul/zuul-jobs master: Rename install-pdk-dependencies to ensure-pdk-dependencies for consistency https://review.opendev.org/716695 | 18:20 |
| openstackgerrit | Albin Vass proposed zuul/zuul-jobs master: Rename install-if-python to ensure-if-python for consistency https://review.opendev.org/716698 | 18:20 |
| openstackgerrit | Albin Vass proposed zuul/zuul-jobs master: Rename install-docker to ensure-docker for consistency https://review.opendev.org/716663 | 18:21 |
| *** diablo_rojo has quit IRC | 18:32 | |
| *** diablo_rojo has joined #opendev | 18:33 | |
| *** hashar is now known as hasharBreak | 18:34 | |
| openstackgerrit | Tobias Henkel proposed zuul/zuul-jobs master: DNM: Debug sphinx message https://review.opendev.org/716722 | 18:40 |
| mordred | infra-root: fwiw, zuul is currently running the base playbook | 18:41 |
| mordred | and previously successfully ran update-system-config | 18:42 |
| mordred | so our new zuul overlords are working | 18:42 |
| openstackgerrit | Albin Vass proposed zuul/zuul-jobs master: Rename install-docker to ensure-docker for consistency https://review.opendev.org/716663 | 18:42 |
| clarkb | exciting | 18:45 |
| mordred | \o/ | 18:46 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Make a new dockerized etherpad.opendev.org https://review.opendev.org/716442 | 18:50 |
| mordred | wow. the base playbook takes 42 minutes | 19:00 |
| corvus | mordred: yeah, i think in the long run we wanted to just do that daily or something? | 19:00 |
| corvus | mordred: now that i look closer at https://review.opendev.org/715957 i'm not sure i understand that second paragraph fully | 19:02 |
| *** xavinux has joined #opendev | 19:03 | |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Run base playbook with 50 forks https://review.opendev.org/716727 | 19:03 |
| mordred | corvus: ^^ also - we ran it differently in zuul than we did in run_all | 19:04 |
| corvus | mordred: why does manage-projects need bridge and base? | 19:04 |
| mordred | it needs bridge because bridge applies any changes to ansible settings (and before it would get those natually as a result of sequencing) | 19:05 |
| corvus | mordred: ideally the answer to that is, it doesn't. but if it does, can we look at putting those tasks in the service playbook? that seems more appropriate. that way they are automatically limited to the hosts involved. | 19:05 |
| mordred | it might not actually need base come to think of it though | 19:05 |
| mordred | corvus: yeah- I think that's a great idea | 19:05 |
| mordred | oh - wait - your second thing I read wrong | 19:06 |
| corvus | so then base is just something that runs infrequently on everything (to maintain stasis) and on new node bringup. then each service playbook encapsulates what's needed to operate that service from start to finish. but if we have lots of playbooks adjusting settings on bridge, that could be a problem (that could be a problem with the current approach too) | 19:07 |
| mordred | but - the first thing is the more telling - I dont think we need base - I think we do need bridge | 19:07 |
| mordred | yeah - I think we could put in a depend on bridge easily - it's a very short playbook | 19:07 |
| mordred | and make it soft - so if nothing touched the trigger files for bridge, it doesn't run and all is good | 19:07 |
| mordred | but if it does, we run bridge real quick then manage-projects, yeah? | 19:08 |
| corvus | that sounds good | 19:08 |
| openstackgerrit | Mohammed Naser proposed zuul/zuul-jobs master: golangci-lint: add job https://review.opendev.org/716452 | 19:08 |
| * corvus lunches | 19:08 | |
| * mordred makes change | 19:08 | |
| mordred | corvus: actually - it's just a bad commit message | 19:09 |
| mordred | corvus: we don't need base for manage-projects - or bridge - no do we depend on them | 19:09 |
| mordred | we need update-system-config for manage-projects, and we do depend on that | 19:09 |
| xavinux | hi, hope everyone is well here | 19:10 |
| clarkb | mordred: that assumes ansible and all that is already in place, which is probably fine (but I think bridge does that technically?) | 19:10 |
| xavinux | have been taking a look at these links https://docs.openstack.org/infra/system-config/ and https://docs.opendev.org/opendev/infra-manual/latest/ | 19:11 |
| xavinux | as i would like to contribute to the infra team | 19:12 |
| openstackgerrit | Albin Vass proposed zuul/zuul-jobs master: Rename install-kubernetes to ensure-kubernetes for consistency https://review.opendev.org/716667 | 19:15 |
| openstackgerrit | Albin Vass proposed zuul/zuul-jobs master: Rename install-openshift to ensure-openshift for consistency https://review.opendev.org/716675 | 19:15 |
| openstackgerrit | Albin Vass proposed zuul/zuul-jobs master: Rename install-podman to ensure-podman for consistency https://review.opendev.org/716682 | 19:15 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Rename bridge.yaml to install-ansible.yaml https://review.opendev.org/716731 | 19:15 |
| openstackgerrit | Albin Vass proposed zuul/zuul-jobs master: Rename install-devstack to ensure-devstack for consistency https://review.opendev.org/716685 | 19:15 |
| openstackgerrit | Albin Vass proposed zuul/zuul-jobs master: Rename install-javascript-packages to ensure-javascript-packages for consistency https://review.opendev.org/716687 | 19:16 |
| mordred | clarkb: yeah - it does - ^^ I just renamed that because it's actually really unclear | 19:16 |
| openstackgerrit | Albin Vass proposed zuul/zuul-jobs master: Rename install-nodejs to ensure-nodejs for consistency https://review.opendev.org/716692 | 19:16 |
| openstackgerrit | Albin Vass proposed zuul/zuul-jobs master: Rename install-yarn to ensure-yarn for consistency https://review.opendev.org/716693 | 19:16 |
| openstackgerrit | Albin Vass proposed zuul/zuul-jobs master: Rename install-pdk-dependencies to ensure-pdk-dependencies for consistency https://review.opendev.org/716695 | 19:16 |
| openstackgerrit | Albin Vass proposed zuul/zuul-jobs master: Rename install-if-python to ensure-if-python for consistency https://review.opendev.org/716698 | 19:16 |
| clarkb | mordred: one small suggestion maybe we should order the jobs in rough "chronological" execution order to show the dependencies that way too? | 19:19 |
| mordred | clarkb: that's a good idea | 19:19 |
| clarkb | -base comes after -ansible at this point basically | 19:20 |
| * clarkb finds lunch now too | 19:21 | |
| clarkb | oh wait cathcing up on scrollback xavinux has questions | 19:21 |
| clarkb | xavinux: welcome! sorry my stomach is hungry so I am distracted :) | 19:21 |
| clarkb | xavinux: currently we've got a few things in progress to give you an idea of the sorts of things happening right now. We are deploying our Gerrit with ansible and docker (migrating from puppet), we are starting to drive our ansible deployments from Zuul (our CI/CD tool) rather than cron, we are deploying a new jitsi meet server to integrate with existing etherpad services (to enable remote meetings and | 19:23 |
| clarkb | collaboration), and we are adding new fedora 31 test nodes (which involves adding mirrors and diskimage-builder image builds) | 19:23 |
| clarkb | xavinux: if any of that is of interest feel free to dive in and start doing reviews or offer to help write changes, and if you need help knowing where to do that you can ask in here (and I'm also happy to help on a more 1:1 basis) | 19:24 |
| clarkb | xavinux: there is typically quite a lot happening so finding something interesting to you that you can focus on is probably a good way to get started | 19:24 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Reorder jobs in job list "chronologicaly" and add files https://review.opendev.org/716734 | 19:26 |
| mordred | clarkb: how's that look now? | 19:26 |
| xavinux | clarkb: no problem! and thanks for your welcome! | 19:26 |
| mordred | oh yay! yeah- welcome xavinux ! | 19:26 |
| mordred | I'm about to be producing a pile of changes for the replace-cron-with-zuul thing he mentioned - eyeballs definitely appreciated! | 19:27 |
| openstackgerrit | Tobias Henkel proposed zuul/zuul-jobs master: Support multiple matchers when parsing tox output https://review.opendev.org/716263 | 19:27 |
| openstackgerrit | Tobias Henkel proposed zuul/zuul-jobs master: Strip source dir from file comments https://review.opendev.org/716264 | 19:27 |
| openstackgerrit | Tobias Henkel proposed zuul/zuul-jobs master: DNM: Debug sphinx message https://review.opendev.org/716722 | 19:27 |
| openstackgerrit | Albin Vass proposed zuul/zuul-jobs master: Rename install-nodejs to ensure-nodejs for consistency https://review.opendev.org/716692 | 19:28 |
| *** ralonsoh has quit IRC | 19:28 | |
| openstackgerrit | Albin Vass proposed zuul/zuul-jobs master: Rename install-yarn to ensure-yarn for consistency https://review.opendev.org/716693 | 19:29 |
| xavinux | mordred thanks for your welcome! | 19:31 |
| clarkb | mordred: the reorder change lgtm and the file matchers on base should speed things up quite a bit | 19:32 |
| openstackgerrit | Tobias Henkel proposed zuul/zuul-jobs master: Keep error status in tox run https://review.opendev.org/716736 | 19:33 |
| xavinux | only a question, what kind of servers do the openstack infra team support? are those server for the openstack website or also other services behind the openstack project? | 19:33 |
| clarkb | xavinux: we run development and collaboration tools to produce the software. code review, ci, wiki and communications tools and so on | 19:34 |
| mordred | clarkb: ++ | 19:37 |
| xavinux | clarkd good, so this infra is used to produce the code behind the openstack services like nova, glance, neutron, etc? | 19:37 |
| clarkb | xavinux: yes | 19:37 |
| xavinux | nice | 19:38 |
| xavinux | to let the team know more about me, i live in argentina, work as a cloud engineer, recently obtain the aws solution architect certification and work as a Linux sysadmin for the last 10 years in an internet service provider | 19:40 |
| xavinux | teach about openstack in an it institute for the las 3 years, where i used to have a small lab with packstack and openstack running an all-in-one version on linux centos | 19:41 |
| openstackgerrit | Tobias Henkel proposed zuul/zuul-jobs master: Keep error status in tox run https://review.opendev.org/716736 | 19:42 |
| *** xavinux has quit IRC | 19:42 | |
| *** xavinux has joined #opendev | 19:47 | |
| openstackgerrit | Tobias Henkel proposed zuul/zuul-jobs master: Strip source dir from file comments https://review.opendev.org/716264 | 19:51 |
| openstackgerrit | Tobias Henkel proposed zuul/zuul-jobs master: DNM: Debug sphinx message https://review.opendev.org/716722 | 19:51 |
| openstackgerrit | Albin Vass proposed zuul/zuul-jobs master: Rename install-pdk-dependencies to ensure-pdk-dependencies for consistency https://review.opendev.org/716695 | 19:52 |
| openstackgerrit | Albin Vass proposed zuul/zuul-jobs master: Rename install-if-python to ensure-if-python for consistency https://review.opendev.org/716698 | 19:52 |
| openstackgerrit | Merged zuul/zuul-jobs master: Keep error status in tox run https://review.opendev.org/716736 | 19:56 |
| *** hasharBreak is now known as hashar | 19:57 | |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Run service-bridge in zuul https://review.opendev.org/716745 | 20:02 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Migrate gitea-lb to zuul https://review.opendev.org/716746 | 20:02 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Run letsencrypt in zuul https://review.opendev.org/716747 | 20:02 |
| mordred | clarkb: ^^ starting to peel services off | 20:04 |
| mordred | xavinux: nice! well - most of us here (other than mnaser) don't spend much time actually running openstack clouds - but we do certainly use them heavily. all of our servers here run as VMs in openstack public clouds | 20:06 |
| mordred | xavinux: we have a static inventory at the moment: https://opendev.org/opendev/system-config/src/branch/master/inventory/openstack.yaml which has all of the servers listed - as well as what region of what cloud they are in | 20:06 |
| openstackgerrit | Albin Vass proposed zuul/zuul-jobs master: Update registry test to use ensure-podman and ensure-docker https://review.opendev.org/716752 | 20:06 |
| mordred | that repo is the root repo that drives our gitops - which is currently transitioning from being puppet run by ansible to being just ansible running things that are increasingly installed via containers | 20:08 |
| mordred | happy to have more friends in channel with linux sysadmin background - you'll fit right in :) | 20:08 |
| clarkb | mordred: the LE one might be awkward because we need it to always run before the cron (until we get services out of cron). We may just want to give people a heads up that ordering there may be weird until we've transitioned | 20:11 |
| clarkb | its eventually consistent which is good | 20:11 |
| mordred | yeah | 20:12 |
| mordred | clarkb: also - I think I can have the rest of these done today | 20:12 |
| mordred | they're not hard patches to write | 20:12 |
| mordred | clarkb: do we want to run letsencrypt early and add soft depends on it from other service playbooks? | 20:13 |
| mordred | I could put it as a soft-depend in that base job and we could just make everything soft-dep on it | 20:13 |
| clarkb | mordred: I think it may need to go after base but before everything else technically | 20:14 |
| mordred | ok. why don't I make that change | 20:14 |
| *** sgw has quit IRC | 20:14 | |
| clarkb | but if it doesn't strictly need base it could be the root too | 20:14 |
| xavinux | mordred good! will take a look at that inventory | 20:16 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Run letsencrypt in zuul https://review.opendev.org/716747 | 20:17 |
| mordred | clarkb: that puts it soft after base - so if we're gonna do base, cool, we'll wait (which is honestly not a bad idea anyway given how many forks base wants) | 20:17 |
| openstackgerrit | Mohammed Naser proposed zuul/zuul-jobs master: golangci-lint: add job https://review.opendev.org/716452 | 20:18 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Run nameserver in zuul https://review.opendev.org/716764 | 20:21 |
| openstackgerrit | Tobias Henkel proposed zuul/zuul-jobs master: Strip source dir from file comments https://review.opendev.org/716264 | 20:23 |
| openstackgerrit | Tobias Henkel proposed zuul/zuul-jobs master: DNM: Debug sphinx message https://review.opendev.org/716722 | 20:23 |
| openstackgerrit | Tobias Henkel proposed zuul/zuul-jobs master: Don't silently ignore exceptions when parsing tox output https://review.opendev.org/716766 | 20:23 |
| openstackgerrit | Tobias Henkel proposed zuul/zuul-jobs master: Support multiple matchers when parsing tox output https://review.opendev.org/716263 | 20:32 |
| openstackgerrit | Tobias Henkel proposed zuul/zuul-jobs master: Don't silently ignore exceptions when parsing tox output https://review.opendev.org/716766 | 20:32 |
| openstackgerrit | Tobias Henkel proposed zuul/zuul-jobs master: Strip source dir from file comments https://review.opendev.org/716264 | 20:32 |
| openstackgerrit | Tobias Henkel proposed zuul/zuul-jobs master: DNM: Debug sphinx message https://review.opendev.org/716722 | 20:32 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Run nodepool in zuul https://review.opendev.org/716770 | 20:34 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Run meetpad in zuul https://review.opendev.org/716771 | 20:34 |
| clarkb | mordred: is there a chagne yet to not load project from system-config in opendev tenant? I Think that is what we need there right? | 20:35 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Run mirror-update in zuul https://review.opendev.org/716772 | 20:38 |
| mordred | clarkb: no - we already do that | 20:38 |
| clarkb | hrm why do we keep getting those -1s | 20:38 |
| mordred | clarkb: I *think* it's more complicated - because I think the opendev tenant has a config error | 20:39 |
| clarkb | ah | 20:39 |
| mordred | https://zuul.opendev.org/t/opendev/config-errors | 20:39 |
| mordred | oh. duh. it's because we reference openstack/project-config - but that's unknown to the opendev tenant | 20:41 |
| mordred | it's the half-transition to opendev issue | 20:41 |
| clarkb | ya we could fix that by not lodaing jobs but I think that is why we have system-config in opendev tenant (for the jobs) | 20:41 |
| mordred | yeah | 20:41 |
| mordred | and this is blocked on the puppet jobs being legacy jobs | 20:41 |
| clarkb | what are we pulling from system-config in opendev specifically? | 20:42 |
| clarkb | maybe we can split that out? | 20:42 |
| mordred | I don't know? | 20:42 |
| mordred | I mean - to be honest, there is very little in the opendev tenant right now | 20:43 |
| mordred | - opendev/gear | 20:44 |
| mordred | - opendev/lodgeit | 20:44 |
| mordred | - openinfralabs/contrib | 20:44 |
| mordred | those are the only "meaningful" repos in there | 20:44 |
| mordred | other than, of course, the inaugust repos | 20:44 |
| clarkb | I41b345246f2012d15d969524b5879c9da32b708d | 20:44 |
| clarkb | "Specifically, we're looking to run third-party-check builds of Gerrit and Zuul on changes to upstream Gerrit repos." | 20:44 |
| mordred | oh ... waiut | 20:45 |
| mordred | yeah | 20:45 |
| clarkb | maybe we should run those out of the openstack tenant for now? | 20:45 |
| mordred | we shoudl really redo the puppet jobs in ozj | 20:45 |
| openstackgerrit | Tobias Henkel proposed zuul/zuul-jobs master: Strip source dir from file comments https://review.opendev.org/716264 | 20:46 |
| openstackgerrit | Tobias Henkel proposed zuul/zuul-jobs master: DNM: Debug sphinx message https://review.opendev.org/716722 | 20:46 |
| *** xavinux has quit IRC | 20:46 | |
| clarkb | another option is to pull in project-config but load only job from it? | 20:46 |
| clarkb | hrm that might not fix the pupept error | 20:47 |
| mordred | yeah | 20:47 |
| clarkb | and the allowed-projects error we could fix that by allowing nothing (just defining the repo in the tenant but nothing else) | 20:47 |
| clarkb | mordred: ya I think we can add openstack/project-config with nothing included and add ozj with just job included? | 20:48 |
| clarkb | or remove system-config from opendev until we can migrate it (and run gerrit integration jobs out of openstack tenant if necessary) | 20:49 |
| corvus | clarkb, mordred: we have suspended the third-party-check builds of gerrit right now; we can ignore them for a bit | 20:51 |
| clarkb | corvus: meaning its safe to undo the system-config inclusion into the opendev tenant (the reason this comes up is we get a -1 from opendev tenant every time we make system-cofnig changes due to project-config not being defined in that tenant) | 20:52 |
| mordred | you know... | 20:53 |
| corvus | clarkb: probably? all i know at this level of involvement is that i disabled the upstream checker config | 20:53 |
| mordred | I think let's disable it for now - since also we have a zuul there now, so us building gerrit with our zuul is less important | 20:54 |
| mordred | we're not even running those from that tenant | 20:56 |
| mordred | we already are running them from openstack | 20:56 |
| mordred | or - hrm. I have no idea where we're running them if we are | 20:57 |
| openstackgerrit | Clark Boylan proposed openstack/project-config master: Revert "Add system-config and zuul to OpenDev tenant for jobs" https://review.opendev.org/716775 | 20:57 |
| clarkb | I think ^ may be all we need. That was prepatory but then we didn't run things from there in the end? | 20:57 |
| clarkb | or it got removed as corvus said | 20:57 |
| mordred | clarkb: I'm game to try it | 20:58 |
| openstackgerrit | Mohammed Naser proposed zuul/zuul-jobs master: golangci-lint: add job https://review.opendev.org/716452 | 21:05 |
| mordred | infra-root: https://review.opendev.org/#/q/topic:infra-prod-zuul <-- I'm slogging through those - which is just making a lovely nice pile of reviews for you all | 21:07 |
| ianw | mordred: not sure if you saw but i got the devel job working; see https://review.opendev.org/#/c/716449/. imo we should merge it because we'll be in a position to validate the ongoing work for automatically routing the old names to their new fully-qualified modes | 21:07 |
| mordred | ianw: yes indeed! I was actually going to follow up with you on that ... I think we could probably take some of it and even start doing it for real now - like the install of the collectinos we are using - and the updating our use of them to being the qualified version | 21:10 |
| mordred | ianw: but I thik what's there is a great step for now | 21:10 |
| *** xavinux has joined #opendev | 21:10 | |
| mordred | ianw: (once ansible-base is out, I'm pretty sure we'll want to just install ansible-base and a couple of collections and call it done) | 21:11 |
| ianw | yeah, if someone feels keen it's just a matter of updating zuul to know about the collections repos and we can do speculative changes for them too, which would be cool | 21:11 |
| mordred | yeah | 21:12 |
| mordred | ianw: in other news - I hope you enjoy the giant pile of infra-prod-zuul topic patches :) | 21:12 |
| mordred | I didn't want you to be bored when you woke up | 21:12 |
| ianw | anyway, would be also be nice to be able to jump into https://github.com/ansible/ansible/pull/67684 when it's later and say "hey, working great for us :)" | 21:12 |
| clarkb | mordred: see comment on https://review.opendev.org/#/c/716764/1 | 21:12 |
| ianw | haha yes my client scrollback overflowed so i knew something was up | 21:13 |
| clarkb | infra-root can we also land and test https://review.opendev.org/#/c/716660/ ? then if that doesn't work we can switch meetpad to etherpad-dev.opendev.org to see if the domains are at fault | 21:13 |
| mordred | clarkb: oh - the zone repos are all in the openstack tenant right now, right/ | 21:13 |
| mordred | ? | 21:13 |
| fungi | ianw: not sure if you saw, but just a heads up that the centos 8 rsync mirror we're pulling from was changed to one which is being updated with more regularity | 21:13 |
| mordred | or did we put zone-zuul in the zuul tenant? | 21:13 |
| fungi | mordred: we've kept them all together afaik | 21:14 |
| clarkb | mordred: they are in openstack and opendev and zuul is also in zuul | 21:14 |
| fungi | oh, got it, so they're all in more than one tenant | 21:14 |
| clarkb | mordred: I think the real trick is that we'd need to add their keys to bridge to have them trigger that job | 21:15 |
| mordred | well - we can't trigger patches in system-config on patches in another tenant no matter how much we wait | 21:15 |
| clarkb | and to do that we basically have to assert that all the zone files are opendev owned | 21:15 |
| mordred | I don't think that'll work atm | 21:15 |
| mordred | we'd have to re-revert the patch you just reverted :) | 21:15 |
| clarkb | mordred: I don't think that is an issue, they are all in openstack/ | 21:15 |
| openstackgerrit | Merged openstack/project-config master: Revert "Add system-config and zuul to OpenDev tenant for jobs" https://review.opendev.org/716775 | 21:15 |
| clarkb | mordred: so we can drive it from openstack/project-config | 21:15 |
| mordred | I though we just said they were in multiple tenants | 21:16 |
| clarkb | the bigger issue is project ssh keys | 21:16 |
| clarkb | mordred: they are, but I don't think that is an issue? | 21:16 |
| ianw | fungi: cool, any manual interventions required? | 21:16 |
| fungi | ianw: not so far at least | 21:17 |
| clarkb | mordred: I think in openstack/project-config/zuul.d/projects.yaml we can add entries for zone* that run the opendev/system-config nameservers playbook job | 21:17 |
| fungi | ianw: seems to have updated fine | 21:17 |
| clarkb | mordred: and since that config is entirely in the openstack tenant it won't complain | 21:17 |
| clarkb | the bigger issue is the project ssh keys aiui | 21:17 |
| mordred | yes. I agree | 21:17 |
| mordred | (just went and checked) | 21:17 |
| ianw | fungi: the new model of sshing and running vos release under localauth on the server seems to have been effective, i'm not sure we've seen any of our locked volumes issues since? | 21:17 |
| mordred | and yes - I agree, the bigger issue would be the project ssh keys - we'd need to be ok with those repos having the ability to run playbooks on bridge | 21:18 |
| clarkb | I don't think corvus wanted to remove https://review.opendev.org/#/admin/groups/2030,members essentially | 21:18 |
| openstackgerrit | Mohammed Naser proposed zuul/zuul-jobs master: golangci-lint: add job https://review.opendev.org/716452 | 21:18 |
| mordred | clarkb: so - I'd say yeah we should just wait - but since there's no actual blockers for any of these, maybe it's time to just make a decision as to how we want that to work? | 21:19 |
| fungi | ianw: i haven't seen anyone raise issues related to it, at least | 21:19 |
| mordred | clarkb: like - it's been a theoretical problem until now since we haven't been doing this :) | 21:19 |
| corvus | clarkb, mordred: i'm having a really hard time keeping up with what you're talking about because i'm working on various other things | 21:20 |
| corvus | clarkb, mordred: do i need to drop that and just hang out with you? | 21:20 |
| mordred | corvus: it's ok - I'm having a tough time keeping up with it too | 21:20 |
| mordred | corvus: nope | 21:20 |
| clarkb | corvus: I don't think its urgent | 21:20 |
| clarkb | mordred: I think the ideal here is to have a repo trigger a job within the context of another repo | 21:20 |
| fungi | mordred: clarkb: looking at the state of the zone repos, they're included in the opendev and zuul tenants without reading any config, they're regular untrusted projects in the openstack tenant right now | 21:20 |
| clarkb | rather than within its own context | 21:20 |
| clarkb | fungi: yup we would use openstack/ to drive things I think | 21:21 |
| mordred | clarkb: yeah - I don't think that triggering concept exists today - it's sort of the very undefined "subscribe to project" feature I keep daydreaming | 21:21 |
| clarkb | mordred: because the real gap here is giving anyone with merge rights to repo A ability to get access to what repo B has access to. What we want to express is that repo A can tell repo B it should do its predefined things | 21:21 |
| corvus | mordred, clarkb: still not sure i'm up to speed but https://review.opendev.org/671637 is relevant | 21:21 |
| mordred | corvus: yes! | 21:22 |
| mordred | corvus: cool. thanks | 21:22 |
| mordred | clarkb: ok - so let's stick that patch at the end to give ourselves time to think on this topic further | 21:22 |
| clarkb | ya I think that change is one way of expressing what I just said | 21:22 |
| openstackgerrit | Merged openstack/project-config master: Add Fedora 31 builds to nb04 https://review.opendev.org/716127 | 21:22 |
| mordred | but it seems like there is a thing that's made some progress | 21:23 |
| clarkb | mordred: ++ | 21:23 |
| corvus | well, that patch may be dead in the water | 21:23 |
| corvus | i have not performed the followup analysis it needs after logan- raised that point | 21:23 |
| clarkb | corvus: ya I think what I head in mind was more of a trigger mechanism internal to zuul | 21:24 |
| clarkb | and that may be less of a security concern (its basically run this predefined thign with no updates) | 21:24 |
| corvus | not sure that's different :) | 21:25 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Run nodepool in zuul https://review.opendev.org/716770 | 21:25 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Run meetpad in zuul https://review.opendev.org/716771 | 21:25 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Run mirror-update in zuul https://review.opendev.org/716772 | 21:25 |
| mordred | clarkb: rebased nameserver out of the stack | 21:26 |
| fungi | oh, the thing we were discussing at the shanghai ptg | 21:26 |
| fungi | now i remember | 21:26 |
| mordred | clarkb: I left a comment on the nameserver patch with a link to the zuul patch and marked it WIP fo rnow | 21:27 |
| openstackgerrit | Mohammed Naser proposed zuul/zuul-jobs master: golangci-lint: add job https://review.opendev.org/716452 | 21:27 |
| clarkb | corvus: I think the key difference is running the job in its original context. Eg this isn't running nameserver-playbook job with zone-zuul's context and a borrowed ssh key. Its zone-zuul emits an event saying "I merged", system-config can then say "run nameserver-playbook in my context on this pipeline after that event" | 21:27 |
| clarkb | its possible that that doesn't chagne things internally | 21:28 |
| corvus | clarkb: yeah, i think we all want the same thing to happen. i think we've established it's a hard change. | 21:28 |
| corvus | clarkb: i think a literal implementation of what you're suggesting would be "allow projects to attach their own jobs to other project pipelines" which is also a *big* change with many pitfalls. | 21:29 |
| openstackgerrit | Mohammed Naser proposed zuul/zuul-jobs master: golangci-lint: add job https://review.opendev.org/716452 | 21:31 |
| clarkb | I keep thinking of it as a new trigger thing. Where maybe we can say its a change-merged trigger in pipeline config, then in project config we say promote subscribes to project 1 project 2 project 3. Then when it runs we get the change info for what merged but we still execute as if it were the "hosting" project | 21:32 |
| clarkb | (and ya thats a big change to things like mergers and executors) | 21:32 |
| mordred | yeah - I keep thinking of it like a new trigger as well - because putting a job into another project's pipeline isn't the thing we want to do - we want to start running a job in our own pipeline but triggered when an external event happens ... and I totally agree, it's likely a huge and spidering change that I can't even begin to comprehend the ramifications of | 21:33 |
| clarkb | in this specific example we don't actaully care about teh triggering project at all | 21:33 |
| corvus | okay i will task switch to talking about this problem | 21:33 |
| clarkb | all we care is that something we know we care about merged and we should run the job | 21:33 |
| clarkb | (which gets tricky in zuul because zuul does care about that state) | 21:34 |
| *** xavinux has quit IRC | 21:34 | |
| mnaser | btw -- thanks for the whole repo with secrets, i am able to centralize it for uploading docker image ssuccessfully! | 21:34 |
| mordred | mnaser: woot! | 21:34 |
| corvus | clarkb: i don't understand the following things you wrote at 21:32: "pipeline config" "project config" "promote subscribes" | 21:34 |
| clarkb | corvus: let me try and mock it up in etherpad. Will be a few minutes | 21:35 |
| mnaser | and we can confirm that the acls were updated too: https://review.opendev.org/#/admin/projects/vexxhost/base-jobs,access | 21:35 |
| corvus | clarkb: i think just using more words there would help. | 21:35 |
| clarkb | corvus: the first bit of it is either a new trigger or modifying existing triggers to apply more broadly than a project:project match | 21:36 |
| mordred | corvus: in my brain this is akin to the SF trigger plugin - there is a trigger source that something can hit that causes one or more jobs to run for a project. except in this case it's not an external party hitting a rest endpoint - it's an internal zuul event - although honest to god it could be a post job in a zone repo pinging a rest endpoint to say "please run this job" | 21:37 |
| corvus | clarkb, mordred: zuul is driven by git events. this is a git event. the event is that a change in project A merged. we want it to run a job which is defined in project B. that is straightforward and there is a model for that. | 21:37 |
| clarkb | corvus: then in your system-config zuul.yaml project: config you can say under promote: job listings that "this job subscribes to this trigger event happening for projects x y and z" | 21:37 |
| corvus | it's not an internal zuul event | 21:37 |
| corvus | the trigger is literally that a commit to a git repo that zuul manages has merged. | 21:37 |
| clarkb | corvus: yes, the main change is updating how we map those unto project: job listings in system-config zuul.yaml | 21:38 |
| corvus | zuul understands things in terms of git commits to projects. to try to bend this into an arbitrary external trigger is the wrong approach. | 21:38 |
| clarkb | I don't know if that needs a new expression or not on the pipeline trigger side | 21:38 |
| corvus | clarkb: yeah, i grok that. it's hard, probably not impossible. | 21:38 |
| clarkb | but then you can basically say "I know this event is for another project but I don't care, run this job in the context of the current "host" sate" | 21:39 |
| corvus | what's a "host" ? | 21:39 |
| clarkb | corvus: in this case system-config's promote: pipeline entry for playbook-nameserver job saying "I want to run when zone-foo mergers" | 21:39 |
| clarkb | corvus: the executors and mergers would prep the job for system-config repos and not use speculative state from zone-foo | 21:39 |
| corvus | i still don't understand what a host is | 21:39 |
| clarkb | corvus: its the context for which the job is running in. We aren't switching that like we would in a setup today | 21:40 |
| corvus | clarkb: running the job is the easy part | 21:40 |
| fungi | host as in the project hosting the job? | 21:40 |
| corvus | clarkb: i get and totally agree that "run this job without the ability for zone-foo to muck it up" is desirable. that's why my change has a WIP-1 on it. | 21:41 |
| clarkb | fungi: yes | 21:41 |
| clarkb | if I understand the security concerns in that job the issue is we're running the job in the context of the "other" | 21:42 |
| corvus | clarkb: let's try this approach. you're suggesting that a merge event on zone-foo should enqueue an item into that pipeline. it also sounds like you think that the project attached to that item should be "system-config", not "zone-foo". what ref would you attach to that item? | 21:42 |
| clarkb | it decides when the job runs | 21:43 |
| clarkb | and all of the zuul dict data in the job is tied to that repo | 21:43 |
| mordred | I thnik the ref should be head of master of system-config | 21:43 |
| clarkb | corvus: yes that is what I'm suggesting | 21:43 |
| mordred | like if it was the periodic pipeline | 21:43 |
| clarkb | mordred: yup | 21:43 |
| clarkb | exactly that | 21:43 |
| clarkb | because what is important here is that the job get run, but not any new or speculative state | 21:44 |
| corvus | why? given that the triggering event is "a change to stable/blarg on zone-foo has merged" why should we enqueue a system-config@master change into a pipeline? | 21:44 |
| corvus | it seems to me that we're really focused on "how to we most expeditiously get this bit of code to run" and not at all thinking about "what is the right way to fit this into the zuul data model" | 21:44 |
| clarkb | corvus: because system-config knows to grab latest zone-foo and apply it, it doesn't need the specifics. I expect this would be a fairly common scenario for more CD type operations | 21:44 |
| corvus | clarkb: sure, but why do we have to break the zuul model for that? why can't we take a minute and see if we can actually encode what's really going on? | 21:45 |
| clarkb | corvus: we don't necessarily need to | 21:45 |
| corvus | because what's really going on is that a change to some branch of zone-foo has merged. *that* is the item that should be enqueued into the pipeline | 21:45 |
| corvus | yes, there's a piece missing because we can't do what we want | 21:46 |
| corvus | but let's see if we can figure out that missing piece in the existing model | 21:46 |
| clarkb | corvus: I agree with the first part. I think the desired end result is that a specific job be run. I'm not sure that the second bit is necessary to achieve the use case. But it would be more consistent with zuul's existing model | 21:46 |
| clarkb | if we can solve the problem within that existing model thats great and I have no objections | 21:47 |
| corvus | how urgent is this? | 21:49 |
| clarkb | I don't think it is critically urgent | 21:50 |
| clarkb | we can either keep nameserver updates happening in cron or add a new periodic pipeline with shorter cycle period and run it there too | 21:50 |
| clarkb | neither option is a regression compared to the current setup and the second allows us to push it into zuul anyway | 21:50 |
| mordred | yeah - this came up because I've been pushing up patches to zuul-ify run_all and clarkb brought up that we should wait on doing the nameserver playbook | 21:51 |
| corvus | re-reading the change, i think logan-'s concern may be fixable with the same fix we made to secrets: allow project-config repos to attach jobs to other projects, and drop the idea of doing this with allowed-projects | 21:51 |
| corvus | clarkb, mordred: let's say we come to agreement that we should have some sort of "trigger on other project" facility that lets zone-foo enqueue a system-config item of some kind | 21:52 |
| corvus | what pipeline should that go into, and where should zuul report the results for that? | 21:52 |
| mordred | corvus: I would imagine it would potentially need to be its own pipeline, and I'd imagine the results would just go to the dashboard | 21:53 |
| clarkb | corvus: to me I think that goes to system-config. The context of the job is there, its where people will look for why the dns update didn't happen etc. All that zone-foo is providing is the trigger that this happen | 21:53 |
| corvus | (i think that it should report the results on the zone-foo change, that way we can look back at that change, and see what happened when it merged; that's a big driver for why i think the current model is the correct one). | 21:53 |
| corvus | clarkb: how does it "go to system-config"? what change in system-config does it report on? | 21:54 |
| clarkb | corvus: right thats where it breaks the zuul model | 21:54 |
| corvus | mordred: earlier you linked to a change which changed manage-projects, and ran manage-projects, and the results were all there. i thought that was cool. | 21:54 |
| corvus | i'd like to keep doing that. | 21:55 |
| mordred | yes - I thnk that's GREAT | 21:55 |
| mordred | but I think that, for some reason, we are not comfortable with that model here | 21:55 |
| mordred | because we don't trust the zone repos the same way we trust the project-config repo | 21:55 |
| corvus | mordred: why are we not comfortable? | 21:55 |
| clarkb | I think where things get weird here is system-config is affecting the state chagne so that is where I expect to observe the state change | 21:55 |
| mordred | corvus: I have no clue | 21:55 |
| corvus | mordred: no i don't think that's it | 21:55 |
| mordred | it's not? | 21:55 |
| corvus | mordred: i'm the one not comfortable with a thing, let me explain it | 21:55 |
| mordred | cool. I think I've misunderstood something fundamental | 21:56 |
| corvus | mordred: the thing from last july that we're not comfortable with (which i raised) was that we wanted to allow the zuul project to maintain its own zone repo, and without doing something that gave them access to prod servers | 21:57 |
| corvus | mordred: it's definitely not the case that we're not comfortable reporting run results on changes to the zuul zone repo. i think that is very desirable. | 21:58 |
| clarkb | right the concern is adding zuul zone's ssh key to bridge | 21:58 |
| mordred | oh - no - that's not the thing that I thought we were uncomfortable with | 21:58 |
| corvus | clarkb: yeah, that's it exactly | 21:58 |
| mordred | right | 21:58 |
| clarkb | while also allowing the zuul project to manage that directly rather than limiting it to infra-root | 21:58 |
| mordred | that's what I meant by "we don't trust the zone repo enough" | 21:58 |
| clarkb | we could do that if we didn't let zuul maange the zone file direclty | 21:58 |
| mordred | we dont' have the same trust relationship between system-config and zone as we do between system-config and project-config - because we ARE willing to put project-config key on bridge, but not zone | 21:59 |
| corvus | mordred: okay. so from a high-level POV, i think we probably all agree that if we can have the zone-foo trigger a system-config job and report the results on zone-foo, that's what we want. *how* to accomplish that is up in the air and probably complicated. | 21:59 |
| mordred | yah. | 22:02 |
| mordred | that said ... | 22:02 |
| corvus | but if we agree that's the ideal result, then i think it's worth looking at something like 671637, and exhausting that solution space before we take the other approach. | 22:02 |
| mordred | for now, the zuul zone repo is not managed by the zuul team | 22:02 |
| *** frickler_ has joined #opendev | 22:03 | |
| mordred | so I think it is safe at the moment to do the job manage-projects-style from an opendev POV - and the work that we're talking about is actually work to enable to use give maint to the zuul team of the zuul-zone repo, right? | 22:03 |
| corvus | mordred: then maybe we should just put its keys on bridge and call it done? :) | 22:03 |
| mordred | yeah. I'm thinking all of this may have just been a long conversation to get us to that point ;) | 22:03 |
| clarkb | it does technically have its own group | 22:03 |
| clarkb | its just the only members of that group are us | 22:03 |
| mordred | it does - but the onlymembers are infra-core | 22:03 |
| mordred | yeah | 22:03 |
| mordred | so - I think we'd have to be the ones to change the membership | 22:03 |
| clarkb | I do think this conversation is an important one to keep thinking on though | 22:04 |
| mordred | so it's still fully withing our trust domain | 22:04 |
| mordred | YES | 22:04 |
| mordred | it's _very_ important | 22:04 |
| clarkb | because this is likely to be fundamental to zuul's cd useablity | 22:04 |
| mordred | but I think we can unblock ourselves on this particular task because of where we happen to be currently | 22:04 |
| corvus | clarkb: in the original zuulv3 design, there's supposed to be a tenant ssh key | 22:05 |
| corvus | clarkb: just doing that would probably solve this for 98% of the world | 22:05 |
| corvus | clarkb: leaving our little corner of "projects that share a tenant in a publicly accessible zuul but don't trust each other" needing a solution | 22:06 |
| corvus | (like if you're in a corp, and your team has a zuul tenant with all your microservices or whatever, just add the tenant ssh key to your prod system) | 22:07 |
| clarkb | ya | 22:07 |
| *** frickler has quit IRC | 22:08 | |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Add zone keys to zuulcd user https://review.opendev.org/716781 | 22:20 |
| mordred | clarkb, corvus : ^^ | 22:20 |
| clarkb | mordred: should we also update the acl for the zone files to explicitly make it infra-root for now? | 22:21 |
| mordred | clarkb: I mean - we could - but since the only member is infra-root - I don't know that it's that much different? | 22:21 |
| * mordred does not have an opinion one way or the other | 22:21 | |
| clarkb | mordred: I think it would mostly just be a reminder that "this repo is in the class of things with root access" | 22:22 |
| mordred | seems reasonable | 22:22 |
| *** DSpider has quit IRC | 22:29 | |
| openstackgerrit | Ian Wienand proposed openstack/project-config master: zuul-worker element: use python3-libselinux for fedora https://review.opendev.org/716783 | 22:34 |
| ianw | infra-root: ^ if we could look at that one, it will stop a broken fedora31 build loop -- i'm going to propose we remove it all in a follow-on anyway | 22:34 |
| openstackgerrit | Merged zuul/zuul-jobs master: local-log-download : role with script to download all log files https://review.opendev.org/715756 | 22:49 |
| openstackgerrit | Merged openstack/project-config master: zuul-worker element: use python3-libselinux for fedora https://review.opendev.org/716783 | 22:50 |
| openstackgerrit | Ian Wienand proposed openstack/project-config master: zuul-worker: remove python-apt & libselinux deps https://review.opendev.org/716785 | 22:58 |
| *** hashar has quit IRC | 22:59 | |
| openstackgerrit | Ian Wienand proposed openstack/project-config master: zuul-worker: remove python-apt & libselinux deps https://review.opendev.org/716785 | 23:01 |
| *** tosky has quit IRC | 23:03 | |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: Debuntu: add apt-transport-https https://review.opendev.org/716788 | 23:11 |
| openstackgerrit | Ian Wienand proposed openstack/project-config master: zuul-worker: remove additional install of apt-transport-https https://review.opendev.org/716789 | 23:14 |
| clarkb | mordred: ianw fungi corvus https://review.opendev.org/#/c/716660/ any chance you want to give that a try on meetpad (use etherpad-dev.openstack.org) | 23:20 |
| clarkb | also Ithink we can remove meetpad from the emergency file? | 23:20 |
| clarkb | I want to say the chagne corvus said it needed has merged | 23:20 |
| mordred | clarkb: did you want to switch that to point to the etherpad-dev.opendev.org ? | 23:21 |
| mordred | (didn't you make a cname for that?) | 23:21 |
| fungi | should i switch ep-dev back to the "no-skin" undefault or did people want to play with the colibris skin some more? | 23:21 |
| clarkb | mordred: I think it would probably be helpful to do it one at a time | 23:21 |
| mordred | kk | 23:21 |
| clarkb | mordred: so that we can identify which issue it is if this or that fixes it | 23:21 |
| clarkb | fungi: I prefer the old one personally | 23:21 |
| fungi | as do i | 23:21 |
| clarkb | I think the old one will be better for meetpad too | 23:21 |
| mordred | +2 from me | 23:22 |
| clarkb | mordred: if new etherpad doesn't work we'll switch to opendev.org and see if domain change fixes it | 23:22 |
| fungi | probably so. i'll yank that line back out of its config and take it back out of the emergency list then | 23:22 |
| clarkb | mordred: but if we land both we won't know which is the thing :) | 23:22 |
| mordred | good point :) | 23:22 |
| mordred | also - if y'all feel like doing more reviews of https://review.opendev.org/#/q/topic:infra-prod-zuul - I can start landing them in the morning | 23:22 |
| fungi | okay, ep-dev is back to no-skin | 23:23 |
| fungi | clarkb: mordred: one problem to note with using etherpad-dev.opendev.org is that it's going to need a vhost change | 23:25 |
| clarkb | fungi: oh do we not * it there? | 23:26 |
| fungi | RewriteRule ^/+(.+)$ https://etherpad-dev.openstack.org/p/$1 [NC,L,R=301] | 23:26 |
| clarkb | I was hoping we'd just need to override the ssl cert validation | 23:26 |
| clarkb | aha | 23:27 |
| clarkb | ok if we get to that point we can maybe just ninja that for a few minutes and test | 23:27 |
| clarkb | (we can probably just add a second vhost and s/openstack/opendev/) | 23:27 |
| fungi | we could probably drop the https://etherpad-dev.openstack.org from that and just rewrite to /p/$1 with no hostname? | 23:27 |
| corvus | clarkb: i will remove meetpad from emergency | 23:27 |
| clarkb | corvus: I'm approving https://review.opendev.org/#/c/716660/ then. thanks! | 23:28 |
| corvus | done | 23:28 |
| fungi | i'll readd etherpad-dev to the emergency file now though in case folks want to fiddle with the vhost config locally on it | 23:28 |
| clarkb | mordred: I suddenly had a worry that the emregency file would stop working with the zuul jobs driving things, but then I remembered we are still running ansible on bridge so that is all fine. | 23:29 |
| clarkb | (this is me talking out loud so that other reviewers can either confirm or reject that statement :) ) | 23:29 |
| clarkb | mordred: re https://review.opendev.org/#/c/716771/2/.zuul.yaml similar to dns, how do we get that to run when docker images have updated. I think this is much simpler since it is all in the same repo | 23:30 |
| ianw | mordred: q inline for https://review.opendev.org/#/c/716745/1 ... why have a semaphore of 10 for service-bridge? | 23:32 |
| clarkb | ianw: its for all the jobs that parent to that base job iirc. That way we limit the number of ansible processes on bridge | 23:32 |
| clarkb | they won't directly conflict with each other but resource consumption may be something to keep in mind | 23:33 |
| ianw | clarkb: hrm, so do we have a lock to ensure service-bridge won't run ontop of itself? | 23:34 |
| clarkb | ianw: the jobs run in supercedent promote pipeline. Though now that you've asked they also run in periodic and we may need to keep promote and periodic from overlapping | 23:35 |
| fungi | it's only triggered from one branchless repo in a supercedent pipeline, right? | 23:35 |
| clarkb | mordred: ^ we may need the per job semaphore afterall | 23:35 |
| fungi | oh, indeed, if periodic is also in use | 23:35 |
| ianw | yeah, i feel like if it can happen, it will :) | 23:36 |
| clarkb | if we only ran in promote it would be fine | 23:36 |
| clarkb | due to supercedent pipeline behvaior | 23:37 |
| fungi | that said, what's the actual risk if it runs twice at the same time? | 23:37 |
| fungi | what's it going to break? | 23:38 |
| fungi | or is it just that we don't know, effectively undefined behavior, better safe than spend a weekend indoors? | 23:38 |
| clarkb | fungi: I think the big risk is in things like service restarts | 23:38 |
| clarkb | with gitea in particular we ensure that we stop a single backend at at time and do the stops in a very coordinated fashion | 23:38 |
| fungi | ahh, yeah, strict ordering | 23:39 |
| clarkb | we could break replication again because we are trying to stop the same backend multiple times (breaking the careful stop start ordering) | 23:39 |
| ianw | also in the later changes, letsencrypt really doesn't want to run over itself .. they would race to update acme.opendev.org records | 23:41 |
| clarkb | I expect what we want to do is add a semaphore per job then it can run in periodic and promote | 23:44 |
| clarkb | and that sempahore will have a count of 1 making it a mutex | 23:44 |
| clarkb | then we can have a second semaphore with a large limit to reduce too many jobs running at once on bridge | 23:44 |
| ianw | yep | 23:46 |
| ianw | as fungi points out many would be pretty close to very idempotent, but still i bet there's plenty of corner cases | 23:47 |
| clarkb | ya and I expect they'll be difficult to debug | 23:48 |
| clarkb | we'll just get really weird behaviors occasionally and not be able to track them back to anything specific | 23:48 |
| openstackgerrit | Merged opendev/system-config master: Switch meetpad to etherpad-dev https://review.opendev.org/716660 | 23:51 |
| openstackgerrit | Merged zuul/zuul-jobs master: golangci-lint: add job https://review.opendev.org/716452 | 23:56 |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!