| *** openstack has joined #opendev | 09:53 | |
| *** ChanServ sets mode: +o openstack | 09:53 | |
| frickler | infra-root: supy-bot seems to have been stuck in a loop waiting for a reply to a ping from yesterday, not sure how it got the idea that might ever be successful http://paste.openstack.org/show/792090/ | 09:54 |
|---|---|---|
| *** dpawlik has quit IRC | 09:55 | |
| *** ysandeep|lunch is now known as ysandeep | 09:55 | |
| frickler | statusbot seems to have gotten stuck at around the same time, restarted them both | 09:56 |
| *** dpawlik has joined #opendev | 09:56 | |
| frickler | #status log restarted statusbot and meetbot which both seem to have stopped working around 2020-04-13 21:40Z | 09:59 |
| openstackstatus | frickler: finished logging | 09:59 |
| openstackgerrit | Marcin Juszkiewicz proposed openstack/diskimage-builder master: Do not try to use MBR on AArch64 https://review.opendev.org/719805 | 10:12 |
| *** rpittau is now known as rpittau|bbl | 10:23 | |
| *** ysandeep is now known as ysandeep|afk | 10:37 | |
| *** ysandeep|afk is now known as ysandeep|rover | 10:53 | |
| openstackgerrit | Sorin Sbarnea proposed zuul/zuul-jobs master: WIP: tox: allow tox to be upgraded https://review.opendev.org/690057 | 10:55 |
| *** drifterza has quit IRC | 11:14 | |
| *** ysandeep|rover is now known as ysandeep|coffee | 11:39 | |
| *** ysandeep|coffee is now known as ysandeep | 11:51 | |
| *** ysandeep is now known as ysandeep|rover | 11:56 | |
| *** hashar has joined #opendev | 12:02 | |
| openstackgerrit | Sorin Sbarnea proposed zuul/zuul-jobs master: tox: allow tox to be upgraded https://review.opendev.org/690057 | 12:28 |
| *** rpittau|bbl is now known as rpittau | 12:31 | |
| *** lpetrut has quit IRC | 12:54 | |
| *** lpetrut has joined #opendev | 13:01 | |
| slittle1 | Yes, it was https://review.opendev.org/#/c/718772 that we need to get on track. | 13:16 |
| slittle1 | and sorry for the long delay. Stuff happens when 'work from home' collides with 'daddy daycare'. :) | 13:18 |
| openstackgerrit | Merged openstack/project-config master: Add kernel to StarlingX https://review.opendev.org/718772 | 13:50 |
| *** ykarel is now known as ykarel|away | 13:50 | |
| *** rpittau has quit IRC | 13:50 | |
| *** rpittau has joined #opendev | 13:51 | |
| slittle1 | thanks | 13:56 |
| slittle1 | Please add myself to starlingx-kernel-core. I'll add the rest. | 13:57 |
| slittle1 | uhmmm... is this normal .... Zuul message "Unable to freeze job graph: Job infra-prod-remote-puppet-else depends on infra-prod-update-system-config which was not run. | 14:01 |
| *** roman_g has joined #opendev | 14:01 | |
| AJaeger | mordred: ^ | 14:02 |
| slittle1 | can't see/clone the new git yet | 14:02 |
| mordred | AJaeger: on project-config? | 14:04 |
| openstackgerrit | Monty Taylor proposed openstack/project-config master: Remove deps from remote-puppet-else https://review.opendev.org/719976 | 14:06 |
| mordred | AJaeger: ^^ | 14:06 |
| mordred | I'll run manage-projects real quick | 14:06 |
| AJaeger | thanks, mordred . config-core, please +2A 719976 | 14:11 |
| *** lpetrut has quit IRC | 14:24 | |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Don't map known_hosts in to manage-project https://review.opendev.org/719986 | 14:26 |
| *** sshnaidm has joined #opendev | 14:29 | |
| *** sshnaidm has quit IRC | 14:30 | |
| corvus | mordred: did you write that change to use the same ansible.cfg in both places? | 14:31 |
| corvus | if not, i can write it | 14:32 |
| mordred | corvus: yes - it's on my list for today | 14:33 |
| corvus | mordred: you want to do it, or want me to? | 14:33 |
| mordred | I want to do it | 14:33 |
| corvus | ok all yours then :) | 14:33 |
| mordred | corvus: I'm ... I'm having some further thoughts about the shape of how this all works based on clarkb's feedback about what to do with /opt/system-config ... and I had another idea this morning | 14:34 |
| mordred | tl;dr - still working on it :) | 14:34 |
| corvus | ack | 14:35 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Fix the hostkey for gerrit https://review.opendev.org/719986 | 14:35 |
| mordred | corvus, fungi: ^^ | 14:35 |
| mordred | that's broken in prod, could use a quick +A | 14:36 |
| corvus | mordred: i don't understand the 'sad' can you leave a comment with a quick summary of what happened? | 14:36 |
| mordred | corvus: you're looking at an old patchset | 14:37 |
| corvus | mordred: oh so i am :) | 14:37 |
| mordred | corvus: the first one was "crap, let's partially revert" - but then I was able to diagnose the actual problem | 14:37 |
| corvus | my wish was granted | 14:37 |
| mordred | \o/ | 14:37 |
| corvus | mordred: https://review.opendev.org/719685 feels like it's in service of making things easier to run manually on bridge -- but what's the process of running manually with that? | 14:40 |
| mordred | corvus: so - on the run ansible from zuul cloned content ... what if we went a slightly different direction, and instead of pushing the content to /home/zuulcd/src we just pushed system-config and project-config to /opt/system-config and /opt/project-config and had the global ansible cfg always apply whether it's us or zuulcd running the playbooks. we'd need to be able to block zuul from syncing, but I'm | 14:40 |
| mordred | starting to worry that we're growing 3 different contexts for running things (manual, zuulcd and run-base test) | 14:40 |
| mordred | corvus: this is jus the thing - it actuallymakes the process harder - but clarkb was worried that we'd run manage-projects without remembering to git pull in /opt/project-config so thought that needing to pass -eproject_config_src=/opt/project-config to ansible-playbook would be a good reminder of that | 14:41 |
| corvus | mordred: "we'd need to block zuul from syncing" you mean: if we want to run something manually, we need to block zuul from doing anything on bridge during that time? | 14:41 |
| mordred | but this is what got me worried that it's getting too complex and we might need to rethink the shape | 14:41 |
| mordred | corvus: yeah - if we have zuulcd push refs to /opt/system-config - we might want to make sure that doesn't happen for a period | 14:42 |
| mordred | we could pretty easily drop a flag file and have the job check for it and bail | 14:42 |
| clarkb | project-config updates are a bit different fwiw. Because zuul and manual use a different path by default | 14:42 |
| mordred | right - that's what I'm getting at ... maybe that's a mistake | 14:43 |
| clarkb | this means it is very easy for zuul to happily keep things moving fowrard then for a human to accidentally sync a 6 month old state | 14:43 |
| clarkb | mordred: ah | 14:43 |
| mordred | like - I think we're creeping towards a system that's brittle when we have to do something manually - which is exactly when we don't want it to be brittle | 14:43 |
| corvus | yeah, i could see a flag file as a solution to that -- we could actually have the job wait in that case (under the assumption that maybe if it waits 15 minutes, we'll be done with whatever manual thing we were doing) | 14:44 |
| mordred | yeah | 14:44 |
| mordred | why don't I code that up and we can see what we think? | 14:44 |
| corvus | and there are also many manual thing we could do without even dropping the flag file (anything where the p-c state isn't absolutely critical) | 14:45 |
| mordred | yeah | 14:45 |
| mordred | that said - what if /home/zuulcd is the new "location" | 14:45 |
| mordred | and we ditch /opt/system-config and /opt/project-config | 14:45 |
| *** mlavalle has joined #opendev | 14:45 | |
| mordred | (keeps the repo sync easy - and if we grow a third or fourth repo it remains consistent) | 14:46 |
| mordred | we could make /opt/system-config a symlink for our muscle memory :) | 14:46 |
| corvus | i have a slight preference for just making it go away so we don't forget it's managed by zuul (but don't object to a symlink if others feel it's good) | 14:47 |
| mordred | yeah - I think that's a good point and you've convinced me | 14:47 |
| mordred | I think it expresses nicely "this is a process owned by zuul and manual operation is the exception" | 14:48 |
| mordred | rather than "zuul is driving an otherwise manual process" or "zuul and manual are equally valid" | 14:48 |
| clarkb | mordred: ya I think "this is a process owned by zuul and manual operation is the exception" is the perspective we should apply to these improvements | 14:49 |
| mordred | cool | 14:49 |
| * mordred will update the stack with this approach | 14:49 | |
| clarkb | then have some methods/tools/process/whatever that allow us to make exceptions to the automated process as necessary | 14:49 |
| *** ysandeep|rover is now known as ysandeep|away | 14:50 | |
| mordred | also - I was thinking it would be neat if we can collapse run-production-playbook.yaml and run-base.yaml more | 14:50 |
| mordred | now that we have run-production-playbook and it's what we're using, it feels like there might be opportunities to do less in run-production-playbook - but I haven't gotten all the way there yet :) | 14:51 |
| mordred | clarkb, corvus: what if we changed the zuulcd user on bridge to just be the zuul user? | 15:01 |
| openstackgerrit | Merged openstack/project-config master: Remove deps from remote-puppet-else https://review.opendev.org/719976 | 15:02 |
| clarkb | mordred: would that change anything? | 15:02 |
| mordred | I say that becuase I'm currently making a symlink from /home/zuul to /home/zuulcd in the test job | 15:02 |
| mordred | since the test job runs in /home/zuul and it's a little harder to change that | 15:03 |
| clarkb | there shouldn't be a /home/zuul? | 15:03 |
| clarkb | oh I see its to match the gate/check jobs to deploy | 15:03 |
| mordred | yeah | 15:03 |
| mordred | otherwise we need to add a variable and set it in the test job but not in prod | 15:03 |
| clarkb | I think at the time we called it zuulcd the idea was to differentiate it from zuul direct login | 15:03 |
| clarkb | but maybe that was a mistake given the test setups | 15:03 |
| mordred | yeah - totally, and it was a reasonable choice at the time I think | 15:04 |
| mordred | but yeah - I think maybe we've learned something | 15:04 |
| mordred | I can do the symlink for now for this stack | 15:04 |
| mordred | and we can take that as a follow on cleanup | 15:04 |
| mordred | ALSO - I had this thought that we should make inventory/groups.yaml into inventory/groups/{foo}.yaml - so that we can better do file matchers in zuul | 15:05 |
| mordred | (and the gate-groups.yaml wouldn't be a special-case for gate jobs either) | 15:05 |
| slittle1 | please add 'scott little' is first core of starlingx-kernel-core . I'll add the others | 15:13 |
| clarkb | slittle1: done | 15:13 |
| fungi | clarkb is the fastest draw in the west | 15:13 |
| clarkb | its an easy task while listening to the board meeting | 15:14 |
| clarkb | :) | 15:14 |
| fungi | that's what i thought too, but i only got as far as the group filter page ;) | 15:15 |
| fungi | anybody else having trouble reaching cacti.openstack.org? | 15:19 |
| clarkb | fungi: not me | 15:20 |
| clarkb | I'll be hitting it via ipv4 fwiw | 15:20 |
| fungi | it just started responding for me | 15:20 |
| fungi | ahh, yep, that was a v6 timeout | 15:20 |
| fungi | looks like something has killed my v6 routes | 15:21 |
| *** JasonF is now known as JayF | 15:25 | |
| fungi | yeah, apparently sometimes my provider's dhcp6 server doesn't respond to my dhcpcd and it expires my delegation. i should probably force a longer grace period in an override | 15:26 |
| clarkb | fungi: I think a lot of dhcp renewals happen at the halfway point through a lease (so you can renew very early) | 15:27 |
| fungi | yeah, it starts trying according to the log and goes multiple attempts with a "no carrier" | 15:28 |
| fungi | it may be that the broadband connection itself was down at the time, i was sound asleep when it happened | 15:29 |
| fungi | in fact, my v4 dhclient was complaining about "host is down" too, so i bet that was it | 15:30 |
| *** drifterza has joined #opendev | 15:33 | |
| openstackgerrit | Clark Boylan proposed opendev/system-config master: DNM Test docker-compose upgrade https://review.opendev.org/719682 | 15:43 |
| openstackgerrit | Clark Boylan proposed opendev/system-config master: Install docker-compose from pypi https://review.opendev.org/719589 | 15:50 |
| openstackgerrit | Clark Boylan proposed opendev/system-config master: Use HUP to stop gerrit in docker-compose https://review.opendev.org/719051 | 15:50 |
| *** rpittau is now known as rpittau|afk | 16:02 | |
| *** drifterza has quit IRC | 16:07 | |
| clarkb | infra-root ^ fwiw I think testing is showing that that is safe, but I want it to go green and not just me assume that based on what I'm seeing :) I've also got a couple general test job improvements related to that that I'll push up once the current round of results are available | 16:09 |
| clarkb | this has me thinking that it might be a good idea to more properly formalize a before and after upgrade process in our test jobs. I'm not quite sure how to make that less hacky though | 16:09 |
| clarkb | I'll ponder on it further as it does give you that extra bit of confidence going from state A to state B won't break anything | 16:10 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Use project-config from zuul instead of direct clones https://review.opendev.org/719343 | 16:11 |
| mordred | clarkb, corvus: how does that look now ^^? | 16:11 |
| mordred | clarkb: I agree - a before and after upgrade would be really nice | 16:11 |
| clarkb | mordred: what I'm thinking is we want to run everything on HEAD^ then go ahead to HEAD and run everything at that state | 16:12 |
| mordred | clarkb: yeah. | 16:13 |
| clarkb | but that makes the jobs run logner and I'm sure there are corner cases where doing that won't work as I expect (like if HEAD^ is just broken compared to state of the world) | 16:13 |
| mordred | clarkb: we'll need to special case "this patch added a completely new playbook" | 16:13 |
| mordred | clarkb: that's a very good pint | 16:13 |
| mordred | point | 16:13 |
| clarkb | mordred: ya it makes me wonder if we can make it opt in somehow | 16:14 |
| mordred | clarkb: maybe we write a general upgrade job but put it in an on-demand only pipeline | 16:14 |
| mordred | because there are plenty of changes where an upgrade test would be overkill | 16:14 |
| clarkb | ya maybe check opendev upgrade comemnts are the way to make it opt in | 16:15 |
| clarkb | I had a thought we could have a "UPGRAGE" file that we edit to trigger jobs too | 16:15 |
| clarkb | (but thats the hacky stuff I'm talking about ) | 16:15 |
| mordred | clarkb: I think it would be important to the design to name it UPGRAGE | 16:19 |
| clarkb | I didn't catch that typo but I agree it should stay :) | 16:20 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Rename zuulcd to zuul https://review.opendev.org/720029 | 16:45 |
| mordred | clarkb: ^^ | 16:46 |
| *** hashar has quit IRC | 16:49 | |
| clarkb | https://review.opendev.org/#/c/719682 passes testing \o/ I think that means we are good to land the production change | 16:52 |
| openstackgerrit | Clark Boylan proposed opendev/system-config master: DNM Test docker-compose upgrade https://review.opendev.org/719682 | 16:53 |
| openstackgerrit | Clark Boylan proposed opendev/system-config master: Run jobs prod test jobs when docker images update https://review.opendev.org/720030 | 16:53 |
| clarkb | however new ps to simplify some things | 16:53 |
| clarkb | infra-root https://review.opendev.org/720030 in particular is a simple job files update to ensure jobs run when we update docker images if you have a moment | 16:53 |
| clarkb | that tripped me up when I started thsi testing process | 16:53 |
| mordred | clarkb: I'm good landing that whenver | 16:59 |
| noonedeadpunk | hi everyone. | 17:01 |
| mordred | hi noonedeadpunk | 17:01 |
| noonedeadpunk | have small question about how required-projects works. Am I right that they got cloned `/home/zuul/src/` on master branch and patch being tested don't affect it? | 17:03 |
| noonedeadpunk | So like if patch is for stein, cloned roles in /home/zuul/src/ will still be for master? | 17:03 |
| AJaeger | noonedeadpunk: no, they are for stein as well | 17:03 |
| AJaeger | noonedeadpunk: unless you use override-checkout | 17:03 |
| AJaeger | Why do you ask? | 17:03 |
| noonedeadpunk | Actually investigating why https://zuul.opendev.org/t/openstack/build/3e730fa1cd7143daac1d2957270ab749 fails... So we actually linking cloned project here https://zuul.opendev.org/t/openstack/build/3e730fa1cd7143daac1d2957270ab749/log/job-output.txt#2739 | 17:05 |
| noonedeadpunk | But it feels like os_tempest is from master branch... | 17:05 |
| noonedeadpunk | but maybe issue is somewhere else then | 17:05 |
| fungi | noonedeadpunk: does this match what you think should have been checked out there? https://zuul.opendev.org/t/openstack/build/3e730fa1cd7143daac1d2957270ab749/log/job-output.txt#1811-1812 | 17:07 |
| mordred | noonedeadpunk: zuul will prepare the state for all branches ... | 17:07 |
| mordred | so that if there are depends-on or whatnot across branches, they'll all be stacked as needed into the appropriate branches - git checkout will work if you need to switch between them - but as AJaeger mentions, you can also set override-checkout in the job to control which one of them zuul checks out by default | 17:08 |
| noonedeadpunk | fungi: yeah, it's correct | 17:08 |
| noonedeadpunk | mordred: oh, it's pretty useful info thanks | 17:09 |
| noonedeadpunk | didn't know that's possible | 17:09 |
| mordred | noonedeadpunk: yeah - so you can test things like "does this change to stable/train still work with the upgrades currently in master" | 17:11 |
| noonedeadpunk | yeah, actually was fighting with the way of upgrades test | 17:12 |
| noonedeadpunk | we have implemented one, but at the moment it don't respect patches | 17:13 |
| fungi | noonedeadpunk: so if you see evidence that there's a different version installed than what zuul checked out, the usual culprits are something like a pip install not pointed at that source tree, or a distro package slipping in overwriting things, or some condition causing a script to grab its own copy of the source out across the internet | 17:13 |
| openstackgerrit | Sorin Sbarnea proposed zuul/zuul-jobs master: Improve 404 error message on download-logs.sh https://review.opendev.org/720035 | 17:13 |
| fungi | i'm not spotting any of those in the job log you linked, but there's presumably a lot more going on than winds up in that log | 17:13 |
| *** roman_g has quit IRC | 17:19 | |
| *** roman_g has joined #opendev | 17:19 | |
| *** roman_g has quit IRC | 17:20 | |
| *** roman_g has joined #opendev | 17:20 | |
| *** roman_g has quit IRC | 17:20 | |
| *** roman_g has joined #opendev | 17:21 | |
| *** roman_g has quit IRC | 17:21 | |
| *** roman_g has joined #opendev | 17:22 | |
| *** roman_g has quit IRC | 17:22 | |
| *** roman_g has joined #opendev | 17:23 | |
| *** roman_g has quit IRC | 17:23 | |
| *** roman_g has joined #opendev | 17:23 | |
| *** roman_g has quit IRC | 17:24 | |
| clarkb | infra-root ok https://review.opendev.org/#/c/719682/ has passed for a second time. Ithink that implies https://review.opendev.org/719051 and its parent are safe to go now. https://review.opendev.org/720030 is another good one that fixes job defs related to this | 17:49 |
| clarkb | yall should look over the testing in that first change though in order to convince yourself that the second is good | 17:49 |
| clarkb | I need to get late breakfast/early lunch now before our meeting but will be back in an hour for that | 18:00 |
| *** dpawlik has quit IRC | 18:02 | |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Use project-config from zuul instead of direct clones https://review.opendev.org/719343 | 18:06 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Rename zuulcd to zuul https://review.opendev.org/720029 | 18:06 |
| *** tobiash has quit IRC | 18:16 | |
| *** tobiash has joined #opendev | 18:17 | |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Use project-config from zuul instead of direct clones https://review.opendev.org/719343 | 18:22 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Rename zuulcd to zuul https://review.opendev.org/720029 | 18:22 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Fix the hostkey for gerrit https://review.opendev.org/719986 | 18:25 |
| mordred | infra-root: ^^ sorry - typo | 18:25 |
| fungi | aha, missing ' | 18:25 |
| fungi | sorry i didn't spot that earlier | 18:26 |
| frickler | mordred: still one ' missing in 719986 | 18:34 |
| mordred | frickler: doh | 18:41 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Fix the hostkey for gerrit https://review.opendev.org/719986 | 18:41 |
| mordred | frickler, fungi : ^^ fixed (thanks!) | 18:42 |
| * fungi must need glasses | 18:42 | |
| *** ralonsoh has quit IRC | 18:49 | |
| clarkb | I know I need them :) | 18:55 |
| openstackgerrit | Jeremy Stanley proposed opendev/irc-meetings master: Update OpenDev meeting location and name https://review.opendev.org/720060 | 19:15 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Rename zuulcd to zuul https://review.opendev.org/720029 | 19:16 |
| openstackgerrit | Jeremy Stanley proposed opendev/irc-meetings master: Not all meetings are OpenStack https://review.opendev.org/720063 | 19:21 |
| *** DSpider has quit IRC | 19:22 | |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Rename zuulcd to zuul https://review.opendev.org/720029 | 19:44 |
| ianw | fungi: could you look over https://review.opendev.org/#/c/719707/, which i noticed yesterday while debugging xenail venv failures | 19:50 |
| ianw | removes an old pip install from wheel build | 19:50 |
| openstackgerrit | Merged opendev/system-config master: Fix the hostkey for gerrit https://review.opendev.org/719986 | 19:51 |
| openstackgerrit | Merged opendev/system-config master: Publish docs updates to docs.opendev.org https://review.opendev.org/718762 | 19:51 |
| ianw | infra-root: another smallish one -> https://review.opendev.org/718600 ; this sends stats on vos release timing for mirror updates to graphite. it will be nice to graph that, alongside the timings we track for the other repos from the periodic release script | 19:54 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Update install-ansible away from /opt/system-config https://review.opendev.org/719186 | 19:55 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Run playbooks out of zuul checkout https://review.opendev.org/719190 | 19:55 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Use project-config from zuul instead of direct clones https://review.opendev.org/719343 | 19:55 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Rename zuulcd to zuul https://review.opendev.org/720029 | 19:55 |
| AJaeger | config-core, please review https://review.opendev.org/718676 - grafana neutron update | 20:01 |
| fungi | okay, i need to step away to start preparing dinner, but will jump back on to get some stuff done once that's finished | 20:01 |
| clarkb | mordred: can you check my comment on https://review.opendev.org/#/c/719192/1 | 20:07 |
| clarkb | mordred: and comment on https://review.opendev.org/#/c/719484/2 | 20:08 |
| openstackgerrit | Merged openstack/project-config master: Revert "wheel-build : temporary pip with checksum output" https://review.opendev.org/719707 | 20:09 |
| AJaeger | mnaser: did you see the comment on https://review.opendev.org/#/c/717507/ by tobiash to rename the role? | 20:11 |
| *** mtreinish has joined #opendev | 20:12 | |
| mnaser | AJaeger: i did, but i'm a little overwhelemd with a lot of stuff going on :< | 20:15 |
| openstackgerrit | Merged openstack/project-config master: Update Grafana dashboards for stable Neutron releases https://review.opendev.org/718676 | 20:16 |
| *** fdegir4 has quit IRC | 20:19 | |
| *** fdegir4 has joined #opendev | 20:19 | |
| *** fdegir4 has quit IRC | 20:20 | |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Write out db config for root user https://review.opendev.org/719192 | 20:21 |
| *** moppiner has joined #opendev | 20:21 | |
| mordred | clarkb: ^^ fixed | 20:21 |
| *** moppy has quit IRC | 20:21 | |
| mordred | clarkb: for the other - it shouldn't matter - but maybe we should update that to ansible first just in case? | 20:21 |
| mordred | clarkb: also - what's the deal with the other bup server? it seems like we have one for puppet and one for ansible | 20:22 |
| openstackgerrit | Merged opendev/gerritbot master: Build docker images https://review.opendev.org/715635 | 20:23 |
| clarkb | mordred: I think that is due to them being managed separately and weneed to transition from one to the other | 20:29 |
| clarkb | ianw ^ should know for sure | 20:29 |
| mordred | clarkb: I'm going to have to merge teh "run from project-config" and the "Rename to zuul" changes - because otherwise it actually gets really srewey | 20:29 |
| clarkb | it? | 20:29 |
| clarkb | oh you mean squash them together? | 20:29 |
| mordred | I can't make the run from project-config change work in test | 20:29 |
| mordred | yeah | 20:29 |
| ianw | clarkb / mordred : it's been a long while since i looked in on that. i'll have to context switch, but we should be migrated to the new server? | 20:30 |
| mordred | it's gonna make the change bigger - but I dont think it's worth making the middle one work | 20:30 |
| mordred | ianw: non-urgent, so don't know that you need to context switch this moment | 20:30 |
| ianw | mordred: where are you seeing references to the old server? | 20:32 |
| openstackgerrit | Merged opendev/system-config master: mirror-update: stats for vos release of mirrors https://review.opendev.org/718600 | 20:33 |
| openstackgerrit | Merged opendev/system-config master: Back up a single gitea backend https://review.opendev.org/719465 | 20:33 |
| mordred | ianw: in the puppet - one sec | 20:33 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Rename zuulcd to zuul https://review.opendev.org/720029 | 20:33 |
| mordred | ianw: bup::site { 'ord.rax': | 20:34 |
| mordred | ianw: is in a bunch of puppet modules in modules/openstack_project | 20:34 |
| mordred | ianw: while the ansible-driven bup stuff uses backup01.ca-ymq-1.vexxhost.opendev.org | 20:35 |
| ianw | mordred: oh yeah, i feel like that's right, the idea was to cut over as we removed puppet | 20:40 |
| mordred | ianw: kk. in that case I'll wait on eavesdrop | 20:41 |
| ianw | mordred: although now i notice etherpad say isn't in the backup group | 20:43 |
| mordred | it is | 20:44 |
| mordred | backup: | 20:45 |
| mordred | - etherpad[0-9]*.opendev.org | 20:45 |
| mordred | infra-root: would you mind +Aing this: https://review.opendev.org/#/c/718559/ | 20:46 |
| ianw | oh, sorry, was on the wrong branch | 20:46 |
| mordred | *phew* | 20:47 |
| ianw | mordred: on a related note some debug removals -> https://review.opendev.org/#/c/717874/ | 20:47 |
| ianw | that change is from the prehistoric era; april 7 | 20:48 |
| mordred | ianw: that's so long ago | 20:48 |
| clarkb | a whole week | 20:50 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Use project-config from zuul instead of direct clones https://review.opendev.org/719343 | 20:55 |
| clarkb | ianw: couple of things on https://review.opendev.org/#/c/717882/19 | 20:56 |
| ianw | clarkb: on the change ... yeah i feel like it's very unlikely; it's already documented wrong? | 20:58 |
| clarkb | ianw: oh I see its documented wrong but then we still change the var | 20:59 |
| clarkb | ya in that case I think its fine | 20:59 |
| ianw | clarkb: on the checking, that was somewhat deliberate, thinking it should be totally idempotent | 20:59 |
| ianw | i had being toying with this; in a way perhaps we should put it in the base role, and roles like this should say "assume a working pip"? | 21:00 |
| clarkb | ianw: ya, its mostly just concern with individuals like tristanC being very concered about job runtime | 21:00 |
| clarkb | ianw: and fetch-zuul-cloner, fetch-subunit, and ensure-tox are all likely to run in the same job | 21:01 |
| ianw | it feels like you can't have it both ways -- either you run a role that's supposed to ensure it and make sure it's idempotent, or leave it up to the role to assume things are there | 21:01 |
| ianw | but then take the chance the role is not usable "out of the bxo" | 21:01 |
| ianw | box even | 21:01 |
| clarkb | ianw: I think the difference here is we are hiding those details in the roles and not asking jobs to deal with it | 21:02 |
| clarkb | we make it idempotent anyways so it can be used that way, but since these set of roles are expected to be used together we can optimize a bit? | 21:02 |
| ianw | i just feel a bit icky making ansible skip like that, it feels counter to the way you're supposed to let the cfg mgmt tool figure out what to run | 21:04 |
| ianw | does it actually run it twice anyway, if no arguments have changed? | 21:04 |
| clarkb | ianw: yes | 21:04 |
| clarkb | puppet wouldn't because it generates its big graph of unique resources and collapses includes. But ansible tasks are run each time they are listed | 21:05 |
| clarkb | fwiw I'm happy to go with it as is and then we can optimize later if we find we need to | 21:06 |
| tristanC | well it seems more efficient to have smaller role that can be combined efficiently from a play instead of a multi-purpose roles that does everything | 21:06 |
| clarkb | I just half expect that to be the case | 21:06 |
| ianw | clarkb: it definitely doesn't run duplicates twice -- https://docs.ansible.com/ansible/latest/user_guide/playbooks_reuse_roles.html#role-duplication-and-execution | 21:07 |
| clarkb | tristanC: the problem with that appraoch is then users fail to get all the bits composed in their plays | 21:07 |
| ianw | but ... i don't know if across playbooks etc we are fooling it | 21:07 |
| *** roman_g has joined #opendev | 21:08 | |
| clarkb | ianw: in this case we aren't using the role: directive on a play but instead are using the include_role task | 21:08 |
| clarkb | and it is my understanding that since it is a task it runs every time | 21:08 |
| openstackgerrit | Merged opendev/system-config master: Don't log the public loop on master-nameserver https://review.opendev.org/718559 | 21:08 |
| openstackgerrit | Merged opendev/system-config master: Switch documentation to alabaster theme https://review.opendev.org/718790 | 21:08 |
| tristanC | clarkb: isn't ianw suggestion of adding assumption, like "assume a working pip", a solution to incorect play? | 21:10 |
| clarkb | tristanC: no that doesn't fix the play. It forces you to know to add that to your plays | 21:11 |
| clarkb | I think it is an option. I don't think it is a good option for most zuul users | 21:11 |
| ianw | tristanC: in this case, it's using a variable exported by "ensure-pip" ... so it's a harder dependency of "ensure-pip" was run | 21:12 |
| ianw | so, the question is sort of -- do you want people to have to incrementally debug why the role doesn't work as they use it (it's wrong if you just try and call it, without reading the documentation in detail) | 21:13 |
| ianw | or, have it work out of the box | 21:13 |
| clarkb | ianw: ya and not only that what the end user wants is `tox` to run and work because that is the tool they are interfacing with | 21:13 |
| clarkb | in depth debugging of ansible isn't necessarily a part of their skillset | 21:13 |
| ianw | i keep coming back to http://sweng.the-davies.net/Home/rustys-api-design-manifesto which i mentioned to mordred the other day | 21:13 |
| ianw | it's about a -7 v a +7 :) | 21:13 |
| clarkb | basically I think what you've done is correct. I'm just wodnering if we should optimize it within the context of those related and controled roles | 21:14 |
| ianw | maybe a 7 to 3 actually -- "read the documentation and you'll get it right." v "The obvious use is (probably) the correct one" :) | 21:15 |
| clarkb | I've +2'd the stack I think we can optimize later as we learn more | 21:15 |
| ianw | clarkb: how about i do a follow-on to gate on the variable and see how it goes? | 21:15 |
| clarkb | ianw: sure | 21:16 |
| *** tobiash has quit IRC | 21:18 | |
| ianw | tristanC: i'd sure like it if you could look over https://review.opendev.org/#/q/topic:ensure-pip anyway ... | 21:18 |
| tristanC | clarkb: ianw: for what it worth, when using kubectl connection, the runtime likely already have the build requirements like tox/pip/python baked in, and in that situation i find it wasteful to have the job spend time (and fill the log with useless task) ensure things | 21:20 |
| *** tobiash has joined #opendev | 21:20 | |
| clarkb | tristanC: yes that is why I brought this up as I knew you'd not like :) | 21:21 |
| tristanC | but since it's already the case with ensure-tox, then i think the only way to be really efficient is to provide custom play that goes straight to running the actual job like tox | 21:21 |
| clarkb | ultimately I think zuul-jobs role (no pun intended) is to provide generic tools that work well together and address common problems | 21:21 |
| clarkb | I think that means we can't make you happy in all cases because your needs are so environment specific | 21:21 |
| ianw | i'm totally up for incrementally improving it ... i think this more conservative approach is probably good for the initial switch off of pip-and-virtualenv | 21:21 |
| tristanC | yes, thus i don't mind having "heavy" ensure-role in the current zuul-jobs that works for most case, even when the user somehow got the play wrong | 21:22 |
| clarkb | tristanC: in the case of docker image backed tasks I would expect that all tools are already present and the job shoudl simply just execute commands | 21:22 |
| clarkb | tristanC: and so zuul-jobs isn't very valuable for the runtime of the job there (but would be for setting up the use of those images) | 21:22 |
| tristanC | clarkb: well we still want to use the `tox` role as it takes care of sibbling installation, but it seems like we have to create new tox job without the pre-run that ensure things | 21:23 |
| tristanC | as long as the `tox` role doesn't include the `ensure-tox` role, we are in the clear :-) | 21:26 |
| ianw | this stack has not added ensure-tox to the tox role :) | 21:27 |
| ianw | it has added ensure-pip to the ensure-tox role, though | 21:28 |
| ianw | maybe that's a compromise, that ensure-roles include other ensure roles, but the "runtime" roles can assume the setup has been done? | 21:28 |
| corvus | tristanC: the ensure-tox should just be a no-op if tox is already installed -- but you're saying that even the no-op wastes too much time? | 21:28 |
| corvus | ianw: yes, i think that is or should be the understanding -- these roles are split so that they can be put in different playbooks. ensure- roles should generally be in pre-run. runtime roles should be in run. | 21:29 |
| corvus | ianw: that happens to make it so that tristanC can avoid running the ensure role too if he wants to define a new tox job. defining new jobs with different combinations of roles is also an explicit goal -- that's why we put all the work in roles and not playbooks. | 21:30 |
| clarkb | corvus: I think you just said what I tried to say better. Particularly the bit about images starting with what they need (so the ensure-* goes aay) | 21:32 |
| ianw | i think we're in agreement. we're not crossing runtime roles with setup roles in this stack | 21:34 |
| corvus | i think so too | 21:34 |
| ianw | however, if you would like to review the stack with that explicitly in mind, that would be welcome :) | 21:34 |
| tristanC | corvus: i haven't looked in detail, but the existing tox job, when doing a simple flake8 linter may spend more time in pre/post than doing the actual tox command | 21:35 |
| clarkb | tristanC: which we identified was an odd ansible behavior that needed debugging | 21:36 |
| *** roman_g has quit IRC | 21:53 | |
| ianw | znc listservers | 22:32 |
| mordred | clarkb, ianw, corvus : https://review.opendev.org/#/c/719343/ is green! | 22:38 |
| mordred | I had to squash it with the zuulcd -> zuul change, because getting just half of it to work was more work than it was worth | 22:39 |
| clarkb | k Im heading out on a bike ride then will review | 22:41 |
| ianw | i'm still getting my head around 719186 :) | 22:44 |
| openstackgerrit | James E. Blair proposed opendev/system-config master: Meetpad: proxy through meetpad to etherpad.opendev.org https://review.opendev.org/720095 | 22:45 |
| corvus | do we think that's the best way to use the new setup? (still proxying through the meetpad server?) or should we have it go directly to the new server like we did earlier? | 22:45 |
| corvus | mordred: what happens if we "modify" the zuul user in test? | 22:46 |
| corvus | mordred: other than that curiosity, that looks nice :) | 22:47 |
| openstackgerrit | Goutham Pacha Ravi proposed openstack/project-config master: Add devstack-plugin-ceph notifications to manila channel https://review.opendev.org/720097 | 22:56 |
| mordred | corvus: it fails with a "user in use" error - lemme see if I can find one | 23:17 |
| corvus | mordred: what's it trying to do? i'm just wondering why it doesn't noop | 23:17 |
| mordred | corvus: https://84cacd386095a03eb330-320386062a8fef96051148fe5e7af6b1.ssl.cf2.rackcdn.com/720029/1/check/system-config-run-base/c6018fb/bridge.openstack.org/ara-report/result/0260d718-f9cc-48b0-8371-75d69c6304ca/ | 23:18 |
| mordred | corvus: it's trying to change the uid | 23:19 |
| mordred | corvus: because we set uid and gid in users in system-config | 23:19 |
| corvus | ah that'll do it | 23:19 |
| mordred | yeah | 23:19 |
| openstackgerrit | Goutham Pacha Ravi proposed openstack/project-config master: Add devstack-plugin-ceph notifications to manila channel https://review.opendev.org/720097 | 23:20 |
| *** tosky has quit IRC | 23:20 | |
| *** mlavalle has quit IRC | 23:28 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!