| clarkb | ianw: both updated changes lgtm. It might be a good idea to try and get corvus and/or mordred to look over https://review.opendev.org/c/opendev/system-config/+/820320/ and double check our brainstorm didn't miss anything important. They wrote a bunch of that original code there and also corvus can probably chime in re doing stuff in a zuul context | 00:09 |
|---|---|---|
| *** rlandy|ruck is now known as rlandy|out | 00:09 | |
| clarkb | and ya computers. Its amazing we get them to do anything useful at all :) | 00:10 |
| ianw | clarkb: minor one but i assume you're ok with the updated tag for system-config @ https://review.opendev.org/c/openstack/project-config/+/819715/2/gerrit/projects.yaml ? | 00:13 |
| clarkb | yup approved | 00:15 |
| opendevreview | Jeremy Stanley proposed opendev/system-config master: Block outbound SMTP connections from test jobs https://review.opendev.org/c/opendev/system-config/+/820900 | 00:24 |
| clarkb | swift is updating jobs to use centos stream instead of centos 8 | 00:24 |
| clarkb | Its seems we may not have stream set up for arm64. That might be something we need toc onsider? | 00:24 |
| corvus | clarkbianw lgtm, but i have covid booster brain | 00:24 |
| clarkb | corvus: ha, ok | 00:24 |
| fungi | apparently my egress filtering change was working, i just needed to know what the rule normalized to so i could match it in the test | 00:25 |
| clarkb | There is a board meeting tonight so I'll probably checkout here shortly then try and be awake in a few hours when that starts | 00:25 |
| opendevreview | Merged openstack/project-config master: Update the opendev/system-config tag https://review.opendev.org/c/openstack/project-config/+/819715 | 00:26 |
| clarkb | I think I preferred everyone being jetlagged in a room over staying up all night or getting up early :) | 00:26 |
| fungi | corvus: yeah, i got boosted yesterday, and the main thing which seems not to be boosted in the process is my energy level. here's hoping tomorrow is better | 00:26 |
| *** timburke__ is now known as timburke | 00:33 | |
| opendevreview | Merged openstack/project-config master: Fix Neutron periodic dashboard https://review.opendev.org/c/openstack/project-config/+/820912 | 00:34 |
| ianw | hrm, i thought we were testing 8-stream arm64, let me see | 00:37 |
| ianw | huh, maybe not | 00:38 |
| opendevreview | Merged openstack/project-config master: Add rights to neutron-dynamic-routing-stable-maint https://review.opendev.org/c/openstack/project-config/+/820351 | 00:41 |
| opendevreview | Ian Wienand proposed openstack/diskimage-builder master: Test 8-stream aarch64 build https://review.opendev.org/c/openstack/diskimage-builder/+/820970 | 00:41 |
| opendevreview | Ian Wienand proposed openstack/diskimage-builder master: Test 8-stream aarch64 build https://review.opendev.org/c/openstack/diskimage-builder/+/820970 | 02:30 |
| opendevreview | Jeremy Stanley proposed opendev/system-config master: Block outbound SMTP connections from test jobs https://review.opendev.org/c/opendev/system-config/+/820900 | 02:43 |
| opendevreview | Jeremy Stanley proposed opendev/system-config master: Copy Exim logs in system-config-run jobs https://review.opendev.org/c/opendev/system-config/+/820899 | 03:19 |
| opendevreview | Jeremy Stanley proposed opendev/system-config master: DNM: Confirm outbound E-mail is blocked https://review.opendev.org/c/opendev/system-config/+/820392 | 03:27 |
| opendevreview | Merged opendev/system-config master: Add zuul-client config to schedulers https://review.opendev.org/c/opendev/system-config/+/820951 | 03:38 |
| opendevreview | Ian Wienand proposed openstack/diskimage-builder master: Use OpenDev mirrors for 8-stream CI builds https://review.opendev.org/c/openstack/diskimage-builder/+/820978 | 04:17 |
| clarkb | ianw: thats running an x86-64 build not arm64 right? | 04:31 |
| clarkb | oh wait I see it runs on a debian arm64 node | 04:32 |
| clarkb | I guess the arch is by default inherited from the host then | 04:32 |
| clarkb | that makes sense since dib doesn't really cross build | 04:32 |
| opendevreview | Jeremy Stanley proposed opendev/system-config master: DNM: Confirm outbound E-mail is blocked https://review.opendev.org/c/opendev/system-config/+/820392 | 04:40 |
| *** raukadah is now known as chandankumar | 04:43 | |
| *** ysandeep|out is now known as ysandeep | 04:50 | |
| ianw | clarkb: yep; i think it will "just work". going to fiddle with the mirror setups though as that seems to have slipped through | 04:56 |
| clarkb | ianw: ya I guess we have the label but no nodeset defined for it and that is what swift tripped over | 04:57 |
| clarkb | however I don't see stream-9 on arm so that may actually be missing | 04:57 |
| clarkb | one thing at a time | 04:57 |
| ianw | yeah 9-stream is also missing, i had that on my todo list | 05:01 |
| fungi | so unfortunately, while redirecting stdin from /dev/null does bypass newlist's wait for the confirmation to send a list admin notification, it doesn't appear to actually cause it to send the notification: https://zuul.opendev.org/t/openstack/build/4690bb1000244222baff80e20edd987c/log/lists.openstack.org/exim4/mainlog | 05:10 |
| fungi | so i'm back to square 1 | 05:10 |
| clarkb | are we sure that exim and mailman are properly configured to talk to each other? | 05:12 |
| clarkb | I wonder if we need to send it a 'yes\n' or similar | 05:12 |
| fungi | they seem to do so in production | 05:12 |
| fungi | i wonder if piping /bin/yes into it would suffice | 05:13 |
| clarkb | ya I'm just wondering outloud if the special exim for mailman isn't configured on the test node beacuse we're not matching the case for some reason | 05:13 |
| fungi | well, that's only for inbound delivery anyway though, right? | 05:14 |
| fungi | outbound delivery should be working for all our servers | 05:14 |
| clarkb | I'm not sure. I guess mailman sends email like any other server? so ya maybe that is true | 05:15 |
| fungi | tomorrow i'll set an autohold on a broken revision of 820392 and fiddle with the test node to confirm some theories | 05:16 |
| *** poojajadhav is now known as pojadhav|rover | 05:38 | |
| opendevreview | Ian Wienand proposed openstack/diskimage-builder master: Use OpenDev mirrors for 8-stream CI builds https://review.opendev.org/c/openstack/diskimage-builder/+/820978 | 05:55 |
| clarkb | I guess gerrit 3.5.0 has released today | 06:06 |
| clarkb | yesterday? what day is it anyway :) | 06:06 |
| *** pojadhav is now known as pojadhav|rover | 06:35 | |
| opendevreview | yatin proposed openstack/project-config master: Fix Neutron periodic dashboard https://review.opendev.org/c/openstack/project-config/+/820980 | 06:37 |
| *** marios is now known as marios|ruck | 06:43 | |
| *** bhagyashris_ is now known as bhagyashris | 06:57 | |
| opendevreview | Michal Nasiadka proposed opendev/irc-meetings master: kolla: Update agenda url https://review.opendev.org/c/opendev/irc-meetings/+/820981 | 07:14 |
| *** bhagyashris_ is now known as bhagyashris | 07:19 | |
| *** ysandeep is now known as ysandeep|lunch | 07:23 | |
| opendevreview | Merged openstack/project-config master: Fix Neutron periodic dashboard https://review.opendev.org/c/openstack/project-config/+/820980 | 08:29 |
| *** ysandeep|lunch is now known as ysandeep | 08:35 | |
| opendevreview | Ian Wienand proposed openstack/diskimage-builder master: Use OpenDev mirrors for 8-stream CI builds https://review.opendev.org/c/openstack/diskimage-builder/+/820978 | 08:48 |
| *** pojadhav|rover is now known as pojadhav|lunch | 09:11 | |
| *** ykarel_ is now known as ykarel | 09:21 | |
| *** pojadhav|lunch is now known as pojadhav|rover | 09:30 | |
| *** ysandeep is now known as ysandeep|afk | 10:11 | |
| *** pojadhav|rover is now known as pojadhav|rover|afk | 10:37 | |
| *** pojadhav|rover|afk is now known as pojadhav|rover | 11:05 | |
| *** rlandy|out is now known as rlandy|ruck | 11:05 | |
| *** ysandeep|afk is now known as ysandeep | 11:16 | |
| opendevreview | Merged opendev/irc-meetings master: kolla: Update agenda url https://review.opendev.org/c/opendev/irc-meetings/+/820981 | 11:24 |
| *** pojadhav|rover is now known as pojadhav|rover|brb | 12:17 | |
| *** pojadhav|rover|brb is now known as pojadhav|rover | 12:40 | |
| *** ysandeep is now known as ysandeep|brb | 12:49 | |
| *** outbrito_ is now known as outbrito | 13:02 | |
| *** ysandeep|brb is now known as ysandeep | 13:07 | |
| *** ysandeep is now known as ysandeep|dinner | 13:49 | |
| *** ykarel is now known as ykarel|away | 14:07 | |
| *** pojadhav|rover is now known as pojadhav|rover|afk | 14:46 | |
| opendevreview | Jeremy Stanley proposed opendev/system-config master: DNM: Confirm outbound E-mail is blocked https://review.opendev.org/c/opendev/system-config/+/820392 | 14:52 |
| *** artom__ is now known as artom | 15:08 | |
| *** ysandeep|dinner is now known as ysandeep | 15:45 | |
| fungi | omg, how did i not notice before? i guess newlist --help is out of sync with the manpage these days | 16:08 |
| fungi | there's a -a/--automate option to newlist now, which is supposed to automatically send the list admin notification without prompting | 16:09 |
| opendevreview | Jeremy Stanley proposed opendev/system-config master: DNM: Confirm outbound E-mail is blocked https://review.opendev.org/c/opendev/system-config/+/820392 | 16:09 |
| opendevreview | Jeremy Stanley proposed opendev/system-config master: Correct Python interpreter in mailman initscript https://review.opendev.org/c/opendev/system-config/+/821095 | 16:09 |
| amorin | hey team | 16:11 |
| amorin | I proposed this few days ago: | 16:11 |
| amorin | https://review.opendev.org/c/openstack/project-config/+/820369 | 16:11 |
| amorin | not sure I am on the right chan | 16:12 |
| amorin | cc fungi clarkb | 16:12 |
| fungi | amorin: oh, thanks for the heads up! i didn't notice that on friday, but yes this is the appropriate channel | 16:13 |
| clarkb | amorin: thank you for the heads up | 16:15 |
| amorin | I have no idea who I should ping for this, you were in my memory :) | 16:16 |
| fungi | we're both fine choices, thanks so much | 16:17 |
| clarkb | fungi: yay manpages being out of date | 16:17 |
| amorin | so, we will upgrade one of our region (BHS1) to a new openstack release | 16:18 |
| clarkb | fungi: we could just -q or -a depending on the flag if we wanted. Still not great for testing but probably close enough that its fine? | 16:18 |
| amorin | in our procedure, we close the API for few hours (like 2/3 hours) | 16:18 |
| amorin | so I think it better to avoid spawning instances, right? | 16:18 |
| clarkb | amorin: yes, we'll gracefully fallback but if we know it is happening in advance that sort of change is a good way to avoid problems | 16:18 |
| amorin | we are going to start this at 9 UTC, which is maybe too early for you? Should we merge this the day before? | 16:19 |
| clarkb | amorin: yes the day before is probably a good idea | 16:20 |
| clarkb | though frickler is in a european timezone and may be able to land it day of. But I'm fine with day before | 16:20 |
| amorin | ack, so, I will come back on that channel the day before to ping of of you | 16:20 |
| amorin | one* of you | 16:21 |
| clarkb | sounds good | 16:22 |
| fungi | clarkb: well, the bad news is that i still can't seem to get newlist to actually attempt to send the list admin notifications through the mta (according to exim's logs), even the normal way with no newlist cli options at all. i may need to dig into its source code | 16:22 |
| clarkb | fungi: is it possible there is some other log we need to look at where it is recording things? | 16:23 |
| fungi | i did at least confirm that if i try to send mail locally (e.g. with the `mail` utility) then exim logs the rejected outbound smtp connections (both ipv4 and v6) courtesy of the new firewall rules in 820900 | 16:24 |
| fungi | mailman itself is only creating one log that i could find | 16:24 |
| fungi | and doesn't mention anything about the newlist or notification sending | 16:24 |
| fungi | though it might also be related to 821095, i've got another autohold set with that added to the stack | 16:25 |
| clarkb | fungi: uh isn't mailman python2 only? | 16:26 |
| clarkb | wouldn't it be better to install python2? | 16:26 |
| clarkb | fungi: https://packages.ubuntu.com/focal/mailman it hard depends on python2 | 16:27 |
| clarkb | fungi: looks like that may only create /usr/bin/python2.7 on focal | 16:29 |
| clarkb | I think that should be what we test for not python3 | 16:29 |
| fungi | oh, weird, i wonder why it wasn't installed? | 16:32 |
| clarkb | fungi: I think it is installed but it doesn't install /usr/bin/python anymore | 16:32 |
| clarkb | only /usr/bin/python2.7 | 16:32 |
| fungi | and indeed, mailman was dropped from debian after buster (so not included in bullseye, which has only mailman3) | 16:34 |
| fungi | looks like on lists.o.o /usr/bin/python is a symlink to /usr/bin/python2 which is a symlink to /usr/bin/python2.7 | 16:36 |
| fungi | the /usr/bin/python symlink on it is being provided by the python-is-python2 package | 16:38 |
| fungi | we can just add that to the dependencies | 16:38 |
| *** pojadhav|rover|afk is now known as pojadhav|rover | 16:44 | |
| opendevreview | Jeremy Stanley proposed opendev/system-config master: Make sure /usr/bin/python is present for mailman https://review.opendev.org/c/opendev/system-config/+/821095 | 16:44 |
| opendevreview | Jeremy Stanley proposed opendev/system-config master: DNM: Confirm outbound E-mail is blocked https://review.opendev.org/c/opendev/system-config/+/820392 | 16:44 |
| *** marios|ruck is now known as marios|out | 16:56 | |
| opendevreview | Clark Boylan proposed zuul/zuul-jobs master: Try to fix broken stestr command discovery https://review.opendev.org/c/zuul/zuul-jobs/+/821101 | 17:04 |
| *** rlandy|ruck is now known as rlandy|ruck|mtg | 17:09 | |
| *** ysandeep is now known as ysandeep|out | 17:10 | |
| fungi | okay, so some progress... for whatever reason the server ends up with the mailman-openinfra service in a dead state with no running processes, but if i stop and start it then it does start the expected 9 processes, maybe some sort of startup race? | 17:12 |
| fungi | unfortunately, running newlist to create new lists still does not attempt to send any actual notification to the provided list admin address even going through the normal confirmation prompt manually | 17:13 |
| fungi | held node is 172.99.67.72 | 17:13 |
| fungi | /var/log/exim4/mainlog does not indicate the mta received any messages to deliver | 17:15 |
| fungi | oh! mailman is sending directly, not through exim | 17:15 |
| fungi | now that services are running, it logs the rejections in /srv/mailman/openinfra/logs/smtp-failure | 17:16 |
| clarkb | cool so things are working as expected now? | 17:17 |
| clarkb | which changes do we want to keep ? | 17:17 |
| fungi | aha! i think it's because mailman is sending through 127.0.0.1:25 and that's being rejected by iptables | 17:17 |
| fungi | should we also collect all of /srv/mailman in the system-config-run-lists job? or somehow grab just /srv/mailman/*/logs/* instead? | 17:19 |
| fungi | i'm amending the firewalling change to allow localhost smtp but reject remote smtp | 17:20 |
| clarkb | probably better to not grab all the mailman site contets if we can avoid it | 17:21 |
| clarkb | there is a lot of not helpful data in there iirc (like all the templates) | 17:21 |
| jrosser | when I do a depends-on, does the current branch in the modified repo include that patch, or is the modified repo left in a 'detached' state from the original branch? (i.e if I locally cloned the on-disk repo using the branch name would I get the additional change) | 17:22 |
| clarkb | jrosser: the repos branches are all updated to the appropriate commit. They won't be detached. But it doesn't cherry pick across branches either | 17:25 |
| fungi | jrosser: it's included as long as you use the branch states or checked-out state of the on-disk copy of the repositories | 17:25 |
| clarkb | that means if you have depends-on repo foo branch A change then repo foo's branch A will be checked out to that proposed state | 17:26 |
| clarkb | s/check out/set/ | 17:26 |
| clarkb | you might have to check it out yourself | 17:26 |
| jrosser | I am looking at how to make ansible-galaxy use collections from the zuul repos | 17:26 |
| jrosser | and it by default clones them from the place you tell it they are | 17:27 |
| fungi | but check it out as 'foo' not 'origin/foo' (the latter represents the state prior to the change being applied) | 17:27 |
| jrosser | so I have to give some refspec to ansible-galaxy which doesn't just throw away any depends-on | 17:27 |
| fungi | what branch does ansible-galaxy try to checkout by default? | 17:28 |
| fungi | or does it try to use a tag? | 17:28 |
| clarkb | you would checkout the branch for your depends on | 17:29 |
| jrosser | hah well that's where it gets fun, it expects to be given a value of some sort, which can be a tag or a branch | 17:29 |
| jrosser | or even a SHA I expect | 17:29 |
| clarkb | and zuul will automatically update that branch with the depends on content | 17:29 |
| fungi | okay, so any refname git supports, in essence | 17:29 |
| jrosser | ok cool, so if I can give it 'master' and that will bring the depends-on with it, it's all good | 17:29 |
| fungi | so yes, specifying the local branch name (not a remote branch) should do what you want | 17:29 |
| clarkb | The idea is that zuul is setting the repo state so you don'y have to figure that out (historically many many yaers ago, this was a very common source of bugs in gating, jobs weren't testing what they thought they were testing and then things got broken). Then you just interact with the repo on disk and checkout the branches you need | 17:34 |
| clarkb | as fungi points out the origin/ refs are set to the non modified commits and can be used if you need to compare deltas for linting or similar | 17:34 |
| *** pojadhav|rover is now known as pojadhav|out | 17:35 | |
| fungi | okay, so zuul_copy_output doesn't seem to support any sort of wildcarding or regular expressions, making it hard to get /srv/mailman/*/logs from the test node when that * is determined by the mailman_sites list in host_vars | 17:46 |
| fungi | clarkb: any suggestion there? | 17:46 |
| clarkb | fungi: I would use a post-run playbook to copy the files into a better directory that can be copied wholesale | 17:47 |
| fungi | ahh, yeah can do that, just a lot more complexity | 17:47 |
| clarkb | then you can use bash/find/rsync whatever to do the richer thing | 17:47 |
| fungi | i'll see if there's a suitable playbook already i can just add it to | 17:47 |
| clarkb | for X in `ls /srv/mailman` ; do cp -R /srv/mailman/$X/logs /other/location ; done then tell zuul_copy_output to copy /other/location | 17:48 |
| fungi | i guess i can do it in a post-run: playbooks/zuul/run-lists-post.yaml | 17:51 |
| clarkb | or append it to the end of the run playbook but post run seems better structurally | 17:53 |
| clarkb | 821101 is still waiting for a tumbleweed image. We deleted those right? I'm going to go ahead and propose the removal of tumbleweed testing from zuul-jobs | 17:54 |
| fungi | i should be able to loop over mailman_sites if i'm doing it in a playbook | 17:56 |
| opendevreview | Clark Boylan proposed zuul/zuul-jobs master: Remove tumbleweed jobs https://review.opendev.org/c/zuul/zuul-jobs/+/821111 | 18:01 |
| *** rlandy|ruck|mtg is now known as rlandy|ruck | 18:05 | |
| opendevreview | Jeremy Stanley proposed opendev/system-config master: Block outbound SMTP connections from test jobs https://review.opendev.org/c/opendev/system-config/+/820900 | 18:11 |
| opendevreview | Jeremy Stanley proposed opendev/system-config master: Copy Exim logs in system-config-run jobs https://review.opendev.org/c/opendev/system-config/+/820899 | 18:11 |
| opendevreview | Jeremy Stanley proposed opendev/system-config master: Make sure /usr/bin/python is present for mailman https://review.opendev.org/c/opendev/system-config/+/821095 | 18:11 |
| opendevreview | Jeremy Stanley proposed opendev/system-config master: DNM: Confirm outbound E-mail is blocked https://review.opendev.org/c/opendev/system-config/+/820392 | 18:11 |
| opendevreview | Jeremy Stanley proposed opendev/system-config master: Collect mailman logs in deployment testing https://review.opendev.org/c/opendev/system-config/+/821112 | 18:11 |
| corvus | infra-root: would you mind looking at https://review.opendev.org/820954 and https://review.opendev.org/820956 -- then we can start playing with keycloak + zuul | 18:29 |
| clarkb | corvus: can anyone create a keycloak user and say thare in the infra-root group or the openstack group to get zuul admin ? | 18:33 |
| corvus | clarkb: no, groups are managed by admins | 18:34 |
| corvus | (also, i have disabled new user creation for the zuul realm, so only admins can create users) | 18:34 |
| clarkb | got it | 18:34 |
| clarkb | last question should you restrict those admin rules to the keycloak.opendev.org issuer? | 18:35 |
| corvus | i don't even see groups in the self-serve interface | 18:35 |
| clarkb | I guess right now its equivalent since we only haev one issuer handing out groups | 18:36 |
| clarkb | and that is the keycloak issuer | 18:36 |
| corvus | clarkb: that's probably a safer thing to do (more explicit) but not strictly necessary because ^ | 18:36 |
| clarkb | ok +2'd both but didn't approve project-config change as I think we want the system-config change in first to define the issuer | 18:37 |
| fungi | in https://review.opendev.org/821112 i reference the mailman_sites var which is set in the inventory, but jinja2 seems to think that "'mailman_sites' is undefined" | 18:38 |
| fungi | do i need to do something special to reference inventory vars in a playbook? | 18:39 |
| corvus | fungi: it's defined in system-config-ansible's inventory, but not zuul-ansible's inventory | 18:39 |
| fungi | oh, wait, this is crossing the boundary between the nested ansible and zuul | 18:39 |
| corvus | yep that | 18:39 |
| fungi | yeah, that just dawned on me | 18:39 |
| * fungi sighs... rethinking | 18:39 | |
| fungi | i may have to do clarkb's original suggestion of just doing it in shell script | 18:40 |
| clarkb | Ansible can list directories like that too, but I always find dealing with ansible loops to be so confusing | 18:43 |
| fungi | oh, i'll see if i can't figure that out | 18:45 |
| corvus | i would totally do that in shell script; ansible isn't adding anything to that task | 18:47 |
| fungi | yeah, i guess i if i inline the script in the playbook i can still use jinja variable substitution to know the target dir | 18:49 |
| corvus | ++ best of both worlds | 18:49 |
| opendevreview | Clark Boylan proposed zuul/zuul-jobs master: Try to fix broken stestr command discovery https://review.opendev.org/c/zuul/zuul-jobs/+/821101 | 18:52 |
| opendevreview | Jeremy Stanley proposed opendev/system-config master: Collect mailman logs in deployment testing https://review.opendev.org/c/opendev/system-config/+/821112 | 18:56 |
| opendevreview | Jeremy Stanley proposed opendev/system-config master: Make sure /usr/bin/python is present for mailman https://review.opendev.org/c/opendev/system-config/+/821095 | 18:56 |
| opendevreview | Jeremy Stanley proposed opendev/system-config master: DNM: Confirm outbound E-mail is blocked https://review.opendev.org/c/opendev/system-config/+/820392 | 18:56 |
| fungi | should we be mirroring or retrying docker installation? https://80c9b8c03fffae2c70dd-4e6e3c7c56acb1be7502a51f02e38917.ssl.cf2.rackcdn.com/820900/7/check/system-config-run-zuul/2e806cb/bridge.openstack.org/ara-report/results/601.html | 19:10 |
| clarkb | fungi: https://mirror.bhs1.ovh.opendev.org/deb-docker/ we do mirror it | 19:10 |
| clarkb | but convincing our production playbooks to use the mirror might be more trouble than it is worth | 19:10 |
| fungi | aha, we just don't use it in our tests, i guess plumbing it to the nested ansible is nontrivial | 19:10 |
| clarkb | ya it would take some effort to detect running under test then swap in the different urls | 19:11 |
| fungi | retries may not be a terrible idea there though | 19:11 |
| clarkb | ya retries seem reasonable | 19:11 |
| clarkb | fungi: if you have time for https://review.opendev.org/c/zuul/zuul-jobs/+/821111/ that would be good. Then I'll keep trying to get tristanC's input on the child for why SF failed third party ci | 19:15 |
| fungi | oh, sure | 19:15 |
| clarkb | fungi: looks like tristanC corrected the SF ci if you have another minute to look at the child (821101) | 19:20 |
| clarkb | I'm happy to approve 821101 when I can watch it land in case it does anything unexpected though I think we have decent code coverage in the testing | 19:20 |
| clarkb | (I'm going to try and go on another bike ride today as the forecast says more rain and maybe even snow for the next week so this is my opportunity) | 19:20 |
| fungi | on https://review.opendev.org/821112 should i not be using zuul.executor.log_root on the job node? i see run-base-post.yaml uses it on bridge, but i guess i need to get it from the lists node to the bridge node for it to be collected automatically? | 19:22 |
| clarkb | I thought the autocollection collected from the nodes in the zuul homedir somewhere | 19:23 |
| clarkb | fungi: you shouldn't use the zuul executor log root path on the test node though | 19:24 |
| clarkb | the contexts are different | 19:24 |
| clarkb | fungi: fetch-output is the role that does the automagic | 19:25 |
| clarkb | it says it copies from {{ ansible_user_dir }}/zuul-output by default | 19:25 |
| opendevreview | Jeremy Stanley proposed opendev/system-config master: Collect mailman logs in deployment testing https://review.opendev.org/c/opendev/system-config/+/821112 | 19:25 |
| opendevreview | Jeremy Stanley proposed opendev/system-config master: Make sure /usr/bin/python is present for mailman https://review.opendev.org/c/opendev/system-config/+/821095 | 19:25 |
| opendevreview | Jeremy Stanley proposed opendev/system-config master: DNM: Confirm outbound E-mail is blocked https://review.opendev.org/c/opendev/system-config/+/820392 | 19:25 |
| fungi | i took a different approach for now and just hard-coded a path consistent with the other single-site mailman node, for simplicity | 19:27 |
| clarkb | ya I think taht will work | 19:28 |
| opendevreview | Merged zuul/zuul-jobs master: Remove tumbleweed jobs https://review.opendev.org/c/zuul/zuul-jobs/+/821111 | 19:29 |
| clarkb | fungi: re the docker fetch, we should be careful that didn't fail due to the firewall rule update | 19:29 |
| fungi | it shouldn't unless docker is running a webserver on 25/tcp | 19:30 |
| fungi | i rechecked it to find out whether it's consistent though | 19:30 |
| clarkb | ya it seems unlikely, but its an outbound tcp connection and we're blocking some outbound tcpconnections | 19:30 |
| fungi | as for 821101, i feel like we've already worked around that somewhere at least once, maybe tempest? i recall it was the cause of a significant percentage of failures at one point | 19:31 |
| clarkb | fungi: ya its come up before. There are tasks in there to emit the value for logsatsh tracking | 19:33 |
| clarkb | https://zuul.opendev.org/t/openstack/build/d8217da86e3746bd812f30d9c77914b1/console and https://zuul.opendev.org/t/openstack/build/1754d6b32850472e96fbe0af1414d633/console show this happening though and it is relatively infrequent | 19:34 |
| clarkb | I think simply trimming off the extra whitespace and running the comamnd should be a straighforward workaround | 19:34 |
| clarkb | it must be something with how type -p works but I've yet to reproduce it unfortunately | 19:35 |
| opendevreview | Merged zuul/zuul-jobs master: Try to fix broken stestr command discovery https://review.opendev.org/c/zuul/zuul-jobs/+/821101 | 19:43 |
| clarkb | I think 814783's tox jobs may have started with ^ in place | 19:48 |
| clarkb | ya looking at the logs for that it seems happy though hard to be sure we used the new code since the console doesn't show us the actual ansible run | 19:54 |
| clarkb | I don't see anything going super sideways after 821101. I'll get that bike ride in now. Back in a bit | 19:59 |
| opendevreview | Jeremy Stanley proposed opendev/system-config master: Collect mailman logs in deployment testing https://review.opendev.org/c/opendev/system-config/+/821112 | 20:02 |
| opendevreview | Jeremy Stanley proposed opendev/system-config master: Make sure /usr/bin/python is present for mailman https://review.opendev.org/c/opendev/system-config/+/821095 | 20:02 |
| opendevreview | Jeremy Stanley proposed opendev/system-config master: DNM: Confirm outbound E-mail is blocked https://review.opendev.org/c/opendev/system-config/+/820392 | 20:02 |
| *** tobias-urdin3 is now known as tobias-urdin | 20:10 | |
| fungi | i'm beginning to realize that all the log collection in the world isn't going to do much good when the mailman site initscripts aren't started | 20:42 |
| opendevreview | Jeremy Stanley proposed opendev/system-config master: DNM: Confirm outbound E-mail is blocked https://review.opendev.org/c/opendev/system-config/+/820392 | 20:45 |
| opendevreview | Merged opendev/system-config master: Rename install-ansible to bootstrap-bridge https://review.opendev.org/c/opendev/system-config/+/820282 | 20:49 |
| opendevreview | Jeremy Stanley proposed opendev/system-config master: DNM: Confirm outbound E-mail is blocked https://review.opendev.org/c/opendev/system-config/+/820392 | 21:26 |
| clarkb | on my bike ride I was owrried that we might have been taking the first testr command found beacuse the script can output multiple. But it stops looking after the first is found | 21:59 |
| clarkb | fungi: hrm for this exercise I don't know that we need the services running? or does newlist depend on them to be running to send email? | 22:01 |
| fungi | i suspect the latter, yes, which is what i'm verifying now | 22:02 |
| fungi | yep, that seems to have done it: https://zuul.opendev.org/t/openstack/build/5f3655828b3342b18ca9f9ad84f0d5a3/log/lists.openstack.org/mailman/openinfra/smtp-failure | 22:05 |
| fungi | also proves that my last edit to the firewall change isn't actually allowing mailman to connect to exim | 22:06 |
| clarkb | didn't you fix the localhost problem? | 22:07 |
| fungi | i thought the last revision had | 22:08 |
| fungi | syslog doesn't seem to be logging the rejections though | 22:10 |
| fungi | yeah, nothing logged for DPT=25 | 22:12 |
| fungi | i have a feeling we disable logging by applying our ruleset | 22:12 |
| fungi | all the iptables log entries are early, like before the job starts for the most part | 22:12 |
| opendevreview | Jeremy Stanley proposed opendev/system-config master: DNM: Confirm outbound E-mail is blocked https://review.opendev.org/c/opendev/system-config/+/820392 | 22:20 |
| fungi | that ^ should turn on logging for those rules | 22:20 |
| *** artom_ is now known as artom | 23:01 | |
| fungi | https://zuul.opendev.org/t/openstack/build/aa179b4022284cfeafe6a9c8bede6168/log/lists.openstack.org/syslog.txt#1970-1971 | 23:01 |
| fungi | that's it logging the smtp connections, they're definitely localhost | 23:01 |
| clarkb | and we want those to go through | 23:01 |
| clarkb | (just making sure I understand) | 23:01 |
| fungi | right | 23:01 |
| fungi | maybe iptables works on a last-match basis not first-match | 23:02 |
| fungi | everything i see says iptables is first-match though | 23:03 |
| fungi | so in theory the -i lo -j ACCEPT should be passing it through per https://zuul.opendev.org/t/openstack/build/aa179b4022284cfeafe6a9c8bede6168/log/lists.openstack.org/rules.v4.txt | 23:04 |
| fungi | i'm going to try dropping the first log statement and leaving the second in | 23:05 |
| fungi | to make sure we're hitting the reject rule | 23:05 |
| opendevreview | Jeremy Stanley proposed opendev/system-config master: DNM: Confirm outbound E-mail is blocked https://review.opendev.org/c/opendev/system-config/+/820392 | 23:05 |
| fungi | logging in openbsd's pf is one of the things i really appreciate, you can decorate any rule with the log keyword and it logs not only the packet details but also the action taken and the rule it matched, including its position in the ruleset | 23:07 |
| clarkb | I like that pf rules are interface oriented and not chain oriented | 23:08 |
| clarkb | I find that much easier to reason about | 23:08 |
| clarkb | ianw: apparently I had previously written a user conversion for lodgeit https://review.opendev.org/c/opendev/system-config/+/818606 | 23:16 |
| clarkb | I've got on my todo list to look at the other irc and matrix bots next though since they cohabitate | 23:16 |
| clarkb | https://review.opendev.org/c/opendev/gerritbot/+/818494/ and parent are related to the gerritbot user switch. Probably worth getting thsoe in at this point now as I think we've learned stuff | 23:17 |
| ianw | clarkb: happy to try that; it doesn't seem like it will have any issues with the db | 23:21 |
| clarkb | ianw: ya since it all seems to be db focused it should be fine. But let me double check it listens on a high port | 23:22 |
| clarkb | ya port 9000 | 23:22 |
| clarkb | I'll try to approve that tomorrow morning when I can watch it more properly and revert if necessary | 23:22 |
| *** rlandy|ruck is now known as rlandy|ruck|bbl | 23:35 | |
| opendevreview | Ian Wienand proposed openstack/diskimage-builder master: Use OpenDev mirrors for 8-stream CI builds https://review.opendev.org/c/openstack/diskimage-builder/+/820978 | 23:40 |
| fungi | okay, i think https://zuul.opendev.org/t/openstack/build/3051b8a7d0eb421680fb867441e24fd1/log/lists.openstack.org/syslog.txt#1957-1958 confirms we're matching the reject rule for 25/tcp and not the accept rule for the lo interface | 23:49 |
| fungi | the relevant rules: https://zuul.opendev.org/t/openstack/build/3051b8a7d0eb421680fb867441e24fd1/log/lists.openstack.org/rules.v4.txt#23-25 | 23:50 |
| *** ysandeep|out is now known as ysandeep | 23:51 | |
| fungi | what nuance of iptables am i missing? | 23:51 |
| fungi | the logged entry includes OUT=lo | 23:52 |
| fungi | so why does that packet not match -i lo? | 23:53 |
| fungi | ohhh, man iptables says i need -o there | 23:53 |
| fungi | -i is short for --in-interface, there's a separate -o/--out-interface | 23:54 |
| opendevreview | Jeremy Stanley proposed opendev/system-config master: DNM: Confirm outbound E-mail is blocked https://review.opendev.org/c/opendev/system-config/+/820392 | 23:54 |
| fungi | i wrongly assumed -i was --interface (direction agnostic) | 23:55 |
| ianw | so it matches incoming lo0 packets, but not outgoing? | 23:55 |
| ianw | is the input already covered by "-A openstack-INPUT -i lo -j ACCEPT" ? | 23:56 |
| clarkb | btu then on the output chain we blocked it | 23:56 |
| clarkb | thsi is why I don't like iptables chains. Too many places to catch things and be confused. | 23:57 |
| fungi | the new egress rules are on an output chain, because we want to explicitly prevent the server from sending to specific remote addresses/ports with it | 23:57 |
| fungi | so yes, i had it allowing in on the lo interface but not allowing out from the lo interface | 23:58 |
| fungi | and one does not imply the other | 23:58 |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!