| *** ryohayakawa has joined #opendev | 00:02 | |
| *** ysandeep|away is now known as ysandeep|rover | 00:11 | |
| openstackgerrit | Merged openstack/diskimage-builder master: update gentoo-releng gpg key https://review.opendev.org/740604 | 00:39 |
|---|---|---|
| ianw | something is up, here's teh same ssh key error again https://zuul.opendev.org/t/openstack/build/2ba031ca6b4a41f193c98269be220499 | 01:19 |
| ianw | https://zuul.opendev.org/t/openstack/build/b77c70e6be764f7887bce430a4eb76fa/log/job-output.txt here it is with two bionic hosts; so it's not xenial/bionic/etc related | 01:21 |
| ianw | five of the jobs failed with it @ https://review.opendev.org/#/c/740609/ | 01:22 |
| fungi | the review01 backup finally completed, and ~root/.bup on it is now only 3.8gb after getting recreated | 01:22 |
| *** rh-jelabarre has quit IRC | 01:23 | |
| fungi | ianw: what generates the keys? | 01:23 |
| ianw | fungi: that is a good question ... we're behind layers of zuul jobs and moving keys around so i'm not 100% sure | 01:24 |
| ianw | one thing people seem to report is if the .pub file is missing you can get this message as a red-herring error | 01:25 |
| ianw | bridge.openstack.org | Data could not be sent to remote host "23.253.159.123". Make sure this host can be reached over ssh: Load key "/root/.ssh/id_rsa": invalid format | 01:25 |
| ianw | it's the nested ansible run on bridge | 01:25 |
| ianw | must be roles/root-keys | 01:29 |
| ianw | root_rsa_key: "{{ lookup('file', zuul.executor.work_root + '/' + zuul.build + '_id_rsa') }}" | 01:31 |
| ianw | ... we put ze01 into operation ... do these all have an executor in common ... | 01:31 |
| ianw | https://zuul.opendev.org/t/openstack/build/32240c4ec737448b8768aee92aed9b8d/log/job-output.txt | 01:33 |
| ianw | https://zuul.opendev.org/t/openstack/build/43e54b33e8ca4c09946cf10fe5d52cfe/log/job-output.txt | 01:33 |
| ianw | https://zuul.opendev.org/t/openstack/build/2ba031ca6b4a41f193c98269be220499/log/job-output.txt | 01:33 |
| ianw | all ze01 ... i think we have a smoking gun ... | 01:33 |
| ianw | i guess the lookup must pass ... but hrm ... that is maybe generated the container which is now some later debian? | 01:35 |
| ianw | # openssl rsa -noout -text < 252efb6f46874363a3da17cafa4eca53_id_rsa | 01:37 |
| ianw | unable to load Private Key | 01:37 |
| ianw | 139691867600536:error:0906D06C:PEM routines:PEM_read_bio:no start line:pem_lib.c:701:Expecting: ANY PRIVATE KEY | 01:37 |
| ianw | ok ... that doesn't work *in* the container either | 01:39 |
| ianw | -----BEGIN OPENSSH PRIVATE KEY----- | 01:41 |
| ianw | the other executors have | 01:42 |
| ianw | -----BEGIN RSA PRIVATE KEY----- | 01:42 |
| *** ysandeep|rover is now known as ysandeep|afk | 01:42 | |
| ianw | so ... now i learn what an openssh private key is | 01:42 |
| ianw | looks like we want a "-m PEM" | 01:44 |
| openstackgerrit | Ian Wienand proposed zuul/zuul-jobs master: add-build-sshkey: Generate PEM format key https://review.opendev.org/740841 | 01:52 |
| ianw | fungi: ^ if you happen to pop in. it's probably a limited problem, only for jobs like system-config that copy that key | 01:54 |
| corvus | ianw: that role is used in every job, so we should be careful testing that | 02:02 |
| corvus | ianw: is there a pattern to the client side that was rejecting this (ie, specific old operating system)? | 02:03 |
| corvus | (i understand that the executor where this is failing is newer than the others) | 02:04 |
| *** cloudnull has quit IRC | 02:13 | |
| *** cloudnull has joined #opendev | 02:13 | |
| ianw | corvus: i guess it's bridge.o.o refusing to open it, so at least on bionic | 02:18 |
| ianw | seem openssh version 7.8p1-1 is where it started | 02:20 |
| ianw | OpenSSH_7.9p1 Debian-10+deb10u2, OpenSSL 1.1.1d 10 Sep 2019 inside the container | 02:21 |
| *** sgw1 has quit IRC | 02:21 | |
| *** ysandeep|afk is now known as ysandeep|rover | 02:22 | |
| ianw | yep, https://www.openssh.com/txt/release-7.8 to be concrete | 02:23 |
| openstackgerrit | Ian Wienand proposed zuul/zuul-jobs master: add-build-sshkey: Generate PEM format key https://review.opendev.org/740841 | 02:25 |
| ianw | better commit message | 02:25 |
| ianw | although it does say that it has been supported since 2014 | 02:27 |
| *** cloudnull6 has joined #opendev | 02:34 | |
| *** cloudnull has quit IRC | 02:35 | |
| *** cloudnull6 is now known as cloudnull | 02:35 | |
| ianw | i've rechecked and am trying to catch it on ze01 | 02:47 |
| ianw | SSLQ2aEEZ2PGr2DBAAAAEXp1dWwtYnVpbGQtc3Noa2V5AQ== | 03:04 |
| ianw | -----END OPENSSH PRIVATE KEY-----root@bridge:~/.ssh | 03:04 |
| ianw | it looks like it's because it's missing a trailing newline! | 03:04 |
| *** sgw1 has joined #opendev | 03:07 | |
| *** ysandeep|rover is now known as ysandeep|afk | 03:10 | |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: run-base : don't strip root ssh private key https://review.opendev.org/740854 | 03:16 |
| ianw | corvus/fungi: ^ i think that's the magic | 03:19 |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: Copy generated inventory to bridge logs https://review.opendev.org/740605 | 03:33 |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: Add host keys to inventory; give host key in launch-node script https://review.opendev.org/739412 | 03:33 |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: Add host keys on bridge https://review.opendev.org/739414 | 03:33 |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: testinfra: silence yaml.load() warnings https://review.opendev.org/740608 | 03:33 |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: Fix junit error, add HTML report https://review.opendev.org/740609 | 03:33 |
| fungi | ianw: genius! | 03:37 |
| fungi | yes, it has been picky about properly terminatibg keyfiles | 03:38 |
| *** iurygregory has quit IRC | 03:54 | |
| ianw | kevinz: this is happening a lot on the arm64 system-config testing nodes "fatal: unable to access 'https://github.com/infraly/k8s-on-openstack/': gnutls_handshake() failed: Error in the pull function." | 03:55 |
| ianw | i feel like it must be something networkish in the cloud | 03:55 |
| *** shtepanie has quit IRC | 04:23 | |
| *** sgw1 has quit IRC | 04:25 | |
| *** marios has joined #opendev | 04:53 | |
| *** ysandeep|afk is now known as ysandeep | 04:57 | |
| *** elod is now known as elod_off | 06:15 | |
| *** halali_ has quit IRC | 06:30 | |
| mnasiadka | morning | 06:38 |
| *** boyvinall has joined #opendev | 06:46 | |
| *** halali_ has joined #opendev | 06:51 | |
| *** SotK has quit IRC | 06:54 | |
| *** SotK has joined #opendev | 06:55 | |
| *** DSpider has joined #opendev | 06:55 | |
| *** boyvinall has quit IRC | 06:59 | |
| *** iurygregory_ has joined #opendev | 07:10 | |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: Fix junit error, add HTML report https://review.opendev.org/740609 | 07:12 |
| *** kevinz has joined #opendev | 07:26 | |
| *** iurygregory_ is now known as iurygregory | 07:31 | |
| *** halali_ has quit IRC | 07:34 | |
| *** ysandeep is now known as ysandeep|brb | 07:34 | |
| *** tosky has joined #opendev | 07:37 | |
| *** bhagyashris|afk is now known as bhagyashris | 07:44 | |
| *** dtantsur|afk is now known as dtantsur | 07:56 | |
| *** moppy has quit IRC | 08:01 | |
| *** moppy has joined #opendev | 08:03 | |
| *** ysandeep|brb is now known as ysandeep|rover | 08:10 | |
| *** boyvinall has joined #opendev | 08:44 | |
| *** donnyd has quit IRC | 08:53 | |
| *** donnyd has joined #opendev | 08:53 | |
| *** boyvinall has quit IRC | 08:54 | |
| *** fressi has joined #opendev | 09:03 | |
| *** auristor has quit IRC | 09:03 | |
| *** auristor has joined #opendev | 09:09 | |
| *** frickler is now known as frickler_pto | 09:44 | |
| *** frickler_pto is now known as frickler | 09:47 | |
| *** dtantsur is now known as dtantsur|bbl | 09:49 | |
| *** avass has joined #opendev | 10:09 | |
| *** tkajinam has quit IRC | 10:12 | |
| *** ShadowJonathan has quit IRC | 10:22 | |
| *** gouthamr has quit IRC | 10:22 | |
| *** ShadowJonathan has joined #opendev | 10:22 | |
| *** gouthamr has joined #opendev | 10:23 | |
| openstackgerrit | Albin Vass proposed zuul/zuul-jobs master: Add option to install kuberenetes with kind https://review.opendev.org/740935 | 10:50 |
| openstackgerrit | Albin Vass proposed zuul/zuul-jobs master: Add option to install kuberenetes with kind https://review.opendev.org/740935 | 10:51 |
| *** ysandeep|rover is now known as ysandeep|afk | 10:53 | |
| openstackgerrit | Albin Vass proposed zuul/zuul-jobs master: Add option to install kuberenetes with kind https://review.opendev.org/740935 | 10:58 |
| openstackgerrit | Albin Vass proposed zuul/zuul-jobs master: Add option to install kuberenetes with kind https://review.opendev.org/740935 | 11:05 |
| *** fressi has quit IRC | 11:21 | |
| *** fressi has joined #opendev | 11:24 | |
| *** iurygregory has quit IRC | 11:46 | |
| *** ysandeep|afk is now known as ysandeep | 11:47 | |
| openstackgerrit | Albin Vass proposed zuul/zuul-jobs master: Add option to install kuberenetes with kind https://review.opendev.org/740935 | 11:54 |
| *** iurygregory has joined #opendev | 12:00 | |
| openstackgerrit | Albin Vass proposed zuul/zuul-jobs master: Add option to install kubernetes with kind https://review.opendev.org/740935 | 12:06 |
| *** rh-jelabarre has joined #opendev | 12:09 | |
| openstackgerrit | Albin Vass proposed zuul/zuul-jobs master: Add option to install kubernetes with kind https://review.opendev.org/740935 | 12:14 |
| *** fressi has quit IRC | 12:18 | |
| *** ryohayakawa has quit IRC | 12:25 | |
| openstackgerrit | Albin Vass proposed zuul/zuul-jobs master: Add option to install kubernetes with kind https://review.opendev.org/740935 | 12:36 |
| frickler | infra-root: I'll be ptoing starting later today for 2.5 weeks, I won't be completely offline, but don't expect me to be around much | 12:42 |
| *** fressi has joined #opendev | 12:57 | |
| openstackgerrit | Albin Vass proposed zuul/zuul-jobs master: Add option to install kubernetes with kind https://review.opendev.org/740935 | 12:59 |
| *** ysandeep is now known as ysandeep|rover | 13:03 | |
| fungi | ianw: i wonder if we should add that project to zuul and have it pushed onto the node so we can take advantage of caches on the executors | 13:12 |
| fungi | frickler: thanks for the heads up, and i hope you're able to enjoy your time off! | 13:12 |
| *** sgw1 has joined #opendev | 13:14 | |
| mordred | fungi: we could also stop cloning it - we're not actually using it anywhere currently | 13:21 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Stop cloning k8s-on-openstack https://review.opendev.org/740956 | 13:22 |
| mordred | fungi, frickler: ^^ | 13:22 |
| *** dtantsur|bbl is now known as dtantsur | 13:45 | |
| *** frickler is now known as frickler_pto | 13:50 | |
| fungi | oh, neat | 13:51 |
| fungi | good idea ;) | 13:51 |
| *** mlavalle has joined #opendev | 13:58 | |
| openstackgerrit | Oleksandr Kozachenko proposed openstack/project-config master: Add openstack/horizon to the vexxhost tenant https://review.opendev.org/740969 | 14:06 |
| Open10K8S | Hi team | 14:21 |
| Open10K8S | Please check this PS | 14:21 |
| Open10K8S | Add openstack/horizon to the vexxhost tenant https://review.opendev.org/740969 | 14:21 |
| Open10K8S | Needed-By: https://review.opendev.org/740822 | 14:21 |
| *** knikolla has joined #opendev | 14:21 | |
| openstackgerrit | Merged openstack/project-config master: Add openstack/horizon to the vexxhost tenant https://review.opendev.org/740969 | 14:43 |
| openstackgerrit | Merged openstack/project-config master: update-constraints: Install pip for all versions https://review.opendev.org/738926 | 14:43 |
| *** ysandeep|rover is now known as ysandeep|food | 14:47 | |
| *** manfly000 has joined #opendev | 14:49 | |
| AJaeger | frickler_pto: enjoy your vacation! | 14:54 |
| *** manfly000 is now known as xiaoguang | 14:54 | |
| *** fressi has quit IRC | 14:54 | |
| *** xiaoguang is now known as manfly000 | 14:55 | |
| *** manfly000 has left #opendev | 15:03 | |
| clarkb | corvus: do you want to review https://review.opendev.org/#/c/739876/2 I think that affects zuul-web publishing? | 15:12 |
| clarkb | I mean it should be a noop but if it isn't then it would affect zuul-web | 15:12 |
| corvus | clarkb: will do | 15:13 |
| *** davidlenwell has joined #opendev | 15:23 | |
| openstackgerrit | Merged openstack/project-config master: maintain-github-mirror: add requests dependency https://review.opendev.org/740711 | 15:26 |
| *** ysandeep|food is now known as ysandeep | 15:38 | |
| *** ysandeep is now known as ysandeep|away | 15:43 | |
| *** cloudnull6 has joined #opendev | 15:44 | |
| *** cloudnull has quit IRC | 15:46 | |
| *** cloudnull6 is now known as cloudnull | 15:46 | |
| openstackgerrit | Albin Vass proposed zuul/zuul-jobs master: Add option to install kubernetes with kind https://review.opendev.org/740935 | 15:47 |
| openstackgerrit | Albin Vass proposed zuul/zuul-jobs master: Add option to install kubernetes with kind https://review.opendev.org/740935 | 15:51 |
| *** marios is now known as marios|out | 16:01 | |
| *** sshnaidm is now known as sshnaidm|afk | 16:08 | |
| openstackgerrit | Albin Vass proposed zuul/zuul-jobs master: Add option to install kubernetes with kind https://review.opendev.org/740935 | 16:09 |
| openstackgerrit | Albin Vass proposed zuul/zuul-jobs master: Add option to install kubernetes with kind https://review.opendev.org/740935 | 16:25 |
| clarkb | I'm popping out for a bike ridebeforesummer happens later today. Back well before our meeting | 16:37 |
| corvus | clarkb: i don't understand why we're making the change in https://review.opendev.org/739876 ? | 16:43 |
| corvus | there's no 'why' in the commit message :/ | 16:43 |
| corvus | ianw, zbr: ^ ? | 16:43 |
| zbr | well, my fix had a better comment: https://review.opendev.org/#/c/739674/4 | 16:44 |
| zbr | if i understood correctly, the final outcome is the same | 16:45 |
| corvus | (i did go look at the docs for the upload-afs-synchronize role, and there is no indication in the readme of why that role should be used. and i looked at the commit message which added it as well, and while it had more text, it didn't answer my question) | 16:45 |
| zbr | we need to be sure we do not attempt to chown/chgrp | 16:45 |
| corvus | zbr: ah, thanks :) | 16:45 |
| zbr | ianw pinged me when i raised mine, he was addressing the same problem in parallel | 16:47 |
| *** sshnaidm|afk is now known as sshnaidm | 16:48 | |
| zbr | meeting is in about 2h, time to get a break or i will be tired to attend it | 16:48 |
| corvus | i'm curious what this has to do with afs | 16:51 |
| corvus | my understanding is we needed to avoid user/group changes across the board | 16:51 |
| corvus | so i see zbr's change to fix that, which makes sense; but i still don't understand what the addition of the upload-afs-synchronize role gets us | 16:52 |
| avass | corvus: looks like the same problem we're linting for in zuul-jobs | 17:10 |
| avass | corvus: I guess the synchronize in upload-afs-synchronize does the same thing, but the file task before it makes sure the parent directories exist in case they don't? | 17:13 |
| fungi | has anybody tried requests-html, say as an alternative to beautifulsoup4? | 17:25 |
| *** dtantsur is now known as dtantsur|afk | 17:31 | |
| *** marios|out has quit IRC | 17:35 | |
| corvus | avass: yeah, and i'm unfamiliar with the problem of parent dirs not existing | 17:44 |
| corvus | fungi: nope | 17:44 |
| fungi | just had a case where i might need to parse html, and discovered that the original author of requests had also written an html parser library | 17:45 |
| fungi | looks remarkably usable | 17:45 |
| ShadowJonathan | What library? | 18:06 |
| ShadowJonathan | Also, why not bs4 with html5lib? | 18:07 |
| *** qchris has quit IRC | 18:08 | |
| fungi | ShadowJonathan: sorry, i was talking about requests-html and asking if anyone had used it and could compare it to bs4 | 18:13 |
| ShadowJonathan | Oh yeah, sorry, I joined in the conversation with no context | 18:13 |
| fungi | given the popularity of the requests library, i thought perhaps requests-html might have seen similar levels of popularity | 18:13 |
| ShadowJonathan | Yeah, I don't have a comparison to that, sorry | 18:13 |
| *** qchris has joined #opendev | 18:22 | |
| openstackgerrit | Clark Boylan proposed opendev/system-config master: Increase parallelism of gitea project creation https://review.opendev.org/738064 | 18:44 |
| openstackgerrit | Matthew Thode proposed openstack/diskimage-builder master: add openrc init system support to serial console element https://review.opendev.org/741028 | 18:47 |
| openstackgerrit | Matthew Thode proposed openstack/diskimage-builder master: add openrc init system support to serial console element https://review.opendev.org/741028 | 18:59 |
| fungi | weekly opendev infra irc meeting is underway now in #opendev-meeting, btw | 19:04 |
| ianw | corvus: i responded, i had mentioned in the prior change i expected it to be squashed if we wanted to go with it. basically centralise the place where we have the rsync upload caveats into one role. i will update the cll | 19:12 |
| openstackgerrit | Ian Wienand proposed opendev/base-jobs master: promote-deployment: use upload-afs-synchronize https://review.opendev.org/739876 | 19:21 |
| openstackgerrit | Merged opendev/system-config master: run-base : don't strip root ssh private key https://review.opendev.org/740854 | 19:34 |
| openstackgerrit | Andreas Jaeger proposed opendev/base-jobs master: promote-deployment: use upload-afs-synchronize https://review.opendev.org/739876 | 19:41 |
| fungi | and now that the meeting has concluded, i have yardwork i need to get to | 20:03 |
| fungi | but will be around intermittently as i need breaks | 20:04 |
| ianw | infra-root: https://review.opendev.org/#/q/status:open+topic:host-keys should be ready now ... | 20:04 |
| ianw | it manages our host keys in git, and eliminates forgetting to manually add them when you start a new host | 20:04 |
| openstackgerrit | Merged opendev/system-config master: Add Zuul to backups group https://review.opendev.org/740824 | 20:04 |
| ianw | reviews appreciated | 20:05 |
| clarkb | k will try and catch up with the various reviews after lunch | 20:05 |
| clarkb | looks like the ssh key fix in zuul-jobs has merged | 20:05 |
| corvus | ianw: do we want to do that vs sshfp? | 20:23 |
| corvus | (the latter means we benefit from that on our own workstations) | 20:23 |
| fungi | well, benefit from it if we turn it on in our local ssh configs. openssh doesn't rely on it by default | 20:26 |
| corvus | insert word "can" :) | 20:26 |
| ianw | i guess we'd need to configure ansible to obey too? | 20:27 |
| corvus | yeah | 20:27 |
| ianw | i also just put in the rsa key in the inventory, as that was what ansible uses to connect | 20:28 |
| corvus | re ansible, i'm not sure if we would just set the global config on bridge, or add a "-o" option to ansible's use of ssh; i imagine at least one of those would work | 20:29 |
| corvus | (set the global openssh config) | 20:29 |
| corvus | i keep leaving out critical words :) | 20:29 |
| fungi | for posterity, if you want to rely on sshfp set VerifyHostKeyDNS=yes in your ssh client configuration or pass -o VerifyHostKeyDNS=yes | 20:30 |
| fungi | assuming openssh here | 20:30 |
| fungi | also dnssec is a must for it to work | 20:31 |
| corvus | want to try this out for a host real quick-like? maybe something not critical and also a singleton? grafana.opendev.org? | 20:31 |
| fungi | if there's no dnssec validation with the lookup, openssh will ignore it | 20:31 |
| AJaeger | corvus: do you want to review https://review.opendev.org/739876 (ianw's afs change) again, please? | 20:34 |
| openstackgerrit | James E. Blair proposed opendev/zone-opendev.org master: Add SSHFP records for grafana01 https://review.opendev.org/741047 | 20:35 |
| corvus | fungi, ianw: ^ ? | 20:35 |
| openstackgerrit | Ian Wienand proposed opendev/zone-opendev.org master: Add sshfp records for grafana https://review.opendev.org/741048 | 20:35 |
| fungi | duelling changes | 20:35 |
| ianw | do we want to add it for the cname too? | 20:36 |
| fungi | you can't | 20:36 |
| fungi | "cname and other data" | 20:36 |
| fungi | the cname is an alias, your lookup will still get you the same sshfp records | 20:37 |
| fungi | so you don't need to anyway | 20:37 |
| corvus | presumably clients should just resolve it; good thing to test though :) | 20:37 |
| corvus | (i think that makes this a better test than our non-cnamed hosts) | 20:37 |
| fungi | yeah, i mean, bind wouldn't allow you to set an sshfp record for a name which also has a cname record, instead it will refuse to load the new zone and throw the classic "cname and other data" error | 20:38 |
| corvus | AJaeger, ianw: re 739876 -- the only effective change is the extra parent directory thing... what's that about? | 20:38 |
| *** auristor has quit IRC | 20:38 | |
| fungi | though maybe in recent years they've replaced that error with something more descriptive | 20:38 |
| corvus | honestly, i'm not sure i'm on board with the whole "we need a special role to copy a file into afs"; sounds like lots of overhead | 20:39 |
| ianw | corvus: well, my thinking was that there's already been a lot of overhead with people getting the permissions flags wrong that the synchronize role already got right for you | 20:39 |
| ianw | (because i got it wrong in that role initially too i think) | 20:40 |
| *** auristor has joined #opendev | 20:40 | |
| ianw | the docs on upload-afs-synchronize i agree are thin, i'm adding something there | 20:41 |
| openstackgerrit | Merged opendev/zone-opendev.org master: Add SSHFP records for grafana01 https://review.opendev.org/741047 | 20:41 |
| corvus | at base, the observed issue is not unique to afs: don't try to set perms on filesystems you don't have access to :) | 20:41 |
| corvus | ianw: +3 assuming all will be explained with docs update :) | 20:44 |
| openstackgerrit | Merged opendev/base-jobs master: promote-deployment: use upload-afs-synchronize https://review.opendev.org/739876 | 20:50 |
| openstackgerrit | Ian Wienand proposed zuul/zuul-jobs master: upload-afs-synchronize: expand documentation https://review.opendev.org/741051 | 20:52 |
| clarkb | if we're looking at sshfp should I wait on upates to https://review.opendev.org/#/q/status:open+topic:host-keys ? | 20:57 |
| clarkb | the inventory thing seems generally useful so maybe I'll review that anyway | 20:58 |
| ianw | there's a couple stacked ontop for testinfra things that i can move out too | 21:01 |
| corvus | when i add "VerifyHostKeyDNS yes" and ssh, it still asks me but then says "Matching host key fingerprint found in DNS." | 21:02 |
| corvus | i was expecting that to auto-accept | 21:02 |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: Copy generated inventory to bridge logs https://review.opendev.org/740605 | 21:03 |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: testinfra: silence yaml.load() warnings https://review.opendev.org/740608 | 21:03 |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: Fix junit error, add HTML report https://review.opendev.org/740609 | 21:03 |
| ianw | clarkb: ^ that stack is really independent | 21:03 |
| clarkb | thanks that gives me a good place to focus | 21:04 |
| clarkb | ianw: would you include the inventory change or should I hold off on that one? | 21:04 |
| ianw | it's just copying the generated inventory, which is nice to have anyway | 21:04 |
| clarkb | https://review.opendev.org/#/c/739892/ that one I mean | 21:05 |
| ianw | oh, you mean the write-inventory bit -- i guess that could be useful in another context too? it allows arbitrary data to be passed into write-inventory on a per-host basis | 21:05 |
| clarkb | ya I think it would be useful to have anyway | 21:05 |
| clarkb | I'll review it too | 21:05 |
| ianw | it is unit tested, so i think stands alone | 21:05 |
| corvus | oh, i think i need to set edns0 locally | 21:06 |
| ianw | i see that too | 21:10 |
| ianw | the permissions | 21:10 |
| ianw | so glibc isn't reporting to ssh that the results come from dnssec? | 21:11 |
| clarkb | edns0? | 21:11 |
| corvus | it looks like it works on bridge, since bridge is running a local validating recursive resolver | 21:11 |
| corvus | but locally i think i need to do work to convince ssh that the resolver on my wrt is secure enough | 21:11 |
| clarkb | oh I see | 21:12 |
| ianw | https://sourceware.org/glibc/wiki/DNSSEC | 21:17 |
| ianw | https://sourceware.org/git/?p=glibc.git;a=commit;h=446997ff1433d33452b81dfa9e626b8dccf101a4 | 21:17 |
| ianw | so with recent enough glibc, you can mark a dns server as trusted | 21:17 |
| ianw | however, i guess to be practical, networkmanger has to plumb that through | 21:21 |
| clarkb | anyone else have a moment for https://review.opendev.org/#/c/740716/ ? if I can get another +2 on that I'll land it tomorrow morning. The upgrade delta is fairly minimal but I should have plenty of time tomorrow to keep an eye on it | 21:22 |
| ianw | it gets pretty complicated when you start throwing a corporate vpn in the mix too | 21:22 |
| clarkb | it would just work on bridge though right? but I guess if it doesn't work for our laptops its not necessarily better than the originally proposed idea? | 21:23 |
| clarkb | and in that case picking the simpler option may be better? I guess with sshfp you'd still get an indication if it checks out locally even if it doesn't trust it ? | 21:23 |
| ianw | yeah for most non-unbind installs you get "i found this key in dns and it matches but i don't trust it" which is kind of the same impedance as "the server gave me this key and i don't trust it" | 21:25 |
| ianw | s/unbind/unbound/ | 21:26 |
| openstackgerrit | Merged zuul/zuul-jobs master: write-inventory: add per-host variables https://review.opendev.org/739892 | 21:26 |
| corvus | okay, it turns out my dns config on my wrt was wrong and i wasn't passing through validation | 21:27 |
| corvus | i have now flipped the order so it's unbound -> dnsmasq, and things are looking better now | 21:27 |
| corvus | hopefully my leases renew :) | 21:28 |
| corvus | clarkb: i think "works just as well on bridge, but has the potential to improve things for folks with a dnssec-validating-resolver" still gives the sshfp plan the edge | 21:29 |
| clarkb | corvus: ya I guess sshfp has the ability to work in more places if not everywhere | 21:30 |
| clarkb | whereas the other solution has less ability to work anywhere else | 21:30 |
| corvus | yeah, and fwiw, edns0 wasn't necessary on my side; it really was just me failing to configure my server correctly | 21:36 |
| fungi | corvus: still catching back up, but yeah the ssh_config(5) bit on VerifyHostKeyDNS states "the user will still need to confirm new host keys according to the StrictHostKeyChecking option" and i'm not aware of nor can i find any option to auto-accept new host keys only when sshfp records match | 21:56 |
| ianw | fungi: i think it just accepts it *if* it knows the response came from a fully trusted DNS lookup | 21:57 |
| fungi | corvus: er, nevermind, that was for VerifyHostKeyDNS=ask, VerifyHostKeyDNS=true should do what you want (if dnssec is good) | 21:57 |
| ianw | ... and looking at unifi, it seems you *can* set this up with dns masq and manually adding in the trust-anchor | 21:57 |
| fungi | er, s/true/yes/ | 21:57 |
| ianw | ... with a pretty big caveat that most people seem to have managed to lock themselves out due to time issues | 21:57 |
| fungi | and yes, if i `ssh -o VerifyHostKeyDNS=yes grafana.opendev.org` i get straight in without being asked about the host key. it doesn't even get added to known_hosts | 22:01 |
| fungi | so i guess that confirms my dnssec is set up correctly | 22:01 |
| fungi | without -o VerifyHostKeyDNS=yes the same command prompts me about an unknown host key | 22:01 |
| fungi | so seems like this is working the way we want | 22:04 |
| fungi | also i don't do anything fancy with local dns resolution on debian, and my first hop resolver is a recursive unbound on my openbsd firewall which isn't set up as a forwarder | 22:05 |
| fungi | i'm not running nsd or any similar local cache on my clients | 22:06 |
| ianw | fungi: if you have a sec for https://review.opendev.org/#/c/740827/ i can watch those hosts get new ansible backups | 22:13 |
| fungi | ianw: +2 but didn't approve, see comment, not sure if that will pose a problem | 22:23 |
| ianw | fungi: hrm i just grepped for bup:: and i thought i looked up that host ... sorry i've lost track of where it's at | 22:24 |
| fungi | don't be. wiki.openstack.org is a cname to wiki-upgrade-test.openstack.org (which was an emergency cut-over after an unfortunate firewall mishap during an ubuntu release upgrade exposed the elasticsearch api and the server got pwn3d) | 22:25 |
| fungi | multiple iterations of wiki-dev* servers have happened under config management, but none fully functional yet, mostly due to trying to wrangle all the persistent data and git checkouts of somewhere near a hundred extensions | 22:26 |
| ianw | ohhh, right ok ... and wiki-upgrade-test is disabled | 22:27 |
| fungi | yup | 22:27 |
| ianw | hrrmm, that causes a little hiccup in the removal of the old server | 22:29 |
| ianw | the new backup roles are pretty much written to assume the backed-up host is online because they generate keys there and then put them in the authorized hosts on the server | 22:30 |
| fungi | and yeah, the firewall mishap was that the iptables-persistent package wanted to reverse the direction of a symlink when upgrading (from... precise to trusty i think?) but we were creating the symlink with puppet, so we ended up with a circular symlink where the ruleset should have been and it just went ahead with no firewall rules at all... which would have been fine except for the fact that the search | 22:30 |
| fungi | extension we're using relies on a local elasticsearch daemon which just listens on the public interface by default... you can guess the rest | 22:30 |
| ianw | just your usual set of minor problems resulting in a catastrophic confluence | 22:32 |
| fungi | but yeah, like i said in the review comment, i'm happy^H^H^H^H^Hwilling to do a manual dance with installing keys or whatever for that one server | 22:32 |
| ianw | i'll remove it to avoid confusion | 22:33 |
| fungi | until it eventually gets fixed or, more likely, catches fire and burns to the ground while we all stand around cheering it on | 22:33 |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: Backup all hosts with Ansible https://review.opendev.org/740827 | 22:34 |
| *** DSpider has quit IRC | 22:42 | |
| *** tosky has quit IRC | 22:50 | |
| *** tkajinam has joined #opendev | 22:58 | |
| *** mlavalle has quit IRC | 23:01 | |
| openstackgerrit | Merged opendev/system-config master: Backup all hosts with Ansible https://review.opendev.org/740827 | 23:02 |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!