| mordred | infra-root: ^^ first of those is green now - I'm hoping the second two will be to - I think the tests are testing a good potion of that, but we should carefully review those. I also want to do corvus' suggestion of writing out the ansible.cnf to /opt/system-config too ... but I'm gonna do that as a followup | 00:01 |
|---|---|---|
| *** ysandeep|off is now known as ysandeep|rover | 04:07 | |
| *** ykarel|away is now known as ykarel | 04:58 | |
| openstackgerrit | Albin Vass proposed opendev/gerritlib master: Use ensure-* roles https://review.opendev.org/719404 | 07:36 |
| openstackgerrit | Albin Vass proposed opendev/system-config master: Use ensure-* roles https://review.opendev.org/717833 | 07:38 |
| openstackgerrit | Albin Vass proposed zuul/zuul-jobs master: Remove install-* roles https://review.opendev.org/719322 | 07:52 |
| openstackgerrit | Albin Vass proposed zuul/zuul-jobs master: Remove install-* roles https://review.opendev.org/719322 | 07:52 |
| *** ysandeep|rover is now known as ysandeep|lunch | 08:07 | |
| *** ykarel is now known as ykarel|lunch | 08:52 | |
| frickler | corvus: just saw this on a jitsi channel, not sure if it affects us or if we are safe because we run on lo only https://github.com/jitsi/docker-jitsi-meet/commit/a015710e547e8746023f12ee33d2482dabf8ce0f | 09:00 |
| *** ysandeep|lunch is now known as ysandeep|rover | 09:05 | |
| *** DSpider has joined #opendev | 09:14 | |
| yoctozepto | morning folks; is etherpad down? https://etherpad.opendev.org/p/KollaWhiteBoard - Internal Server Error | 09:20 |
| *** tosky has joined #opendev | 09:25 | |
| *** ykarel|lunch is now known as ykarel | 09:38 | |
| *** lpetrut has joined #opendev | 10:30 | |
| *** ysandeep|rover is now known as ysandeep|coffee | 11:02 | |
| *** ysandeep|coffee is now known as ysandeep|rover | 11:35 | |
| *** drifterza has joined #opendev | 12:10 | |
| *** rosmaita has joined #opendev | 12:36 | |
| rosmaita | yoctozepto: it | 12:39 |
| rosmaita | s happening for me too -- https://etherpad.opendev.org/p/cinder-ussuri-meetings | 12:39 |
| rosmaita | giving internal server error | 12:39 |
| * mordred is checking | 12:40 | |
| mordred | rosmaita, yoctozepto : sorry for the issue - should be fixed | 12:41 |
| rosmaita | mordred: ty, working now | 12:41 |
| mordred | I have a theory on what's happening there and will work on a fix as soon as coffee is in the mouth | 12:41 |
| rosmaita | mordred: no rush, now that i know the content hasn't been lost! | 12:42 |
| yoctozepto | thanks mordred | 12:58 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Build and use our own etherpad image https://review.opendev.org/719455 | 13:08 |
| mordred | rosmaita, yoctozepto: fwiw, that should prevent the issue from occurring again ^^ | 13:09 |
| mordred | frickler: I believe it doesn't affect us because those services are only talking across lo | 13:11 |
| mordred | frickler: (we, in fact, set passwords for those, but we set them publically, so I'm assuming corvus did not think they were real passwords) | 13:12 |
| openstackgerrit | Albin Vass proposed zuul/zuul-jobs master: Test base-test https://review.opendev.org/719457 | 13:13 |
| openstackgerrit | Albin Vass proposed zuul/zuul-jobs master: Test base-test https://review.opendev.org/719457 | 13:14 |
| rosmaita | mordred: makes sense to me & should prevent future coronaries | 13:17 |
| mordred | rosmaita: coronary prevention is job 1 | 13:24 |
| rosmaita | :) | 13:24 |
| mordred | rosmaita: so you're saying we should back up that database? | 13:24 |
| fungi | heh | 13:25 |
| rosmaita | mordred: i know that "etherpad" is supposed to imply non-permanence, but i don't think we (openstack) use it like that anymore! | 13:25 |
| rosmaita | +1 on db backups | 13:25 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Back up a single gitea backend https://review.opendev.org/719465 | 13:35 |
| *** ykarel is now known as ykarel|afk | 13:39 | |
| corvus | frickler, mordred: yeah, i think we'd need to secure that if we exposed the xmpp service (which we talked about doing) | 13:41 |
| mordred | rosmaita: (we're already backing it up - that was a bad attempt at dry humor on my part) :) | 13:43 |
| rosmaita | :D | 13:44 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Back up a single gitea backend https://review.opendev.org/719465 | 13:50 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Back up eavesdrop https://review.opendev.org/719484 | 13:50 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Back up a single gitea backend https://review.opendev.org/719465 | 13:53 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Back up eavesdrop https://review.opendev.org/719484 | 13:53 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Don't back up docker containers https://review.opendev.org/719488 | 13:53 |
| mordred | fungi: ^^ | 13:53 |
| mordred | fungi: we should probably _not_ back up docker containers | 13:53 |
| openstackgerrit | Albin Vass proposed zuul/zuul-jobs master: Adds roles to install and run hashicorp packer https://review.opendev.org/709292 | 14:04 |
| *** ykarel|afk is now known as ykarel | 14:04 | |
| openstackgerrit | Albin Vass proposed zuul/zuul-jobs master: Adds roles to install and run hashicorp packer https://review.opendev.org/709292 | 14:10 |
| openstackgerrit | Albin Vass proposed zuul/zuul-jobs master: Adds roles to install and run hashicorp packer https://review.opendev.org/709292 | 14:11 |
| *** ysandeep|rover is now known as ysandeep|away | 14:12 | |
| fungi | big storms are rolling through here, so electricity and internet are rather unstable at the moment. i'll be around whenever meteorologically possible | 14:14 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Use project-config from zuul instead of direct clones https://review.opendev.org/719343 | 14:20 |
| mordred | fungi: try to not get washed away | 14:20 |
| fungi | i'll hold onto the keyboard | 14:21 |
| openstackgerrit | Albin Vass proposed zuul/zuul-jobs master: Test base-test https://review.opendev.org/719457 | 14:27 |
| openstackgerrit | Merged opendev/system-config master: Build and use our own etherpad image https://review.opendev.org/719455 | 14:37 |
| *** lpetrut has quit IRC | 14:39 | |
| openstackgerrit | Merged opendev/system-config master: Don't back up docker containers https://review.opendev.org/719488 | 14:41 |
| *** ykarel is now known as ykarel|away | 14:41 | |
| mordred | corvus: for our deploy jobs - shoudl we have them provides/requires on themselves? | 14:43 |
| mordred | corvus: so taht in the deploy pipeline zuul will be sure to schedule them in order? | 14:43 |
| mordred | or do we not need to worry about that | 14:44 |
| openstackgerrit | Merged opendev/system-config master: Use SafeLoader in irc_checks https://review.opendev.org/718558 | 14:44 |
| corvus | mordred: pipelines are already ordered, so we don't need to worry about it | 14:44 |
| openstackgerrit | Albin Vass proposed zuul/zuul-jobs master: Adds roles to install and run hashicorp packer https://review.opendev.org/709292 | 14:44 |
| corvus | mordred: (so in this case, the mutex of 1 gives us the same behavior) | 14:45 |
| mordred | corvus: kk | 14:46 |
| mordred | etherpad is now running from our image | 14:49 |
| * fungi tests | 14:49 | |
| fungi | pads are still loading fine for me, including style headers | 14:50 |
| mordred | woot | 14:50 |
| fungi | looks good! | 14:50 |
| mordred | we should not have the issue with things going to 500 after ansible runs any more :) | 14:50 |
| corvus | i am not used to this being so fast :) | 14:50 |
| mordred | it's very exciting | 14:50 |
| fungi | indeed. thanks mordred! | 14:50 |
| fungi | we seem to be getting a pattern down for these too | 14:51 |
| mordred | ++ | 14:51 |
| mordred | fungi: if you haven't seen https://review.opendev.org/#/c/719343/ and its two parents yet - that's the next step in speed and correctness | 14:51 |
| mordred | (once it works) | 14:51 |
| corvus | it's pretty cool you're getting -1s on that :) | 14:52 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Use project-config from zuul instead of direct clones https://review.opendev.org/719343 | 14:52 |
| mordred | corvus: yes indeed! | 14:53 |
| fungi | multiple times a day, the track-upstream cron on review-dev is emitting this: /usr/local/lib/python3.7/dist-packages/paramiko/client.py:837: UserWarning: Unknown ssh-rsa host key for [review-dev.opendev.org]:29418: b'03cb1cc7df724fe7d53cc06a8d9cef5c' key.get_name(), hostname, hexlify(key.get_fingerprint()) | 14:53 |
| mordred | fungi: well that's annoying | 14:53 |
| fungi | i think it started when we updated how track-upstream was being run | 14:53 |
| fungi | or maybe we just started running track-upstream on review-dev | 14:53 |
| mordred | yeah. maybe we need to bind-mount the .ssh dir in more? | 14:53 |
| mordred | eyah - that too | 14:53 |
| mordred | fungi: yeah - we should probably accept the hosts host key and then add .ssh/known_hosts to the bindmounts? | 14:54 |
| fungi | i think elsewhere we used configuration management to install the known hosts entry | 14:57 |
| fungi | especially for well-known host keys like the gerrit service | 14:57 |
| mordred | fungi: yeah. that's probably the better idea | 14:58 |
| clarkb | what was causing etherpad to crash? | 15:03 |
| mordred | clarkb: something about re-installing the npm module in ansible was causing etherpad to get confused with the bindmount | 15:05 |
| clarkb | and that is why switchign to an image with it built in fixes things | 15:06 |
| mordred | not 100% _why_ - but since we wanted ot replace that with installing the npm module during an image build, I didn't dig too deeply | 15:06 |
| mordred | yeah | 15:06 |
| clarkb | mordred: fungi one thing I saw scroll past over the weekend was we ignore the disabled list for backups? | 15:09 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Add self host keys to known_hosts on gerrit https://review.opendev.org/719568 | 15:11 |
| mordred | clarkb: fixed | 15:11 |
| mordred | fungi, clarkb: ^^ how's that look for the host key thing? | 15:12 |
| clarkb | mordred: fixed meaning now we don't ignore the disabled list? | 15:12 |
| mordred | clarkb: https://review.opendev.org/#/c/719309/ | 15:12 |
| mordred | that's right | 15:12 |
| clarkb | got it, thanks for clearing that up. I didn't have time to dig into it over the weekend but it caught my eye | 15:13 |
| fungi | clarkb: the disable list now affects whether or not the backups cronjob gets installed, not whether it runs | 15:13 |
| mordred | clarkb: oh - https://review.opendev.org/#/c/719465/ is from you mentioning we should do that, btw | 15:14 |
| clarkb | mordred: TIL [] escapes are valid in fqdn and ipv4 address contexts. At least for ssh hostkey listings | 15:15 |
| mordred | clarkb: I totally copy-pastad that syntax from manifests/site.pp - we do it for accepting gerrit host keys in zuul | 15:15 |
| *** mtreinish has quit IRC | 15:18 | |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Use project-config from zuul instead of direct clones https://review.opendev.org/719343 | 15:22 |
| corvus | i'm having trouble adding avass to zuul-jobs-core | 15:33 |
| corvus | https://review.opendev.org/#/admin/groups/1771,members | 15:33 |
| corvus | when i type in "albin" i get two popups that are identical | 15:33 |
| corvus | and when i select either one, and hit add, i get an error | 15:34 |
| *** mlavalle has joined #opendev | 15:34 | |
| corvus | i don't see anything in the gerrit error log | 15:34 |
| fungi | sounds like the typical gerrit "dulpicate e-mail address" failure | 15:34 |
| fungi | i'll check the db | 15:35 |
| corvus | oh yeah, they both have the same email | 15:35 |
| fungi | that causes gerrit to pretend the account isn't valid when adding to groups or as a requested reviewer | 15:35 |
| corvus | so we need to figure out which is correct, then remove or alter the email address for the other? | 15:36 |
| fungi | two active accounts (29671 created 2018-12-13, 31007 created 2019-09-19) | 15:36 |
| clarkb | I think if you use the specific account id then you can getaround the error adding to the group, but the accounts should probably be combined | 15:37 |
| clarkb | or disambiguated if there is a need for two | 15:37 |
| corvus | looks like the 29671 is what's used | 15:37 |
| fungi | yeah, 31007 looks like it may have been the result of logging into ubuntuone with a gmail address | 15:37 |
| fungi | the older account has a username of albin_vass while the newer has a username of vass | 15:38 |
| fungi | the challenge i sometimes run into is when a user has gotten themselves into a situation where they're using one account in the webui and associated another with their ssh access | 15:38 |
| corvus | good point, so we need to ask avass if he's (even inadvertantly) using both accounts | 15:40 |
| fungi | yep | 15:40 |
| corvus | i'll hold off on using the numeric workaround for now; hopefully he can drop by and we can work out a better solution soon | 15:41 |
| fungi | unfortunately it's become harder to look up ssh keys since recent gerrit started sticking that somewhere other than in the sql db | 15:41 |
| *** avass has joined #opendev | 15:41 | |
| fungi | here's avass! | 15:41 |
| avass | Hey :) | 15:41 |
| corvus | avass: hi! we found you have two accounts in gerrit | 15:41 |
| avass | corvus: right, I think I might have accidentally created another | 15:42 |
| avass | let me check | 15:42 |
| corvus | they both have the same email address, but they have different ssh user names; account id 29671==albin_vass and 31007==vass | 15:42 |
| corvus | it looks like you're using 29671==albin_vass to upload changes | 15:43 |
| avass | should be albin_vass | 15:43 |
| avass | yeah | 15:43 |
| corvus | avass: can you visit https://review.opendev.org/#/settings/ | 15:43 |
| corvus | avass: with whatever account you usually use to use the web interface, and tell us what "Account ID" says there? | 15:43 |
| fungi | (also if you're using gertty, it would be good to double-check the username configured in it too) | 15:44 |
| avass | id=29671 | 15:44 |
| corvus | okay, so it looks like 31007==vass is unused | 15:44 |
| fungi | sounds like we should be safe to just mark that newer account inactive | 15:44 |
| corvus | fungi: can you do that? | 15:44 |
| fungi | yep, gimme one sec | 15:45 |
| fungi | and done | 15:45 |
| corvus | looks good! | 15:45 |
| *** olaph has quit IRC | 15:45 | |
| corvus | avass: this should make it easier for folks to add you as a reviewer | 15:45 |
| corvus | and it makes it easier for me to add you to the zuul-job-core group | 15:46 |
| fungi | #status log set unused gerrit account 31007 inactive for avass | 15:46 |
| openstackstatus | fungi: finished logging | 15:46 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Use project-config from zuul instead of direct clones https://review.opendev.org/719343 | 15:48 |
| fungi | clarkb: corvus: not sure if you saw me mention yesterday, but etherpad 1.8.0 seems to have fixed line number alignment for us: https://etherpad.opendev.org/p/tANNPB4DN0J936odiiBj | 15:49 |
| corvus | and that's with the style plugin too, nice! | 15:51 |
| *** drifterza has quit IRC | 15:52 | |
| avass | I guess it's done, thanks! | 15:52 |
| corvus | avass: yep, let us know if there's any unexpected fallout from this :) | 15:53 |
| fungi | also, this looks promising: https://github.com/ether/etherpad-lite/issues/3861 | 15:53 |
| clarkb | TIL that docker-composes file formats are not actually fixed in time :/ stop_grace_period is a valid v2 directive but not in the version of docker-compose we install | 15:54 |
| avass | corvus: will do! :) | 15:54 |
| mordred | clarkb: boo | 15:54 |
| clarkb | mordred: kinda makes you wonder why bother with the file versions at all... | 15:54 |
| mordred | clarkb: I guess docker-compose isn't in the docker apt repo we install :( | 15:54 |
| clarkb | 1.10 docker-compose added stop_grace_period. On bionic we have 1.5 | 15:54 |
| clarkb | mordred: I don't think so but its a pypi package so we could in theory install it from pypi | 15:55 |
| clarkb | I'll work on a change to do that and the whole thing should be self testing | 15:55 |
| mordred | ++ | 15:56 |
| mordred | clarkb: it seems like a reasonable example of a real reason to use a more recent version than what's in distro | 15:56 |
| mordred | oh - you know ... | 15:56 |
| openstackgerrit | Clark Boylan proposed opendev/system-config master: Use HUP to stop gerrit in docker-compose https://review.opendev.org/719051 | 15:59 |
| openstackgerrit | Clark Boylan proposed opendev/system-config master: Install docker-compose from pypi https://review.opendev.org/719589 | 15:59 |
| clarkb | we can see if that ^ makes things happy at least | 15:59 |
| *** rosmaita has left #opendev | 16:02 | |
| slittle1 | Could we get eyes on https://review.opendev.org/#/c/713968/ ... there is a bit of urgency to get this new git set up. Thanks | 16:02 |
| fungi | slittle1: wrong change? or wrong irc channel? | 16:03 |
| mordred | nope. (I was checking to see if there was a newer docker-compose in a backports repo) | 16:06 |
| AJaeger | slittle1: do you mean https://review.opendev.org/718772 ? | 16:07 |
| openstackgerrit | Merged zuul/zuul-jobs master: ensure-tox: make idempotent and update testing https://review.opendev.org/718284 | 16:12 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Use project-config from zuul instead of direct clones https://review.opendev.org/719343 | 16:22 |
| AJaeger | infra-root, team, FYI, we have errors in openstack tenant, see https://zuul.opendev.org/t/openstack/config-errors - the repo openstack/openstack-ansible-pip_install was retired but some jobs still have required-projects setup. Seems also that the role is still used for older branches, so shouldn't have been retired as is ;( I raised this already on #openstack-ansible and expect that mnaser will | 16:27 |
| AJaeger | look into it. | 16:27 |
| corvus | AJaeger: i'm guessing we don't remove .zuul.yaml from old branches, which is why no one saw the problem before the project was removed from the config? | 16:28 |
| corvus | oh, but if it's included as a "role" even that may not have shown us the problem | 16:29 |
| AJaeger | corvus: it's a job in another repo. The deleted repo was used as a role but it breaks since the job in the other repo uses required-projects to check it out. | 16:29 |
| AJaeger | corvus: there're no self-checks to warn us about these AFAIK | 16:30 |
| corvus | yep, i think that's the case | 16:30 |
| openstackgerrit | Merged zuul/zuul-jobs master: Fix check_jobs_documented linter https://review.opendev.org/719054 | 16:31 |
| AJaeger | corvus: https://review.opendev.org/719029 is one of those changes to remove the required-projects. | 16:31 |
| AJaeger | But team needs to figure out now how to handle these branches | 16:31 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Add self host keys to known_hosts on gerrit https://review.opendev.org/719568 | 16:33 |
| clarkb | mordred: infra-root I believe that installing docker-compose from pypi has fixed the gerrit stop_grace_period issue. Now we need to decide if that is how we want to install docker-compose I guess | 16:35 |
| clarkb | current change only affects gerrit installs as we don't have a common docker-compose role | 16:36 |
| clarkb | and now I need to pop out nad prep for school at home today. Back in a bit | 16:36 |
| mordred | clarkb: I think we should move installing docker-compose into the install-docker role | 16:38 |
| mordred | clarkb: and then just install it the same way everywhere | 16:38 |
| mordred | (there's basically nowhere that we're using docker in prod where we're not using docker-compose - or where having docker-compose installed would be weird) | 16:39 |
| mordred | clarkb, corvus, fungi : booyah. I think https://review.opendev.org/#/c/719343 and its two parents are now ready for review | 16:52 |
| clarkb | mordred: are we comfortable using the pypi version considering we now have a need for newer features? | 16:58 |
| clarkb | also I've realized that my keyboard might be too loud for class time. I may need to switch to laptop | 17:00 |
| mordred | clarkb: I am - but we should take temperatures from folks | 17:00 |
| mordred | clarkb: becuase I think that is a legit worthwhile feature | 17:00 |
| mordred | not just a nice-to-have | 17:00 |
| clarkb | ya | 17:01 |
| clarkb | I'll start on a docker-compose role in a bit under the assumption we'll move forard like that | 17:02 |
| mordred | clarkb: ++ - and then we can just include_role: docker-compose in install-docker? | 17:03 |
| clarkb | ya | 17:03 |
| clarkb | mordred: ci is still failing on the project-config from zuul change | 17:04 |
| mordred | dammit | 17:05 |
| mordred | ah - it's a new gitea error - cool | 17:05 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Use project-config from zuul instead of direct clones https://review.opendev.org/719343 | 17:08 |
| *** prometheanfire has quit IRC | 17:14 | |
| mordred | clarkb: our testing is doing a good job | 17:20 |
| *** olaph has joined #opendev | 17:22 | |
| clarkb | mordred: looking at docker-compose more, really the only generic thing is the package install. I almost wonder if we should just put that in the docker role and not bother with a separate role | 17:22 |
| mordred | clarkb: yeah. probably so | 17:23 |
| mordred | clarkb: it's really not a thing we do independently | 17:23 |
| mordred | clarkb, fungi : https://review.opendev.org/#/c/719568/ is green - should fix the track_upstream issue from earlier | 17:27 |
| clarkb | mordred: isn't that the comment field? why did we haev to change it? | 17:28 |
| mordred | clarkb: the module does a validation - the name is actually supposed to be one of the hosts in the entry | 17:29 |
| clarkb | thats ok | 17:29 |
| mordred | so it writes the entry to a file, then does an ssh-keygen -F {{ name }} on it | 17:30 |
| openstackgerrit | Clark Boylan proposed opendev/system-config master: Install docker-compose from pypi https://review.opendev.org/719589 | 17:38 |
| openstackgerrit | Clark Boylan proposed opendev/system-config master: Use HUP to stop gerrit in docker-compose https://review.opendev.org/719051 | 17:38 |
| clarkb | something like that maybe for docker compose | 17:38 |
| mordred | clarkb: ++ | 17:40 |
| fungi | sorry, was on a call, catching back up on the flurry of patches | 17:50 |
| mordred | clarkb, fungi, corvus : https://review.opendev.org/#/c/719343 is finally fully green | 17:53 |
| clarkb | k will take a look at it momentarily | 17:54 |
| mordred | (and I have good confidence in that green, given all the various reasons it was red before) | 17:54 |
| clarkb | mordred: reading https://review.opendev.org/#/c/719186/6//COMMIT_MSG does PWD need to be /opt/system-config or do we just need to run a playbook from /opt/system-config/playbooks for that chagne of context to apply? | 17:58 |
| mordred | clarkb: PWD needs to be /opt/system-config for the ansible.cfg to get picked up | 17:58 |
| mordred | clarkb: and I'm going to do a followup that will write that ansible.cfg out from the same template as /etc/ansible/ansible.cfg but with different inputs | 17:59 |
| mordred | but - too many spinning plates | 17:59 |
| clarkb | mordred: thinking out loud here why not keep the structure the same but with different paths pointed to in config? that way we don't accidentaly use the zuul version if not cd'd into /opt/sytem-config? | 18:01 |
| clarkb | basically make the global install not work | 18:01 |
| fungi | mordred: i approved 719568 though noted we may want to consider if using /etc/ssh/ssh_known_hosts for these keys in the future makes any of this easier (compared to dotdirs in homedirs) | 18:01 |
| clarkb | and force all commands to be explicit about the install they want | 18:01 |
| mordred | clarkb: yeah - I think I can ponder that in the followup | 18:02 |
| mordred | clarkb: for the most part it shouldn't matter - but I think that'sa. good improvement | 18:02 |
| mordred | fungi: good thought | 18:03 |
| mordred | fungi: would it be useful at all to do sshfp dns records? | 18:03 |
| fungi | probably not as that requires additional client configuration | 18:03 |
| fungi | assuming you mean auto-accept host keys if they have valid sshfp records | 18:04 |
| fungi | by default the openssh client behavior is to report whether the sshfp record matches, but it won't auto-add without explicitly telling it that's the behavior we want | 18:04 |
| *** prometheanfire has joined #opendev | 18:05 | |
| mordred | fungi: is auto-add ust about adding to known-hosts? | 18:05 |
| fungi | yep | 18:05 |
| fungi | and accepting the host key | 18:05 |
| mordred | also I guess we have paramiko in places too | 18:05 |
| fungi | you need to set VerifyHostKeyDNS=yes if you want it to implicitly trust sshfp records | 18:06 |
| mordred | that said - we could probably enable that in /etc/ssh/ssh_config globally if we wanted - assuming paramiko can grok sshfp records too | 18:06 |
| mordred | just seems a little lame that we have to write host keys out when we also have a dnssec signed dns | 18:06 |
| fungi | also we would need to be sure that all our resolvers are doing dnssec validation, because if there's no dnssec chain of trust to the root for those sshfp records then the client will ignore them anyway | 18:06 |
| fungi | though i think we're probably okay on that at this point | 18:07 |
| fungi | at least for anything in domains we're hosting | 18:07 |
| mordred | yeah | 18:07 |
| fungi | if we don't want to fiddle with ssh client configuration then we could pass -oVerifyHostKeyDNS=1 on the cli, but that's probably at least as hard to update everywhere | 18:09 |
| mordred | yeah - I think setting known_hosts is easier than that | 18:10 |
| mordred | HOWEVER - it looks like paramiko doesn't support sshfp without a custom policy like this: https://www.xpra.org/trac/attachment/ticket/2097/_sshfp_policy.py | 18:10 |
| fungi | neat | 18:10 |
| mordred | so it doesn't seem like it would be a huge win anyway | 18:10 |
| mordred | fungi: that said - we _could_ publish sshfp records anyway :) | 18:11 |
| fungi | yes, that's still valuable. i've used VerifyHostKeyDNS=ask in my client configs because maintain sshfp records for my own systems | 18:11 |
| fungi | it solves the tofu problem | 18:11 |
| fungi | i wish ask was the default there | 18:12 |
| mordred | yeah. | 18:12 |
| mordred | if I don't have the key, but there is one in dns that's a much more reasonable thing to ask me if it's opk | 18:12 |
| mordred | clarkb: yes to your question on https://review.opendev.org/#/c/719190 | 18:13 |
| fungi | right, it shifts the problem from "the odds that someone is monkeying with my initial connection are low and at least if i accept this key then i'll detect when it does happen later" to "whoever maintains dns says this is the correct key" | 18:14 |
| fungi | you still get the usual behavior with key changes too, where you'll need to delete the old conflicting key | 18:14 |
| clarkb | mordred: ok the last change inthe stack is the one that has my brain most melted. It seems we've sort of punted on the actual git repo updating and are just synchronizing what we've currently got. Comments with more details | 18:17 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Update install-ansible away from /opt/system-config https://review.opendev.org/719186 | 18:20 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Run playbooks out of zuul checkout https://review.opendev.org/719190 | 18:20 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Use project-config from zuul instead of direct clones https://review.opendev.org/719343 | 18:20 |
| mordred | clarkb: o - I missed your second comment - one sec | 18:20 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Use project-config from zuul instead of direct clones https://review.opendev.org/719343 | 18:25 |
| mordred | clarkb: ok - I think I addressed your second comment - good catch | 18:25 |
| clarkb | mordred: we still need something to update the non zuul side? | 18:26 |
| clarkb | also slightly worried this is complicated enough we are likely to run manage projects with the wrong version of projects.yaml at some point | 18:27 |
| clarkb | mordred: maybe we should have the manual side use /home/zuul/src too? | 18:27 |
| clarkb | then we are always in sync with the last zuul run | 18:27 |
| clarkb | (that could be racy though) | 18:27 |
| mordred | clarkb: I don't think we should have anything update the non-zuul side | 18:28 |
| mordred | I think that's purposefully manual | 18:28 |
| mordred | set system-config (and optionally project-config) to the ref you want | 18:28 |
| clarkb | mordred: I worry that we'll run manage projects manually assumign its up to date | 18:28 |
| mordred | I think we shoudl not do that :) | 18:28 |
| mordred | but - I mean - we can certainly add a sync | 18:29 |
| mordred | easy enough to do | 18:29 |
| clarkb | mordred: maybe we should invert the default path. Make the default zuul then force manual users to point at what they know to be accurate? | 18:29 |
| mordred | I'm just arguing we shouldn't do it because it's most of the time work that's not needed, and it only causes us to need to be careful to put bridge into the disabled list if we want to do something manually | 18:29 |
| mordred | clarkb: that's a good idea | 18:29 |
| mordred | clarkb: and I can get behind that | 18:29 |
| mordred | default empty basically | 18:30 |
| clarkb | ya | 18:30 |
| mordred | if you want to run manage-projects by hand, pass -eproject_config_src | 18:30 |
| mordred | one sec, fixing | 18:30 |
| clarkb | that way its clear you need to do the extra step | 18:30 |
| clarkb | because the current default is almost never what you want unless you've taken extra steps | 18:30 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Use project-config from zuul instead of direct clones https://review.opendev.org/719343 | 18:32 |
| mordred | clarkb: totally agree | 18:32 |
| corvus | mordred: are you saying we would need to install https://www.xpra.org/trac/attachment/ticket/2097/_sshfp_policy.py in the ansible running on bridge in order for sshfp to help? | 18:35 |
| mordred | clarkb: or - do you think, if it's not set, we should clone latest to project-config and then use that? | 18:35 |
| mordred | corvus: yeah - I think so | 18:35 |
| mordred | corvus: which means I think it's not very useful | 18:35 |
| mordred | corvus: oh - well, not ansible - doesn't ansible use openssh? | 18:36 |
| mordred | corvus: I think we'd have to install it in jeepyb if we wanted to use sshfp for jeepyb scripts instead of installing host keys for them | 18:36 |
| mordred | for openssh we could just set the config value in /etc/ssh | 18:36 |
| corvus | mordred: oh, yeah. well, that seems like a big win then. | 18:37 |
| corvus | and the jeepyb thing is fixable by us to, so that should catch the last 5% | 18:37 |
| mordred | corvus: should we add that to zuul? | 18:37 |
| corvus | mordred: where? | 18:38 |
| mordred | in gerritlib I guess | 18:38 |
| corvus | ah yeah, that's a good idea | 18:38 |
| clarkb | mordred: I think if it isn't set thats a good cue to the human to be specific about what they want | 18:38 |
| mordred | clarkb: kk | 18:38 |
| clarkb | mordred: we can even add a note about it in the readme so that when it fails and they go looking they get that info | 18:38 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Add asserts about project_config_src https://review.opendev.org/719665 | 18:38 |
| mordred | clarkb: ^^ there's an assert | 18:38 |
| openstackgerrit | Merged opendev/system-config master: Add self host keys to known_hosts on gerrit https://review.opendev.org/719568 | 18:42 |
| clarkb | mordred: k I'm figuring out lunch and will rerevie stack after | 18:42 |
| mordred | cool | 18:42 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Update install-ansible away from /opt/system-config https://review.opendev.org/719186 | 19:22 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Run playbooks out of zuul checkout https://review.opendev.org/719190 | 19:22 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Use project-config from zuul instead of direct clones https://review.opendev.org/719343 | 19:22 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Add asserts about project_config_src https://review.opendev.org/719665 | 19:22 |
| openstackgerrit | Tristan Cacqueray proposed zuul/zuul-jobs master: use-buildset-registry: wait for kubernetes service to be available https://review.opendev.org/719673 | 19:27 |
| *** olaph has quit IRC | 20:01 | |
| *** olaph has joined #opendev | 20:01 | |
| *** olaph has quit IRC | 20:02 | |
| clarkb | mordred: stack lgtm though I left another ocmment on https://review.opendev.org/#/c/719343/9 | 20:14 |
| clarkb | basically I think the current setup of setting the var always but only consuming (and having the repo present) in a small subset of cases may be confusing | 20:15 |
| clarkb | and if we add more of those cases chances of getting it wrong are higher than just having the repo update each time | 20:15 |
| openstackgerrit | Merged zuul/zuul-jobs master: use-buildset-registry: wait for kubernetes service to be available https://review.opendev.org/719673 | 20:19 |
| clarkb | https://zuul.opendev.org/t/openstack/build/5622bbf19fa54ad1ac448778f2846a09/log/job-output.txt#48525 the joys of upgrading. Good thing we have tests | 20:19 |
| clarkb | now to see if there are other places where we rely on the names being static | 20:19 |
| clarkb | mordred: also any idea if that is dangerous for the running services? | 20:19 |
| clarkb | ugh | 20:19 |
| openstackgerrit | Clark Boylan proposed opendev/system-config master: Install docker-compose from pypi https://review.opendev.org/719589 | 20:27 |
| openstackgerrit | Clark Boylan proposed opendev/system-config master: Use HUP to stop gerrit in docker-compose https://review.opendev.org/719051 | 20:27 |
| clarkb | I'm going to WIP ^ until we can think about if/how the container names will affect our running services. | 20:28 |
| clarkb | mordred: corvus ^ you might have existing setup that can test that quickly? Basically I think we want to know if `docker-compose up` on new version of docker-compose will properly stop the old container then start the new containers with the name transition (and everything else that ahs changed) | 20:29 |
| *** olaph has joined #opendev | 20:30 | |
| clarkb | otherwise we might have to do a stop on the old version and a start on the new? | 20:30 |
| clarkb | I'm trying to construct a change that will test it in zuul now | 20:31 |
| openstackgerrit | Clark Boylan proposed opendev/system-config master: DNM Test docker-compose upgrade https://review.opendev.org/719682 | 20:44 |
| clarkb | I think ^ may cover this for us, its weird because we really need old and new verification to be split but we should be able to infer if its generally working or not even if the test cases fail | 20:44 |
| mordred | clarkb: was on phone - reading scrollback | 21:00 |
| mordred | clarkb: oh fun! (re: naming) | 21:00 |
| mordred | clarkb: I don't know the answer to the upgrade question | 21:00 |
| clarkb | mordred: I think I hacked together a thing that will test it reasonably well at https://review.opendev.org/719682 | 21:01 |
| clarkb | so we can watch that space and see what happens? | 21:01 |
| clarkb | its possible I need to edit more files so that more jobs run but one step at a time | 21:01 |
| mordred | clarkb: I agree - I think that should tel us some things | 21:01 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Write project_config_src in a vars file per job https://review.opendev.org/719685 | 21:13 |
| mordred | clarkb: ^^ what do you think about the shape of that? | 21:13 |
| openstackgerrit | Clark Boylan proposed opendev/system-config master: DNM Test docker-compose upgrade https://review.opendev.org/719682 | 21:17 |
| clarkb | mordred: ^ I needed to make mroe changes so that more jobs would run | 21:17 |
| mordred | clarkb: ++ | 21:18 |
| mordred | clarkb: the thing above is my stab at addressing your issue with the var being set on the base job but only used sometimes | 21:18 |
| clarkb | mordred: +2 I like that makes it explicit to those jobs | 21:18 |
| clarkb | rather than half global | 21:18 |
| mordred | yeah - and gives us a mechanism ot pass arbitrary vars, rather than hardcoding that one | 21:19 |
| mordred | clarkb, corvus : hrm. the change I made to update the gitea homepage template is resulting in failed run-gitea job - and I don't understnad what's broken yet | 21:28 |
| mordred | wouldn't mind an extra set of eyes | 21:29 |
| clarkb | mordred: have a link? | 21:29 |
| mordred | https://zuul.opendev.org/t/openstack/build/07034064f75e43ee94c567e6e0297c89 | 21:29 |
| clarkb | mordred: https://zuul.opendev.org/t/openstack/build/07034064f75e43ee94c567e6e0297c89/log/gitea99.opendev.org/docker/giteadocker_gitea-web_1.txt#129 it can't find its cert | 21:30 |
| clarkb | mordred: did you remove the LE test machinery maybe? | 21:31 |
| clarkb | or possible that le dev just failed | 21:31 |
| mordred | hrm. I don't _think_ I removed it | 21:31 |
| clarkb | mordred: the LE test machinery talks to LE's dev servers and expects everything to "work" then it provides a self signed cert from the server itself | 21:32 |
| clarkb | if LE dev servers fail then I think we may fail in that way | 21:32 |
| mordred | nod | 21:32 |
| mordred | clarkb: how do we see that failure? | 21:33 |
| mordred | clarkb: https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_070/718764/1/gate/system-config-run-gitea/0703406/bridge.openstack.org/ara-report/result/61eccc7c-e562-420d-960f-87795f5e20d1/ | 21:34 |
| mordred | there's the running of acme.sh | 21:34 |
| clarkb | mordred: https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_070/718764/1/gate/system-config-run-gitea/0703406/bridge.openstack.org/ara-report/reports/e6f93bce-d23d-4b7b-87af-c95f0a41bdda.html | 21:35 |
| clarkb | thats green though | 21:35 |
| clarkb | so maybe that didn't fail | 21:35 |
| mordred | yeah- but cert.pem not being there does seem like a red flag | 21:39 |
| mordred | since we should be bind-mounting it in | 21:39 |
| clarkb | mordred: ya LE creates the file in /etc/lesomething | 21:39 |
| clarkb | then we run a handler that is supposed to copy it to /var/gitea/something | 21:39 |
| clarkb | maybe that handler idnd't fire | 21:39 |
| mordred | clarkb: well - actually | 21:39 |
| mordred | oh, you already said | 21:39 |
| mordred | clarkb: it doesn't look like handlers log very well | 21:39 |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!