| opendevreview | Ian Wienand proposed zuul/zuul-jobs master: encrypt-file : role to encrypt a file https://review.opendev.org/c/zuul/zuul-jobs/+/828818 | 00:06 |
|---|---|---|
| opendevreview | Ian Wienand proposed zuul/zuul-jobs master: encrypt-file : role to encrypt a file https://review.opendev.org/c/zuul/zuul-jobs/+/828818 | 00:14 |
| opendevreview | Ian Wienand proposed opendev/system-config master: [WIP] Base work for exporting encrypted logs https://review.opendev.org/c/opendev/system-config/+/828810 | 00:18 |
| clarkb | ok meeting agenda has been sent | 00:26 |
| mordred | fungi: awesome. | 00:34 |
| mordred | fungi: also - considering the wiki is like the second oldest service ... it's kind of mind blowing that it's alive to the extent that it is with zero config mgmt :) | 00:35 |
| mordred | (eavesdrop being the first) | 00:35 |
| mordred | although IIRC, we started on a different wiki and migrated - so it's not really the second oldest service | 00:36 |
| fungi | yeah, it was moinmoin previously | 00:36 |
| fungi | though just about everything on eavesdrop has been replaced with different software over time as well | 00:36 |
| mordred | it has indeed | 00:37 |
| mordred | and oh yeah - I remember the moinmoin now | 00:37 |
| opendevreview | Ian Wienand proposed zuul/zuul-jobs master: encrypt-file : role to encrypt a file https://review.opendev.org/c/zuul/zuul-jobs/+/828818 | 00:49 |
| opendevreview | Ian Wienand proposed opendev/system-config master: Base work for exporting encrypted logs https://review.opendev.org/c/opendev/system-config/+/828810 | 01:46 |
| opendevreview | Ian Wienand proposed opendev/system-config master: run-production-playbook: return encrypted logs https://review.opendev.org/c/opendev/system-config/+/829147 | 01:46 |
| opendevreview | Ian Wienand proposed zuul/zuul-jobs master: encrypt-file : role to encrypt a file https://review.opendev.org/c/zuul/zuul-jobs/+/828818 | 01:51 |
| ianw | ^ fungi/clarkb: that stack is ready ... interested to see what you think of it. it took a *lot* of tweaking, but the final result is pretty straight forward | 03:14 |
| opendevreview | Merged opendev/system-config master: Stop mirroring Fedora 34 https://review.opendev.org/c/opendev/system-config/+/829115 | 03:53 |
| fungi | #status log Updated OpenID provider for the refstack.openstack.org service from openstackid.org to id.openinfra.dev | 04:29 |
| opendevstatus | fungi: finished logging | 04:29 |
| *** pojadhav|afk is now known as pojadhav | 04:41 | |
| opendevreview | Merged opendev/system-config master: Switch refstack's IDP to OpenInfraID https://review.opendev.org/c/opendev/system-config/+/824065 | 04:52 |
| *** ysandeep|out is now known as ysandeep | 05:30 | |
| *** amoralej|off is now known as amoralej | 07:00 | |
| *** ysandeep is now known as ysandeep|lunch | 08:02 | |
| *** jpena|off is now known as jpena | 08:34 | |
| dpawlik | clarkb: hey, sure | 08:41 |
| *** ysandeep|lunch is now known as ysandeep | 09:12 | |
| amusil | gtema: Hi, could you please take a look at https://review.opendev.org/c/openstack/openstacksdk/+/761933 | 10:14 |
| *** pojadhav- is now known as pojadhav | 10:24 | |
| *** pojadhav is now known as pojadhav|afk | 10:24 | |
| dtantsur | FYI folks, it was brought to my attention that "gitweb" link on our gerrit pages leads to "Not Found" | 10:44 |
| *** rlandy|out is now known as rlandy|ruck | 11:16 | |
| *** dviroel|out is now known as dviroel | 11:19 | |
| *** pojadhav|afk is now known as pojadhav | 12:31 | |
| *** ysandeep is now known as ysandeep|brb | 12:40 | |
| *** ykarel_ is now known as ykarel | 12:49 | |
| *** ysandeep|brb is now known as ysandeep | 13:10 | |
| *** amoralej is now known as amoralej|lunch | 13:20 | |
| fungi | dtantsur: yeah, we're in the process of pointing those to gitea but have run into a bug in gerrit which we're trying to get fixed. the commit id link should take you to the gitiles view for now until we've got the gitea patch in | 13:48 |
| dtantsur | ack, thanks | 13:48 |
| fungi | dtantsur: https://review.opendev.org/825339 is the config update we'll land as soon as we have a patched gerrit | 13:49 |
| fungi | the depends-on there is the patch in the gerrit community's gerrit we're testing across different gerrits | 13:50 |
| lourot | fungi o/ I believe since we have landed https://review.opendev.org/c/openstack/project-config/+/825089 , https://github.com/openstack/charm-nova-compute-nvidia-vgpu should have been created. Do you have some insights about what could have gone wrong? Thanks! | 13:58 |
| *** amoralej|lunch is now known as amoralej | 14:02 | |
| fungi | lourot: looks like the maintain-github-openstack-mirror should be taking care of creating repositories in github, and it last succeeded 2021-12-07 according to https://zuul.opendev.org/t/openstack/builds?job_name=maintain-github-openstack-mirror&skip=0&limit=100 | 14:03 |
| fungi | looks like the repo listing mechanism in it bitrotted around some structureal change in zuul vars: https://zuul.opendev.org/t/openstack/build/69feba2382c5419685a7a807e64736a8/log/job-output.txt#381-389 | 14:04 |
| lourot | fungi, thanks for the pointers! do you have a timeline as to when this could get fixed? | 14:18 |
| fungi | lourot: no clue, someone will need to look into how the job needs to be adjusted. if that someone is me, it won't be until this afternoon | 14:18 |
| fungi | er, that's vague, sorry. won't be for at least a few more hours due to meetings | 14:19 |
| fungi | i can probably take a closer look at it after today's opendev meeting ends around 20 utc | 14:20 |
| lourot | thanks a lot, that's a more granular/precise timeline than what I expected actually :D | 14:20 |
| fungi | i strive for precision ;) | 14:29 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/project-config master: Move missed repos under openstack-ansible-roles ACL https://review.opendev.org/c/openstack/project-config/+/829278 | 14:48 |
| *** pojadhav is now known as pojadhav|dinner | 15:04 | |
| *** ysandeep is now known as ysandeep|out | 15:25 | |
| clarkb | dpawlik: hey, so I'm trying to get up to speed on where the ELK stuff with opensearch is in order to plan the removal of the old services | 15:36 |
| clarkb | I think we are very close to being able to do that? What do you think are the remaining TODOs before we can tell people to use the new location and stop using the old one? | 15:37 |
| *** dviroel is now known as dviroel|lunch | 15:37 | |
| dpawlik | clarkb: right. So today I rebuild logscraper + logearman container images and all seems to be fine but.. Probably the logstash service is down. I don't have access to listen cloud formation projects in AWS. I send an email right now to Reed to check it | 15:41 |
| clarkb | thanks. I think we also probably want to start asking people to use the new system to try it out? | 15:42 |
| clarkb | people like melwitt and dansmith and gmann are probably good early users if they are interested | 15:42 |
| dpawlik | clarkb: besides that, index-pattern seems to visible to readonly user, so it's good. I was thinking to change the kibana login page to show user credentials to login, when session will be gone or maybe add to the Openstack page somewhere a link that is automatically injecting header with basic credentials so user does not need to login | 15:43 |
| dpawlik | clarkb: let's wait 1 day or 2 | 15:44 |
| dpawlik | clarkb: until logstash is not up and running. Hope it will be done soon | 15:44 |
| dpawlik | or maybe Reed will set proper access to my account so I can check it | 15:44 |
| reed | Me? | 15:45 |
| fungi | the other reed ;) | 15:45 |
| reed | 😄 | 15:45 |
| dpawlik | sorry reed :) | 15:46 |
| clarkb | dpawlik: sounds like a plan. Thanks for the update | 15:46 |
| dpawlik | clarkb: it was working ok, before my PTO, but I compare it with old elasticsearch metrics and seems that some of them were not available in opensearch. I did not have time before pto to check whats going on. When I take random missing build on my local instance it seems that issue was on remote host where the logs are stored (I spotted an issue on | 15:48 |
| dpawlik | OVH host) | 15:48 |
| dpawlik | clarkb: but as I said, I did not check it fully. Maybe it is an issue with gearman (logscraper catch that build and process it) | 15:49 |
| clarkb | ok, if the old system was able to pull the logs the new one should be able to as well. There is no authentication for log access for example | 15:50 |
| clarkb | basically no extra privileges are involved | 15:50 |
| dpawlik | clarkb: I have the Opensearch opendev as prio. Will notify you and fungi what is the current progress | 15:50 |
| fungi | dpawlik: i thought rt had set you up with aws admin access before you were off | 15:52 |
| dpawlik | clarkb: so in tl;dr: logstash seems to be overloaded (or is down right now); autologin for kibana will be an awesome feature; replace kibana login page with readonly user credentials info; create new visualization/dashboard + port them to yaml and store in ci-log-processing repo | 15:52 |
| dpawlik | fungi: he fix errors that I have for opensearch, but I did not check cloud formation service | 15:53 |
| dpawlik | fungi: I was not aware how it is running, until I did not check the documentation :) https://review.opendev.org/c/openstack/ci-log-processing/+/826405/4/opensearch-config/deploy_opensearch.sh | 15:53 |
| fungi | ahh, cool | 15:54 |
| *** amoralej is now known as amoralej|off | 16:08 | |
| *** pojadhav|dinner is now known as pojadhav|afk | 16:16 | |
| *** pojadhav|afk is now known as pojadhav | 16:16 | |
| jrosser | would it be ok to save a copy of the contents of the `zuul` variable into the zuul-info directory? (https://zuul-ci.org/docs/zuul/latest/job-content.html#zuul-variables) | 16:17 |
| clarkb | jrosser: we do, it is in the inventory file | 16:18 |
| jrosser | ah! | 16:18 |
| fungi | the zuul vars are exposed as ansible vars, so are included i that faux "inventory" view | 16:18 |
| fungi | (it's not the exact ansible inventory ansible uses i the builds, just an approximation, but the vars should be representative) | 16:19 |
| jrosser | ok thats great, thanks | 16:19 |
| fungi | i can't seem to type "in" today | 16:19 |
| jrosser | part two would be, is that anywhere on the disk i can read with include_vars? | 16:19 |
| clarkb | only as the file we log I think. Not sure if you can include_vars an inventory file safely | 16:21 |
| *** marios is now known as marios|run | 16:21 | |
| clarkb | you could write it out explicitly yourself though and then include that later without the inventory stuff | 16:21 |
| jrosser | i had started hacking up an osa specific thing for the embedded ansible to be able to read a copy of the `zuul` yaml | 16:23 |
| jrosser | but it felt like that might also be more widely applicable for anything with an embedded ansible | 16:23 |
| opendevreview | Clark Boylan proposed opendev/system-config master: Haproxy http checks for Gitea https://review.opendev.org/c/opendev/system-config/+/829141 | 16:30 |
| *** ysandeep|out is now known as ysandeep | 16:33 | |
| *** dviroel|lunch is now known as dviroel | 16:52 | |
| *** marios|run is now known as marios | 16:54 | |
| *** ysandeep is now known as ysandeep|out | 17:11 | |
| clarkb | that was fun, rebooted to pick up updates and turns out simpledrm driver completely breaks booting on amd gpu devices? | 17:16 |
| fungi | um, wow | 17:29 |
| fungi | i guess it's "too simple"? | 17:29 |
| clarkb | https://bugzilla.opensuse.org/show_bug.cgi?id=1195887 | 17:36 |
| clarkb | I think they just don't know what is going on yet :/ | 17:36 |
| fungi | ahh | 17:37 |
| *** jpena is now known as jpena|off | 17:38 | |
| clarkb | fungi: the haproxy change is still failing. The issue now is that we haven't told haproxy how to verify the ssl certs. In testing we don't really want to verify the ssl certs because it is all self signed (and that file is on another server os hard to add it to the local trust change). But in production it could be beneficial. Do you have an opinion on splitting testing from | 17:57 |
| fungi | clarkb: the https check needs to point to a ca file (or set verify none or ssl-server-verify none globally) | 17:57 |
| clarkb | production here (and hoping we don't instantly down all gitea when this change lands) or should we just not verify for now in both places? | 17:57 |
| clarkb | ha yup | 17:57 |
| fungi | seems we were both looking it over at the same exact moment | 17:57 |
| fungi | i agree in production it could be nice if invalid certs caused members to be excluded from the available pool, but in practice that's probably not super necessary (we didn't have it before anyway) | 17:58 |
| fungi | i'd go with the simple solution for now | 17:59 |
| clarkb | right the tcp checks didn't have verification either | 17:59 |
| opendevreview | Clark Boylan proposed opendev/system-config master: Haproxy http checks for Gitea https://review.opendev.org/c/opendev/system-config/+/829141 | 18:00 |
| fungi | in reality we have some belts (periodic le jobs) and braces (certcheck warning us a month ahead of expirations), so i'm not super worried one of the gitea servers is going to spontaneously have a bad cert. odds are they'll all have it or none will | 18:00 |
| fungi | so i doubt it's worth the added complexity for something we can't have proper test coverage of anyway | 18:01 |
| fungi | or, i guess we could have test coverage, but that would be even more complex still | 18:01 |
| fungi | i imagine it would entail having test and prod versions of ca certs to verify against | 18:03 |
| clarkb | yes, we'd have to slurp the cert from the gitea host and then splat it out onto the haproxy host. Then bind mound that and set the ca-file and restart haproxy after the deployment | 18:03 |
| fungi | and somehow plumbing the gitea server cert in the job so that it's also the ca | 18:03 |
| clarkb | in prod we'd use /etc/ssl/certs/ca-certificates.crt | 18:03 |
| fungi | right that | 18:03 |
| fungi | also, complex health checks can represent a liability if it means more that can go wrong with the checking itself, unnecessarily evacuating the entire pool and taking the service completely offline | 18:04 |
| clarkb | it is doable and if anyone feels strongly about that we can add it in. However, I don't think the lack of verification is a regression compared to the old tcp checks so may make a good followon | 18:05 |
| fungi | yep, i concur | 18:05 |
| clarkb | Do we know if https://mirror.bhs1.ovh.opendev.org/wheel/centos-8-x86_64/ can be safely cleaned up? I suspect that the stream nodes are using the stream mirror properly due to the issueswe had when things were shared previously. I don't think the wheels are a big storage consumer but noticed we've still got centos-8 there | 18:15 |
| clarkb | fungi: I've updated the gerrit gitea changes upstream to what I'm hoping are pretty close to final iterations. Might be worthwhile respinning a hold on your change to test that? | 18:17 |
| fungi | yeah, i can put one in now | 18:54 |
| fungi | clarkb: do we need to add the other change as a depends-on in 825339? | 18:56 |
| clarkb | fungi: I don't think so. It seems to have worked somehow | 18:57 |
| fungi | (for the gitiles patch) | 18:57 |
| clarkb | it won't hurt though | 18:57 |
| clarkb | https://gerrit-review.googlesource.com/c/plugins/gitiles/+/330361 is the url | 18:57 |
| fungi | yeah, that's the one i just found | 18:58 |
| fungi | it depends-on the other change, so i can simply update the depends-on in ours to that | 18:58 |
| fungi | assuming we have plugins/gitiles in our projects list | 18:59 |
| fungi | (we do, right?) | 18:59 |
| * fungi checks | 18:59 | |
| fungi | yeah, it's in the tenant | 18:59 |
| clarkb | we should | 18:59 |
| opendevreview | Jeremy Stanley proposed opendev/system-config master: Use Gitea for Gerrit's code browser URLs https://review.opendev.org/c/opendev/system-config/+/825339 | 19:01 |
| opendevreview | Jeremy Stanley proposed opendev/system-config master: DNM: Fail our Gerrit testing for an autohold https://review.opendev.org/c/opendev/system-config/+/825396 | 19:01 |
| fungi | autohold has been set for 825396,4 | 19:01 |
| fungi | and i've dropped the old autohold | 19:03 |
| clarkb | thanks | 19:03 |
| fungi | huh, gertty complains with a syntax error when querying for a topic which starts with a digit | 19:27 |
| opendevreview | Steve Baker proposed openstack/diskimage-builder master: Replace kpartx with qemu-nbd in extract-image https://review.opendev.org/c/openstack/diskimage-builder/+/828617 | 20:12 |
| opendevreview | Steve Baker proposed openstack/diskimage-builder master: Move grub-install to the end, and skip for partition images https://review.opendev.org/c/openstack/diskimage-builder/+/826976 | 20:12 |
| opendevreview | Jeremy Stanley proposed openstack/project-config master: Read tenant config with care in github_manager https://review.opendev.org/c/openstack/project-config/+/829402 | 20:27 |
| fungi | lourot: ^ that should fix the github repo creation job | 20:27 |
| fungi | turned out to be a simple fix | 20:27 |
| ianw | NeilHanlon/clarkb: re: 828435 we'll want to update several matches in https://opendev.org/openstack/project-config/src/branch/master/nodepool/elements first I think | 20:47 |
| clarkb | ianw: ah thanks for catching that | 20:54 |
| ianw | we could probably add that into the dib boot tests | 21:00 |
| ianw | it does expand the list of things that can go wrong in that job though, and that list is already fairly long | 21:00 |
| clarkb | I think we can take this one step at a time. Address what we know needs addressing, land it and see if the images build and go from there | 21:01 |
| clarkb | glean is another thing that may not work | 21:01 |
| clarkb | (I don't know if glean will recognize it is on a rhel machine) | 21:01 |
| NeilHanlon | ianw: same change, or a different one/ | 21:03 |
| NeilHanlon | ?* | 21:03 |
| ianw | i'd probably stack it before that change as a new one. if you can run a build with those elements all the better | 21:04 |
| opendevreview | Merged openstack/project-config master: Read tenant config with care in github_manager https://review.opendev.org/c/openstack/project-config/+/829402 | 21:04 |
| opendevreview | Neil Hanlon proposed openstack/project-config master: Add Rocky Linux to nodepool elements tooling https://review.opendev.org/c/openstack/project-config/+/829405 | 21:05 |
| opendevreview | Neil Hanlon proposed openstack/project-config master: Add rockylinux-8 to nodepool configuration https://review.opendev.org/c/openstack/project-config/+/828435 | 21:06 |
| opendevreview | Neil Hanlon proposed openstack/project-config master: Add Rocky Linux to nodepool elements tooling https://review.opendev.org/c/openstack/project-config/+/829405 | 21:09 |
| *** dviroel is now known as dviroel|out | 21:35 | |
| opendevreview | Neil Hanlon proposed openstack/project-config master: Add Rocky Linux to nodepool elements tooling https://review.opendev.org/c/openstack/project-config/+/829405 | 22:31 |
| *** rlandy|ruck is now known as rlandy|ruck|bbl | 22:46 | |
| opendevreview | Ian Wienand proposed openstack/diskimage-builder master: Update platform support to describe stable testing https://review.opendev.org/c/openstack/diskimage-builder/+/418204 | 23:20 |
| opendevreview | Ian Wienand proposed openstack/diskimage-builder master: Update platform support to describe stable testing https://review.opendev.org/c/openstack/diskimage-builder/+/418204 | 23:40 |
| *** rlandy|ruck|bbl is now known as rlandy|ruck | 23:46 | |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!