| opendevreview | Ian Wienand proposed opendev/system-config master: make-tarball: role to archive directories https://review.opendev.org/c/opendev/system-config/+/865784 | 00:09 |
|---|---|---|
| opendevreview | Ian Wienand proposed opendev/system-config master: tools/make-backup-key.sh https://review.opendev.org/c/opendev/system-config/+/866430 | 00:09 |
| clarkb | kevinz: are you around? | 00:26 |
| clarkb | ianw: kevinz: re works on arm I think the firs thting I would start with is pointing out that the hardware didn't work for the vast majority of the last 6 months | 00:26 |
| clarkb | But then on top of that we can talk about the crpytography package builds and kolla work? | 00:27 |
| ianw | yeah probably the fact it's fairly constantly busy is probably confirmation enough. i feel like we could pull stats from zuul | 00:30 |
| clarkb | I can work on drafting something if ya'll prefer but I won't be able to get to it until tomorrow. I do think its a bit unfair to act as if the hardware was available for those 6 months though. Since aiui there was hardware issues at equinix | 00:30 |
| clarkb | ianw: ya grafana has info | 00:30 |
| clarkb | https://grafana.opendev.org/d/391eb7bb3c/nodepool-linaro?orgId=1&from=now-30d&to=now | 00:30 |
| clarkb | anyway I think we should carefully call out that we don't have 6 months of data because the hardware was really only available for the last month or two | 00:30 |
| clarkb | but then point at the good things that are happening anyway | 00:31 |
| ianw | ++ | 00:32 |
| ianw | i wonder if i should just manually delete the linaro-us-regionone graphite stats | 00:34 |
| ianw | the reason that shows in the graph is we use linaro-* | 00:34 |
| clarkb | aha https://grafana.opendev.org/d/391eb7bb3c/nodepool-linaro?orgId=1&from=now-30d&to=now&var-region=linaro-regionone is a better graph | 00:34 |
| ianw | our sweep will clear them out eventually, after they don't get updates for a while | 00:35 |
| clarkb | https://etherpad.opendev.org/p/3DcVXw0PBOknv1bgyZWh | 00:35 |
| clarkb | working on a response there | 00:35 |
| clarkb | ianw: something like that maybe? | 00:55 |
| clarkb | feel free to edit and send or discard and send etc. Otherwise I'll probably send it out tomorrow | 00:55 |
| clarkb | and now I need to get our meeting agenda out before dinner | 00:55 |
| ianw | that looks pretty good to me. although we're building on osuosl (for now, anyway, until building becomes a special case of a regular test) the fact that we're producing working arm64 images is maybe another good angle. i'm not sure if they're used, but we have a tool provably doing it | 00:58 |
| clarkb | oh ya ++ to that | 00:58 |
| clarkb | maybe add that to the paragraph about zuul + ansible | 00:59 |
| clarkb | ok agenda is out and I'm being told dinner is ready. ianw if you want to send that email feel free. Otherwise I can send it out tomorrow morning | 01:02 |
| ianw | i think about it a bit more, not quite what i want to say. i'll re-read it all in a bit but i think maybe sleep on it and send tomorrow | 01:02 |
| clarkb | Looks like they are also asking about plans for the future. Which I haven't really addressed so far | 01:02 |
| clarkb | ianw: if you can think of future plans ^ feel free to add those too | 01:03 |
| ianw | wheel builds is another thing we do | 01:03 |
| opendevreview | Ian Wienand proposed opendev/system-config master: make-tarball: role to archive directories https://review.opendev.org/c/opendev/system-config/+/865784 | 01:11 |
| opendevreview | Ian Wienand proposed opendev/system-config master: tools/make-backup-key.sh https://review.opendev.org/c/opendev/system-config/+/866430 | 01:11 |
| opendevreview | Ian Wienand proposed opendev/system-config master: make-tarball: role to archive directories https://review.opendev.org/c/opendev/system-config/+/865784 | 02:43 |
| opendevreview | Ian Wienand proposed opendev/system-config master: tools/make-backup-key.sh https://review.opendev.org/c/opendev/system-config/+/866430 | 02:43 |
| opendevreview | Ian Wienand proposed opendev/system-config master: make-tarball: add some extraction instructions https://review.opendev.org/c/opendev/system-config/+/875587 | 03:49 |
| ianw | clarkb: ^ i think that's now ready to go again | 03:50 |
| ianw | you can validate the process with https://4bf453712ef03cadf4d7-91acd78cd46015ce54b5f888f723113e.ssl.cf2.rackcdn.com/865784/15/check/system-config-run-base/7e31121/bridge99.opendev.org/make-tarball/backup_2023-02-28T03:04:56.tar | 03:50 |
| ianw | the key can be reconstructed from https://review.opendev.org/c/opendev/system-config/+/865784/15/playbooks/zuul/templates/group_vars/bastion.yaml.j2 | 03:51 |
| ianw | 875587 has a little session example | 03:51 |
| *** yadnesh|away is now known as yadnesh | 04:19 | |
| *** thuvh1 is now known as thuvh | 07:15 | |
| *** thuvh1 is now known as thuvh | 07:28 | |
| *** jpena|off is now known as jpena | 08:24 | |
| *** elodilles is now known as elodilles_afk | 10:28 | |
| *** yadnesh is now known as yadnesh|away | 11:07 | |
| *** elodilles_afk is now known as elodilles | 12:44 | |
| bbezak | Hi. I got ssh key issue in one of job, when connecting to zuul@172.99.67.80 - https://zuul.opendev.org/t/openstack/build/2965f8f2f4c84560b1e5ffeaf3ac6c1c | 13:08 |
| bbezak | looks weird :) | 13:09 |
| fungi | bbezak: it happens from time to time, when nova loses track of a vm in a cloud provider's network for some reason and then neutron allocates the same ip address to us for a new vm without realizing something on the network still answers to it, so ansible randomly ends up getting routed to the old vm | 13:12 |
| fungi | if we see a high incidence of it, we usually correlate the ip addresses and then provide a list to the provider, but most of the time they run background cleanup processes to find and delete those automatically anyway | 13:13 |
| fungi | looks like that one is in rackspace's iad region | 13:15 |
| bbezak | ok, so vm is not in openstack, but somehow still running on the host | 13:15 |
| bbezak | ok, thx for clarification fungi | 13:17 |
| fungi | yes, usually you end up (as the operator) having to do something like track its mac down to a specific hypervisor host through your bridge tables and then use virsh to get a listing of things running there and check them against nova's db | 13:19 |
| fungi | but obviously an automated process there could just periodically dump a list of all virtual machines on the host and then see if any don't match a db entry | 13:19 |
| fungi | i think there are some popular openstack ops tools around stuff like that, but since i'm not a production cloud operator these days my understanding of this is mostly theoretical | 13:20 |
| *** kopecmartin_ is now known as kopecmartin | 15:00 | |
| clarkb | oh wow gitea09 sync is still not completed | 16:15 |
| clarkb | I'm thinking for the next giteas I'll try to do them assembly line style all at once and maybe we can have the syncs run over a weekend | 16:16 |
| frickler | somethings's wrong with meetbot, for the recent #-qa meeting only one file was saved, also no links were posted to the channel https://meetings.opendev.org/meetings/qa/2023/ | 16:22 |
| frickler | the neutron-ci meeting that ended just a bit earlier was recorded fine | 16:22 |
| frickler | nothing obvious in the logs that I could find | 16:23 |
| clarkb | could it be afs quotas? | 16:24 |
| clarkb | though I would've expected messages back to the channel and onl the file issues in that case | 16:24 |
| clarkb | doesn't look like that volume is on our afs dashboard | 16:24 |
| frickler | I don't think that thas is on afs even? | 16:27 |
| frickler | ha, it just missed the earlier #endmeeting | 16:27 |
| frickler | just tried again and now all looks well | 16:27 |
| clarkb | I think it is in afs | 16:27 |
| clarkb | or at least the site is, maybe not the raw logs | 16:28 |
| fungi | nothing is wrong. people sometimes forget that the chair changing their nick after starting the meeting causes them to no longer appear to be the meeting chair | 16:37 |
| fungi | meetbot simply doesn't track nick changes. possible feature addition for the future i suppose | 16:38 |
| fungi | the obvious workarounds are to add a second chair before changing your nick, or changing your nick back to what you started the meeting with | 16:41 |
| clarkb | fungi: frickler: have you had a chance to look at https://etherpad.opendev.org/p/3DcVXw0PBOknv1bgyZWh ? This is an email response to arm about the use of our Works on Arm hardware hosted by equinix running a linaro cloud for arm CI | 16:56 |
| clarkb | Looking to send that out today and any feedback you might have is appreciated | 16:56 |
| clarkb | the git gc on gitea09 is starting to pack those objects into more space effiicent pack files | 17:16 |
| * clarkb is a bit annoyed the replication sync is still happening. I thought I'd be asking for reviews to add gitea09 to haproxy by now | 17:16 | |
| fungi | clarkb: the writeup lgtm, i left a couple of comments | 17:21 |
| *** jpena is now known as jpena|off | 17:23 | |
| clarkb | fungi: the point about linking to the image build logs and images themselves is good. I'll get that added. For the second item I think ianw was trying to convey that we're busy on keeping things running which makes it difficult to find time for writing blogposts | 17:45 |
| opendevreview | Clark Boylan proposed opendev/system-config master: Publish raw images on our nodepool builders https://review.opendev.org/c/opendev/system-config/+/875775 | 17:50 |
| clarkb | fungi: ^ that comes out of this too | 17:51 |
| clarkb | I'll let ianw respond to the other comment then we can hopefully ship this out in a couple hours | 17:52 |
| clarkb | there are about 200 more replication tasks before we get into the ~200 or so retries that are queued up | 18:10 |
| fungi | i don't remember replication being so slow in the past | 18:10 |
| clarkb | fungi: its slow for the initial sync | 18:11 |
| clarkb | but also made slower due to gerrit putting all the data in git | 18:11 |
| fungi | i remember it taking hours, but not this long | 18:11 |
| clarkb | ya there are a lot more refs now with notedb | 18:12 |
| fungi | i wonder if an rsync initially would make the process faster | 18:12 |
| clarkb | it might, but gitea relies on git hooks receiving the data to populate its knowledge of the repo iirc | 18:13 |
| clarkb | I didn't attempt an rsync due to ^ | 18:13 |
| clarkb | also rsync would not apply the gerrit acls (which mostly doesn't matter for us outside of all-projects and all-users) | 18:14 |
| fungi | true. i couldn't remember if you were copying the database in already | 18:16 |
| clarkb | I did, but since we didn't prevent people from pushing new changes in the interim | 18:17 |
| clarkb | Its probably possible, its just this route is likely the most safe | 18:17 |
| clarkb | fungi: for the next servers I think what I want to do after gitea09 proves the process is deploy three new servers all together then run a big sync over the weekend | 18:20 |
| fungi | that makes sense | 18:24 |
| clarkb | fungi: the gerrit trouble yesterday would've impacted replication too | 18:45 |
| clarkb | though its still taking quite a bit of time considering | 18:45 |
| fungi | oh, yeah i suppose that did chuck a wrench into the gears | 18:46 |
| clarkb | fungi: and between gitea09 being done and adding more new servers I think we should land https://review.opendev.org/c/opendev/system-config/+/875533 to update gitea. I don't want to do that until gitea09 is behind haproxy though | 18:47 |
| clarkb | just to avoid moving parts | 18:47 |
| fungi | sounds good, i agree | 18:59 |
| ianw | clarkb/fungi: yeah was just trying to say we're probably not going to be writing full on blog posts, etc., but if there's a marketing/comms person driving that sort of thing we can certainly help | 19:02 |
| fungi | ianw: got it, i included some suggested rephrasing to make that clearer | 19:03 |
| opendevreview | Merged opendev/system-config master: Publish raw images on our nodepool builders https://review.opendev.org/c/opendev/system-config/+/875775 | 19:55 |
| clarkb | ianw: I think that edit looks good | 19:58 |
| clarkb | ianw: do we want to wit for nb04 to list raw images then update the footnotes before sending? | 19:58 |
| clarkb | *wait | 19:58 |
| ianw | we can do for our own sake of being technically correct, but i doubt anyone will double check it really :) | 19:59 |
| clarkb | and did you have a preference for who sends it? I'm happy either way. But you've put a lot of effort into the arm stuff so you may want to take some of the credit here :) | 20:00 |
| ianw | heh, i'm happy for you to send it | 20:00 |
| clarkb | ok, I'll wait for nb04 to list images then send it out | 20:01 |
| fungi | i struck through the opening to paragraph #7 since it was somewhat repetitove with the start of paragraph #8 (typo notwithstanding) | 20:01 |
| clarkb | fungi: thanks | 20:02 |
| fungi | so yeah, clean up line 59 once the vhost fix deploys and it's good to go in my opinion | 20:03 |
| clarkb | gitea09 is down to replicating tripleo-heat-templates | 20:04 |
| clarkb | once that completes I'll trigger a full replication of everything? | 20:04 |
| clarkb | everything == all repos and all giteas | 20:04 |
| clarkb | are we comfortable with that at this point after yesterday's fun? | 20:04 |
| ianw | i think so, i didn't get any particular response, but also it doesn't seem we're still getting hit with that | 20:07 |
| fungi | yeah, no new hits on my grep of apache logs | 20:09 |
| clarkb | alright I'll trigger that once I see gitea09 finish up | 20:14 |
| clarkb | and then I'll remove my WIP from https://review.opendev.org/c/opendev/system-config/+/874175 | 20:15 |
| clarkb | I had to restart/reload apache to pick up the vhost changes https://nb04.opendev.org/images/ loads interesting info now though | 20:17 |
| clarkb | will put the email together and send it now | 20:17 |
| clarkb | full replication has been enqueued | 20:23 |
| clarkb | and arm email is sent | 20:25 |
| fungi | thanks! | 20:34 |
| clarkb | I think the full sync is done already | 20:52 |
| clarkb | which makes sense since it should noop most things | 20:52 |
| clarkb | I have unwiped https://review.opendev.org/c/opendev/system-config/+/874175 to put gitea09 behind the load balancer | 20:53 |
| ianw | lgtm, i didn't +w in case you want to watch it or something, but gitea09:3000 seems fine | 20:54 |
| clarkb | I think we can probably send it in. Worst case I'll manually remove it from pools and push a revert | 20:55 |
| clarkb | I've approved it | 20:56 |
| opendevreview | Ian Wienand proposed opendev/system-config master: make-tarball: add some extraction instructions https://review.opendev.org/c/opendev/system-config/+/875587 | 20:59 |
| clarkb | fungi: whats up with this email from gmann on openstack-discuss "Returned mail: Data format error" | 21:13 |
| fungi | where did you see it? | 21:15 |
| gmann | clarkb: fungi: I just checked, not sure how it coming | 21:15 |
| gmann | fungi: https://lists.openstack.org/pipermail/openstack-discuss/2023-February/032494.html | 21:15 |
| gmann | it was not from my sent item | 21:15 |
| fungi | oh, i see, my mailserver rightly detected it as spam | 21:16 |
| clarkb | oh someone spoofing then | 21:16 |
| gmann | humm | 21:16 |
| fungi | spoofed post, originated from an ip address in korea | 21:17 |
| fungi | it made it through because it was spoofed for an address which is a subscriber and, unlike usually happens with these, it was under the 40kb message limit | 21:18 |
| fungi | if more of these start coming in, i'll put the list in emergency moderation mode, but would prefer not to do that since it would mean every post gets held for moderation and i think i'm the only active moderator these days | 21:23 |
| gmann | fungi: anything I need to do on this? | 21:23 |
| fungi | i don't think so, no | 21:24 |
| gmann | ack, thanks | 21:24 |
| opendevreview | Merged opendev/system-config master: Add gitea09 to the gitea load balancer https://review.opendev.org/c/opendev/system-config/+/874175 | 21:56 |
| clarkb | that got in ahead of the hourly runs | 22:01 |
| clarkb | I'm still getting balanced to gitea01 so hard for me to test that gitea09 is happy through the lb. But cacti shows that it appears to be in use | 23:22 |
| fungi | that's probably the bets you can do without trying random machines with different source addresses | 23:28 |
| fungi | s/bets/best/ | 23:30 |
| clarkb | ianw: if you get a chance to look at https://review.opendev.org/c/opendev/system-config/+/875533 I'll try to land that tomorrow when I can keep an eye on it now that gitea09 is just like the other 8 gitea servers | 23:45 |
| ianw | np will do | 23:46 |
| clarkb | ianw: in https://review.opendev.org/c/opendev/system-config/+/865784 what caused the rax dns backup contents to be present and now things pass? | 23:53 |
| ianw | umm i think that directory was always there, maybe it was something else? I had a few issues in the tar generation but i think it's all quoted and a bit simpler now | 23:54 |
| clarkb | ah. I seem to recall it complaining specifically about that but maybe it was a quoting problem and I didn't notice that | 23:54 |
| ianw | i just wanted to do two fairly separate directories, but also not always upload too much to the logs | 23:55 |
| ianw | i did double check by manually getting into the generated output | 23:55 |
| ianw | i thought about including a script or something, but i just ended up making some notes -> https://review.opendev.org/c/opendev/system-config/+/875587 | 23:56 |
| clarkb | ianw: ya just left some comments on that (minor things) | 23:58 |
| ianw | clarkb: yeah, you'll need the password to import the private key, and decrypt with it. although if you use gnome it may be cached in between | 23:59 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!