| *** rlandy is now known as rlandy|out | 00:03 | |
| opendevreview | Merged opendev/system-config master: Add (lack of) license information for org logos https://review.opendev.org/c/opendev/system-config/+/834343 | 00:06 |
|---|---|---|
| opendevreview | Ian Wienand proposed openstack/project-config master: grafana afs : remove centos 8 volume tracking https://review.opendev.org/c/openstack/project-config/+/834412 | 01:15 |
| opendevreview | Merged zuul/zuul-jobs master: [multi-node-bridge] Allow to skip openvswitch installation https://review.opendev.org/c/zuul/zuul-jobs/+/832497 | 03:26 |
| *** persia is now known as Guest2697 | 03:53 | |
| *** pojadhav is now known as pojadhav|rover | 03:53 | |
| *** ysandeep|out is now known as ysandeep | 04:13 | |
| *** chkumar|rover is now known as chandankumar | 05:33 | |
| *** jpena|off is now known as jpena | 08:35 | |
| *** ysandeep is now known as ysandeep|lunch | 09:22 | |
| opendevreview | Albin Vass proposed zuul/zuul-jobs master: mirror git repos: cleanup submodule checkouts https://review.opendev.org/c/zuul/zuul-jobs/+/834432 | 09:23 |
| opendevreview | Albin Vass proposed zuul/zuul-jobs master: mirror git repos: cleanup submodule checkouts https://review.opendev.org/c/zuul/zuul-jobs/+/834432 | 09:24 |
| opendevreview | Albin Vass proposed zuul/zuul-jobs master: mirror git repos: cleanup submodule checkouts https://review.opendev.org/c/zuul/zuul-jobs/+/834432 | 09:26 |
| *** pojadhav|rover is now known as pojadhav|lunch | 09:36 | |
| dpawlik | ianw: some issue with zuul? | 09:54 |
| opendevreview | Merged openstack/project-config master: grafana afs : remove centos 8 volume tracking https://review.opendev.org/c/openstack/project-config/+/834412 | 10:02 |
| dpawlik | frickler, fungi: hey, do you have some issue with zuul? | 10:09 |
| *** ysandeep|lunch is now known as ysandeep | 10:16 | |
| *** pojadhav|lunch is now known as pojadhav|rover | 10:18 | |
| frickler | dpawlik: none that I know of, do you have more details? | 10:19 |
| dpawlik | frickler: sorry, false alarm. | 10:22 |
| dpawlik | chrome got some issue, after restart works :) | 10:22 |
| *** rlandy|out is now known as rlandy | 10:24 | |
| *** dviroel|out is now known as dviroel | 11:17 | |
| opendevreview | Marios Andreou proposed openstack/project-config master: Update channel ops for oooq (tripleo ci) channel https://review.opendev.org/c/openstack/project-config/+/834199 | 11:35 |
| *** ysandeep is now known as ysandeep|break | 11:39 | |
| fungi | dpawlik: did you have a zuul status page open over the weekend and found it showing something broken today? | 12:07 |
| fungi | wondering if that's related to saturday's service upgrades. we have multiple zuul-web services with a load balancer directing traffic to them, but i don't think we've tested to see how seamless that is with the status page autorefresh | 12:08 |
| dpawlik | fungi: It stuck on refreshing status builds web page, but it seems to be issue on my side | 12:53 |
| fungi | well, if it had been up refreshing constantly since before the upgrade on saturday, it's possible it got a bad api response at one point when we were failing over between zuul-web servers. that could persist on the screen, but i would also have expected a hard refresh to clear it if that were the cause | 12:57 |
| dpawlik | aha, good to know | 13:01 |
| dpawlik | I see in Opensearch that there is low amount of logs from yeasterday 5 PM UTC, but now it seems to be ok | 13:02 |
| * bhagyashris brb | 13:04 | |
| *** ysandeep|break is now known as ysandeep | 13:05 | |
| fungi | dpawlik: it's possible we weren't running many jobs. weekends tend to be especially quiet times for openstack development work, sundays in particular | 13:06 |
| fungi | you can check the zuul jobs launched graph to see: https://grafana.opendev.org/d/21a6e53ea4/zuul-status?orgId=1&viewPanel=16&from=now-4d&to=now | 13:09 |
| *** iurygregory_ is now known as iurygregory | 13:10 | |
| *** needssleep is now known as TheJulia | 13:13 | |
| fungi | those tall spikes are when all the daily periodic and periodic-stable jobs run | 13:20 |
| fungi | but you can see that the activity around emea/americas "business hours" are missing on saturdays and sundays | 13:21 |
| fungi | looks like sunday actually does pick up a little compared to saturday, i guess that's apac monday mornings showing up in the mix | 13:22 |
| clarkb | fungi: looks like the change to build new gerrit images did land | 14:50 |
| clarkb | And zuul reports the promote image jobs were successful | 14:50 |
| clarkb | any opinions on when we should try and restart for that? | 14:51 |
| fungi | i'm up for it any time, just didn't want to do it until we'd had a chance to discuss | 14:51 |
| fungi | maybe when activity starts to die down later in our day? | 14:51 |
| clarkb | sure that should work for me | 14:52 |
| clarkb | I'll let the openstack release team know due to proximity to the openstack release | 14:56 |
| *** pojadhav|rover is now known as pojadhav|afk | 14:57 | |
| *** dviroel is now known as dviroel|lunch | 15:29 | |
| fungi | thanks! | 15:57 |
| *** ysandeep is now known as ysandeep|out | 16:05 | |
| *** pojadhav|afk is now known as pojadhav|rover | 16:30 | |
| *** dviroel|lunch is now known as dviroel | 16:38 | |
| *** pojadhav|rover is now known as pojadhav|out | 16:40 | |
| *** marios is now known as marios|out | 16:50 | |
| clarkb | https://github.com/go-gitea/gitea/issues/19118 some updates on that gitea partial clone bug. I just responded as it doesn't quite make sense to me yet | 16:52 |
| fungi | i have a feeling something changed in client-side handling of the filter feature between when it was first added and the versions we saw working with gitea | 17:25 |
| fungi | it's really the only explanation which makes sense | 17:25 |
| fungi | clarkb: one detail you elided in the description, which may have led readers to an incorrect assumption, is that these same git clients/options were working fine prior to our upgrade to gitea 1.16 | 17:28 |
| fungi | it's possible for someone reading that report to guess (incorrectly) that this is the first version of gitea we've tried to run, rather than that we observed a regression when upgrading | 17:29 |
| clarkb | ah yup I meant to mention that when I indiated the commit that chagned the bheuavor but it got missed | 17:31 |
| *** jpena is now known as jpena|off | 17:32 | |
| clarkb | This is fun. I just had rpm fail because cpio woudln't replace a directory with a file. Package was updated to change a path from a dir to a file. Had to manually move the dir aside and rerun the installation. | 17:51 |
| *** lajoskatona_ is now known as lajoskatona | 17:59 | |
| fungi | interesting it doesn't have logic to deal with that | 18:28 |
| *** dviroel is now known as dviroel|afk | 19:52 | |
| corvus | the zuul docs publishing job had an error preventing a docs builds for a tagged release today; i'm going to manually copy in a build (detailed discussion in #zuul earlier) | 19:58 |
| clarkb | thanks for the heads up | 19:58 |
| corvus | #status log manually published /afs/openstack.org/project/zuul-ci.org/www/docs/zuul/5.1.0 | 20:01 |
| opendevstatus | corvus: finished logging | 20:01 |
| fungi | thanks! | 20:02 |
| *** outbrito_ is now known as outbrito | 20:50 | |
| *** dviroel|afk is now known as dviroel | 21:03 | |
| fungi | okay, i'll double-check we have the latest image on review | 21:16 |
| clarkb | fungi: thanks I was lookign at zuul status and there are some changes that we might wait a few minutes for but then go for it I think | 21:16 |
| fungi | looks like there was a mariadb container update to pull | 21:16 |
| fungi | opendevorg/gerrit 3.4 86c3bcd9fafe 2 days ago 800MB | 21:17 |
| fungi | mariadb 10.4 fd1ed4499d48 2 days ago 403MB | 21:17 |
| clarkb | if we want to go a little longer we could update dockerd and do a reboot too. Not sure if we want to try and sneak that in | 21:17 |
| clarkb | that will definitely be more noticeable | 21:18 |
| fungi | yeah, it's probably not a huge deal to add those in, should go quickly | 21:18 |
| clarkb | ok | 21:18 |
| clarkb | process for that would be to apt-get update && docker-compose down && apt-get dist-upgrade && reboot then docker-compose up -d | 21:18 |
| fungi | there's new kernel images to install as well | 21:18 |
| fungi | i'll get those installed first and hold on the docker package updates just yet while gerrit's still running | 21:19 |
| clarkb | ok | 21:19 |
| fungi | just so we don't waste time during the outage | 21:20 |
| clarkb | I think at some point we should consider a docker image prune on that server. But that can happen later not when we're doing a short downtime | 21:20 |
| clarkb | maybe after we've run the current images for a few days happily we can manually do that | 21:20 |
| clarkb | fungi: the tripleo chagnes that were close to merging are really close now. Just finished up the paused job then a few changes should be able to land there. Then I think we can proceed? | 21:25 |
| fungi | status notice The Gerrit service on review.opendev.org will be offline momentarily for a Gerrit patch upgrade and kernel update, but should return again shortly | 21:26 |
| fungi | that lgty? | 21:26 |
| clarkb | lgtm | 21:26 |
| clarkb | the tripleo job is in their collect logs phase which iirc can take a few minutes :/ | 21:27 |
| clarkb | I'm putting together the agenda for tomorrow's meeting. Looks like ianw added an item. Anything else to add before I snd that out? I'll probably send it out once gerrit work is done | 21:29 |
| fungi | there's a repo rename request i thinl | 21:29 |
| fungi | think | 21:29 |
| clarkb | ah so there is. Let me make sure that is on the agenda | 21:29 |
| clarkb | the tripleo job is uploading logs to swift now | 21:31 |
| clarkb | should be really close | 21:31 |
| fungi | #status notice The Gerrit service on review.opendev.org will be offline momentarily for a Gerrit patch upgrade and kernel update, but should return again shortly | 21:31 |
| opendevstatus | fungi: sending notice | 21:31 |
| fungi | i'll get that circulating | 21:31 |
| -opendevstatus- NOTICE: The Gerrit service on review.opendev.org will be offline momentarily for a Gerrit patch upgrade and kernel update, but should return again shortly | 21:31 | |
| clarkb | those chagnes are out of the zuul queue now | 21:34 |
| clarkb | maybe give it a few seconds just to make sure the gerrit stuff is reported but then I think we can proceed | 21:34 |
| clarkb | gerrit merged changes list shows them as merged. I think we can proceed | 21:34 |
| clarkb | fungi: were you going to drive? you were alraedy doing stuff on the server | 21:35 |
| fungi | yep, on it | 21:35 |
| fungi | gerrit down, docker upgrading | 21:35 |
| clarkb | I see the containers have stopped. That lgtm I think we can proceed wtih docker updates | 21:35 |
| fungi | will reboot as soon as that's done | 21:35 |
| clarkb | sounds good | 21:35 |
| fungi | rebooting | 21:35 |
| clarkb | and then we'll have to docker-compose up -d when it is up again | 21:36 |
| fungi | it's back | 21:36 |
| clarkb | yup I'm in | 21:36 |
| fungi | Linux review02 5.4.0-105-generic #119-Ubuntu SMP Mon Mar 7 18:49:24 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux | 21:36 |
| fungi | starting gerrit | 21:36 |
| corvus | (btw i rebooted the schedulers on saturday to get the new kernel) | 21:37 |
| fungi | webui is up | 21:37 |
| fungi | thanks corvus! | 21:37 |
| fungi | Powered by Gerrit Code Review (3.4.4-14-g76806c8046-dirty) | 21:38 |
| clarkb | there it goes took a minute before web ui would load for me but it is there now and I see the version we expected | 21:38 |
| clarkb | I'm not seeing files listed in the changes. I seem to recall I noticed this last time too and it took a little time to prime the diff caches? | 21:39 |
| fungi | oh maybe | 21:39 |
| clarkb | yup just refreshed a change and now the files show up | 21:39 |
| clarkb | I kidna wish they wouldn't report the server is ready before stuff like that is happy | 21:40 |
| fungi | i see them in a change i just opened | 21:40 |
| clarkb | But noting it here as I think this is expected and related to caching | 21:40 |
| clarkb | ya it doesn't take long but is just long enough to be noticeable | 21:40 |
| *** dviroel is now known as dviroel|afk | 21:40 | |
| clarkb | all seems happy from what I'm looking at. We're a few commits ahead of 3.4.4 proper. Looks like some older schema changes (shouldn't affect us as we're already ahead of those schema changes) to GC repos as they migrated into notedb for stuff, a comment got clarified and an exception gets logged more verbosely | 21:44 |
| clarkb | I don't see anything of concern there | 21:44 |
| clarkb | fungi: we should probably #status log Updated Gerrit to 3.4.4-14-g76806c8046-dirty ? | 21:45 |
| fungi | go for it, yeah | 21:45 |
| clarkb | #status log Updated Gerrit on review.o.o to 3.4.4-14-g76806c8046-dirty | 21:45 |
| opendevstatus | clarkb: finished logging | 21:45 |
| *** rlandy is now known as rlandy|bbl | 21:47 | |
| clarkb | alright time to work on sending teh agenda now | 22:27 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!