| clarkb | oh its the very grandparent change that complains | 00:00 |
|---|---|---|
| clarkb | I don' | 00:00 |
| clarkb | er I don't understand why that change in particular would trip that check. Does the propose-updates job use that playbook maybe? | 00:00 |
| clarkb | or trigger on it? something like that is my best guess | 00:01 |
| clarkb | oh you know what | 00:01 |
| clarkb | Zuul had a change to check this stuff better before merging | 00:01 |
| clarkb | before it would fail when trying to run the job iirc | 00:01 |
| clarkb | but now zuul can catch this stuff early. This may just be fallout from that. Before it would've failed to run the job at all. But now (since the upgrade over the weekend) we catch it when updating the config itself | 00:02 |
| opendevreview | Ian Wienand proposed openstack/project-config master: Remove publish-service-types-authority dependency https://review.opendev.org/c/openstack/project-config/+/849764 | 00:02 |
| clarkb | corvus: ^ fyi I think we theorized this could happen but figured it was ok since things would've been broken previously | 00:02 |
| clarkb | Did that get a release note in zuul? If not we may want one | 00:02 |
| *** dviroel|rover is now known as dviroel|out | 00:14 | |
| ianw | looks like we don't put a space next to the pin mark on the notice tweets | 00:25 |
| ianw | https://twitter.com/opendevinfra/status/1547233065038135299 | 00:25 |
| ianw | this is triggering my whitespace sensibilities | 00:25 |
| opendevreview | Ian Wienand proposed opendev/statusbot master: twitter: ensure we have a space after the status icon https://review.opendev.org/c/opendev/statusbot/+/849766 | 00:27 |
| opendevreview | Merged openstack/project-config master: Remove publish-service-types-authority dependency https://review.opendev.org/c/openstack/project-config/+/849764 | 00:31 |
| ianw | fungi/clarkb: https://review.opendev.org/q/topic:upload-pypi-api should be ready for review now, to switch the pypi uploads to api token. everything stacks ontop of https://review.opendev.org/c/zuul/zuul-jobs/+/849589 | 00:45 |
| opendevreview | Merged opendev/statusbot master: twitter: ensure we have a space after the status icon https://review.opendev.org/c/opendev/statusbot/+/849766 | 00:49 |
| *** ysandeep|out is now known as ysandeep | 02:46 | |
| opendevreview | Merged opendev/system-config master: Update Gitea to 1.16.9 https://review.opendev.org/c/opendev/system-config/+/849754 | 04:07 |
| opendevreview | Ian Wienand proposed openstack/diskimage-builder master: Revert "containerfile: use focal for testing" https://review.opendev.org/c/openstack/diskimage-builder/+/849274 | 05:16 |
| ianw | infra-prod-service-eavesdrop: timed_out | 06:03 |
| ianw | that's a weird one | 06:03 |
| ianw | afaics, from the log file on bridge, that ran to completion | 06:05 |
| ianw | this is not particularly uncommon :/ https://zuul.opendev.org/t/openstack/builds?job_name=infra-prod-service-eavesdrop&result=TIMED_OUT&skip=0 | 06:09 |
| opendevreview | Ian Wienand proposed opendev/system-config master: production-playbook logs : don't use ansible_date_time https://review.opendev.org/c/opendev/system-config/+/849784 | 06:27 |
| opendevreview | Ian Wienand proposed opendev/system-config master: production-playbook logs : move to post-run step https://review.opendev.org/c/opendev/system-config/+/849785 | 06:34 |
| *** ysandeep is now known as ysandeep|afk | 06:46 | |
| marios | morning folks anyone know whats up with gerrit getting 500 internal server error trying to post comments | 07:39 |
| jm1 | in addition to marios: trying to do a "git review" on opendev projects causes an error "error: remote unpack failed: error No space left on device" | 07:39 |
| jm1 | "fatal: Unpack error, check server log" | 07:40 |
| gtema | same for me - can't push new changes | 07:48 |
| ianw | ah, that's annoying, the disk is full | 07:49 |
| ianw | ok, sorry about that, should be ok | 07:54 |
| jm1 | ianw++ pushing reviews works again. thank you :) | 07:55 |
| gtema | thanks for prompt support ianw, works now | 07:55 |
| ianw | #status log freed up some space on gerrit partition review.opendev.org after full disk errors | 07:57 |
| opendevstatus | ianw: finished logging | 07:57 |
| marios | thanks ianw | 08:04 |
| *** mrunge_ is now known as mrunge | 09:12 | |
| opendevreview | Jonathan Rosser proposed opendev/base-jobs master: Separate swift provider selection from the swift log upload task https://review.opendev.org/c/opendev/base-jobs/+/848881 | 09:21 |
| *** soniya29|ruck is now known as soniya29|ruck|afk | 10:33 | |
| opendevreview | Ian Wienand proposed openstack/diskimage-builder master: Revert "containerfile: use focal for testing" https://review.opendev.org/c/openstack/diskimage-builder/+/849274 | 10:55 |
| *** soniya29|ruck|afk is now known as soniya29|ruck | 11:01 | |
| *** ysandeep|afk is now known as ysandeep | 11:05 | |
| *** dviroel__ is now known as dviroel|rover | 11:15 | |
| *** soniya29|ruck is now known as soniya29|ruck|afk | 11:27 | |
| opendevreview | Merged openstack/project-config master: Add a new openinfra/way project https://review.opendev.org/c/openstack/project-config/+/849576 | 12:13 |
| *** ysandeep is now known as ysandeep|break | 12:38 | |
| *** soniya29|ruck|afk is now known as soniya29|ruck | 12:46 | |
| *** dasm|off is now known as dasm | 12:53 | |
| *** ysandeep|break is now known as ysandeep | 12:56 | |
| *** ysandeep is now known as ysandeep|afk | 13:01 | |
| opendevreview | Merged openstack/project-config master: Add openinfra/way to Zuul https://review.opendev.org/c/openstack/project-config/+/849577 | 14:54 |
| corvus | <clarkb> "Did that get a release note in..." <- I think we tried in https://zuul-ci.org/docs/zuul/latest/releasenotes.html#bug-fixes but maybe we didn't describe it 100%? | 14:58 |
| clarkb | corvus: ya I think there is a third case which is "you may discover new errors when pushing chagnes to previously "happy" repos" | 14:59 |
| corvus | clarkb: i suspect our focus at the time was on in-repo configs of untrusted projects; but config projects may see more errors because they have lots of project stanzas. | 15:01 |
| opendevreview | Clark Boylan proposed opendev/system-config master: Fix system-config-run-review file triggers https://review.opendev.org/c/opendev/system-config/+/849879 | 15:59 |
| opendevreview | Clark Boylan proposed opendev/system-config master: DNM forcing test failure to hold test gerrit https://review.opendev.org/c/opendev/system-config/+/849880 | 15:59 |
| *** ysandeep|afk is now known as ysandeep | 16:04 | |
| opendevreview | Clark Boylan proposed opendev/system-config master: Explicitly disable large Gerrit disk caches https://review.opendev.org/c/opendev/system-config/+/849886 | 16:25 |
| *** ysandeep is now known as ysandeep|out | 16:30 | |
| *** dviroel|rover is now known as dviroel|rover|brb | 16:35 | |
| *** dviroel|rover|brb is now known as dviroel|rover | 17:35 | |
| *** dasm is now known as Guest5013 | 17:59 | |
| *** Guest5013 is now known as dasm | 18:01 | |
| *** rlandy is now known as rlandy|biab | 19:36 | |
| clarkb | infra-root: Likely not complete yet but here is the bionic server upgrade listing with some thoughts on how we can appraoch them https://etherpad.opendev.org/p/opendev-bionic-server-upgrades | 19:39 |
| clarkb | feel free to add entries or notes too | 19:39 |
| *** rlandy|biab is now known as rlandy | 20:29 | |
| corvus | clarkb: added notes about dns | 20:37 |
| clarkb | thanks just saw them | 20:38 |
| ianw | fungi / clarkb: could you have a look at https://review.opendev.org/q/project:opendev%252Fsystem-config+status:open+topic:log-timestamp; i think it will help diagnose some timeouts on the jobs i saw late yesterday | 20:48 |
| clarkb | ianw: yes | 20:55 |
| clarkb | ianw: second change in that topic needs updating | 21:04 |
| clarkb | (details on the chagne itself) | 21:05 |
| opendevreview | Merged opendev/system-config master: Fix system-config-run-review file triggers https://review.opendev.org/c/opendev/system-config/+/849879 | 21:06 |
| *** dasm is now known as dasm|off | 21:23 | |
| ianw | re 849886 seems a bit of deficiency in h2 if it can't clean itself up :/ | 21:56 |
| opendevreview | Ian Wienand proposed opendev/system-config master: production-playbook logs : move to post-run step https://review.opendev.org/c/opendev/system-config/+/849785 | 21:58 |
| ianw | clarkb: ^ doh!, thanks :) | 21:58 |
| *** dviroel|rover is now known as dviroel|rover|afk | 21:59 | |
| *** rlandy is now known as rlandy|bbl | 22:04 | |
| clarkb | ianw: ya according to upstream gerrit it is a known issue with h2 because h2 will only spend 200ms compacting before giving up (and that isn't long enough for the larger files) | 22:09 |
| clarkb | ianw: fungi: I think 849886 is something we keep in our back pocket after we see if we can stabilize with larger disk. | 22:09 |
| clarkb | I think our next steps are: Delete the older of the two index backups currently in the filesystem (that should free another 14GB). Then either replace the cinder volume with a new larger one or add a second volume to increase the size of the lv then extend the ext4 fs on top of that. I think I have a slight preference for replacing the volume since fungi has done that quite a bit | 22:10 |
| clarkb | recently and it seems to work well and is fewer moving parts | 22:10 |
| clarkb | But then we monitor and see if we stabilize consumption on the larger disk and if not proceed iwth 849886 | 22:11 |
| ianw | for my own reference when i'm searching this conversation, the cache docs are | 22:11 |
| ianw | https://gerrit-documentation.storage.googleapis.com/Documentation/3.5.2/config-gerrit.html#cache_names | 22:11 |
| ianw | are we sure that the storage from backup01 is the same type as the one attached to /home/gerrit2? | 22:12 |
| clarkb | Before we can add mor edisk we need to free up the old backup server's cinder volumes so that our quota can be repurposed. Or we can ask mnaser kindly for a quota bump but it sounds like we are ok to clean up those cinder volumes | 22:12 |
| ianw | istr some discussion about about that | 22:13 |
| clarkb | that == more quota or freeing up cinder volumes? | 22:13 |
| ianw | sorry, the storage type selected for /home/gerrit2 | 22:13 |
| clarkb | its currently an nvme ovlume right? I think that was chosen to match the old rax ssd volumetype as closely sa pssible | 22:14 |
| clarkb | considering the amount of disk io that gerrit does I think that is still warranted | 22:14 |
| clarkb | I marked 849886 WIP to make it more explicit that this is our fallback option | 22:20 |
| clarkb | looks like we have quota now after some pruning. I think that means ew can proceed with adding a newer bigger volume. fungi were you still interested in driving that since you've done a number of them recently ( and have a lot more familiarity with lvm than I do) | 22:26 |
| clarkb | /home/gerrit2/index.backup.1643061960 is the older of the two index backups on that filesystem and represents another 14GB that I think we can cleanup too. If you want to rm that before you migrate the data | 22:27 |
| clarkb | (its block level replication though right so not sure if that makes it go faster or not) | 22:27 |
| clarkb | I'm going to try and catch up on the pypi token changes now that we've got something of a plan for gerrit | 22:28 |
| fungi | yeah, i can add an nvme volume (hopefully our quota for that isn't separate), did we want to double or quadruple the size? | 22:29 |
| clarkb | I was thinking 1TB because we ideally also need room for index backups when we do project renames and server upgrades | 22:30 |
| clarkb | I think if we double things may still end up tight | 22:30 |
| fungi | that's fair, yeah | 22:35 |
| ianw | clarkb: i know you're still in catchup/firefight mode but https://review.opendev.org/q/topic:selfsigned-shared-ca is a stack that's also ready | 22:35 |
| ianw | https://review.opendev.org/c/opendev/system-config/+/845316 is tangentially related | 22:35 |
| clarkb | ianw: noted | 22:37 |
| clarkb | I just discovered that gerrit can show you git blame info by accident | 22:53 |
| clarkb | ianw: left a couple of comments on the pypi token changes. The major one is on the last testing change (idea for making it more reliable without comments and commit dances | 23:22 |
| ianw | is it updating the version number with the unix timestamp and always uploading? :) that was a thought i had yesterday | 23:23 |
| clarkb | yes | 23:24 |
| clarkb | using sed in the repo t oset it | 23:24 |
| ianw | haha i guess great minds think alike :) | 23:24 |
| clarkb | ianw: looking at the CA topic briefly I wonder if this could be made more generic and put in zuul-jobs. Specifically the CA bits and maybe the "give me a cert" part. Then have our LE testing tie into that somehow | 23:34 |
| clarkb | bit hand wavy and I ddon't think we need to solve that here now. But something to ponder | 23:34 |
| ianw | maybe if it would be useful for zuul/zuul-registry/etc? | 23:38 |
| clarkb | well more generally I know that devstack also creates a CA (it probably can't farm that out to zuul jobs due to being expected to run locally too) so seems like something that is generally useful | 23:40 |
| opendevreview | Ian Wienand proposed zuul/zuul-jobs master: upload-pypi: always test upload https://review.opendev.org/c/zuul/zuul-jobs/+/849903 | 23:47 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!