| *** tosky has quit IRC | 00:22 | |
| *** DSpider has quit IRC | 00:26 | |
| *** brinzhang_ has quit IRC | 02:39 | |
| fungi | meh, i'm really not going to catch it any more idle, going ahead with the restart momentarily | 02:57 |
|---|---|---|
| fungi | #status log restarted zuul scheduler, now gerrit wip state should prevent zuul from attempting to merge | 03:03 |
| openstackstatus | fungi: finished logging | 03:03 |
| fungi | i'll reenqueue the previously running changes as soon as it's done collecting configuration | 03:04 |
| fungi | looks like it's done, but zuul-web seems unhappy now... i just get proxy errors from apache | 03:08 |
| fungi | i'll give it a minute before i restart it too | 03:09 |
| fungi | no response on http://127.0.0.1:9000/ | 03:11 |
| fungi | restarting it | 03:11 |
| fungi | now it works again | 03:12 |
| fungi | #status log restarted zuul-web because 9000/tcp became unresponsive after the scheduler restart and did not recover on its own | 03:12 |
| openstackstatus | fungi: finished logging | 03:12 |
| fungi | reenqueuing now | 03:13 |
| fungi | and done (there were only 6 changes in total anyway) | 03:13 |
| fungi | i'll try to keep an eye on things for a little while before i disappear for the night | 03:13 |
| *** paladox has quit IRC | 03:30 | |
| *** paladox has joined #opendev | 03:30 | |
| *** jrosser has quit IRC | 04:29 | |
| *** jrosser has joined #opendev | 04:29 | |
| openstackgerrit | Dr. Jens Harbott proposed zuul/zuul-jobs master: Temporarily stop running Gentoo base role tests https://review.opendev.org/c/zuul/zuul-jobs/+/771105 | 07:20 |
| *** brinzhang has joined #opendev | 07:32 | |
| *** paladox has quit IRC | 08:04 | |
| *** dmellado has quit IRC | 08:04 | |
| *** paladox has joined #opendev | 08:04 | |
| *** dmellado has joined #opendev | 08:04 | |
| *** slaweq has joined #opendev | 08:20 | |
| prometheanfire | fungi: I did find a small issue it looks like, a couple | 08:31 |
| prometheanfire | 2021-01-17 08:17:44.333 | Image file test-nodepool-image-20210117.qcow2 created... | 08:31 |
| openstackgerrit | Matthew Thode proposed openstack/diskimage-builder master: add missing packages for python reinstall on gentoo https://review.opendev.org/c/openstack/diskimage-builder/+/771139 | 08:35 |
| openstackgerrit | Matthew Thode proposed openstack/project-config master: allow create-repo-list to be run without six https://review.opendev.org/c/openstack/project-config/+/771140 | 08:40 |
| openstackgerrit | Matthew Thode proposed openstack/project-config master: add gnupg2 to package map for gentoo https://review.opendev.org/c/openstack/project-config/+/771141 | 08:41 |
| prometheanfire | those three reviews should fix it, I think | 08:41 |
| prometheanfire | fixed it for me at least | 08:41 |
| *** brinzhang has quit IRC | 09:16 | |
| *** slaweq has quit IRC | 09:57 | |
| *** DSpider has joined #opendev | 09:59 | |
| *** slaweq has joined #opendev | 10:27 | |
| *** slaweq has quit IRC | 10:29 | |
| *** zoharm has joined #opendev | 10:36 | |
| zoharm | Hi all, I would like to ask for some pointers regarding Cinder driver 3rd party CI. | 10:48 |
| zoharm | We have devstack streamlined to use our storage backend and are able to launch successful tempest runs. My question/request is, what are useful resources documenting the integration points needed for Gerrit, initiating assigned CI runs, and publishing results? | 10:48 |
| zoharm | For example, is a hand crafted CI server a viable option? That would listen to the Gerrit event stream, pick up patchset-create events, launch devstack with the commit, run tempest, upload artifacts, and leave a gerrit review comment. | 10:48 |
| zoharm | Any more recommendatinos or any feedback please? | 10:48 |
| *** slaweq has joined #opendev | 11:04 | |
| *** tosky has joined #opendev | 11:50 | |
| *** zoharm has quit IRC | 12:34 | |
| *** zoharm1 has joined #opendev | 12:34 | |
| *** zoharm1 has quit IRC | 13:52 | |
| *** zoharm has joined #opendev | 13:52 | |
| *** avass has quit IRC | 14:08 | |
| *** yourname has joined #opendev | 14:09 | |
| *** yourname has quit IRC | 14:14 | |
| *** yourname has joined #opendev | 14:18 | |
| fungi | zoharm: yes, you could create a ci system from scratch. note that we don't dictate what you should test or report on, that's up to the cinder folks. but be aware that there are a lot of nuances when it comes to testing proposed changes, especially when a change can depend on another change, perhaps in a completely different project, and testing them together on current branch states rather than the states | 14:29 |
| fungi | the branches were in for the developer's local checkout... we solved a lot of this by creating https://zuul-ci.org/ | 14:29 |
| fungi | but if you want to make a ci system yourself from scratch, we mainly just ask that you make sure it doesn't generate an unnecessary volume of queries against gerrit, that it only reports on changes for projects which have requested that of you, and you'll probably want to make sure it reports results in the same format as zuul does within review comments because we have integration to be able to parse and | 14:30 |
| fungi | summarize those for reviewers | 14:31 |
| *** iurygregory has joined #opendev | 14:31 | |
| fungi | prometheanfire: i've approved those, though the dib change is going to need a release before we can use it, i think? | 14:37 |
| openstackgerrit | Merged openstack/project-config master: allow create-repo-list to be run without six https://review.opendev.org/c/openstack/project-config/+/771140 | 14:47 |
| openstackgerrit | Merged openstack/project-config master: add gnupg2 to package map for gentoo https://review.opendev.org/c/openstack/project-config/+/771141 | 14:47 |
| *** SWAT has joined #opendev | 15:11 | |
| openstackgerrit | Merged openstack/diskimage-builder master: add missing packages for python reinstall on gentoo https://review.opendev.org/c/openstack/diskimage-builder/+/771139 | 15:31 |
| *** zoharm has quit IRC | 15:32 | |
| *** zoharm1 has joined #opendev | 15:32 | |
| *** tosky_ has joined #opendev | 16:05 | |
| *** tosky has quit IRC | 16:05 | |
| *** tosky_ is now known as tosky | 16:06 | |
| *** sshnaidm|afk has quit IRC | 16:12 | |
| *** sshnaidm|afk has joined #opendev | 16:27 | |
| *** sshnaidm|afk has quit IRC | 16:32 | |
| fungi | ianw: frickler: ^ when you're around next, opinions on a quick dib release for that? | 16:33 |
| *** sshnaidm has joined #opendev | 16:54 | |
| *** stevebaker has joined #opendev | 19:00 | |
| *** slaweq has quit IRC | 19:04 | |
| *** slaweq has joined #opendev | 20:55 | |
| *** slaweq has quit IRC | 20:56 | |
| ianw | i'll check out the queue to see what state we're in | 21:16 |
| ianw | ok, mirror-update has a bunch of stuck publish-mirror-logs | 21:25 |
| ianw | reprepro/ubuntu-ports.log.1.1.1.1.1.1.1.1.1.1.1.1 | 21:25 |
| ianw | that's an odd file | 21:25 |
| fungi | indeed, could it be a side effect of me holding locks for all the mirrors | 21:26 |
| fungi | ? | 21:26 |
| ianw | not sure, something must have gone crazy with rotation | 21:26 |
| ianw | ubuntu-ports-focal-updates.log.1.1.1.2.gz.2.gz.2.gz.2.gz.1 | 21:26 |
| fungi | btw, there's another project.tarballs vos release in progress since 01:55 utc, doesn't claim to be a full release this time, probably it's whatever changed recently, but i expect it's slowed by the other transactions still lingering | 21:27 |
| fungi | vos status on afs01.dfw claims there are presently 8 transactions in progress | 21:28 |
| fungi | that almost looks like there were multiple logrotate invocations in quick succession or something | 21:30 |
| fungi | or at the same time? | 21:30 |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: reprepro: only rotate .log files https://review.opendev.org/c/opendev/system-config/+/771153 | 21:31 |
| ianw | fungi: ^ i think that might be it | 21:31 |
| ianw | i've killed all those and will remove anything that's not a .log file | 21:33 |
| fungi | oh, nice. thanks | 21:34 |
| fungi | i wonder why it's only happening recently... i guess we redid the logrotate there not too long ago? | 21:35 |
| ianw | not sure yet ... | 21:37 |
| ianw | running it manually | 21:37 |
| ianw | rsync: mkstemp "/afs/.openstack.org/mirror/logs/reprepro/.debian-docker-bionic-mirror.log.hwc1ik" failed: File too large (27) | 21:37 |
| ianw | ohh, it might make sense why things were hanging if that volume is full | 21:39 |
| fungi | oof | 21:41 |
| ianw | mirror.logs 100000000 58824 0% 82% | 21:41 |
| fungi | debian...bionic | 21:41 |
| fungi | um... | 21:41 |
| ianw | i wonder though if it's file name/path length though | 21:41 |
| * fungi will assume docker folks just don't understand the relationship between debian and ubuntu | 21:42 | |
| fungi | also we can probably try to end any in-progress transactions if they're not for project.tarballs | 21:43 |
| ianw | i feel like this might be related to the large number of very long filenamed files | 21:43 |
| fungi | oh, yeah, there's a file record limit which varies depending on the length of the filenames | 21:43 |
| ianw | i'm just going to rm from the afs partition too | 21:43 |
| ianw | ok a manual run of /usr/local/bin/publish-mirror-logs works now | 21:46 |
| ianw | so i see 8 busy volumes | 21:47 |
| fungi | yup | 21:49 |
| fungi | one of which is the project.tarballs volume with a release underway | 21:49 |
| fungi | the other 7 are associated with vos releases for mirror volumes which were manually killed, but the transactions weren't ended because at the time we had no available slots to execute rpc calls | 21:51 |
| ianw | interestingly more than that are showing as locked | 21:52 |
| ianw | all the following are locked http://paste.openstack.org/show/801705/ | 21:53 |
| ianw | several of those are locked, but don't show up in the busy list of volume | 21:54 |
| fungi | yeah, likely we'll need to delete some locks on volumes with no running transactions | 21:56 |
| ianw | http://paste.openstack.org/show/801706/ | 21:58 |
| ianw | i'll just try clearing all these afs locks and see if we can get things back in sync | 22:01 |
| ianw | i will leave tarballs alone | 22:01 |
| fungi | thanks, tarballs should be locked, there's an actual vos release running (it's just fighting for bandwidth with those other vestigial transactions which haven't ended on their own yet) | 22:03 |
| ianw | they're unlocked now, but the volumes are still busy | 22:07 |
| *** zoharm has joined #opendev | 22:08 | |
| *** zoharm1 has quit IRC | 22:11 | |
| ianw | fungi: are you sure the tarballs release is currently doing anything? rxdebug seems to show afs02.dfw idle | 22:12 |
| fungi | interesting | 22:13 |
| fungi | it's possible the tarballs release is waiting on other transactions to end | 22:13 |
| ianw | vos status shows the active transactions | 22:17 |
| fungi | yeah, mapping those to the volume ids mentioned by vos listvldb, 536871090 is the read-only replica for project.tarballs | 22:18 |
| fungi | that's the first transaction returned for me by vos status | 22:19 |
| ianw | i feel like we should endtrans the rest of them ... | 22:19 |
| ianw | it seems to me the volume list was not in sync with the transactions anyway | 22:20 |
| ianw | like volumes were locked that did not have busy volumes (and presumably transactions running) | 22:20 |
| fungi | the others are read-only volumes for mirror.ubuntu, mirror.fedora, mirror.centos, mirror.ubuntu-ports, mirror.opensuse, mirror.debian, and mirror.yum-puppetlabs | 22:21 |
| fungi | those kicked off via cron before i thought to hold the locks for them | 22:21 |
| ianw | that correlates with http://paste.openstack.org/show/801706/ | 22:21 |
| fungi | also i don't think afs02.dfw is idle, based on the eth0 traffic graph for it in cacti at least | 22:22 |
| ianw | no i agree, rxdebug must just be saying it can respond | 22:22 |
| *** DSpider has quit IRC | 22:42 | |
| fungi | i've submitted a vos endtrans on those other 7 transactions | 22:55 |
| ianw | i think it can take 10+ minutes for anything to happen, if it does. something about slots timing out | 22:55 |
| fungi | though i don't know whether it will abort them | 22:55 |
| fungi | yeah | 22:56 |
| ianw | i'm looking at the ord server with an eye to trying our 1.8 manual update | 22:56 |
| ianw | and moving bits into ansible as i can | 22:56 |
| fungi | might as well, sure | 22:56 |
| ianw | the only thing we have on there is docs | 22:57 |
| ianw | it has a bunch of other readonly volumes that the volume database doesn't think it has | 22:58 |
| ianw | i.e. vos listvol -server afs01.ord.openstack.org shows them, but vos listvldb doesn't show them | 22:59 |
| ianw | i'm going to zap the volumes of afs01.ord that don't have vldb entries to clean this up before attempting an upgrade | 23:11 |
| fungi | sounds fine to me, than | 23:12 |
| fungi | ks | 23:12 |
| fungi | fwiw, none of the transactions i endtrans'd have budged, still showing up in vos status | 23:13 |
| ianw | mirror.deb-hammer.readonly 536870959 RO 1940675 K On-line | 23:14 |
| ianw | mirror.npm.readonly 536870941 RO 609696982 K On-line | 23:14 |
| ianw | mirror.pypi.readonly 536870932 RO 298510820 K On-line | 23:14 |
| ianw | mirror.ubuntu.readonly 536870950 RO 156858708 K On-line | 23:14 |
| ianw | mirror.wheel.cent7x64.readonly 536870956 RO 443230 K On-line | 23:14 |
| ianw | mirror.wheel.trustyx64.readonly 536870947 RO 450824 K On-line | 23:14 |
| ianw | for reference it is ^ on ORD | 23:14 |
| *** tkajinam has quit IRC | 23:17 | |
| *** tkajinam has joined #opendev | 23:17 | |
| ianw | i'm half feeling we should pull 1.8.7 openafs into our ppa | 23:20 |
| ianw | just to start there. but not sure | 23:20 |
| *** tosky has quit IRC | 23:21 | |
| *** tosky_ has joined #opendev | 23:21 | |
| *** tosky_ is now known as tosky | 23:21 | |
| fungi | seems like a reasonable choice. i guess 1.9 is still a bit green | 23:22 |
| ianw | salsa doesn't have it, so might as well stick with what's there | 23:27 |
| fungi | yeah, salsa's only going to have what the debian package maintainer for openafs has pulled to work on packaging | 23:30 |
| fungi | wouldn't be surprised if 1.9 isn't even on their radar yet from that perspective | 23:30 |
| ianw | yep, 1.8.6-5 was pushed as the emergency update into debian | 23:32 |
| ianw | so may as well stick there | 23:32 |
| ianw | removing these old npm and pypi volumes on ord is taking it's time | 23:33 |
| fungi | the other 7 transactions at least have "transactionFlags: delete" now according to vos status | 23:36 |
| openstackgerrit | Merged opendev/system-config master: reprepro: only rotate .log files https://review.opendev.org/c/opendev/system-config/+/771153 | 23:49 |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!