| *** DSpider has quit IRC | 00:02 | |
| *** tosky has quit IRC | 00:02 | |
| fungi | zuul' | 00:04 |
|---|---|---|
| fungi | s almost caught up on node requests | 00:04 |
| fungi | down to about 500 pending | 00:05 |
| ianw | ok afs02.dfw upgraded and built, has the key material, going to reboot it | 00:13 |
| ianw | ok, it's up, seems to be running ok | 00:16 |
| ianw | fungi: want to try the mirror.deb-nautilus? | 00:16 |
| fungi | yeah, will start it momentarily | 00:17 |
| fungi | and it's off to the races | 00:17 |
| fungi | Starting ForwardMulti from 536871072 to 536871072 on afs02.dfw.openstack.org (full release). | 00:17 |
| fungi | hopefully will only take a few minutes to complete | 00:18 |
| fungi | and done | 00:19 |
| fungi | not even a few minytes | 00:19 |
| fungi | Released volume mirror.deb-nautilus successfully | 00:19 |
| fungi | can do octopus too if you want, it's a few times larger | 00:19 |
| ianw | nice, yeah i think so. i think i might work on the db servers and leave afs01 for last | 00:19 |
| ianw | after all that, i think try an in-place distro upgrade on afs01.ord to see how that goes. then we can roll that through, theoretically with zero downtime | 00:21 |
| fungi | Released volume mirror.deb-octopus successfully | 00:24 |
| openstackgerrit | Merged opendev/system-config master: Fix review01's fqdn in infratesting https://review.opendev.org/c/opendev/system-config/+/771242 | 00:30 |
| fungi | ianw: shall i go ahead with some of the larger mirror volumes or is that going to conflict with upgrade work? | 00:31 |
| fungi | opensuse and centos are the next smallest at around 200GiB | 00:32 |
| fungi | each | 00:32 |
| ianw | fungi: the only question i guess is if we want to try stopping the queue to update afs01. i note that thanks to clarkb's rebase i've also approved in those gerrit changes too | 00:33 |
| ianw | the one i'd be most worried about having in sync mirrors is tarballs, did that get done? | 00:34 |
| fungi | yeah, tarballs is up to date | 00:41 |
| fungi | my main concern is when services stop on afs01.dfw, it's the only place serving current mirror volumes, so those will all go offline globally and break almost every running job | 00:41 |
| fungi | which given we're running at full tilt, will probably be a lot of collateral damage | 00:42 |
| ianw | agree, my main concern on the other side is that we don't really understand what running 1.6 means at this point. i did think we were thinking it would result in randomish failures | 00:44 |
| ianw | running a 1.6 that has been restarted post timestamp rollover -- which this now has | 00:44 |
| auristor | 1.6 won't result in random failures until a restart | 00:44 |
| auristor | and then it either fails and you restart it again or it doesn | 00:45 |
| ianw | it has been restarted, due to a failure of the cloud provider | 00:45 |
| auristor | then either it failed or it didn't | 00:45 |
| ianw | ok, well i guess it didn't! :) | 00:45 |
| ianw | i still wonder if a gate reset might be worth it to get us completely out of worrying about it all might be a trade-off that is worth it | 00:46 |
| fungi | thanks for the clarification auristor! i guess we rolled lucky dice this time | 00:46 |
| auristor | by the 31st the risk of failure after restart will be reduced to where it was on Sunday | 00:47 |
| fungi | and increasingly less as the timestamp has more and more non-zero bits i guess | 00:47 |
| auristor | the best practice is that before maintenance that the fileserver be vacated, wait for two hours for the client volume location cache to expire, and then perform the maintenance. | 00:48 |
| fungi | yeah, at the moment we have a number of very large volumes which, due to unrelated failures in the cloud service provider we're using, are only current on one server and so we're weighing the cost/benefit of getting those replicas back to normal first before upgrading, so that we don't cause an outage for the volumes in question | 00:50 |
| fungi | but the replication time for them is going to be on the order of a week to complete | 00:50 |
| auristor | I would get those volumes replicated to protect against data loss if the OS upgrade goes south | 00:50 |
| fungi | well, for now we're upgrading the openafs packages, os upgrades will come later once that's finished | 00:51 |
| fungi | i think we're less worried about waiting for the os upgrades | 00:51 |
| ianw | yeah, os upgrades we can round-robbin once this is all over | 00:51 |
| fungi | we need to get operating systems upgraded before ~april | 00:52 |
| auristor | since this is all cloud, I would stand up a third fileserver using the new OS version, then when that is operational, move the RWs to that server and add RO sites to that server, then remsite the volumes from afs01.dfw, wait two hours, turn off the afs01.dfw fileserver, reinstall the OS and deploy a new fileserver, then addsite to that server, release to that server, then decommission afs02.dfw | 00:53 |
| fungi | at the moment the focus is on upgrading from openafs 1.6 to 1.8 w/ the timestamp bug fixes | 00:53 |
| auristor | at the moment there exists a single point of failure that that IMO should be the priority | 00:54 |
| fungi | yeah, that's not a terrible idea for the os upgrades, we're mostly concerned there about the pain of changing ip addresses when replacing servers (it's like you say, basically adding and removing servers from the cluster rather than straight up replacing a server with a new build) | 00:54 |
| auristor | IP addresses can change willy nilly for fileservers. its the location servers that cannot change | 00:55 |
| fungi | and yes, i think the current spof is the greater risk, especially what with a service provider whose iron (both for storage and virtual machines) likes to act up frequently | 00:55 |
| auristor | A client is told where volumes reside with a list of fileserver UUIDs not IP addresses. The clients query the location service to resolve UUID to IP address. The fileserver registers it current IP addresses for its UUID each time it restarts. | 00:56 |
| fungi | so i'm inclined to not restart services on afs01.dfw if we can avoid it while trying to get the remaining volumes replicated again | 00:56 |
| ianw | fungi: my issue there is that if it shuts down again, we again roll dice restarting it. if we took a short outage to update it, we can do the replications with more confidence IMO | 00:57 |
| auristor | and it it fails to restart (by which the failure is an inability to register with the location service), you restart it again until it works | 00:58 |
| fungi | ianw: can you estimate the service downtime you saw for the other fileservers? maybe if it's just a minute or so then the number of job failures won't be too great | 00:58 |
| ianw | fungi: it's really just the time to reboot the server; on that order of a minute or so | 00:58 |
| fungi | yeah, the risk with restarts, as we've seen, is that the provider might have a problem and none of us is awake/paying attention for some hours | 00:59 |
| ianw | also, just from a code-motion POV, that allows us to cull the puppet bits, simplify the groups as i work on the db servers | 00:59 |
| fungi | so it could come up with the service failing to start and then sit like that for hours until someone discovers everything's broken | 00:59 |
| ianw | well yeah, i wouldn't like fricker to have to parse all this almost cold turkey :) | 01:00 |
| fungi | we're down to around a 250 node request backlog in zuul, maybe once all builds have their node assignments we give it a few minutes and then upgrade? hopefully that minimizes the impact on serving mirror volumes | 01:01 |
| fungi | running jobs tend to only hit package mirrors in the first few minutes anyway | 01:01 |
| ianw | fungi: if that's the general process we're agreeing on, i should install the PPA, build the modules and get the system-config change ready | 01:02 |
| ianw | the only other thing we might like to queue up is if we stop afs01 and then quickly recreate the LVM snapshot. that should ensure at the worst case we can go back to that. but we've now done two other servers that are working fine | 01:03 |
| fungi | yeah, not a bad idea | 01:04 |
| fungi | basically i just want to avoid undue disruption to the running builds, while also getting sync back on track for the read-only replicas on afs02.dfw, but not at the expense of creating or blocking more work for any of us if we can help it | 01:05 |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: Move afs01.dfw into afs-1.8 group https://review.opendev.org/c/opendev/system-config/+/771293 | 01:05 |
| fungi | since the only thing we're more strapped for than build resources right now is people | 01:06 |
| openstackgerrit | Merged opendev/system-config master: run-selenium: run selenium on a node https://review.opendev.org/c/opendev/system-config/+/767078 | 01:07 |
| ianw | fungi: yeah, i think if we get things ready, we've got only a small window of reboot time. i then feel better about letting the replications run as they need to with the fileservers at least consistent | 01:09 |
| ianw | i need to just feed people here, bib | 01:09 |
| openstackgerrit | Merged opendev/system-config master: gerrit: Initalize in testing https://review.opendev.org/c/opendev/system-config/+/765224 | 01:11 |
| openstackgerrit | Merged opendev/system-config master: gerrit: move plugins to common code https://review.opendev.org/c/opendev/system-config/+/767269 | 01:12 |
| openstackgerrit | Merged opendev/system-config master: openafs-server: ensure vos_release keys installed on new servers https://review.opendev.org/c/opendev/system-config/+/771284 | 01:12 |
| fungi | oh, i've also been holding a lock for updates to the mirror.deb-docker volume, which is tiny so we could also use that as initial testing after the afs01.dfw restart, before trying to resume updates for larger mirror volumes | 01:21 |
| fungi | i guess we're not tracking that one in grafana either | 01:21 |
| openstackgerrit | Merged opendev/system-config master: bazelisk-build: specify targets as list https://review.opendev.org/c/opendev/system-config/+/767272 | 01:23 |
| openstackgerrit | Merged opendev/system-config master: Move afs02.dfw.openstack.org to afs-1.8 group https://review.opendev.org/c/opendev/system-config/+/771285 | 01:23 |
| ianw | infra-prod-service-afs (3. attempt) wasn't happy, let me debug | 01:32 |
| ianw | warning: failed to remove playbooks/filter_plugins/__pycache__/getaddrinfo.cpython-36.pyc: Permission denied | 01:32 |
| ianw | sigh, i guess that ansible running out of system-config puts in .pyc files | 01:33 |
| ianw | hang on, if we git ignore these, then "git clean" will ignore them ...? | 01:34 |
| ianw | hrm, already ignore *.pyc | 01:35 |
| ianw | no that's silly, git clean is explicitly removing things we ignore as they're untracked | 01:37 |
| openstackgerrit | xinliang proposed openstack/diskimage-builder master: Add rhel support for iscsi-boot https://review.opendev.org/c/openstack/diskimage-builder/+/770702 | 01:39 |
| fungi | yeah, untracked files should be okay as they wouldn't prevent pulling | 01:45 |
| fungi | or being pushed into by zuul | 01:45 |
| ianw | this is not really what i wanted to be debugging :/ | 01:47 |
| fungi | we could consider reverting that in zuul-jobs as it's had unanticipated side effects | 01:48 |
| ianw | i do agree it's right; but maybe it needs to become: yes | 01:49 |
| fungi | at first blush, that seems like a suitable compromise | 01:51 |
| ianw | although this got reverted https://opendev.org/zuul/zuul-jobs/commit/1e92a67db6f5fa3f3284d5b1928f104c428187f3 ; it does show that there's some history of become: yes in there | 01:52 |
| fungi | i suppose we only want to become for a git clean? | 01:58 |
| ianw | yeah i'm trying splitting it | 01:58 |
| *** ysandeep|away is now known as ysandeep | 02:07 | |
| openstackgerrit | Ian Wienand proposed zuul/zuul-jobs master: mirror-workspace-git-repos: run clean as root https://review.opendev.org/c/zuul/zuul-jobs/+/771297 | 02:17 |
| ianw | fungi: it looks like node requests have really dropped. i feel like we should put afs01.dfw in emergency and do the upgrade soonish, and get to releasing volumes with everything consistent | 02:21 |
| fungi | yeah, i'm good with that. i'll add it to the emergency disable list now | 02:22 |
| ianw | i'm going to install the ppa and build the modules. we can look at the snapshot too | 02:22 |
| fungi | and added | 02:22 |
| fungi | i presume we can blow away the previous lvm snapshot | 02:24 |
| ianw | right, so lvremove it, shutdown afs server and recreate it | 02:26 |
| fungi | do you have the lvcreate command you used last time handy? if not i can whip something up | 02:28 |
| ianw | oh, unmount it first | 02:28 |
| fungi | yup | 02:29 |
| fungi | i'll do that now | 02:29 |
| ianw | Do you really want to remove and DISCARD active logical volume vicepa_snap? [y/n]: | 02:29 |
| ianw | agree yes? | 02:30 |
| fungi | oh, no need to umount, it's not mounted. probably because of the reboot | 02:31 |
| fungi | and yeah, you can agree with that | 02:32 |
| fungi | or you could lvchange -an that volume first | 02:32 |
| ianw | lvcreate -l100%FREE -s -n vicepa_snap /dev/main/vicepa | 02:32 |
| ianw | is the command to make it | 02:32 |
| fungi | which avoids the prompt | 02:32 |
| ianw | modules are built | 02:32 |
| fungi | for future reference | 02:32 |
| ianw | ok, i'm going to stop afs, create snapshot and reboot as fast as possible | 02:33 |
| fungi | i guess we want to stop afs services and umount /vicepa before creating the snapshot? | 02:33 |
| ianw | ok, yeah unmount it too | 02:33 |
| fungi | yeah | 02:33 |
| fungi | that helps ensure it's quiescent | 02:33 |
| ianw | done, rebooting | 02:36 |
| fungi | in retrospect we probably should have also touched /fastboot | 02:40 |
| *** hemanth_n has joined #opendev | 02:40 | |
| ianw | it should all be back, poking | 02:42 |
| ianw | fungi: i think we're good. do you want to try a smaller release? | 02:43 |
| fungi | yup, on it | 02:43 |
| fungi | Released volume mirror.deb-docker successfully | 02:46 |
| fungi | several times because that one's a loop | 02:46 |
| fungi | i can also do mirror.apt-puppetlabs quickly, another we're not tracking in the dashboard | 02:47 |
| ianw | yeah, i need a script to keep things in order | 02:47 |
| fungi | Released volume mirror.apt-puppetlabs successfully | 02:49 |
| fungi | looks like things are still working | 02:50 |
| ianw | nice. well at this point our deployment jobs are borked | 02:50 |
| ianw | so even if we wanted to merge the sequential release change i have, it wouldn't get onto the servers ATM | 02:51 |
| ianw | so i think fungi, if you want to just manually sequentially release volumes we can hopefully get back into sync | 02:51 |
| fungi | i can still access volumes which are only current on afs01.dfw so i think we can move forward | 02:52 |
| fungi | and yeah, i'll queue some up to run sequentially in screen now | 02:53 |
| ianw | at this point afs01|02.dfw are in emergency. i think that's the best way to leave it. when deployment happens we can double check again afs02.ord is happy and then remove them from emergency | 02:56 |
| ianw | i'll cleanup system-config and write the afsdb ansible bits now | 02:56 |
| fungi | reprepro-mirror-update /etc/reprepro/ubuntu mirror.ubuntu 2>&1 | tee /var/log/reprepro/ubuntu.log ; reprepro-mirror-update /etc/reprepro/ubuntu-ports mirror.ubuntu-ports 2>&1 | tee /var/log/reprepro/ubuntu-ports.log ; reprepro-mirror-update /etc/reprepro/ubuntu-cloud-archive mirror.ubuntu-cloud 2>&1 | tee /var/log/reprepro/ubuntu-cloud-archive.log ; centos-mirror-update mirror.centos 2>&1 | tee | 03:04 |
| fungi | /var/log/rsync-mirrors/centos.log ; opensuse-mirror-update mirror.opensuse 2>&1 | tee /var/log/rsync-mirrors/opensuse.log ; fedora-mirror-update mirror.fedora 2>&1 | tee /var/log/rsync-mirrors/fedora.log ; reprepro-mirror-update /etc/reprepro/debian mirror.debian 2>&1 | tee /var/log/reprepro/debian.log ; reprepro-mirror-update /etc/reprepro/debian-security mirror.debian-security 2>&1 | tee | 03:04 |
| fungi | /var/log/reprepro/debian-security.log | 03:05 |
| fungi | that's what i'm running | 03:05 |
| fungi | apologies for the length | 03:05 |
| fungi | okay, starting that now | 03:07 |
| ianw | ++ | 03:07 |
| *** jhesketh_ has joined #opendev | 03:40 | |
| *** jhesketh has quit IRC | 03:41 | |
| *** jhesketh_ is now known as jhesketh | 03:43 | |
| fungi | looks like i forgot the epel mirror in that, but otherwise should be all of them | 03:45 |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: Remove afs-1.8 group https://review.opendev.org/c/opendev/system-config/+/771293 | 03:52 |
| *** ysandeep is now known as ysandeep|pto | 03:55 | |
| openstackgerrit | Ian Wienand proposed zuul/zuul-jobs master: mirror-workspace-git-repos: run clean as root https://review.opendev.org/c/zuul/zuul-jobs/+/771297 | 04:01 |
| *** lbragstad has quit IRC | 04:16 | |
| *** ykarel has joined #opendev | 04:18 | |
| *** hemanth_n has quit IRC | 04:40 | |
| *** hemanth_n has joined #opendev | 04:44 | |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: Manage afsdb servers with Ansible https://review.opendev.org/c/opendev/system-config/+/771340 | 04:53 |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: Manage afsdb servers with Ansible https://review.opendev.org/c/opendev/system-config/+/771340 | 05:14 |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: Remove AFS puppet https://review.opendev.org/c/opendev/system-config/+/771342 | 05:14 |
| openstackgerrit | Ian Wienand proposed zuul/zuul-jobs master: mirror-workspace-git-repos: run clean as root https://review.opendev.org/c/zuul/zuul-jobs/+/771297 | 05:21 |
| openstackgerrit | Merged opendev/system-config master: gerrit: get files from bazel build dir https://review.opendev.org/c/opendev/system-config/+/767433 | 05:29 |
| openstackgerrit | Merged opendev/system-config master: gerrit: Install zuul-summary-results plugin https://review.opendev.org/c/opendev/system-config/+/767079 | 05:30 |
| *** guillaumec has quit IRC | 05:40 | |
| ianw | since it's quiet, i might deploy ^ to get the plugin | 05:43 |
| *** guillaumec has joined #opendev | 05:44 | |
| ianw | #status log restarted gerrit to get zuul-summary-results; see also http://lists.openstack.org/pipermail/openstack-discuss/2021-January/019885.html\ | 06:00 |
| openstackstatus | ianw: finished logging | 06:00 |
| ykarel | ianw, you mean summary visible around Files tab? i don't see it | 06:04 |
| ykarel | i see two tabs: Files and Findings | 06:05 |
| ykarel | zuul summary will be around those only, not sure /me looking correctly | 06:05 |
| ianw | ykarel: you might need a shift-reload | 06:05 |
| ykarel | ianw, i did that but still don't see | 06:06 |
| ykarel | let me try on different browser | 06:06 |
| ykarel | okk i see new tab in chrome | 06:07 |
| ykarel | may be some issue on my firefox | 06:07 |
| ianw | i use firefox and am seeing it there, and also just checked on chrome | 06:07 |
| ianw | it's very basic so wouldn't imagine browser would matter much | 06:08 |
| ykarel | ok, i see SyntaxError: "invalid regexp group" likely that causing it | 06:14 |
| *** ykarel_ has joined #opendev | 06:16 | |
| ykarel_ | ok, i see SyntaxError: "invalid regexp group" likely that causing it | 06:17 |
| *** ykarel has quit IRC | 06:19 | |
| *** hemanth_n has quit IRC | 06:23 | |
| *** marios has joined #opendev | 06:23 | |
| *** hemanth_n has joined #opendev | 06:23 | |
| *** ykarel_ is now known as ykarel | 06:29 | |
| ianw | hrm, what version of firefox? | 06:33 |
| *** sboyron has joined #opendev | 06:33 | |
| *** brinzhang has quit IRC | 06:50 | |
| *** brinzhang has joined #opendev | 06:51 | |
| ykarel | firefox-70.0-1.fc29.x86_64 | 07:04 |
| *** fressi has joined #opendev | 07:24 | |
| *** DSpider has joined #opendev | 07:24 | |
| *** ralonsoh has joined #opendev | 07:27 | |
| *** fressi has quit IRC | 07:31 | |
| *** DSpider has quit IRC | 07:31 | |
| *** brinzhang has quit IRC | 07:32 | |
| *** hemanth_n has quit IRC | 07:32 | |
| *** DSpider has joined #opendev | 07:32 | |
| *** hemanth_n has joined #opendev | 07:33 | |
| *** fressi has joined #opendev | 07:38 | |
| *** lpetrut has joined #opendev | 07:39 | |
| *** eolivare has joined #opendev | 07:47 | |
| *** slaweq has joined #opendev | 07:48 | |
| *** fressi has quit IRC | 07:55 | |
| *** fressi has joined #opendev | 07:55 | |
| *** fressi has quit IRC | 07:56 | |
| *** fressi has joined #opendev | 08:05 | |
| *** fressi has quit IRC | 08:06 | |
| *** rpittau|afk is now known as rpittau | 08:07 | |
| *** brinzhang has joined #opendev | 08:07 | |
| *** andrewbonney has joined #opendev | 08:13 | |
| *** hashar has joined #opendev | 08:22 | |
| *** fressi has joined #opendev | 08:26 | |
| *** tosky has joined #opendev | 08:39 | |
| *** jpena|off is now known as jpena | 08:58 | |
| *** sboyron has quit IRC | 09:06 | |
| *** sboyron_ has joined #opendev | 09:06 | |
| openstackgerrit | Lee Yarwood proposed opendev/elastic-recheck master: Add query for bug 1912310 https://review.opendev.org/c/opendev/elastic-recheck/+/771388 | 09:13 |
| openstack | bug 1912310 in OpenStack Compute (nova) "libvirt.libvirtError: unable to connect to server at " [Undecided,New] https://launchpad.net/bugs/1912310 | 09:13 |
| ianw | 03:27:38 -> 08:59:54 for a full ubuntu release | 09:23 |
| *** brinzhang has quit IRC | 09:29 | |
| *** sboyron has joined #opendev | 09:43 | |
| *** sboyron_ has quit IRC | 09:43 | |
| openstackgerrit | Hervé Beraud proposed openstack/project-config master: Adding irc notification for missing oslo projects https://review.opendev.org/c/openstack/project-config/+/771392 | 09:49 |
| guillaumec | ianw, ykarel https://developer.mozilla.org/en-US/docs/Mozilla/Firefox/Releases/78#javascript "Named capture groups" | 09:50 |
| *** hashar is now known as hasharOut | 09:59 | |
| *** sboyron_ has joined #opendev | 10:05 | |
| *** sboyron has quit IRC | 10:08 | |
| openstackgerrit | Martin Kopec proposed opendev/system-config master: WIP Deploy refstack with ansible docker https://review.opendev.org/c/opendev/system-config/+/705258 | 10:17 |
| *** brinzhang has joined #opendev | 10:28 | |
| ykarel | guillaumec, Thanks, so will need to update firefox | 10:59 |
| ykarel | after updating firefox no longer see that syntax error and Zuul Summary is visible | 11:04 |
| *** ysandeep|pto is now known as ysandeep | 11:14 | |
| openstackgerrit | Martin Kopec proposed opendev/system-config master: WIP Deploy refstack with ansible docker https://review.opendev.org/c/opendev/system-config/+/705258 | 11:18 |
| *** marios has quit IRC | 11:57 | |
| *** dpawlik has quit IRC | 11:57 | |
| *** logan- has quit IRC | 11:57 | |
| *** avass has quit IRC | 11:57 | |
| *** paladox has quit IRC | 11:57 | |
| *** paladox has joined #opendev | 11:57 | |
| *** avass has joined #opendev | 11:58 | |
| *** lpetrut_ has joined #opendev | 11:58 | |
| *** marios has joined #opendev | 11:58 | |
| *** logan- has joined #opendev | 12:00 | |
| *** lpetrut has quit IRC | 12:01 | |
| openstackgerrit | Gorka Eguileor proposed zuul/zuul-jobs master: Fix CentOS wheel mirror URL https://review.opendev.org/c/zuul/zuul-jobs/+/771428 | 12:02 |
| *** eolivare_ has joined #opendev | 12:14 | |
| *** dpawlik has joined #opendev | 12:15 | |
| *** eolivare has quit IRC | 12:16 | |
| *** artom has quit IRC | 12:28 | |
| *** artom has joined #opendev | 12:28 | |
| *** jpena is now known as jpena|lunch | 12:34 | |
| *** hasharOut is now known as hashar | 12:45 | |
| *** ttx has quit IRC | 12:51 | |
| *** hemanth_n has quit IRC | 12:52 | |
| *** eolivare_ has quit IRC | 12:53 | |
| *** fressi has quit IRC | 12:56 | |
| *** ykarel has quit IRC | 13:01 | |
| *** ttx has joined #opendev | 13:04 | |
| openstackgerrit | Luigi Toscano proposed openstack/project-config master: cursive: prepare to move the jobs in-tree https://review.opendev.org/c/openstack/project-config/+/771443 | 13:12 |
| openstackgerrit | Gorka Eguileor proposed zuul/zuul-jobs master: Fix CentOS wheel mirror URL https://review.opendev.org/c/zuul/zuul-jobs/+/771428 | 13:13 |
| *** ykarel has joined #opendev | 13:24 | |
| openstackgerrit | Guillaume Chauvel proposed opendev/system-config master: Increase comment log text width to avoid line wrap https://review.opendev.org/c/opendev/system-config/+/771445 | 13:25 |
| *** eolivare_ has joined #opendev | 13:26 | |
| *** jpena|lunch is now known as jpena | 13:34 | |
| *** lbragstad has joined #opendev | 13:37 | |
| *** _mlavalle_1 has quit IRC | 13:58 | |
| *** mlavalle has joined #opendev | 13:59 | |
| *** fressi has joined #opendev | 14:04 | |
| *** artom has quit IRC | 14:10 | |
| openstackgerrit | Guillaume Chauvel proposed opendev/system-config master: Increase comment log text width to avoid line wrap https://review.opendev.org/c/opendev/system-config/+/771445 | 14:25 |
| *** slittle1 has quit IRC | 14:32 | |
| *** artom has joined #opendev | 14:32 | |
| *** artom has quit IRC | 14:32 | |
| *** slittle1 has joined #opendev | 14:33 | |
| *** artom has joined #opendev | 14:33 | |
| slittle1 | Morning all | 14:35 |
| slittle1 | could we get eyes on https://review.opendev.org/c/openstack/project-config/+/771235 ? | 14:36 |
| slittle1 | If acceptable, please set up Cole Walker and myself as cores. Thanks, Scott Little | 14:36 |
| *** rosmaita has left #opendev | 14:48 | |
| *** fressi has left #opendev | 14:51 | |
| *** chandankumar is now known as raukadah | 15:08 | |
| fungi | slittle1: oh, yep, i saw that go by yesterday but was firefighting, i should have a moment to look now. thanks for the reminder! | 15:09 |
| openstackgerrit | Merged openstack/project-config master: Add PTP Notification app to StarlingX https://review.opendev.org/c/openstack/project-config/+/771235 | 15:22 |
| *** hashar is now known as hasharKids | 15:28 | |
| *** sshnaidm|ruck is now known as sshnaidm|afk | 15:37 | |
| *** ysandeep is now known as ysandeep|dinner | 15:37 | |
| openstackgerrit | Merged opendev/gerritbot master: Skip notifications about WIP changes https://review.opendev.org/c/opendev/gerritbot/+/765130 | 15:46 |
| *** ysandeep|dinner is now known as ysandeep | 15:47 | |
| *** brinzhang has quit IRC | 15:48 | |
| *** brinzhang has joined #opendev | 15:51 | |
| *** ykarel is now known as ykarel|away | 16:00 | |
| *** diablo_rojo has joined #opendev | 16:01 | |
| clarkb | docker closed my ipv6 bug says it is too complicated and risky | 16:03 |
| clarkb | I guess that makes it official | 16:03 |
| fungi | that is, docker thinks ipv6 raw addresses aren't worth supporting? | 16:03 |
| clarkb | apparently | 16:03 |
| *** sshnaidm|afk is now known as sshnaidm|ruck | 16:03 | |
| slittle1 | I think zuul is unhappy with the merge of https://review.opendev.org/c/openstack/project-config/+/771235, check out https://zuul.opendev.org/t/openstack/build/ea9b8fbb985b456fa0721f71de6f4f44 | 16:04 |
| clarkb | fungi: I think you were looking at something like ^ yesterday? | 16:05 |
| clarkb | (I haven't quite caught up on that yet) | 16:05 |
| fungi | clarkb: ahh, yes, there was a change which merged to zuul-jobs yesterday to force a git clean, but when we run python scripts (either directly or ansible modules) from the zuul-deployed git repo as root it breaks zuul's ability to clean the repo even though the files are untracked | 16:07 |
| clarkb | and I guess it is zuul trying to remove a rile owned by root? | 16:07 |
| openstackgerrit | Guillaume Chauvel proposed opendev/system-config master: Increase comment log text width to avoid line wrap https://review.opendev.org/c/opendev/system-config/+/771445 | 16:07 |
| fungi | ianw has proposed a compromise to run the git clean as root using become, but obviously that needs help on nodes where zuul lacks root privs | 16:07 |
| fungi | yeah, zuul's git clean is trying to remove bytecode files created by the python interpreter which were created as root when run | 16:08 |
| clarkb | fungi: have a link to the original change? I'm curious to see what precipitated that | 16:08 |
| clarkb | (I can dig it up if not) | 16:08 |
| fungi | clarkb: https://review.opendev.org/771220 | 16:09 |
| fungi | maybe the git clean could be best effort and the task would just ignore nonzero exit from that | 16:10 |
| clarkb | ah we were always cleaning but now a failed clean is an error. Got it (and I guess that makes esnse overall) | 16:10 |
| clarkb | ya maybe we need to bring that up with tobiash? | 16:10 |
| fungi | i think discussion is happening in ianw's change, finding... | 16:10 |
| fungi | clarkb: https://review.opendev.org/771297 | 16:11 |
| clarkb | hrm that will fail in a lot of cases I bet (like when using containers without root?) | 16:12 |
| fungi | i've made that suggestion in a review comment now | 16:14 |
| *** ykarel|away has quit IRC | 16:17 | |
| *** CWalker has joined #opendev | 16:29 | |
| *** lbragstad_ has joined #opendev | 16:46 | |
| openstackgerrit | Jeremy Stanley proposed zuul/zuul-jobs master: mirror-workspace-git-repos: ignore errors from clean https://review.opendev.org/c/zuul/zuul-jobs/+/771297 | 16:47 |
| *** lpetrut_ has quit IRC | 16:48 | |
| *** slaweq has quit IRC | 16:48 | |
| *** DSpider has quit IRC | 16:48 | |
| *** DSpider has joined #opendev | 16:48 | |
| *** slaweq has joined #opendev | 16:49 | |
| *** lbragstad has quit IRC | 16:49 | |
| *** lbragstad_ is now known as lbragstad | 16:58 | |
| fungi | with python 3.8 and later there's an envvar you can use to control the bytecode path | 16:59 |
| fungi | https://github.com/python/cpython/commit/fc96437 | 17:00 |
| fungi | maybe we could set PYTHONPYCACHEPREFIX in /etc/environment on bridge.o.o? but not until we're using 3.8 of course | 17:01 |
| clarkb | it is 3.6 currently right? | 17:02 |
| fungi | yeah, 3.6.9 on bionic | 17:03 |
| fungi | there is a python3.8 package in bionic-updates we could try | 17:04 |
| fungi | but it won't be the default python3 | 17:04 |
| *** brinzhang_ has joined #opendev | 17:09 | |
| *** mlavalle has quit IRC | 17:11 | |
| *** sshnaidm|ruck is now known as sshnaidm|afk | 17:11 | |
| *** brinzhang has quit IRC | 17:12 | |
| *** mlavalle has joined #opendev | 17:12 | |
| *** zbr3 has joined #opendev | 17:18 | |
| *** zbr has quit IRC | 17:20 | |
| *** zbr3 is now known as zbr | 17:20 | |
| openstackgerrit | James E. Blair proposed opendev/base-jobs master: Force ownership of zuul repo dirs in opendev CD jobs https://review.opendev.org/c/opendev/base-jobs/+/771496 | 17:24 |
| *** marios is now known as marios|out | 17:30 | |
| openstackgerrit | James E. Blair proposed opendev/base-jobs master: Force ownership of zuul repo dirs in opendev CD jobs https://review.opendev.org/c/opendev/base-jobs/+/771496 | 17:33 |
| openstackgerrit | James E. Blair proposed opendev/base-jobs master: Force ownership of zuul repo dirs in opendev CD jobs https://review.opendev.org/c/opendev/base-jobs/+/771496 | 17:36 |
| *** hamalq has joined #opendev | 17:40 | |
| clarkb | corvus: fungi I've +2'd it but not approved it because I don't think I'm in a good spot for monitoring it | 17:43 |
| clarkb | I've got to prep for the meeting momentarily and have a couple of other things to finish up | 17:43 |
| corvus | clarkb: any reason not to +w and just check back in a couple hours? | 17:44 |
| corvus | it should either break like now or work like before? | 17:44 |
| clarkb | well its modifying bridge as well? | 17:45 |
| clarkb | maybe tahts just me being overly apranoid | 17:45 |
| mordred | clarkb, corvus : base-jobs patch lgtm | 17:45 |
| *** d34dh0r53 has quit IRC | 17:47 | |
| fungi | i'm happy to approve it and keep an eye on things | 17:47 |
| fungi | the sooner we see if we're unblocked the sooner we can get other stuff deployed again | 17:47 |
| clarkb | fungi: thanks | 17:47 |
| *** d34dh0r53 has joined #opendev | 17:50 | |
| openstackgerrit | Merged opendev/base-jobs master: Force ownership of zuul repo dirs in opendev CD jobs https://review.opendev.org/c/opendev/base-jobs/+/771496 | 17:55 |
| fungi | okay, now that's merged, fingers crossed. i'll try to reenqueue 771235 into deploy... that should be safe right? | 17:56 |
| clarkb | fungi: as long as no other changes have merged after it (you'll want to use the most recent one I think) | 17:56 |
| clarkb | on the system-config side I believe we always checkout master but for project-config I think it is tied to the commit | 17:56 |
| fungi | yeah, that's the most recent project-config change to merge | 17:57 |
| fungi | sudo zuul enqueue --tenant openstack --pipeline deploy --project openstack/project-config --change 771235,2 | 17:59 |
| fungi | trying that | 17:59 |
| fungi | it's waiting for the semaphore | 18:00 |
| *** eolivare_ has quit IRC | 18:02 | |
| *** jpena is now known as jpena|off | 18:18 | |
| *** dtantsur is now known as dtantsur|afk | 18:23 | |
| *** artom has quit IRC | 18:30 | |
| openstackgerrit | Jeremy Stanley proposed opendev/base-jobs master: Use root to correct ownership of repos https://review.opendev.org/c/opendev/base-jobs/+/771503 | 18:33 |
| fungi | corvus: clarkb: mordred: ^ the failure looks like https://zuul.opendev.org/t/openstack/build/5e954c5436c241979305560cc8826f9d/console#1/2/2/bridge.openstack.org | 18:33 |
| corvus | fungi: so sorry! i had that in there but must have accidentally deleted it in a revision :( | 18:34 |
| fungi | oh no worries | 18:34 |
| fungi | i should have spotted it in review | 18:34 |
| fungi | will try again once that's in | 18:34 |
| clarkb | approved | 18:34 |
| corvus | in my defense, i didn't get much sleep last night due to unseasonal howling winds | 18:34 |
| fungi | thanks! | 18:34 |
| *** hasharKids has quit IRC | 18:34 | |
| corvus | it's basically fire season again here (hot winds) right in the middle of what's supposed to be rainy season | 18:35 |
| fungi | i can completely sympathize wrt howling winds. here we can feel the house swaying with every gust | 18:35 |
| mordred | corvus: are you having fire-rain yet? | 18:35 |
| corvus | mordred: i think that's yosemite in early februrary. and the fire-tornadoes are still farther north. | 18:35 |
| fungi | between there being no appreciable wind break when you're right on the water, and the house resting on 4m tall pilings | 18:35 |
| mordred | corvus: I mean - that's in a normal year. who knows - maybe you'll get lucky and get fire-tornadoes this year? | 18:36 |
| corvus | fungi: i can imagine. i felt the house move just a little with some of the winds last night. i'm used to it only moving when there's a quake on the fault up the street. | 18:36 |
| corvus | mordred: if 2021 wants to one-up 2020 (which it's starting to look like) i wouldn't put it past it. | 18:37 |
| fungi | i just imagine the house is trying to rock me to sleep | 18:37 |
| mordred | corvus: if the west coast is starting to suffer from damaging high winds, I worry that we're going to start having earthquakes here | 18:37 |
| fungi | 2021, the year of the fire hurricanes | 18:37 |
| corvus | fungi: oh that's so much better than rocking awake | 18:37 |
| mnaser | clarkb: for the upcoming infra meeting, i have another meeting from 2 to 2:30, would it be possible to have my discussion point earlier in the start of the meeting (i don't think it will take much time) | 18:38 |
| clarkb | the only time I realized I was in an earthquake it was because my bed felt like it had transported to the ocean for about 20 seconds at 2am | 18:38 |
| clarkb | mnaser: would it be easier at the end from 2:30- 3? | 18:39 |
| fungi | maybe he meant he had a meeting from 2:30-3 | 18:39 |
| mnaser | clarkb: sigh i goofed, i meant my meeting is 2:30 to 3, so if we could discuss it 2:00 to 2:30 (et) | 18:39 |
| mnaser | yes, sorry, slow day :) | 18:39 |
| fungi | welcome to the club ;) | 18:39 |
| clarkb | mnaser: yes we can make that work | 18:39 |
| *** rpittau is now known as rpittau|afk | 18:41 | |
| openstackgerrit | Merged opendev/base-jobs master: Use root to correct ownership of repos https://review.opendev.org/c/opendev/base-jobs/+/771503 | 18:45 |
| fungi | okay, trying again | 18:45 |
| fungi | it's been running a while, so not failing at the start like before at least. good sign | 18:52 |
| *** artom has joined #opendev | 18:53 | |
| *** marios|out has quit IRC | 18:53 | |
| fungi | succeeded! | 18:54 |
| clarkb | woot | 18:54 |
| avass | fungi: \o/ | 18:56 |
| CWalker | Thank you for fixing that! :) | 18:57 |
| fungi | CWalker: i've added you (author of 771235) as the initial member for the starlingx-ptp-notification-armada-app-core group in gerrit now | 18:59 |
| fungi | slittle1: ^ | 18:59 |
| fungi | CWalker: thanks go to corvus for getting the deployment tooling back on track | 19:00 |
| CWalker | Excellent, much appreciated | 19:00 |
| openstackgerrit | Jeremy Stanley proposed opendev/gerritbot master: Revert "Skip notifications about WIP changes" https://review.opendev.org/c/opendev/gerritbot/+/771506 | 19:27 |
| openstackgerrit | Jeremy Stanley proposed opendev/gerritbot master: Revert "Revert "Skip notifications about WIP changes"" https://review.opendev.org/c/opendev/gerritbot/+/771507 | 19:27 |
| frickler | clarkb: which docker ipv6 bug was that? /me is always looking for more reasons not to like docker ;) | 19:35 |
| clarkb | frickler: you cannot pull or push images using an ipv6 address literal as the source/target of the operation | 19:37 |
| clarkb | frickler: you have to use a dns name or proxy ipv4 to ipv6 (zuul does the proxy step using socat) | 19:37 |
| corvus | fungi, zbr: i expressed my thoughts in full-sentence form on https://review.opendev.org/771507 | 19:38 |
| zbr | imho what we lack more is people doing reviews, not notifications about what everyone is working on. | 19:41 |
| *** andrewbonney has quit IRC | 19:42 | |
| zbr | that was why i wanted to keep the notifications lower, less noise. but i will update it. | 19:42 |
| *** slaweq has quit IRC | 19:43 | |
| openstackgerrit | Merged opendev/gerritbot master: Revert "Skip notifications about WIP changes" https://review.opendev.org/c/opendev/gerritbot/+/771506 | 19:50 |
| frickler | clarkb: ah, found it, thx. that issue was indeed as old as I seemed to remember it to be | 19:51 |
| *** Jeffrey4l has quit IRC | 20:04 | |
| clarkb | ianw: https://opendev.org/opendev/system-config/src/branch/master/roles/openafs-client/tasks/openafs-client/Debian.yaml#L11-L12 is the condition in openafs-client I was talking about | 20:06 |
| clarkb | ianw: my read of that is xenial hosts using that role will install the xenial 1.6 packages? | 20:08 |
| fungi | so on the ppa addition on our executors, even ze12 has several copies which all add the same ppa in /etc/apt/sources.list.d: openstack-ci-core-ubuntu-openafs-amd64-hwe-xenial.list and ppa_openstack_ci_core_openafs_xenial.list both from 2020-05-07, and openafs.list from 2020-05-26 | 20:08 |
| fungi | 2020-05-07 looks like it might be the date the server was built | 20:08 |
| fungi | no, my bad, the server seems to have been built 2019-05-08 (a year earlier) | 20:09 |
| clarkb | I'm due for lunch and a bike ride, back in a bit to dig into that xenial stuff more if necessary | 20:09 |
| fungi | and yeah, the two files with the same 2020-05-07 date actually have the same precise timestamp (down to a nanosecond) | 20:10 |
| ianw | clarkb: umm, yeah i think that whole when: statement should probably just go, i think it's all old | 20:10 |
| fungi | so i guess that's what ianw meant by deploying two copies at one point | 20:10 |
| clarkb | ianw: that was my read on it too, but good to have someone else confirm. I can push that up a bit later today | 20:10 |
| fungi | but anyway, i expect we ansibilified the executors after last may, so they're getting packages from that ppa because puppet configured it | 20:11 |
| ianw | fungi: hrm, the hwe xenial thing ... didn't we have memory issues on the executor and switched to the HWE kernels at some point in the past? | 20:11 |
| clarkb | ianw: ya | 20:11 |
| clarkb | it had to do with swappiness iirc | 20:11 |
| clarkb | or how the kernel calculated when to swap | 20:11 |
| *** openstackgerrit has quit IRC | 20:12 | |
| ianw | that's right, something like that | 20:12 |
| mordred | clarkb: my IRC client tells me there was a nick highlight in #opendev-meeting during the opendev meeting - but does not have the skill to show me _where_ | 20:12 |
| clarkb | and it was doing it super aggressively because it didn't proeprly calulcate its free memory | 20:12 |
| clarkb | mordred: 19:48:04* fungi | in the case of storyboard, we already have docker images publishing on each new change merged (thanks mordred!) | 20:12 |
| mordred | \o/ | 20:12 |
| mordred | yay I was helpful | 20:12 |
| *** Jeffrey4l has joined #opendev | 20:13 | |
| fungi | indeed, that's going to go a long way to us being able to upgrade the sb server | 20:13 |
| *** openstackgerrit has joined #opendev | 20:13 | |
| openstackgerrit | Merged opendev/bindep master: ArchLinux: ignore unrelated warnings from pacman https://review.opendev.org/c/opendev/bindep/+/771108 | 20:13 |
| *** zbr5 has joined #opendev | 20:14 | |
| *** zbr has quit IRC | 20:16 | |
| *** zbr5 is now known as zbr | 20:16 | |
| ianw | so are CD jobs working again? i saw something go by ... | 20:21 |
| clarkb | yes I think so | 20:22 |
| clarkb | ianw: we did a chown in our base job for infra-prod jobs | 20:22 |
| clarkb | and then fungi reenqueued a thing that ran successfully | 20:22 |
| ianw | ahh, right ok https://review.opendev.org/c/opendev/base-jobs/+/771503/1/playbooks/infra-prod/pre.yaml | 20:22 |
| ianw | cool, ok; i'll check on and verify all the afs changes | 20:23 |
| fungi | yahyah | 20:23 |
| fungi | any jobs inheriting from our deployment base job should in theory be mitigated now | 20:24 |
| fungi | root does a recursive chown of the entire project tree on the bastion to zuul:zuul and then the zuul user is able to git clean within those repos after that | 20:25 |
| *** artom has quit IRC | 20:31 | |
| fungi | infra-root: i meant to mention in the meeting, but i've proposed a brush-up for the opendev.org main page here: https://review.opendev.org/769826 | 20:31 |
| *** stevebaker has quit IRC | 20:35 | |
| *** artom has joined #opendev | 20:42 | |
| *** Jeffrey4l has quit IRC | 20:50 | |
| *** Jeffrey4l has joined #opendev | 20:51 | |
| *** stevebaker has joined #opendev | 21:03 | |
| *** artom has quit IRC | 21:17 | |
| *** artom has joined #opendev | 21:17 | |
| *** sboyron_ has quit IRC | 21:26 | |
| *** priteau has quit IRC | 21:35 | |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: openafs-client: cleanup PPA install https://review.opendev.org/c/opendev/system-config/+/771521 | 21:58 |
| ianw | infra-root (but corvus maybe in particular) : is there any reason keeping the executors on Xenial right now, other than just time to replace the hosts? | 21:59 |
| ianw | "time" I mean "finding developer time" | 22:00 |
| corvus | ianw: the only relevant consideration i can think of at this point is afs | 22:04 |
| corvus | other than that, i believe they should be completely agnostic about their underlying os | 22:04 |
| ianw | cool, that was roughly my feeling as well. it should hopefully be fairly mechanical to swap in new hosts to get rid of xenial | 22:08 |
| ianw | infra-root: i should have checked before the meeting, but the vexxhost backup storage situation is now critical | 22:10 |
| ianw | /dev/mapper/main--202010-backups--202010 1007G 1007G 0 100% /opt/backups-202010 | 22:10 |
| ianw | hrm, the problem with re-purposing the bup backup storage quota is that wiki isn't backing up to borg. | 22:13 |
| ianw | i think that has to now become my top priority issue | 22:14 |
| ianw | we are still running to the rax server, which has 3tb. but it's not the redundancy we desire | 22:14 |
| clarkb | ianw: and thats doing append only backups with borg that is filling up? | 22:22 |
| clarkb | do we need to set a more aggressive pruning scheduling with borg? | 22:23 |
| clarkb | I think I've got mine set up to do like a full week then monthly then annually so it should be tunable | 22:23 |
| ianw | clarkb: hrm, whatever is the default i guess -- do we need to run something asynchronously? and we could probably do with more eyes on what we really need backed up | 22:24 |
| clarkb | ianw: I think borg doesn't prune by default | 22:24 |
| clarkb | I'm digging up my script to see what my local set up does | 22:24 |
| ianw | then i'm sure that's part of the problem :) | 22:25 |
| clarkb | ianw: I run borg prune --verbose --list --prefix '{hostname}-' --show-rc --keep-daily 7 --keep-weekly 4 --keep-monthly 6 | 22:26 |
| clarkb | and that is at the end of my backup script so it does the backups then prunes in one go | 22:26 |
| clarkb | and if I do a borg list I see that it matches that set of keep directives | 22:27 |
| *** iurygregory has quit IRC | 22:28 | |
| ianw | clarkb: hrm, perhaps as easy as adding that to https://opendev.org/opendev/system-config/src/branch/master/playbooks/roles/borg-backup/templates/borg-backup.j2 ? | 22:28 |
| clarkb | ya I think so. May we run it by hand against ethercalc or something like that and double check it does what we expect? | 22:30 |
| clarkb | also maybe keep-monthly 12 ? | 22:30 |
| clarkb | infra-root ^ may have other thoughts on retention though | 22:30 |
| ianw | ok, i've just got to get wiki onto borg so we can have the discussion about retiring all this bup storage (can't retire it while it's still in use :) | 22:31 |
| ianw | i'm thinking i'll manual run a custom playbook because it needs to make and distribute keys | 22:32 |
| clarkb | ianw: thanks for https://review.opendev.org/c/opendev/system-config/+/771521 fungi that is a qucik and easy review if you have a moment | 22:32 |
| fungi | looking | 22:34 |
| ianw | https://opendev.org/opendev/system-config/src/branch/master/playbooks/roles/borg-backup/templates/borg-backup.j2 | 22:34 |
| ianw | bah, sorry meant | 22:34 |
| ianw | https://zuul.opendev.org/t/openstack/build/cbad8b367ae544baa2bbb165ae4a93a0/console#1/2/15/base shows installing the ppa on xenial | 22:35 |
| *** iurygregory has joined #opendev | 22:37 | |
| clarkb | https://borgbackup.readthedocs.io/en/stable/usage/prune.html shows an interesting --save-space option too | 22:41 |
| clarkb | but more importantly there is a dry run option | 22:41 |
| clarkb | ianw: ^ I think we should be good to try running it with the dry run option and see what it tells us then if it looks good run it for real on say ethercalc then apply it to the script? | 22:42 |
| clarkb | then maybe look into --save-space? | 22:42 |
| fungi | we're nearly caught up on mirror full releases, the debian volume has been running for a couple of hours and once it finishes we need to do debian-security and then epel which i missed queuing up earlier, and then a second pass to bring them all up to date which will hopefully not take long at all | 22:45 |
| fungi | might be finished before i fall asleep tonight | 22:45 |
| ianw | fungi: so is the wiki host not in the inventory? | 22:46 |
| openstackgerrit | Merged openstack/project-config master: cursive: prepare to move the jobs in-tree https://review.opendev.org/c/openstack/project-config/+/771443 | 22:50 |
| openstackgerrit | Merged openstack/project-config master: Adding irc notification for missing oslo projects https://review.opendev.org/c/openstack/project-config/+/771392 | 22:50 |
| openstackgerrit | Merged openstack/project-config master: Combine acl file for all interop source code repo https://review.opendev.org/c/openstack/project-config/+/771066 | 22:50 |
| openstackgerrit | Merged openstack/project-config master: Move snaps ACL to x https://review.opendev.org/c/openstack/project-config/+/770538 | 22:50 |
| fungi | ianw: it's not handled with configuration management, and is still running ubuntu 14.04 | 22:52 |
| ianw | ahh, i didn't realise it wasn't managed at all | 22:52 |
| ianw | hmm | 22:52 |
| fungi | there's a mostly-complete attempt to get it into puppet where i wasn't able to work out all the extension updates | 22:52 |
| fungi | i have a mostly-working wiki-dev server up but was running into some missing theming and openid wasn't working | 22:53 |
| fungi | that one's deployed with puppet fully, just still needs work | 22:53 |
| fungi | the "production" wiki server is called wiki-upgrade-test.openstack.org (long story involving having to tear down the actual production server after the upgrade to trusty disabled iptables and the elasticsearch on the server got immediately hacked) | 22:54 |
| ianw | ok, it just means my ansible idea won't work to get it onto borg. i'll have to do it fully manually. that's ok ... i'll just need a cup of tea first :) | 22:56 |
| *** openstackgerrit has quit IRC | 22:59 | |
| *** brinzhang0 has joined #opendev | 23:17 | |
| *** brinzhang_ has quit IRC | 23:20 | |
| ianw | hrm, borg says that 1.1.15 drops python 3.4 support, but 1.1.14 doesn't seem to be working | 23:25 |
| ianw | File "/opt/borg/lib/python3.4/site-packages/pkg_resources/__init__.py", line 82, in <module> | 23:26 |
| ianw | raise RuntimeError("Python 3.5 or later is required") | 23:26 |
| ianw | interesting ... | 23:26 |
| clarkb | huh maybe they did it a bit early? | 23:27 |
| clarkb | fwiw it sounds likw 1.1.11 does python2 | 23:27 |
| clarkb | Possible that python2 + python3.5 was a thing rather than python 3.4? | 23:27 |
| ianw | it seems like borg is not so much at fault here as ... cue your surprise ... setuptools | 23:27 |
| fungi | whee | 23:28 |
| * fungi tries to stuff his surprise back in the box it accidentally escaped from | 23:28 | |
| clarkb | oh I see what that is saying its setuptools/pkg_resources complaining | 23:28 |
| fungi | might be able to preinstall older setuptools? | 23:29 |
| ianw | i need to find the last setuptools that supproted 3.4 | 23:31 |
| clarkb | ianw: the pypi metadta should help if you view source on the setuptools index paage | 23:32 |
| fungi | well, maybe | 23:33 |
| fungi | the trove classifiers might, but the python_requires metadata didn't really come into vogue until somewhat recently | 23:33 |
| fungi | so some versions before the addition of python_requires=3.5 might have also required 3.5 | 23:34 |
| ianw | https://github.com/pypa/setuptools/commit/dc868755d53520895d96ec0251a66df562a37095 | 23:35 |
| fungi | so pip install setuptools<44 i guess | 23:39 |
| ianw | yep | 23:39 |
| ianw | ok, there is a borg in /opt/borg/ ... we shall never speak of it again | 23:41 |
| *** brinzhang_ has joined #opendev | 23:52 | |
| fungi | heh, thanks! | 23:52 |
| fungi | looks like we're on to the mirror.debian-security volume release now | 23:53 |
| *** brinzhang0 has quit IRC | 23:55 | |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!