| clarkb | 1400 tasks now | 00:03 |
|---|---|---|
| clarkb | fungi: I see quite a few retries for gitea03 | 00:03 |
| clarkb | fungi: did the git repos get created on that host yet? | 00:03 |
| clarkb | looks like docker isn't running there yet, that explains it | 00:04 |
| fungi | yeah, not yet | 00:04 |
| fungi | waiting for the changes we merged to do that to appear in gitea so they'll get applied by ansible | 00:05 |
| clarkb | https://review.opendev.org/#/c/673113/ is not approved yet | 00:05 |
| clarkb | (that is needed to have docker get installed I think) | 00:05 |
| clarkb | should I go ahead and approve it? | 00:06 |
| fungi | oh, yeah, thought it already had been | 00:06 |
| fungi | thanks | 00:06 |
| clarkb | done | 00:06 |
| *** slaweq has joined #openstack-infra | 00:11 | |
| *** rcernin has joined #openstack-infra | 00:15 | |
| *** yamamoto has joined #openstack-infra | 00:15 | |
| *** slaweq has quit IRC | 00:17 | |
| *** rfarr has quit IRC | 00:18 | |
| fungi | replication to 4 backends took roughly 78 minutes | 00:18 |
| fungi | i'll do 01,02,04 now | 00:19 |
| *** igordc has quit IRC | 00:19 | |
| *** igordc has joined #openstack-infra | 00:21 | |
| fungi | queued | 00:23 |
| *** gyee has quit IRC | 00:29 | |
| openstackgerrit | Matthew Thode proposed openstack/diskimage-builder master: support alternate portage directories https://review.opendev.org/671530 | 00:30 |
| fungi | around this time last night replicating to three backends took right at 40 minutes | 00:30 |
| fungi | so completion at 01:03 maybe? | 00:30 |
| openstackgerrit | Merged opendev/system-config master: Add gitea03 replacement to inventory https://review.opendev.org/673113 | 00:40 |
| *** bobh has joined #openstack-infra | 00:51 | |
| *** armax has quit IRC | 01:01 | |
| fungi | 2019-07-27T01:03:05+00:00 1402 tasks | 01:03 |
| fungi | guess not | 01:03 |
| fungi | maybe things are just slower tonight | 01:03 |
| clarkb | I wonder what the cause is | 01:03 |
| *** slaweq has joined #openstack-infra | 01:11 | |
| fungi | wrapped up in 50 minutes (vs 40 last night) | 01:14 |
| *** slaweq has quit IRC | 01:16 | |
| *** yamamoto has quit IRC | 01:19 | |
| clarkb | are you wanting to finish up 03 today? | 01:20 |
| clarkb | I will try to pay attention to changes that need review if so | 01:21 |
| fungi | yeah, lemme see if it's moved on yet | 01:21 |
| fungi | ssh host key accepted on bridge.o.o | 01:23 |
| fungi | gitea's running on it now, time for more replicating | 01:36 |
| *** diablo_rojo has quit IRC | 01:37 | |
| fungi | replication to gitea03 queued now | 01:37 |
| prometheanfire | ianw: actually found something that works... https://review.opendev.org/671530 | 01:38 |
| fungi | oh oops, those replication tasks are going to fail. restoring the db from a copy first | 01:39 |
| *** bobh has quit IRC | 01:39 | |
| *** bobh has joined #openstack-infra | 01:40 | |
| *** bobh has quit IRC | 01:45 | |
| fungi | okay, database restored, missing repos initialized, replication restarted | 01:47 |
| *** yamamoto has joined #openstack-infra | 02:00 | |
| fungi | i had failed to accept the correct hostkey for replication, count finally seems to be falling now | 02:15 |
| *** yamamoto has quit IRC | 02:18 | |
| openstackgerrit | Jeremy Stanley proposed opendev/system-config master: Swap gitea03 into service and bring down gitea04 https://review.opendev.org/673119 | 02:18 |
| fungi | should be safe to approve that ^ once replication completes | 02:19 |
| *** bobh has joined #openstack-infra | 02:25 | |
| *** jamesmcarthur has joined #openstack-infra | 02:26 | |
| *** bobh has quit IRC | 02:38 | |
| *** whoami-rajat has joined #openstack-infra | 02:41 | |
| *** jamesmcarthur has quit IRC | 02:42 | |
| fungi | looks like it's done | 02:47 |
| fungi | i'm gonna knock off for the evening | 02:47 |
| *** bobh has joined #openstack-infra | 02:53 | |
| *** yamamoto has joined #openstack-infra | 02:54 | |
| *** jamesmcarthur has joined #openstack-infra | 02:57 | |
| *** yamamoto has quit IRC | 02:58 | |
| *** yamamoto has joined #openstack-infra | 02:58 | |
| *** yamamoto has quit IRC | 03:02 | |
| *** liuyulong has quit IRC | 03:07 | |
| *** yamamoto has joined #openstack-infra | 03:07 | |
| *** slaweq has joined #openstack-infra | 03:11 | |
| *** markvoelker has quit IRC | 03:12 | |
| *** slaweq has quit IRC | 03:16 | |
| *** bobh has quit IRC | 03:23 | |
| *** jamesmcarthur has quit IRC | 03:23 | |
| *** markvoelker has joined #openstack-infra | 03:28 | |
| *** Lucas_Gray has quit IRC | 03:39 | |
| *** yamamoto has quit IRC | 03:56 | |
| *** yamamoto has joined #openstack-infra | 03:57 | |
| *** bhavikdbavishi has joined #openstack-infra | 04:00 | |
| *** bhavikdbavishi has quit IRC | 04:04 | |
| *** verdurin has quit IRC | 04:14 | |
| *** verdurin has joined #openstack-infra | 04:15 | |
| *** jamesmcarthur has joined #openstack-infra | 04:18 | |
| *** apetrich has quit IRC | 04:20 | |
| *** markvoelker has quit IRC | 04:40 | |
| *** igordc has quit IRC | 04:44 | |
| *** whoami-rajat has quit IRC | 04:47 | |
| *** ricolin has joined #openstack-infra | 04:50 | |
| *** yamamoto has quit IRC | 04:51 | |
| *** altlogbot_2 has quit IRC | 04:58 | |
| *** altlogbot_0 has joined #openstack-infra | 04:59 | |
| *** jamesmcarthur has quit IRC | 05:00 | |
| *** jamesmcarthur has joined #openstack-infra | 05:01 | |
| *** jamesmcarthur has quit IRC | 05:04 | |
| *** jamesmcarthur_ has joined #openstack-infra | 05:04 | |
| *** yamamoto has joined #openstack-infra | 05:09 | |
| *** slaweq has joined #openstack-infra | 05:11 | |
| *** bhavikdbavishi has joined #openstack-infra | 05:12 | |
| *** slaweq has quit IRC | 05:15 | |
| *** dciabrin_ has joined #openstack-infra | 05:32 | |
| *** dciabrin has quit IRC | 05:34 | |
| *** jamesmcarthur_ has quit IRC | 05:40 | |
| evrardjp | corvus: did you find some projects not in required projects? That would be a bug. If we don't have that, our depends-on is basically broken. | 05:54 |
| *** bhavikdbavishi has quit IRC | 05:58 | |
| *** jamesmcarthur has joined #openstack-infra | 06:07 | |
| openstackgerrit | Merged zuul/zuul master: Display artifacts in build page https://review.opendev.org/671865 | 06:08 |
| *** Lucas_Gray has joined #openstack-infra | 06:13 | |
| *** jamesmcarthur has quit IRC | 06:13 | |
| *** Lucas_Gray has quit IRC | 06:18 | |
| *** whoami-rajat has joined #openstack-infra | 06:19 | |
| *** Lucas_Gray has joined #openstack-infra | 06:19 | |
| *** yamamoto has quit IRC | 06:21 | |
| *** yamamoto has joined #openstack-infra | 06:34 | |
| *** kjackal has joined #openstack-infra | 06:39 | |
| *** jamesmcarthur has joined #openstack-infra | 06:41 | |
| *** jamesmcarthur has quit IRC | 06:46 | |
| *** kjackal has quit IRC | 07:02 | |
| *** slaweq has joined #openstack-infra | 07:11 | |
| *** slaweq has quit IRC | 07:16 | |
| *** jamesmcarthur has joined #openstack-infra | 07:21 | |
| *** yamamoto_ has joined #openstack-infra | 07:24 | |
| *** pcaruana has quit IRC | 07:24 | |
| *** jamesmcarthur has quit IRC | 07:25 | |
| *** yamamoto has quit IRC | 07:27 | |
| *** ricolin has quit IRC | 07:29 | |
| *** yamamoto_ has quit IRC | 07:33 | |
| *** pcaruana has joined #openstack-infra | 07:38 | |
| *** yamamoto has joined #openstack-infra | 07:46 | |
| *** pcaruana has quit IRC | 07:47 | |
| *** yamamoto has quit IRC | 07:57 | |
| *** rcernin has quit IRC | 08:06 | |
| *** bhavikdbavishi has joined #openstack-infra | 08:20 | |
| *** jamesmcarthur has joined #openstack-infra | 08:22 | |
| *** jamesmcarthur has quit IRC | 08:26 | |
| *** priteau has joined #openstack-infra | 08:30 | |
| *** priteau has quit IRC | 08:32 | |
| *** yamamoto has joined #openstack-infra | 08:39 | |
| *** yamamoto has quit IRC | 08:51 | |
| *** slaweq has joined #openstack-infra | 08:53 | |
| *** yamamoto has joined #openstack-infra | 08:53 | |
| *** slaweq has quit IRC | 08:57 | |
| *** jamesmcarthur has joined #openstack-infra | 09:01 | |
| *** jamesmcarthur has quit IRC | 09:06 | |
| *** Lucas_Gray has quit IRC | 09:09 | |
| *** yamamoto has quit IRC | 09:12 | |
| *** yamamoto has joined #openstack-infra | 09:22 | |
| *** yamamoto has quit IRC | 09:31 | |
| *** yamamoto has joined #openstack-infra | 09:35 | |
| *** yamamoto_ has joined #openstack-infra | 09:37 | |
| *** yamamoto has quit IRC | 09:40 | |
| *** bhavikdbavishi has quit IRC | 09:47 | |
| *** dpawlik has joined #openstack-infra | 09:52 | |
| *** jamesmcarthur has joined #openstack-infra | 10:03 | |
| *** bhavikdbavishi has joined #openstack-infra | 10:07 | |
| *** jamesmcarthur has quit IRC | 10:07 | |
| *** slaweq has joined #openstack-infra | 10:11 | |
| *** dpawlik has quit IRC | 10:12 | |
| *** trident has quit IRC | 10:13 | |
| *** slaweq has quit IRC | 10:16 | |
| *** trident has joined #openstack-infra | 10:17 | |
| *** dpawlik has joined #openstack-infra | 10:20 | |
| *** bhavikdbavishi has quit IRC | 10:20 | |
| *** jamesmcarthur has joined #openstack-infra | 10:43 | |
| *** jamesmcarthur has quit IRC | 10:48 | |
| *** yamamoto_ has quit IRC | 11:06 | |
| *** yamamoto has joined #openstack-infra | 11:09 | |
| *** yamamoto has quit IRC | 11:09 | |
| *** yamamoto has joined #openstack-infra | 11:09 | |
| *** raissa has joined #openstack-infra | 11:12 | |
| *** Lucas_Gray has joined #openstack-infra | 11:12 | |
| *** jamesmcarthur has joined #openstack-infra | 11:44 | |
| *** e0ne has joined #openstack-infra | 11:45 | |
| *** jamesmcarthur has quit IRC | 11:50 | |
| *** yamamoto has quit IRC | 11:55 | |
| *** whoami-rajat has quit IRC | 11:58 | |
| *** yamamoto has joined #openstack-infra | 12:00 | |
| *** bhavikdbavishi has joined #openstack-infra | 12:01 | |
| *** betherly has joined #openstack-infra | 12:02 | |
| *** trident has quit IRC | 12:06 | |
| *** betherly has quit IRC | 12:07 | |
| *** trident has joined #openstack-infra | 12:09 | |
| *** jamesmcarthur has joined #openstack-infra | 12:14 | |
| *** xek has joined #openstack-infra | 12:18 | |
| *** xek has quit IRC | 12:18 | |
| *** e0ne has quit IRC | 12:30 | |
| *** dpawlik has quit IRC | 12:32 | |
| *** eharney has joined #openstack-infra | 12:34 | |
| *** jamesmcarthur has quit IRC | 12:48 | |
| *** dpawlik has joined #openstack-infra | 12:48 | |
| *** dpawlik has quit IRC | 12:52 | |
| *** betherly has joined #openstack-infra | 12:53 | |
| *** betherly has quit IRC | 12:58 | |
| *** yamamoto has quit IRC | 13:04 | |
| *** dciabrin_ has quit IRC | 13:07 | |
| openstackgerrit | Monty Taylor proposed zuul/nodepool master: WIP Load k8s config on use, not just at startup https://review.opendev.org/673138 | 13:17 |
| fungi | if anybody's around to approve https://review.opendev.org/673119 i can start building the gitea04 replacement | 13:17 |
| *** jamesmcarthur has joined #openstack-infra | 13:19 | |
| *** sgw has quit IRC | 13:20 | |
| mordred | fungi: go for it | 13:20 |
| *** sgw has joined #openstack-infra | 13:23 | |
| *** yamamoto has joined #openstack-infra | 13:23 | |
| *** jamesmcarthur has quit IRC | 13:25 | |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Build gerrit images for 2.16 and 3.0 as well https://review.opendev.org/672273 | 13:25 |
| fungi | thanks mordred! | 13:27 |
| *** whoami-rajat has joined #openstack-infra | 13:27 | |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Build gerrit images for 2.16 and 3.0 as well https://review.opendev.org/672273 | 13:27 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Override-checkout download-commands to v2.16.10 https://review.opendev.org/673107 | 13:27 |
| mordred | fungi: maybe today will be the day that we successfully build gerrit | 13:27 |
| fungi | today is a good day to bazel? | 13:28 |
| mordred | I don't know that I'd go that far | 13:28 |
| fungi | heh | 13:28 |
| fungi | i already weeded the hydrangea and azalea beds, and my only other assigned project for today is to build a coat rack from some old deck planking driftwood and reclaimed dock cleats | 13:30 |
| mordred | that sounds like a fun task | 13:35 |
| fungi | well, first i need to figure out how i'm going to faux rust/weather the cleats, because they'd been cleaned and chrome-painted | 13:37 |
| *** yamamoto has quit IRC | 13:38 | |
| mordred | fungi: just leave them outside? | 13:40 |
| fungi | i'm impatient ;) | 13:40 |
| *** trident has quit IRC | 13:43 | |
| *** yamamoto has joined #openstack-infra | 13:47 | |
| openstackgerrit | Merged opendev/system-config master: Swap gitea03 into service and bring down gitea04 https://review.opendev.org/673119 | 13:48 |
| *** ykarel|away has joined #openstack-infra | 13:50 | |
| *** trident has joined #openstack-infra | 13:52 | |
| *** betherly has joined #openstack-infra | 13:54 | |
| donnyd | I am about to start the transition for FN. The currently running instances will finish, but no new instances will be scheduled until the transition is completed | 13:56 |
| fungi | thanks for the heads up, donnyd! | 13:56 |
| donnyd | Wish me luck | 13:56 |
| donnyd | At least I have proper bgp routing of ipv6 now | 13:56 |
| donnyd | :) | 13:56 |
| *** betherly has quit IRC | 13:59 | |
| mordred | \o/ | 14:04 |
| *** gcerami_ has joined #openstack-infra | 14:07 | |
| *** panda has quit IRC | 14:09 | |
| *** gcerami_ is now known as panda | 14:09 | |
| *** slaweq has joined #openstack-infra | 14:11 | |
| fungi | bgp6 is good stuff | 14:12 |
| * fungi wishes he had the opportunity to make use of openbsd's openbgpd | 14:13 | |
| fungi | i only ever ended up managing bgp peering on ios and junos | 14:14 |
| paladox | mordred you need to do the same to singleusergroup (as that only has a 2.16 branch but not a 3.0 branch) | 14:15 |
| *** slaweq has quit IRC | 14:15 | |
| mordred | paladox: *doh* | 14:16 |
| mordred | ooh! but 2.16 worked!!! | 14:16 |
| paladox | yup | 14:16 |
| fungi | okay, gitea04 is out of the haproxy pools now, so will delete it shortly and start building its replacement | 14:16 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Build gerrit images for 2.16 and 3.0 as well https://review.opendev.org/672273 | 14:18 |
| *** trident has quit IRC | 14:20 | |
| mordred | paladox: it's gonna work this time, I can feel it! | 14:20 |
| *** jamesmcarthur has joined #openstack-infra | 14:20 | |
| paladox | :D | 14:20 |
| mordred | then I just need to actually work on the config management change to use the images ... | 14:21 |
| paladox | mordred our CI is broken until we force merge the gerrit upgrade and rebuild the cache :( | 14:21 |
| mordred | :( | 14:21 |
| *** trident has joined #openstack-infra | 14:24 | |
| *** jamesmcarthur has quit IRC | 14:25 | |
| paladox | bring on bazel 1.0 in September! https://blog.bazel.build/2019/06/06/Bazel-Semantic-Versioning.html | 14:25 |
| *** Lucas_Gray has quit IRC | 14:26 | |
| *** trident has quit IRC | 14:31 | |
| *** trident has joined #openstack-infra | 14:34 | |
| *** yamamoto has quit IRC | 14:40 | |
| *** yamamoto has joined #openstack-infra | 14:42 | |
| *** slaweq has joined #openstack-infra | 14:45 | |
| *** ykarel|away has quit IRC | 14:46 | |
| *** ykarel|away has joined #openstack-infra | 14:47 | |
| *** trident has quit IRC | 14:51 | |
| *** trident has joined #openstack-infra | 14:53 | |
| *** ykarel|away has quit IRC | 14:53 | |
| mordred | paladox: does gerrit build with 0.27 ? | 14:54 |
| paladox | mordred yup! | 14:55 |
| * paladox used bazel 0.27 up till around an hour ago. | 14:55 | |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Try building with 0.27.1 instead https://review.opendev.org/673141 | 14:56 |
| mordred | let's see how that works for us | 14:56 |
| *** betherly has joined #openstack-infra | 14:56 | |
| paladox | mordred i believe we use bazel 0.27 in our bazel image too. | 14:57 |
| mordred | cool. | 14:58 |
| mordred | oh good - the latest build worked for 2.16 and 3.0 but not for 2.15 | 14:58 |
| mordred | well - we're at least getting *somewhere* | 14:59 |
| *** jamesmcarthur has joined #openstack-infra | 15:00 | |
| *** betherly has quit IRC | 15:01 | |
| *** ykarel|away has joined #openstack-infra | 15:10 | |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Split out bazel builder and gerrit base image https://review.opendev.org/673142 | 15:18 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Separate the Dockerfiles for the bazel builds https://review.opendev.org/673143 | 15:19 |
| mordred | corvus: ^^ I got fancy and used zuul image dependencies | 15:19 |
| *** ykarel|away has quit IRC | 15:23 | |
| openstackgerrit | Jeremy Stanley proposed opendev/zone-opendev.org master: Update IP address for gitea04 https://review.opendev.org/673144 | 15:26 |
| *** betherly has joined #openstack-infra | 15:27 | |
| openstackgerrit | Jeremy Stanley proposed opendev/system-config master: Add gitea04 replacement to inventory https://review.opendev.org/673146 | 15:29 |
| *** ykarel|away has joined #openstack-infra | 15:31 | |
| openstackgerrit | Monty Taylor proposed zuul/zuul-jobs master: Don't try to push images when the build failed https://review.opendev.org/673147 | 15:32 |
| mordred | corvus: ^^ and I think that's why we're seeting POST_FAILURE on all of the image build failures above | 15:33 |
| *** betherly has quit IRC | 15:33 | |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Build gerrit images for 2.16 and 3.0 as well https://review.opendev.org/672273 | 15:34 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Try building with 0.27.1 instead https://review.opendev.org/673141 | 15:34 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Split out bazel builder and gerrit base image https://review.opendev.org/673142 | 15:34 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Separate the Dockerfiles for the bazel builds https://review.opendev.org/673143 | 15:34 |
| mordred | and that should exercise whether the zuul-jobs change is good or not | 15:35 |
| mordred | (since that stack does all the things) | 15:35 |
| *** slaweq has quit IRC | 15:36 | |
| *** betherly has joined #openstack-infra | 15:58 | |
| *** whoami-rajat has quit IRC | 16:01 | |
| *** betherly has quit IRC | 16:03 | |
| *** jamesmcarthur has quit IRC | 16:04 | |
| paladox | mordred awesome to see that it builds for you! | 16:14 |
| *** betherly has joined #openstack-infra | 16:19 | |
| *** betherly has quit IRC | 16:24 | |
| *** slaweq has joined #openstack-infra | 16:50 | |
| *** slaweq has quit IRC | 16:55 | |
| *** jamesmcarthur has joined #openstack-infra | 17:00 | |
| *** betherly has joined #openstack-infra | 17:01 | |
| *** ykarel|away has quit IRC | 17:02 | |
| *** ykarel|away has joined #openstack-infra | 17:04 | |
| *** jamesmcarthur has quit IRC | 17:05 | |
| *** betherly has quit IRC | 17:05 | |
| *** pfallenop has quit IRC | 17:06 | |
| *** bhavikdbavishi has quit IRC | 17:06 | |
| *** pfallenop has joined #openstack-infra | 17:08 | |
| *** bdodd has quit IRC | 17:15 | |
| *** whoami-rajat has joined #openstack-infra | 17:19 | |
| donnyd | do passwords for tenants get rotated? | 17:20 |
| *** bobh has joined #openstack-infra | 17:21 | |
| fungi | we generally reset them initially when we get access, and then... infrequently thereafter | 17:21 |
| fungi | initial passwords are often communicated to us over insecure channels, so it's just a general safety precaution | 17:22 |
| donnyd | I C | 17:27 |
| donnyd | I am trying to migrate the mirror, but there doesn't look to be a good way to do it | 17:28 |
| *** ricolin has joined #openstack-infra | 17:28 | |
| donnyd | the keys used to launch it are gone, which is somewhat problematic | 17:28 |
| donnyd | I do however have a fixed port created so we don't have to move any ip addresses | 17:29 |
| donnyd | I am still in waiting for the parts to arrive to bring the nodepool instances back up to their appropriate levels, but the mirror node sits on dedicated hardware. | 17:30 |
| donnyd | I would like to get the mirror back up and running as soon as I can so everything will be ready to roll when i get the final part in (fans for storage server) | 17:30 |
| donnyd | Would it be better to just let ansible rebuild the mirror on its next run and then swap out the port that is created for the one that is already populated in DNS? | 17:31 |
| fungi | i just tested and seem to still be able to authenticate to the api | 17:32 |
| *** jamesmcarthur has joined #openstack-infra | 17:32 | |
| donnyd | I haven't swapped over the api yet | 17:32 |
| donnyd | I can, I guess its all hard down at this point | 17:32 |
| fungi | well, the mirrors aren't auto-built, they're brought up manually | 17:32 |
| fungi | we can rebuild it of course | 17:33 |
| fungi | probably no more than 30 minutes of someone's time | 17:33 |
| fungi | i mean, it's mostly scripted, just needs someone to feed in the appropriate parameters and tweak the volume mounts in some cases | 17:34 |
| clarkb | no volume mounts in this case | 17:36 |
| donnyd | If someone could bring the mirror back up that would be super helpful | 17:37 |
| fungi | i'm not sure what you mean by "the keys used to launch it are gone" since we seem to have usable credentials... if those same credentials are what's needed to do the migration, is there some command we need to run? | 17:37 |
| donnyd | ok, the apis are swapped out | 17:37 |
| donnyd | yea, the keys that were used to launch it aren't in the db, or I would just migrate them | 17:38 |
| donnyd | Its possible they are being stored somewhere else | 17:38 |
| fungi | i assume the keys we have are sufficient to launch an instance, so can we use those to migrate it? | 17:38 |
| fungi | or am i mixing up terminology? | 17:39 |
| *** armax has joined #openstack-infra | 17:39 | |
| donnyd | +----------------------------+ | 17:39 |
| donnyd | | name | | 17:39 |
| donnyd | +----------------------------+ | 17:39 |
| donnyd | | bridge-root-2014-09-15 | | 17:39 |
| donnyd | | infra-root-keys | | 17:39 |
| donnyd | | infra-root-keys-2018-06-15 | | 17:39 |
| donnyd | | precision | | 17:39 |
| donnyd | | bridge-root-2014-09-15 | | 17:39 |
| donnyd | | clarkb-ssh-key | | 17:39 |
| donnyd | | infra-root-keys | | 17:39 |
| donnyd | | infra-root-keys-2018-06-15 | | 17:39 |
| donnyd | | precision | | 17:39 |
| fungi | oh, those are keys used to log into the instance, right? | 17:39 |
| donnyd | however, the one use to launch the mirror are called launch-1562164109 | 17:40 |
| donnyd | All the other keys are there.. just not those ones | 17:40 |
| donnyd | yes | 17:40 |
| fungi | so you're saying if we can log into the instance there's some command we can use to "migrate" it? | 17:40 |
| donnyd | no, migration is manual between controller | 17:41 |
| fungi | but if you had the authentication key we used to log into it when we first launched it, *then* you'd be able to migrate it? | 17:41 |
| donnyd | I was just trying to keep all the same data, IE, pull data from the DB, and repopulate it into the new controller | 17:42 |
| fungi | oh, i see, it's a data integrity thing in nova i guess. surprising that when we delete keys that makes nova migration challenging | 17:43 |
| donnyd | however it would seem like *if* the process isn't a heavy lift from infra's side then maybe it would be faster to get the mirror back up and running by relaunching it | 17:44 |
| donnyd | yea, I don't know what is in the key-pair used to launch the instance | 17:44 |
| donnyd | That is really the only data I am missing | 17:44 |
| fungi | i think those launch-xxxx keys are ephemeral and always deleted after we use them to bootstrap, but yeah we ought to be able to just boot another mirror instance | 17:44 |
| fungi | funny that nova keeps a reference to them even after deleting | 17:45 |
| donnyd | Can either fungi or clarkb give it a whirl if you have time? | 17:45 |
| fungi | i'll see if i can figure it out | 17:45 |
| donnyd | Once the instance is launched I can pop the right interface back on it so we don't have to tinker with DNS | 17:45 |
| fungi | hrm, our api credentials are no longer working | 17:46 |
| fungi | The request you have made requires authentication. (HTTP 401) (Request-ID: req-90c2ac09-b153-4f8a-b635-0c3f4be571a0) | 17:46 |
| fungi | it was working a moment ago | 17:46 |
| donnyd | yes, that is why I asked about the password changing thing | 17:47 |
| fungi | i see | 17:47 |
| fungi | that's the first order of business i guess, before i can boot the mirror | 17:47 |
| fungi | will go through and get credentials redone for the accounts | 17:48 |
| *** bhavikdbavishi has joined #openstack-infra | 17:50 | |
| clarkb | fungi: did gitea03 get GC'd ? | 17:54 |
| clarkb | I think you mentioned ti had been missed on either 02 or 03 and wanted to make sure that got done | 17:54 |
| clarkb | (I can trigger it if it hasn't been done yet) | 17:55 |
| fungi | oh, no i think i didn't do the git gc on it yet, please do | 17:55 |
| *** bobh has quit IRC | 17:56 | |
| fungi | also changes are up to add gitea04 back into the inventory and update dns for it | 17:56 |
| clarkb | gc is running on 03, reviewing 04 changes now | 17:57 |
| clarkb | done and done | 17:59 |
| fungi | thanks | 17:59 |
| *** ricolin has quit IRC | 17:59 | |
| openstackgerrit | Merged opendev/zone-opendev.org master: Update IP address for gitea04 https://review.opendev.org/673144 | 17:59 |
| fungi | i guess https://docs.openstack.org/python-openstackclient/latest/cli/command-objects/user.html#user-set is the admin command... what's the one i want for resetting the password of the account which is authenticating to the api? i never can seem to remember which one that is | 18:02 |
| *** ykarel|away has quit IRC | 18:03 | |
| clarkb | user password set | 18:05 |
| clarkb | fungi: ^ | 18:05 |
| fungi | weird... why is that not in the osc documentation? | 18:06 |
| fungi | that may explain why i always find the wrong one | 18:06 |
| clarkb | I always end up doing something like openstack --help | grep password | 18:08 |
| *** armax has quit IRC | 18:09 | |
| fungi | Request to http://10.0.10.240:5000/v3/users/3c109a4413ca4b68b90560093ff2d79c/password timed out | 18:13 |
| fungi | is that supposed to be an rfc 1918 address? | 18:13 |
| fungi | wrong catalog entry? | 18:13 |
| clarkb | should be https too | 18:13 |
| *** raissa has quit IRC | 18:13 | |
| fungi | yeah | 18:13 |
| fungi | donnyd: ^ is the (new) keystone catalog incorrect? | 18:14 |
| *** jamesmcarthur has quit IRC | 18:14 | |
| donnyd | hrm | 18:14 |
| donnyd | no | 18:14 |
| clarkb | openstack.fortnebula.com does not resolve to that 10.0.10.240 address fwiw (and we set that as the auth_url's fqdn) | 18:14 |
| openstackgerrit | Merged opendev/system-config master: Add gitea04 replacement to inventory https://review.opendev.org/673146 | 18:15 |
| *** wpp has quit IRC | 18:16 | |
| donnyd | you should be getting the public endpoints | 18:16 |
| fungi | does `openstack user password set` try to use internal/admin endpoints for some reason? | 18:18 |
| fungi | the public keystone endpoint in the catalog is https://openstack.fortnebula.com:13000 | 18:20 |
| fungi | which does seem to be reachable | 18:21 |
| donnyd | I am not sure why its feeding you back private APIs | 18:22 |
| fungi | --verbose doesn't seem to report anything useful on the password command either | 18:22 |
| fungi | command: user password set -> openstackclient.identity.v3.user.SetPasswordUser (auth=True) | 18:23 |
| fungi | maybe it's an osc bug and i need to update the one in my virtualenv | 18:24 |
| donnyd | no, its not | 18:24 |
| *** betherly has joined #openstack-infra | 18:24 | |
| donnyd | You can use the --debug flag to see where its calling | 18:24 |
| donnyd | and for some reason it wants to call the private ip address | 18:25 |
| *** ralonsoh has joined #openstack-infra | 18:25 | |
| * donnyd digging into why keystone is not using the catalog apis for public interfaces | 18:29 | |
| fungi | ugh, after upgrading, new problem: | 18:29 |
| fungi | Failed to discover available identity versions when contacting https://openstack.fortnebula.com:13000/v3. Attempting to parse version from URL. | 18:29 |
| fungi | Unable to establish connection to https://openstack.fortnebula.com:13000/v3/auth/tokens: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response',)) | 18:29 |
| *** betherly has quit IRC | 18:29 | |
| fungi | but it at least does seem to be hitting the correct endpoint now | 18:30 |
| fungi | this is with latest python-openstackclient and dependency versions | 18:31 |
| clarkb | try drop the v3/ prefix | 18:31 |
| fungi | well, retried with --debug and suddenly it worked? | 18:32 |
| * fungi sighs at heisenbug, changing its behavior | 18:32 | |
| fungi | yeah, passwords successfully changed now | 18:34 |
| fungi | maybe donnyd adjusted something? | 18:34 |
| fungi | anyway, i saved a pip freeze before upgrading, in case anyone is keen to reproduce | 18:35 |
| donnyd | yea, headers parsing wasn't se | 18:35 |
| donnyd | yea, headers parsing wasn't set | 18:35 |
| fungi | ahh, okay | 18:35 |
| fungi | looks like we want flavor "8cpu-8GBram-200GBdisk" and image "bionic" (those are what the old mirror used) | 18:37 |
| clarkb | yup | 18:37 |
| fungi | are we using --boot-from-volume there? | 18:39 |
| donnyd | no | 18:40 |
| donnyd | there is only 1 hypervisor right now, so it should be scheduled to the proper place | 18:40 |
| fungi | just fiddling with the script now to see if i can get it to include a port name | 18:43 |
| donnyd | if that is too much, then we should be able to do the port swap | 18:43 |
| *** betherly has joined #openstack-infra | 18:45 | |
| *** hwoarang_ has joined #openstack-infra | 18:47 | |
| fungi | i think i just need to work out which parameter that is to openstacksdk | 18:47 |
| *** hwoarang__ has joined #openstack-infra | 18:48 | |
| *** hwoarang has quit IRC | 18:48 | |
| *** jamesmcarthur has joined #openstack-infra | 18:49 | |
| *** betherly has quit IRC | 18:49 | |
| *** hwoarang_ has quit IRC | 18:52 | |
| *** jamesmcarthur has quit IRC | 18:53 | |
| * donnyd very get very angry with fedex @ now | 18:58 | |
| donnyd | so I guess my fan assembly will now be here tuesday... | 18:59 |
| donnyd | so I can't bring any hypervisors for nodepool back online | 18:59 |
| donnyd | however that doesn't effect the mirror | 18:59 |
| *** jamesmcarthur has joined #openstack-infra | 19:02 | |
| fungi | yeah, so our script is calling into https://opendev.org/openstack/openstacksdk/src/branch/master/examples/compute/create.py#L55-L71 which doesn't seem to provide any option for specifying a predefined port | 19:03 |
| fungi | i'll just try to boot it the old fashioned way and see what we end up with | 19:03 |
| fungi | oh, wait, that's an example source, not what we're calling | 19:04 |
| * fungi got lost in the maze of openstacksdk docs | 19:04 | |
| *** betherly has joined #openstack-infra | 19:05 | |
| fungi | here's the method: https://docs.openstack.org/openstacksdk/latest/user/connection.html#openstack.connection.Connection.create_server | 19:05 |
| fungi | still, no port | 19:06 |
| fungi | it does have an ips option, but i think that's for fip | 19:06 |
| fungi | so anyway, it's launching now | 19:07 |
| donnyd | ok cool | 19:08 |
| *** jamesmcarthur has quit IRC | 19:09 | |
| fungi | looks like it got an initial ip address of 2001:470:e045:2:f816:3eff:fef3:93c5 | 19:09 |
| donnyd | so should i swap out the ip address now? | 19:10 |
| *** betherly has quit IRC | 19:10 | |
| fungi | i don't think our script will figure that out. it's still waiting to be able to ssh into 2001:470:e045:2:f816:3eff:fef3:93c5 | 19:11 |
| donnyd | well its reachable from the public interwebs, and the security group rules are wide open | 19:12 |
| fungi | yeah, i think something happened with the keypair its trying to use | 19:12 |
| fungi | `openstack keypair list` is empty | 19:12 |
| fungi | i wonder if i need to rerun our cloud launcher first to readd some | 19:12 |
| fungi | clarkb: ^ does that normally happen automatically or is it manual? | 19:13 |
| fungi | i can't recall | 19:13 |
| fungi | i'll look through our documentation | 19:14 |
| fungi | ahh, yep, it runs in cron hourly from our bridge server | 19:17 |
| fungi | seems it ran at 19:00z | 19:18 |
| fungi | aha, but it's broken | 19:19 |
| donnyd | So are we going to provision and configure and then swap ip's or provision swap ips and then configure? | 19:20 |
| fungi | looks like authentication may be broken for openstackjenkins-vexxhost ca-ymq-1 | 19:20 |
| donnyd | I would think #2 so it gets configured to listen on the right ip addresses for things and what not | 19:21 |
| fungi | donnyd: well, provisioning includes configuring in this case. as i mentioned earlier it's all one scripted process and ansible expects the instance to be reached on the address returned by the server_create() call to openstacksdk | 19:22 |
| fungi | but first i need to figure out how to get the keypairs reinstalled | 19:22 |
| donnyd | ok cool | 19:22 |
| fungi | it's unclear why our script which installs our keypairs in clouds didn't re-add them to the fn tenants... it seems to do each region independently so i don't think the exception raised for openstackjenkins-vexxhost ca-ymq-1 is the cause | 19:24 |
| fungi | i don't actually see it trying to run for the fn tenants at all | 19:24 |
| *** bdodd has joined #openstack-infra | 19:24 | |
| fungi | so reverse-engineering it now | 19:25 |
| donnyd | does it have the same creds as what were just provisioned | 19:26 |
| donnyd | just an fyi I have about 30 more minutes to work on this today, i have to be out of here at 4 | 19:26 |
| fungi | yeah, it should be using the same credentials, i don't see it even mention trying though | 19:27 |
| *** bobh has joined #openstack-infra | 19:36 | |
| fungi | i'm coming back around to suspecting openstackjenkins-vexxhost ca-ymq-1 is causing it to not process any clouds after that one in the yaml list | 19:37 |
| fungi | because i think it's only doing entries listed before that | 19:38 |
| *** jamesmcarthur has joined #openstack-infra | 19:39 | |
| fungi | oh yeah, it stops immediately after failing that one. trying manually with that one omitted this time | 19:41 |
| *** jamesmcarthur has quit IRC | 19:44 | |
| fungi | breaking on openstackjenkins-vexxhost sjc1 too | 19:44 |
| *** bobh has quit IRC | 19:45 | |
| donnyd | maybe move FN above those two inline? | 19:47 |
| donnyd | I'm also interested to see that the images are not yet being populated for nodepool | 19:48 |
| donnyd | is that on a cron? | 19:48 |
| fungi | nodepool also needs appropriate keypairs (added by this same cronjob) | 19:49 |
| fungi | and yeah, looks like it's getting through the rest of the list now with those two removed | 19:49 |
| fungi | it's on limestone now so should only be another couple minutes and then i'll rerun the boot | 19:55 |
| fungi | yep, it's doing fn now | 19:55 |
| donnyd | I also thought it was a little strange when the instance was booted, it only picked up an v6 address | 19:56 |
| donnyd | that network has dualstack on it, so it should have grabbed both | 19:56 |
| *** dciabrin has joined #openstack-infra | 19:56 | |
| fungi | keypairs are showing up now | 19:56 |
| fungi | booting | 19:56 |
| *** betherly has joined #openstack-infra | 19:57 | |
| fungi | it is still just showing openstackci=2001:470:e045:2:f816:3eff:fe60:107d | 19:58 |
| donnyd | hrm, responds to icmp.. | 19:58 |
| fungi | i wonder if i should also attach it to the wan network? | 19:58 |
| donnyd | there is already the correct fip provisioned | 19:59 |
| donnyd | want me to add it? | 19:59 |
| donnyd | not wan | 19:59 |
| donnyd | you want to attach to the openstackci subnet, and then use your FIP | 19:59 |
| donnyd | its it doing anything on your end yet, or still waiting for ssh again? | 20:00 |
| fungi | no, i think part of the problem may be that i'm trying to boot a new instance with a name which is already present in our ansible inventory | 20:01 |
| fungi | booting again as 02 instead of 01 | 20:01 |
| donnyd | Shouldn't ansible just reconfigure the instance so long as it matches in name? | 20:01 |
| fungi | our inventory is static, so we need to update it in git | 20:01 |
| fungi | that is, if nova boots it with a different ip address than is already listed in the inventory | 20:02 |
| *** betherly has quit IRC | 20:02 | |
| donnyd | right, but if I change the port and bounce the box it should just fix the broken bits right? | 20:02 |
| donnyd | on the next run | 20:02 |
| donnyd | IDK how your end of things work, so just curious | 20:02 |
| fungi | well, i don't want to keep you, said you had a hard stop at 20:00z | 20:03 |
| donnyd | I am good for a bit longer | 20:04 |
| donnyd | maybe 20ish minutes | 20:04 |
| fungi | but yeah, a big part of the problem is that the launch script is expecting nova to tell it the correct ip address for the instance, so if the address isn't going to be what nova reports to it, then it can't handle that | 20:04 |
| fungi | though at the moment i'm not sure why it's still not able to ssh into the instance | 20:04 |
| donnyd | how long will it wait> | 20:04 |
| fungi | a couple minutes... maybe 5? i'd have to go digging in the script source | 20:04 |
| donnyd | well its going to use dns to ssh | 20:04 |
| donnyd | well i can swap it out and then reboot in that amount of time to find out | 20:05 |
| fungi | actually it ssh's via ip address because we won't necessarily have instances added to dns until after we know their ip addresses | 20:05 |
| donnyd | Ah i C | 20:05 |
| donnyd | but its trying to use the ip it already has in the inventory? | 20:05 |
| fungi | well, it shouldn't now because i've booted an instance with a name not yet in the ansible inventory | 20:06 |
| fungi | but the script is going to need more debugging added as i don't actually know what it's trying to reach | 20:06 |
| fungi | it just keeps reporting "While testing ssh access: [Errno -2] Name or service not known" | 20:06 |
| donnyd | well that is a dns error | 20:07 |
| fungi | yeah, leads me to wonder if it's not correctly setting the ipv6 address in the ssh command line | 20:07 |
| clarkb | can also change the ip? | 20:08 |
| clarkb | its a small dns update | 20:08 |
| donnyd | also for some reason its not picking up the v4 address when the instance is provisioned | 20:09 |
| donnyd | which is weird | 20:09 |
| fungi | yeah, at the moment i'm not even worried about the ip address, still trying to work out why we're not ssh'ing into it at all. i've rerun the launch script with --verbose in hopes of getting more detail | 20:09 |
| fungi | but --verbose only turns on debug openstacksck responses apparently, doesn't tell me the ssh command it's using to test connectivity | 20:10 |
| clarkb | you can also keep it to try and debug afterlaunch fails | 20:10 |
| clarkb | sorry not more help have friends down fro seattle so am weekending now | 20:10 |
| *** bobh has joined #openstack-infra | 20:11 | |
| fungi | trying again without specifying --network=openstackci this time | 20:11 |
| fungi | no worries clarkb! it *is* saturday | 20:11 |
| fungi | omitting --network=openstackci gives the same result, only an ipv6 address allocated | 20:12 |
| fungi | i've found the ssh test and added some code to print out the arguments | 20:14 |
| fungi | okay, the problem is the ip address it's trying to ssh to is an empty string | 20:17 |
| *** armax has joined #openstack-infra | 20:17 | |
| fungi | ip = server.public_v4 | 20:18 |
| * fungi grumbles | 20:18 | |
| fungi | trying to hardcode it to use server.public_v6 for the moment | 20:19 |
| fungi | that does seem to pass in the ipv6 address according to my debug output | 20:19 |
| fungi | looks like it may have worked | 20:19 |
| fungi | the fun part comes when i try to unwind the layer cake of problems i've found and hacked around the last several hours | 20:20 |
| donnyd | Yet another yak to shave | 20:21 |
| donnyd | LOL | 20:21 |
| donnyd | thanks for the patience in working through these issues on a Saturday | 20:21 |
| fungi | thanks for *your* patience! | 20:21 |
| *** e0ne has joined #openstack-infra | 20:22 | |
| *** bobh has quit IRC | 20:23 | |
| fungi | neat, it took rather a while for the ssh host key to generate | 20:25 |
| fungi | initial ssh timed out once the socket was listening, but the script retried | 20:25 |
| donnyd | so its running now? | 20:26 |
| fungi | yeah, it's configuring the instance, i think | 20:26 |
| *** jamesmcarthur has joined #openstack-infra | 20:26 | |
| fungi | yep, it's creating a swapfile now | 20:27 |
| fungi | running various ansible tasks | 20:27 |
| fungi | seems to be underway | 20:27 |
| fungi | updating packages | 20:27 |
| donnyd | well that is pretty good news | 20:28 |
| donnyd | so after its done, we can swap out the port and move on to funner things | 20:28 |
| donnyd | however not till tuesday... because my fans are stuck in the midwest | 20:29 |
| *** betherly has joined #openstack-infra | 20:29 | |
| donnyd | and I am not at all frustrated or mad about it being late... | 20:29 |
| fungi | Temporary failure resolving 'us.archive.ubuntu.com' | 20:30 |
| fungi | so that eventually broke because i think we set the resolver/forwarder to use ipv4 | 20:30 |
| donnyd | um i see no instance | 20:30 |
| fungi | yes, the launch script deleted it on failure | 20:30 |
| donnyd | ok, why don't we try a different approach | 20:30 |
| donnyd | I will restrict the network to only having one ip available | 20:31 |
| donnyd | the one its supposed to be | 20:31 |
| fungi | yeah, i think the inexplicable lack of ipv4 connectivity is going to continue to bite us in other places otherwise | 20:32 |
| donnyd | yea, that makes no sense at all | 20:33 |
| fungi | if you're restricting the addresses to what we have in dns for mirror01.regionone.fortnebula.opendev.org then i can reuse that hostname because it will jive with what's present in the static ansible inventory | 20:33 |
| donnyd | lemme go look through the neutron logs to see what they say | 20:33 |
| donnyd | yea, I am | 20:33 |
| fungi | at this point we're just asking nova to boot an instance and pick whatever networking it thinks is best, and this is what it's giving us | 20:34 |
| fungi | so curious what's going on there | 20:34 |
| *** betherly has quit IRC | 20:34 | |
| donnyd | yea, that is a super weird issue to have.. if it has two networks... it should dual stack the instance | 20:35 |
| donnyd | almost done | 20:36 |
| donnyd | even more interesting is when i boot a test instance it gets boht | 20:39 |
| donnyd | even more interesting is when i boot a test instance it gets both | 20:39 |
| donnyd | zomg... so neutron is handing out an ip that isn't even in the available pool | 20:41 |
| fungi | yikes | 20:41 |
| donnyd | I am pretty sure that isn | 20:44 |
| donnyd | isn't supposed to happen | 20:44 |
| donnyd | Yea at this point it seems like it may be less work to update the dns record | 20:47 |
| donnyd | do you want to give a spin on your end | 20:47 |
| donnyd | try to figure out why its only getting a v6 address | 20:47 |
| fungi | will do | 20:47 |
| *** e0ne has quit IRC | 20:47 | |
| donnyd | The only thing changed on my end is using bgp to route instead | 20:48 |
| donnyd | everything else has been copy / paste (ish) | 20:48 |
| *** bhavikdbavishi has quit IRC | 20:49 | |
| donnyd | Well this is new | 20:50 |
| donnyd | https://www.irccloud.com/pastebin/6bLcihla/ | 20:50 |
| donnyd | I am thinking its this little fella right here | 20:52 |
| donnyd | 2019-07-27 16:46:55.594 2214 INFO oslo_rootwrap.client [-] Spawned new rootwrap daemon process with pid=2646 | 20:52 |
| donnyd | 2019-07-27 16:46:55.610 2214 ERROR neutron.agent.linux.utils [-] Rootwrap error running command: ['conntrack', '-D', '-f', 'ipv4', '-s', '192.168.133.163', '-w', '4111']: RemoteError: | 20:52 |
| donnyd | ok, try again | 20:52 |
| fungi | yeah, that dunna look good | 20:53 |
| donnyd | it would seem that conntrack wasn't installed... which i am not sure how that even happens... | 20:53 |
| fungi | huhwha? | 20:53 |
| donnyd | Yea, idk fungi | 20:53 |
| donnyd | LOL | 20:53 |
| donnyd | This hypervisor has only been pointed at a different controller... worked fine on the other one | 20:54 |
| donnyd | give it another whirl and we will see if it works this time | 20:54 |
| fungi | that still seems to have only assigned an ipv6 address and no ipv4 address | 20:55 |
| donnyd | did you delete an rebuild? | 20:55 |
| fungi | yep | 20:55 |
| fungi | 6040f911-cc94-4264-8da6-27f256d59c54 is the new instance | 20:55 |
| donnyd | well the error is gone | 20:56 |
| donnyd | can you give it one more spin please | 20:56 |
| fungi | that instance is deleted, server list is now empty | 20:56 |
| fungi | booting another | 20:57 |
| donnyd | yea, same deal | 20:57 |
| *** whoami-rajat has quit IRC | 20:58 | |
| donnyd | so more interesting tidbits | 20:59 |
| donnyd | i remove the interface and reattached a new one and it comes up with both | 20:59 |
| donnyd | I am about out of time today | 20:59 |
| donnyd | I won't have anything useful until tuesday anyways | 21:00 |
| *** betherly has joined #openstack-infra | 21:00 | |
| fungi | indeed, i see it now also shows an rfc 1918 address | 21:00 |
| fungi | okay, well thanks for digging into this! i'm also going to see if i can find the needle of the server create request api call in this haystack of debug output | 21:01 |
| fungi | maybe that will yeild some clues | 21:01 |
| fungi | or, yield | 21:01 |
| fungi | have a good weekend donnyd! | 21:01 |
| donnyd | you too fungi | 21:02 |
| donnyd | thanks for all your hard work today | 21:02 |
| fungi | and yours as well | 21:02 |
| *** betherly has quit IRC | 21:05 | |
| *** jamesmcarthur has quit IRC | 21:05 | |
| *** jamesmcarthur has joined #openstack-infra | 21:18 | |
| *** ralonsoh has quit IRC | 21:31 | |
| *** betherly has joined #openstack-infra | 21:31 | |
| *** betherly has quit IRC | 21:36 | |
| *** jamesmcarthur has quit IRC | 21:47 | |
| *** jamesmcarthur has joined #openstack-infra | 21:48 | |
| *** betherly has joined #openstack-infra | 21:52 | |
| *** jamesmcarthur has quit IRC | 21:53 | |
| fungi | mordred: to wrap up our earlier conversation, my solution to rapidly aging dock cleats is to put them in a mason jar filled with chlorine bleach | 21:56 |
| *** betherly has quit IRC | 21:56 | |
| fungi | "solution" was of course an accidental pun | 21:57 |
| fungi | (as it's sodium hypochlorite in an aqueous solution) | 21:58 |
| fungi | i had wrongly assumed they were cast iron, but they appear to either be nickel plate or cast aluminum, judging from the color of oxidation accumulating on their surface | 22:00 |
| *** jamesmcarthur has joined #openstack-infra | 22:01 | |
| fungi | thinking cast aluminum is more likely | 22:01 |
| fungi | oh, or they *might* be galvanized | 22:02 |
| fungi | zinc oxide would look like this | 22:02 |
| *** trident has quit IRC | 22:03 | |
| *** pkopec has joined #openstack-infra | 22:05 | |
| *** trident has joined #openstack-infra | 22:06 | |
| *** betherly has joined #openstack-infra | 22:13 | |
| *** betherly has quit IRC | 22:18 | |
| *** jamesmcarthur has quit IRC | 22:32 | |
| *** betherly has joined #openstack-infra | 22:33 | |
| *** betherly has quit IRC | 22:38 | |
| *** pkopec has quit IRC | 22:47 | |
| mordred | fungi: good solution! | 23:16 |
| mordred | fungi: (both to the problem, and the liquid substance) | 23:17 |
| fungi | clarkb: looking back through shell history, i may have missed the git gc on 02 as well, so running that now (and then 04, which is nearly ready) | 23:19 |
| *** betherly has joined #openstack-infra | 23:25 | |
| fungi | 02 is taking long enough i suspect this is its first git gc | 23:27 |
| *** altlogbot_0 has quit IRC | 23:29 | |
| *** betherly has quit IRC | 23:30 | |
| *** altlogbot_3 has joined #openstack-infra | 23:30 | |
| fungi | git gc on gitea02 is *still* running | 23:41 |
| *** betherly has joined #openstack-infra | 23:46 | |
| *** betherly has quit IRC | 23:51 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!