| opendevreview | OpenStack Proposal Bot proposed openstack/project-config master: Normalize projects.yaml https://review.opendev.org/c/openstack/project-config/+/809526 | 06:13 |
|---|---|---|
| opendevreview | Merged openstack/project-config master: Normalize projects.yaml https://review.opendev.org/c/openstack/project-config/+/809526 | 07:02 |
| opendevreview | Merged zuul/zuul-jobs master: Pin protobuf<3.18 for Python<3.6 https://review.opendev.org/c/zuul/zuul-jobs/+/809460 | 07:03 |
| fungi | clarkb: if you're around at some point, looks like yuriys is ready for approval on https://review.opendev.org/809897 (i can check in on the status after it deploys to see if things seem to be working as anticipated) | 14:12 |
| yuriys | I can watch cloud state as well. | 14:17 |
| Clark[m] | fungi if you want to approve it that is fine with me. Will be a little bit before I can sit down and approve it myself. Today is drive through flu vaccines for the whole family so trying to get out the door | 15:01 |
| yuriys | It's also totally a Saturday! | 15:13 |
| yuriys | It can wait till Monday if it needs to fungi, no rush. | 15:14 |
| fungi | no sweat, it's a trivial change, just wanted to give Clark[m] a chance to veto. approving it now, should hopefully not take too long to deploy the config update | 15:21 |
| opendevreview | Merged openstack/project-config master: Scaling up Inmotion Cloud workloads. https://review.opendev.org/c/openstack/project-config/+/809897 | 15:33 |
| fungi | looks like the deploy reported success at 15:40:18 so it's been about 45 minutes, we should see some nodes in use | 16:28 |
| fungi | definitely see the quota bump, but i expect we're populating images first before we'll have nodes booting: https://grafana.opendev.org/d/4sdNjeXGk/nodepool-inmotion?orgId=1 | 16:29 |
| fungi | well, no, we have images there in a ready state from earlier | 16:31 |
| fungi | aha, zuul is simply a ghost town right now. there are a handful of changes in openstack's check pipeline, and nothing else going on at the moment in any other tenants: https://zuul.opendev.org/t/openstack/status | 16:33 |
| fungi | yep, just confirmed, no errors in the nl02 log, the inmotion-iad3-main poolworker simply hasn't had an opportunity to grab a node request yet | 16:35 |
| yuriys | Awesome. No worries. | 16:35 |
| yuriys | It's also Saturday! haha. | 16:36 |
| fungi | well, i was just passing time while waiting for a cake to finish baking | 16:38 |
| Clark[m] | The problem with demand based scheduling is without demand we don't see it in action :) | 16:51 |
| fungi | i demand action! | 17:06 |
| fungi | well, that didn't work ;) | 17:06 |
| fungi | 2021-09-18 17:04:31,373 ERROR nodepool.NodeLauncher: [e: d6fd0aea8bec4dee83a23a455a32f63f] [node_request: 900-0015464128] [node: 0026559417] Launch failed for node ubuntu-bionic-inmotion-iad3-0026559417: | 17:15 |
| fungi | nodepool.exceptions.ConnectionTimeoutException: Timeout waiting for connection to 173.231.255.104 on port 22 | 17:15 |
| fungi | so we may have connectivity issues, or we may have nodes booting too slowly, hard to say just from the timeout exception alone | 17:15 |
| fungi | i assume nothing changed with our standard allow all security group | 17:19 |
| Clark[m] | It shouldn't have. And we added the new subnets to the existing External network. | 17:28 |
| Clark[m] | I wonder if the subnets aren't routed properly for some reason | 17:28 |
| fungi | that would be one of the simplest explanations, sure | 17:32 |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!