| openstackgerrit | Mohammed Naser proposed openstack/project-config master: opendev: add openstack/devstack https://review.opendev.org/713123 | 01:04 |
|---|---|---|
| openstackgerrit | Mohammed Naser proposed openstack/project-config master: opendev: add openstack/devstack https://review.opendev.org/713123 | 01:06 |
| openstackgerrit | Mohammed Naser proposed openstack/project-config master: opendev: move vexxhost to seperate tenant https://review.opendev.org/713123 | 01:10 |
| openstackgerrit | Mohammed Naser proposed openstack/project-config master: opendev: move vexxhost to seperate tenant https://review.opendev.org/713123 | 01:11 |
| *** DSpider has joined #opendev | 06:31 | |
| openstackgerrit | Andreas Jaeger proposed zuul/zuul-jobs master: Keep doc/source/roles.rst sorted https://review.opendev.org/713128 | 08:30 |
| openstackgerrit | Sorin Sbarnea proposed zuul/zuul-jobs master: Improve ensure-tox role https://review.opendev.org/708642 | 11:06 |
| openstackgerrit | Sorin Sbarnea proposed zuul/zuul-jobs master: tox: allow tox to be upgraded https://review.opendev.org/690057 | 11:26 |
| openstackgerrit | Sorin Sbarnea proposed zuul/zuul-jobs master: tox: allow tox to be upgraded https://review.opendev.org/690057 | 11:33 |
| openstackgerrit | Jeremy Stanley proposed opendev/system-config master: Set up LE certs for docs.airshipit.org static site https://review.opendev.org/706600 | 14:14 |
| openstackgerrit | Jeremy Stanley proposed opendev/system-config master: Add a new docs.airshipit.org vhost on static01 https://review.opendev.org/706601 | 14:14 |
| openstackgerrit | Mohammed Naser proposed opendev/system-config master: nodepool-builder: mount SSH keys into the container https://review.opendev.org/713136 | 15:43 |
| mnaser | fungi: what special hostname? i wasn't following too closely the conversation over the past few days | 15:46 |
| fungi | looking | 15:46 |
| clarkb | I believe all the changes so far have been captured on host in the compose file including the hostname | 15:48 |
| fungi | looks like we were passing --hostname nb01opendev to podman run? per 20:11-20:13 utc friday | 15:49 |
| clarkb | we arent using podman anymore | 15:49 |
| fungi | but then there was a problem with that name not resolving inside the container | 15:49 |
| clarkb | podman didnt work due to mount propagation issues | 15:49 |
| clarkb | there should be a hostname override in the compose file and docker compose is applying that | 15:50 |
| fungi | ahh, okay, there's ~1k lines of scrollback dealing with this, so still trying to wrap my head around where it ended up | 15:50 |
| clarkb | for the host resolution we edited /etc/hosts and since we usehostnetworking that resolved it | 15:50 |
| clarkb | for this issue I think we need to make sure the ssh key is in place, then add the mount to the compose file then docker compose stop/shutdown/whatever the command is then docker conpose ul | 15:51 |
| clarkb | *up | 15:51 |
| fungi | aha, yep, /etc/hosts on nb01.opendev.org includes a line like | 15:51 |
| fungi | 127.0.1.1 nb01.opendev.org nb01 nb01opendev.opendev.org nb01opendev | 15:51 |
| clarkb | and there is a flag to make it background so you dont have to do the disconnection corvus did | 15:51 |
| clarkb | then trigger an image rebuild | 15:52 |
| clarkb | I can help with that in a bit but not right this second. 15-30 minutes probably | 15:52 |
| fungi | we don't have the convenience script in place to call the rpc client through docker yet either, looks like? | 15:54 |
| fungi | i guess we can do that from another nodepool server? | 15:55 |
| mnaser | fungi: you can probably do that with the docker cli | 15:56 |
| mnaser | fungi: docker exec <container-naem> nodepool list | 15:56 |
| mnaser | _should_ work | 15:56 |
| fungi | cannot access '/home/nodepool/.ssh': No such file or directory | 15:57 |
| clarkb | fungi: ya I would do it from another server for simplicity | 15:57 |
| fungi | ~nodepool does resolve to /home/nodepool | 15:57 |
| fungi | but that doesn't seem to contain a .ssh directory | 15:57 |
| fungi | that does exist on our other builders | 15:58 |
| clarkb | fungi: note becayse we arebind mounting we can actually stash the ssh key wherever we want. But consistency with the other builders is probably good for now | 16:00 |
| fungi | well, i'm trying to figure out what's supposed to be putting the key on the server at all and where it might be doing that currently | 16:00 |
| clarkb | I expect the new ansible just doesnt do it | 16:01 |
| clarkb | puppet does it on the old hosts | 16:01 |
| clarkb | also I've just had a lightbulb | 16:01 |
| clarkb | I think we can set this in the /etc/nodepool/nodepool.yaml config instead and that will avoid needing to restart anything | 16:02 |
| fungi | i guess looks like you're already editing that as i see a .swp file | 16:06 |
| clarkb | ya managed to make it to computer then wanted to do lightbulb idea | 16:07 |
| clarkb | fungi: check git diff on /opt/project-config | 16:07 |
| clarkb | that should be used by the next run so now I think we can trigger a build | 16:08 |
| clarkb | fungi: do you want to do that or should I? | 16:08 |
| clarkb | I'm adding a task to the storyboard story now | 16:09 |
| fungi | sorry, just finished adding the public key | 16:10 |
| fungi | ~nodepool/.ssh/id_rsa.pub is there now and has correct permissions (as does the empty ~nodepool/.ssh/id_rsa and the ~nodepool/.ssh dir) | 16:11 |
| clarkb | cool so we can switch it to the consistent way if we want | 16:11 |
| fungi | copied from what nb01.openstack.org has | 16:11 |
| clarkb | (but the config update should be sufficient for now if we want to wait on restarting services) | 16:11 |
| fungi | oh, got it, now i see what you meant by editing nodepool.yaml | 16:11 |
| fungi | i thought you needed to edit it to add the mount | 16:12 |
| fungi | but that would have been the dockerfile | 16:12 |
| fungi | or docker-compose file or whatever | 16:12 |
| fungi | so you added the literal public key content instead | 16:12 |
| clarkb | fungi: ya, beacuse nodepool rereads its config and will apply that on the next run | 16:13 |
| fungi | honestly i think adding it to the nodepool config makes more sense if no processes running on nodepool actually need it and it's just for embedding into the image | 16:13 |
| clarkb | basically this allows us to bypass restarting things | 16:13 |
| fungi | if we can get by without populating ~nodepool/.ssh at all on the builders then all the better | 16:13 |
| clarkb | fungi: ya I could see us switchign to that form globally (its nice and verbose too rather than hidden in the system setup) | 16:13 |
| clarkb | fungi: https://storyboard.openstack.org/#!/story/2007407 task 39072 (see notes) | 16:14 |
| * fungi makes thumbs-up | 16:15 | |
| clarkb | fungi: should I trigger the build now? | 16:16 |
| fungi | yep please | 16:16 |
| fungi | i guess the argument for keeping it in ~nodepool/.ssh is that the launchers use it? though it doesn't seem like they actually do if the nodes were getting handed over to zuul in this state | 16:16 |
| clarkb | thats a fun nodepool cli behavior | 16:16 |
| fungi | oh? | 16:17 |
| clarkb | I can't trigger the build on the old servers because they have paused that image | 16:17 |
| fungi | hah | 16:17 |
| clarkb | so we do need to sort out the docker exec here | 16:17 |
| clarkb | `sudo docker exec nodepoolbuildercompose_nodepool-builder_1 nodepool image-build fedora-30` seems to be it | 16:18 |
| fungi | oh, cool | 16:18 |
| clarkb | fedora-30-0000000558.log if people want to follow along | 16:18 |
| clarkb | mnaser: ^ fyi | 16:18 |
| clarkb | fungi: when figuring that command out I was sure to run whoami as the command first just to be double sure that I had the right user | 16:19 |
| clarkb | fungi: though it would have simply failed if not | 16:19 |
| fungi | yup | 16:19 |
| openstackgerrit | Merged opendev/system-config master: Set up LE certs for docs.airshipit.org static site https://review.opendev.org/706600 | 16:19 |
| clarkb | fungi: I could also have run `sudo docker exec -it nodepoolbuildercompose_nodepool-builder_1 bash` and dont it interactively | 16:20 |
| clarkb | *done it | 16:20 |
| fungi | i see | 16:20 |
| clarkb | fwiw I'm not 100% sure the builders reload configs like launchers do but am like ~99% sure | 16:22 |
| clarkb | if we still see this failing then next step would be to stop start it via docker-compose | 16:22 |
| clarkb | I'm going to step away from the computer now. It is sunny and dry today and nto snowing anymore so probably going to try and enjoy the weather | 16:23 |
| fungi | enjoy! | 16:23 |
| clarkb | as noted in -infra dnf seems to be on a lunch break. I'm not sure what is going on, but that likely needs more in depth debugging than I can do today | 17:00 |
| clarkb | disks are nowhere near full | 17:00 |
| AJaeger | still nothing for two hours on https://nb01.opendev.org/fedora-30-0000000558.log ;( | 17:42 |
| AJaeger | clarkb, checking https://nb01.opendev.org/fedora-30-0000000557.log - there's also 2hours and 20 mins break during install, so it might continue... | 17:43 |
| AJaeger | and same package - haveged. That looks like something is broken with it ;( | 17:45 |
| clarkb | thabks for checking. I think we just eait then | 17:48 |
| *** jkt has quit IRC | 18:39 | |
| *** corvus has quit IRC | 18:39 | |
| *** jkt has joined #opendev | 18:40 | |
| *** corvus has joined #opendev | 18:40 | |
| AJaeger | so, build continued... | 18:57 |
| fungi | nice | 19:03 |
| AJaeger | "Build completed successfully" | 19:23 |
| AJaeger | how long will it take to upload? | 19:26 |
| clarkb | usually within 10 minutes some cloudshave the images but then depends on the others. I guess we can delete the old image to ensure the new one is the only one used | 19:27 |
| AJaeger | I'm following http://zuul.opendev.org/t/zuul/stream/e1ef0a6b23d445dcaf7be094d62ad21d?logfile=console.log - looks good so far | 19:38 |
| AJaeger | passed \o/ | 19:41 |
| AJaeger | now zuul-jobs-test-install-docker-centos-7 has a post_failure | 19:41 |
| fungi | but at least it's not f30? ;) | 19:41 |
| AJaeger | yep. f30 success :) | 19:42 |
| AJaeger | http://zuul.opendev.org/t/zuul/status/change/712547,5 | 19:42 |
| AJaeger | (second f30 test still running) | 19:42 |
| fungi | zbr: ^ i guess you can recheck your failed builds now | 19:42 |
| * AJaeger rechecked on of zbr's already | 19:44 | |
| AJaeger | 2nd f30 job passed as well, so seems we're goog | 19:45 |
| *** mordred has quit IRC | 20:30 | |
| *** mordred has joined #opendev | 20:31 | |
| *** jaicaa has quit IRC | 21:11 | |
| *** jaicaa has joined #opendev | 21:13 | |
| *** AJaeger has quit IRC | 21:13 | |
| *** AJaeger has joined #opendev | 21:26 | |
| *** factor has joined #opendev | 21:31 | |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: Switch back to docker for gerrit and nodepool-builder https://review.opendev.org/713101 | 21:41 |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: Switch back to docker for gerrit and nodepool-builder https://review.opendev.org/713101 | 22:44 |
| *** DSpider has quit IRC | 23:00 | |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: nodepool-builder: put container configs in /etc https://review.opendev.org/713148 | 23:15 |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: nodepool-builder: put container configs in /etc https://review.opendev.org/713148 | 23:22 |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: Switch back to docker for gerrit and nodepool-builder https://review.opendev.org/713101 | 23:27 |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: bindep: remove lsb-release https://review.opendev.org/713150 | 23:39 |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!