| *** DSpider has joined #opendev | 02:53 | |
| prometheanfire | mordred: might I ask for a review on https://review.opendev.org/717339 ? | 03:36 |
|---|---|---|
| AJaeger | infra-root, we have changes since more than 12 hours in check queue waiting for nodes - and everything idle, see for example http://zuul.opendev.org/t/openstack/status/change/720942,1 | 05:21 |
| AJaeger | corvus: is that the time you were restarting zookeeper? Did we loose events? | 05:23 |
| *** melwitt has quit IRC | 06:13 | |
| *** mugsie has quit IRC | 06:13 | |
| *** cmurphy has quit IRC | 06:13 | |
| *** Open10K8S has quit IRC | 06:13 | |
| *** melwitt has joined #opendev | 06:15 | |
| *** cmurphy has joined #opendev | 06:15 | |
| *** mugsie has joined #opendev | 06:15 | |
| *** Open10K8S has joined #opendev | 06:15 | |
| *** parsley936 has joined #opendev | 11:39 | |
| AJaeger | looks like they've recovered now... | 12:22 |
| corvus | AJaeger: it might be the thing with openstack quotas clarkb has been looking into and will continue next week | 14:26 |
| *** parsley936 has quit IRC | 14:27 | |
| *** Goriya has joined #opendev | 14:36 | |
| Goriya | hey | 14:36 |
| mordred | corvus: so - I relized something yesterday when poking at the nodepool ansible stuff | 14:51 |
| mordred | corvus: which is that nb03 runs on arm | 14:51 |
| mordred | corvus: so we either need to start making arm containers too - or we'll need to keep some ansible for installing directly via pip on some builders | 14:52 |
| corvus | mordred: oh, yep! that's interesting. any idea how to make an arm container? :) | 14:53 |
| mordred | corvus: I have started to read things :) | 14:54 |
| mordred | corvus: https://www.docker.com/blog/getting-started-with-docker-for-arm-on-linux/ | 14:54 |
| mordred | corvus: there's something about docker image manifests involved that their experimental "buildx" command supports - so I think at some point we might need to think about multi-arch support in registry too? | 14:55 |
| mordred | corvus: ugh. their tooling uses qemu | 14:58 |
| mordred | I'm going to need to learn more things | 15:00 |
| mordred | corvus: this: https://dev.to/toolboc/publish-multi-arch-docker-images-to-a-single-repo-with-docker-manifests-329n is more helpful | 15:04 |
| mordred | corvus: if you look down at "Using Docker Manifests in a real project " | 15:05 |
| mordred | corvus: I think we could basically have two sets of jobs that create and publishe zuul/nodepool-builder:x86 and zuul/nodepool-builder:arm images, and then an additional publish job that creates and publishes a manifest pointing to them | 15:07 |
| mordred | corvus: that way we can use arm nodes to build the arm images and x86 nodes to build the x86 images | 15:07 |
| mordred | corvus: biggest issue will be, I think, buildset registry wanting to be in the same cloud as build nodes | 15:08 |
| mordred | so we might need to run an arm buildset registry tied to arm nodes (and thus scheduled in arm clouds) with the arm image build jobs requires-ing it *waves hands* | 15:09 |
| mordred | corvus: of course - for things to be _really_ solid we'll need such a setup to also publish multi-arch manifests in the intermediate registry | 15:13 |
| mordred | corvus: buildah has manifest manipulation support: https://github.com/containers/buildah/blob/master/docs/buildah-manifest.md | 15:17 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Remove puppet-beaker-rspec-puppet-4-infra-system-config https://review.opendev.org/720799 | 15:57 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Remove unused rspec tests https://review.opendev.org/720802 | 15:57 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Make applytest files outside of system-config https://review.opendev.org/720848 | 15:57 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Move puppet apply jobs to system-config repo https://review.opendev.org/720887 | 15:57 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Remove old etherpad.openstack.org https://review.opendev.org/717492 | 15:57 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Remove unused gerrit puppet things https://review.opendev.org/714001 | 15:57 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Remove global variables from manifest/site.pp https://review.opendev.org/720800 | 15:57 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Stop cloning a bunch of puppet modules we don't use https://review.opendev.org/720892 | 15:57 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Remove old etherpad.openstack.org https://review.opendev.org/717492 | 15:59 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Remove unused gerrit puppet things https://review.opendev.org/714001 | 15:59 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Remove global variables from manifest/site.pp https://review.opendev.org/720800 | 15:59 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Stop cloning a bunch of puppet modules we don't use https://review.opendev.org/720892 | 15:59 |
| *** Goriya has quit IRC | 16:40 | |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Stop cloning a bunch of puppet modules we don't use https://review.opendev.org/720892 | 17:34 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Stop using jenkinsuser on eavesdrop https://review.opendev.org/721098 | 17:34 |
| openstackgerrit | Monty Taylor proposed openstack/project-config master: Use zuul deployment keys for yaml2ical https://review.opendev.org/721099 | 17:35 |
| corvus | i just layed in provisions again... the grocery store had a little bit of flour, very little butter, half the usual milk, some sugar, some toilet paper, no dish detergent, almost no sausages or deli meats, but plenty of bread. | 17:46 |
| *** parsley936 has joined #opendev | 18:20 | |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Split eavesdrop into its own playbook https://review.opendev.org/721098 | 18:43 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Stop cloning a bunch of puppet modules we don't use https://review.opendev.org/720892 | 18:43 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Split eavesdrop into its own playbook https://review.opendev.org/721098 | 19:00 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Stop cloning a bunch of puppet modules we don't use https://review.opendev.org/720892 | 19:00 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Remove some extra bits from site.pp https://review.opendev.org/721101 | 19:00 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Split codesearch into its own playbook https://review.opendev.org/721102 | 19:00 |
| mordred | clarkb: ^^ I have started splitting out services | 19:01 |
| mordred | clarkb: as I touch ones that use puppet-project_config, I'm updating them to use sync-project-config instead | 19:01 |
| mordred | hrm. it looks like the only thing openstack_project::server does is install afs client - and we have an ansible role for that | 19:02 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Split eavesdrop into its own playbook https://review.opendev.org/721098 | 19:06 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Stop cloning a bunch of puppet modules we don't use https://review.opendev.org/720892 | 19:06 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Remove some extra bits from site.pp https://review.opendev.org/721101 | 19:06 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Split codesearch into its own playbook https://review.opendev.org/721102 | 19:06 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Split eavesdrop into its own playbook https://review.opendev.org/721098 | 20:39 |
| *** DSpider has quit IRC | 20:42 | |
| * prometheanfire is gonna ask again | 20:48 | |
| prometheanfire | any glean core around that can add a second review to https://review.opendev.org/717339 ? | 20:48 |
| * prometheanfire sees a bunch of old reviews with no action as well | 20:48 | |
| *** tbarron_ has joined #opendev | 21:20 | |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Split eavesdrop into its own playbook https://review.opendev.org/721098 | 21:22 |
| openstackgerrit | Merged opendev/glean master: write one resolv config https://review.opendev.org/717339 | 21:29 |
| openstackgerrit | Merged opendev/glean master: changes for alpine compatibility https://review.opendev.org/619258 | 21:36 |
| prometheanfire | mordred: thanks :D | 21:38 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Split eavesdrop into its own playbook https://review.opendev.org/721098 | 22:16 |
| mordred | corvus, clarkb: any idea why we remove cloud-init in a playbook in launch node and not in the base-server role? | 22:26 |
| * mordred is thinking it's a historical remnant from before we ran a base-server role on all of our servers | 22:27 | |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Move cloud-init removal to its own playbook https://review.opendev.org/721106 | 22:30 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Just move cloud-init removal into base-server https://review.opendev.org/721107 | 22:30 |
| mordred | well - there's two versions of a thing | 22:30 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Split eavesdrop into its own playbook https://review.opendev.org/721098 | 22:44 |
| ianw | hrm, nb04 hasn't enjoyed having those extra plain images added to it ... /usr/local/lib/python3.7/site-packages/diskimage_builder/lib/disk-image-create: line 223: /dev/fd/62: No such file or directory | 22:49 |
| ianw | that is an odd one | 22:49 |
| mordred | ianw: there were some issues we didn't fully track down at the end of the week on nb04 that seemed to get resolved with a restart | 22:52 |
| mordred | ianw: but we were seeing issues trying to do a docker exec into the nodepool builder container - with errors about missing /dev/ptsx or something | 22:52 |
| ianw | hrm, i'll have to scroll back ... sounds similar though | 22:53 |
| mordred | so it's possible there's some sadness lurking still related to the dib+docker games | 22:53 |
| mordred | ianw: https://etherpad.opendev.org/p/PzoWHp44yOP4K8LdXXrK | 22:53 |
| mordred | ianw: that's an etherpad we started to track issues | 22:53 |
| ianw | ok thanks ... let me see if i can spot anything ... | 22:54 |
| mordred | ianw: looks like we didn't capture much info on the nb04 issues | 22:56 |
| ianw | starting container process caused "close exec fds: open /proc/self/fd: no such file or directory": unknown | 22:58 |
| ianw | i guess that was related | 22:58 |
| ianw | a lot of people suggest it's config related | 23:00 |
| mordred | neat | 23:02 |
| mordred | ianw: yeah - I'm guessing something something dib + something something docker == something something | 23:07 |
| ianw | "I have spent a lot of time attempting to track down the root cause, but I keep going in circles." | 23:07 |
| ianw | great comment to see in an issue tracker :) | 23:07 |
| *** parsley936 has quit IRC | 23:23 | |
| *** mrunge has quit IRC | 23:32 | |
| mordred | ++ | 23:34 |
| ianw | see what i can see in https://etherpad.opendev.org/p/8GqzoNcof6_QHo1jfw2V | 23:37 |
| *** mrunge has joined #opendev | 23:49 | |
| ianw | it looks like it's getting commands to map /proc correctly | 23:50 |
| *** icarusfactor has quit IRC | 23:59 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!