| *** mlavalle has quit IRC | 00:00 | |
| *** xiaolin has joined #opendev | 01:35 | |
| *** ysandeep|off is now known as ysandeep | 03:36 | |
| *** xiaolin has quit IRC | 03:37 | |
| *** ravsingh has joined #opendev | 04:58 | |
| *** ravsingh has quit IRC | 06:22 | |
| *** xiaolin has joined #opendev | 06:48 | |
| openstackgerrit | Merged opendev/zone-opendev.org master: Revert "Add three more jvbs to DNS" https://review.opendev.org/733923 | 07:08 |
|---|---|---|
| *** xiaolin has quit IRC | 07:11 | |
| *** avass has quit IRC | 07:45 | |
| *** sgw has quit IRC | 07:56 | |
| *** moppy has quit IRC | 08:01 | |
| *** moppy has joined #opendev | 08:02 | |
| *** xiaolin has joined #opendev | 08:09 | |
| *** slaweq has joined #opendev | 08:46 | |
| *** DSpider has joined #opendev | 09:21 | |
| *** xiaolin has quit IRC | 09:28 | |
| *** elod has quit IRC | 09:38 | |
| *** tosky has joined #opendev | 09:49 | |
| *** slaweq has quit IRC | 09:53 | |
| *** slaweq has joined #opendev | 10:03 | |
| *** slaweq has quit IRC | 10:35 | |
| *** elod has joined #opendev | 11:14 | |
| *** tosky has quit IRC | 11:18 | |
| *** tosky has joined #opendev | 12:17 | |
| *** tosky has quit IRC | 12:17 | |
| *** tosky has joined #opendev | 12:17 | |
| mordred | clarkb, corvus: I found myself down a little rabbit hole this morning and read this: https://github.com/opencontainers/runc/issues/453 as well as found the "no_new_keyring" option in config.json of the OCI spec - which makes me think maybe there's something in there WRT AFS in bubblewrap in a container - like maybe we need to add a seccomp profile that grants the container the ability to create a session | 12:28 |
| mordred | keyring | 12:28 |
| mordred | clarkb, corvus: there's also mention of keyctl and request_key in https://docs.docker.com/engine/security/seccomp/#significant-syscalls-blocked-by-the-default-profile | 12:29 |
| mordred | (which also mentions that the kernel keyring is not namespaced) | 12:29 |
| mordred | corvus: ooh - --security-opt seccomp=unconfined as an option to docker will run without the seccomp profile - might be a quick and easy way to see if the seccomp profile is the thing that's causing bubblewrap+AFS to not work right | 12:30 |
| mordred | corvus: also: https://web.mit.edu/kerberos/krb5-1.12/doc/basic/ccache_def.html might give us some options | 12:46 |
| corvus | mordred: i think that works | 13:39 |
| corvus | mordred: https://etherpad.opendev.org/p/7ObyoG4up0RndQ1ycKNP | 13:39 |
| corvus | that's the test procedure i used | 13:39 |
| corvus | i ran two instances of that simultaneously, and they did not seem to share credentials. i believe that was the failure we saw last time -- that all docker containers shared the same tokens. when i kinit/aklog in the first one, and do nothing in the second, the first has access and the second does not. when i kinit/aklog in the second with a different principal, the first does not switch to operating | 13:42 |
| corvus | with the second principal, it continues using the first. | 13:42 |
| corvus | i also verified neither is available in a third container or the host system. | 13:42 |
| corvus | (zuultest in that etherpad is a local image of zuul/zuul-executor with kinit and aklog installed) | 13:44 |
| corvus | mordred: i think that the important thing here for security is that any container where we allow those syscalls needs to trust all other containers where we allow those syscalls on the same host. since a zuul-executor is the only container we would do that for, i think we can agree to those terms both now and even in the future (if there's a way to do this with k8s, that would probably be fine too). | 13:48 |
| corvus | mordred: i made a custom secconmp profile with add_key, keyctl, and request_key added, and it works | 13:55 |
| mordred | corvus: ah - cool - so just doing the custome seccomp profile is probably a nice way to do that for us | 13:56 |
| corvus | mordred: yep, i think that's going to be a winner -- double checking a few things real quick before i declare the test complete... | 13:57 |
| mordred | corvus: I'm guessing that would probably mean in a k8s sense that one would want to add scheduling hints to prevent any non-zuul-executor from running on a host | 13:57 |
| mordred | corvus: but ... this is a set of things that are only needed for using afs | 13:57 |
| mordred | so it's probably ok to say "if you want to use afs from a zuul-executor in k8s, you need to apply this seccomp profile and these scheduler hints" | 13:58 |
| corvus | mordred: maybe? if you could specify a seccomp profile for a specific container, then i don't think that would be necessary; otherwise, yeah, that would probably be a good idea | 13:58 |
| mordred | corvus: oh - good point | 13:58 |
| fungi | i concur with the security risk assessment around in-kernel key management | 13:59 |
| mordred | corvus: https://kubernetes.io/docs/concepts/policy/pod-security-policy/#seccomp | 14:00 |
| mordred | corvus: doesn't look like it's super-fine-grained in k8s | 14:00 |
| mordred | the k8s admin can control the list of seccomp profiles that pods can use | 14:00 |
| mordred | but it doesn't look like there's a way to say "this pod can use this profile, but others can't" | 14:00 |
| corvus | mordred: that seems like a big hole | 14:01 |
| mordred | corvus: oh - actually | 14:01 |
| mordred | corvus: "unconfined - Seccomp is not applied to the container processes (this is the default in Kubernetes), if no alternative is provided." | 14:01 |
| mordred | corvus: am I reading that right that k8s turns off seccomp by default? | 14:01 |
| corvus | mordred: wow. k8s is all about the just making stuff work out of the box aren't they? | 14:01 |
| mordred | yeah | 14:02 |
| mordred | corvus: so - fwiw, the thing that got me down this rabbit hole is that I was trying to figure out what it would take to make an oci runtime that just used chroot with no other containment technology applied | 14:03 |
| mordred | (because sometimes just using docker as a package manager is really all you want) | 14:03 |
| mordred | corvus: I did learn that there is a DOCKER_RAMDISK env var that if set will cause docker to use chroot and not pivot_root - but I don't think it disables the other isolation stuff | 14:04 |
| corvus | mordred: hrm, i'm having trouble reproducing what i thought was the original error; so let's not call this done yet | 14:05 |
| corvus | mordred: (also cool) | 14:05 |
| mordred | corvus: hrm - sad panda | 14:09 |
| corvus | mordred: okay, this is weird. what i'm seeing is that with the default seccomp profile, all docker containers share the same keyring. however, zuul-bwrap running within one or more of those containers (i tried both ways) get an isolated keyring. | 14:23 |
| corvus | it almost seems like bwrap is somehow able to still make those syscalls? maybe privileged allows it to do that? | 14:24 |
| corvus | (this is all really annoyingly hard to test) | 14:26 |
| corvus | i think it may be worth doing this on a real executor and not my workstation -- that way we can reboot to clear out state, etc. | 14:27 |
| mordred | corvus: yeah. also - I wonder if maybe behavior changed in docker versions and you've upgraded docker since last you tested this | 14:28 |
| mordred | corvus: cause I think you updated something when we were working on multi-arch a few weeks ago yeah? | 14:29 |
| corvus | mordred: hrm, i don't remember, but that doesn't mean it didn't happen | 14:30 |
| mordred | yeah | 14:30 |
| mordred | corvus: we could go ahead and add the install-docker to the executors so we can test this | 14:31 |
| corvus | mordred: ++ | 14:31 |
| corvus | i've got to get started on some saturday stuff now, i'll check back in later | 14:31 |
| mordred | corvus: kk. I'll get the docker stuff installed | 14:33 |
| mordred | corvus: it's already done actually | 14:33 |
| mordred | corvus: I'm going to try your reproducer - without the seccomp stuff first | 14:34 |
| corvus | mordred: you'll need openafs-krb5 installed in the image | 14:35 |
| corvus | mordred: and i'd do this with the executor shut down just so we don't get pollution from any real jobs that use afs | 14:35 |
| mordred | corvus: nod | 14:36 |
| mordred | corvus: I would suppose we'd need to do a kinit/aklog in the container to yeah? | 14:36 |
| mordred | corvus: I'm building a zuultest image with openafs-krb5 in it on ze01 | 14:38 |
| mordred | corvus: alternate strategy: https://blog.tomecek.net/post/kerberos-in-a-container/ | 14:41 |
| mordred | but - I like where we're going currently :) | 14:42 |
| *** ysandeep has quit IRC | 14:49 | |
| *** ysandeep has joined #opendev | 14:51 | |
| mordred | corvus: uhm. I feel like I have lost AFS knowledge. I seem to be able to manipulate things in afs as root on ze01 without any kerberos tickets | 15:03 |
| mordred | corvus: (although as not-root AFS gives me errors when trying to do that) | 15:04 |
| mordred | corvus: also - the current config for krb5 seems to be to put ccache into a file, not into keyring | 15:09 |
| mordred | corvus: https://web.mit.edu/kerberos/krb5-1.12/doc/mitK5defaults.html#paths | 15:10 |
| mordred | corvus: so it's defaulting now to FILE:/tmp/krb5cc_%{uid} | 15:10 |
| mordred | corvus: anywho - I've got a zuultest image on ze01 and the executor shut down - I'm not 100% sure what I should do next to verify | 15:14 |
| mordred | corvus: I also installed krb5-user and openafs-client in the image so the various cli things would be there | 15:15 |
| corvus | mordred: krb tickets go in the file; afs tokens are in the keyring | 15:20 |
| corvus | mordred: root@ze01 has a token in kerenl memory (run 'tokens') | 15:22 |
| corvus | as long as that's around, root will be able to do afs stuff | 15:22 |
| corvus | 'unlog' will remove that | 15:22 |
| corvus | to recap: kinit = get kerberos ticket (save to ccache file in tmp); aklog = turn kerberos ticket into afs token (stored in kernel keyring); unlog = remove token from kernel keyring; kdestroy = delete kerberos credential cache | 15:23 |
| mordred | corvus: ah | 15:38 |
| mordred | cool. I have unlogged | 15:39 |
| mordred | and I can now confirm that I can't touch the private dir I made in /afs/openstack.org/users/mordred even as root | 15:39 |
| mordred | corvus: fwiw- zuul-bwrap in that container is producing a --ro-bind None None | 15:58 |
| mordred | in the middle of the command line | 15:58 |
| mordred | ah - it's ssh_auth_sock - silly me, I skipped the ssh-agent invocation | 16:01 |
| mordred | corvus: on ze01 I ahve used your reproducer and I have kinit'd and akloged in one bubblewrap | 16:04 |
| mordred | I have started a second container and I do not see tokens | 16:04 |
| mordred | nor can I access AFS resources in the second container that I can in the first | 16:04 |
| mordred | (also I do not see any AFS tokens as root on the host) | 16:05 |
| mordred | corvus: I'm leaving the executor off on ze01 in case you want to poke | 16:09 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Run zuul-executor using docker https://review.opendev.org/733967 | 16:19 |
| mordred | corvus: I think that ^^ is what we'll want to do once we're comfortable that this will work | 16:20 |
| prometheanfire | it looks like the daily generate-constraints update failed again | 16:20 |
| prometheanfire | it's running fine manually | 16:20 |
| *** slaweq has joined #opendev | 17:12 | |
| *** slaweq has quit IRC | 17:17 | |
| *** roman_g has joined #opendev | 17:38 | |
| *** roman_g has quit IRC | 17:44 | |
| *** roman_g has joined #opendev | 17:45 | |
| *** roman_g has quit IRC | 17:45 | |
| *** slaweq has joined #opendev | 18:05 | |
| *** slaweq has quit IRC | 18:31 | |
| *** Dmitrii-Sh has quit IRC | 20:47 | |
| *** Dmitrii-Sh has joined #opendev | 20:47 | |
| *** roman_g has joined #opendev | 20:54 | |
| *** Dmitrii-Sh has quit IRC | 20:56 | |
| *** Dmitrii-Sh has joined #opendev | 20:56 | |
| *** roman_g has quit IRC | 21:34 | |
| *** wrenchyfrenchy has joined #opendev | 22:02 | |
| *** noonedeadpunk has quit IRC | 22:15 | |
| *** noonedeadpunk has joined #opendev | 22:15 | |
| *** dtantsur|afk has quit IRC | 22:42 | |
| *** dtantsur has joined #opendev | 22:42 | |
| *** dtantsur has quit IRC | 22:49 | |
| *** tosky has quit IRC | 23:35 | |
| *** DSpider has quit IRC | 23:46 | |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!