bot-bottle/docs/research/rootless-docker-in-apple-container-spike.md

Rootless Docker inside Apple Container bottles

Spike branch: spike/rootless-docker-macos (a4d8461)

Outcome: the podman recommendation below shipped as the nested_containers bottle flag — see docs/prds/prd-new-nested-containers.md. The docker_access name used throughout the spike text was renamed on the way in; it granted no access to anything on the host.

Summary

Negative result. Rootless Docker cannot run inside an Apple Container bottle without granting the bottle CAP_SYS_ADMIN. This is a kernel constraint on writing multi-range uid_map, not a packaging gap we can close with a better init script, a different base image, or more careful /etc/subuid handling.

The spike was built on the premise — stated in bot_bottle/backend/macos_container/rootless_docker.py — that it would "deliberately refuse to compensate for missing prerequisites with outer capabilities, a privileged container, or a host Docker socket." That premise is exactly what the experiment falsified. The two ways forward are to abandon the premise (add CAP_SYS_ADMIN to the bottle, and with it most of the isolation the bottle exists to provide) or to abandon rootless Docker.

Recommendation: abandon rootless Docker. Podman does not have this problem — see Podman is not blocked by this below.

Local environment

Tested on 2026-07-21:

$ sw_vers
ProductName:		macOS
ProductVersion:		26.5.1
BuildVersion:		25F80

$ container --version
container CLI version 1.0.0 (build: release, commit: ee848e3)

$ uname -a   # inside the bottle
Linux ... 6.18.15 #1 SMP Tue Mar 17 01:36:53 UTC 2026 aarch64 GNU/Linux

The failure

tests/integration/test_macos_rootless_docker_spike.py builds the image, launches the bottle, and dies in rootless_docker.start:

+ exec rootlesskit --net=slirp4netns --mtu=65520 ... dockerd-rootless.sh
[rootlesskit:parent] error: failed to setup UID/GID map:
  newuidmap 1100 [0 1000 1 1 100000 65536] failed:
  newuidmap: write to uid_map failed: Operation not permitted

Why it fails

Every prerequisite you would normally suspect is present and correct in the guest:

Check	Result
/usr/bin/newuidmap	-rwsr-xr-x root root — setuid bit intact, survived the OCI export
/ mount options	rw,relatime — not nosuid
NoNewPrivs	0
Seccomp	0, no filters
/etc/subuid, /etc/subgid	node:100000:65536 in both
user namespace	user:[4026531837], identical to pid 1 — the initial userns
unshare -U -r true	succeeds
/proc/sys/user/max_user_namespaces	4505

The one thing that is missing is in the capability bounding set that Apple Container gives the container:

CapBnd: 00000000a80425fb
= chown, dac_override, fowner, fsetid, kill, setgid, setuid, setpcap,
  net_bind_service, net_raw, sys_chroot, mknod, audit_write, setfcap

No CAP_SYS_ADMIN. That is the whole story, and the chain is:

1. The kernel's map_write() gates writing a uid_map on file_ns_capable(file, ns, CAP_SYS_ADMIN) — capability over the new user namespace, evaluated against the credentials that opened /proc/<pid>/uid_map.
2. newuidmap is setuid-root, so it runs with euid 0 — but its capability sets are clamped by the bounding set, which has no CAP_SYS_ADMIN.
3. cap_capable() has a shortcut that grants all capabilities when the caller's userns is the new namespace's parent and ns->owner == cred->euid. It does not apply: the namespace was created by node (uid 1000) while newuidmap runs as euid 0.
4. So the check falls through to the effective-set test in the initial userns, which fails. EPERM.

Note that the single-line unprivileged path (unshare -U -r) works precisely because it does not go through newuidmap and does not need CAP_SYS_ADMIN. Only the multi-range subuid mapping that rootless Docker requires does.

This is the same constraint that makes upstream's dind-rootless image require --privileged. It is not specific to Apple Container, except that Apple Container gives us no bounding set that includes CAP_SYS_ADMIN by default.

It does work with the capability — which is the point

Adding the capability clears the failure immediately, and exposes one further, much smaller blocker: /dev/net/tun exists (the kernel has tun; /proc/misc lists 200 tun) but Apple Container creates it crw------- root root, so uid 1000 cannot open it and slirp4netns fails with open: Permission denied. A chmod 0666 /dev/net/tun as root inside the bottle fixes that, and needs no capability beyond what the bottle already has.

With both applied by hand, the daemon comes up completely:

$ container run --rm -u root --cap-add CAP_SYS_ADMIN \
    bot-bottle-claude:latest-rootless-docker sh -c '...'
Server Version: 20.10.24+dfsg1
Storage Driver: fuse-overlayfs
Cgroup Driver: none
Cgroup Version: 2
API listen on /tmp/rt/docker.sock

So rootless-docker-init.sh and rootless_docker.py are correct. The spike did not fail on a bug. It failed on its own premise.

Two secondary findings from that run, relevant if anyone revisits this:

• Debian's docker.io package pins Docker 20.10 (EOL), not the 28.x implied by the docker:28-cli compose plugin the image copies in.
• Cgroup Driver: none — no resource limits on nested containers.

Why we should not just add the capability

CAP_SYS_ADMIN is close to a superset of "root" in practical terms — mount, pivot_root, namespace manipulation, and a long tail of subsystem-specific powers. Granting it to the agent bottle would undercut the containment argument the rest of the backend is built around, including the deliberately narrow choices immediately adjacent to it in launch.py (--cap-drop CAP_NET_RAW, no NET_ADMIN, a host-only agent network). Trading all of that for nested docker compose is a bad exchange.

Podman is not blocked by this

Sanity-checked on the same host, same kernel, same runtime, so the comparison is apples to apples:

Scenario	Result
Podman rootless, /etc/subuid populated	Fails identically — newuidmap: write to uid_map failed: Operation not permitted
Podman rootless, no subuid ranges, --network=host	Works, no added capabilities
Podman rootless, no subuid ranges, default netns, /dev/net/tun at 0600	Fails — slirp4netns: open("/dev/net/tun"): Permission denied
Podman rootless, no subuid ranges, default netns, /dev/net/tun at 0666	Works, no added capabilities

The difference is that podman degrades gracefully when no subuid range is available: it falls back to a single-UID self-mapping, which an unprivileged process may write itself, so newuidmap is never invoked and CAP_SYS_ADMIN is never needed. Docker's rootless mode has no equivalent fallback.

The cost of that fallback is real and should be weighed before building on it: with a single-UID mapping, every UID inside a nested container collapses onto the bottle's own uid 1000. There is no UID separation between the agent and anything it runs — root in a nested container is the agent user outside it. It also requires ignore_chown_errors on the storage driver. Whether that is acceptable depends on whether the bottle boundary (which is unchanged) or the nested-container boundary (which is effectively nil) is the one we are relying on.

What the podman spike then needed

The podman implementation that replaced the Docker one on this branch turned up two more device-node blockers of the same shape as /dev/net/tun — Apple Container creates the node, but 0600 root:root:

• /dev/fuse — blocks the fuse-overlayfs storage driver (fuse: failed to open /dev/fuse: Permission denied). Without it the only working driver is vfs, which copies whole layers per container.
• /dev/net/tun — blocks slirp4netns, which rootless podman uses for the default bridge network.

Both are fixed by chmod 0666 as root inside the bottle, which needs no capability the bottle does not already hold. This is categorically different from the CAP_SYS_ADMIN requirement: it is a permission on a node that already exists, not an outer privilege grant.

One design note worth recording: the agent-facing surface stays docker and docker compose, pointed at podman's Docker-compatible API socket via DOCKER_HOST. Setting netns="host" in containers.conf does not propagate through that compat API — stock docker run and compose files request bridge networking explicitly — so slirp4netns (and therefore the /dev/net/tun chmod) is required for ordinary compose files to work at all. Host networking remains available per-workload via --network=host.

Verified working in a bottle with zero added capabilities: fuse-overlayfs storage, the compat API socket, docker run on both bridge and host networking, and published ports.

Nested pulls collide with our own egress DLP

The first live run got podman up and docker compose running, then failed on the image pull:

web Pulling
initializing source docker://python:3.12-alpine: reading manifest ...
StatusCode: 403, egress DLP: Generic Bearer JWT found in body

This is bot-bottle's own egress scanner, not a podman problem. The Docker registry auth flow carries a bearer JWT by protocol, and the token_patterns detector's Generic Bearer JWT rule (Bearer\s+[A-Za-z0-9._\-]{50,}) matches it on every pull. Any bottle that pulls images will hit this.

The fix is per-route detector scoping, which the egress config already supports — drop token_patterns on the registry hosts and keep known_secrets:

{"host": "registry-1.docker.io",
 "dlp": {"outbound_detectors": ["known_secrets"]}}

That is the right trade rather than a grudging one: known_secrets matches the bottle's actual credential values, so real exfil through a registry host is still caught. token_patterns on a registry route only ever produces protocol noise.

Worth generalising later: any manifest enabling nested_containers needs this on its registry routes, so it probably belongs in a shared registry-route snippet rather than being copy-pasted per bottle.

And then registry auth collides with the Authorization strip

With DLP scoped, the pull failed differently: unauthorized: authentication required. This one is architectural.

egress_addon.py strips agent-set Authorization unconditionally before forwarding — deliberately, so an agent cannot smuggle a credential out in a header the DLP detectors don't recognise. A route may carry gateway-injected auth instead, but only from a static token in an env var (auth_scheme + token_env).

Docker registry auth doesn't fit that shape. The client fetches a short-lived, per-repository-scope bearer token from auth.docker.io and presents it to registry-1.docker.io. There is no static token to inject, and the token the client legitimately obtained is stripped.

Measured inside a bottle, by hand:

Step	Result
Fetch token from auth.docker.io	200, 5409-byte token body
Manifest request with that valid token	401
Manifest request with no Authorization	401 — identical

A valid token behaves exactly like sending none, which is direct evidence the header never arrives. Any nested-container workflow that pulls from a registry is blocked on this, so it is not a detail that can be deferred: pulling base images is most of what nested containers are for.

Registries that skip the token dance work today

Not every registry needs the stripped header. Measured directly:

Registry	Manifest request with no Authorization
quay.io	200
mcr.microsoft.com	200
registry.k8s.io	307 (redirect, no auth)
ghcr.io	401
registry-1.docker.io	401

So "just add the registry to the bottle config" genuinely works — for quay, MCR, registry.k8s.io, or any unauthenticated internal registry. Docker Hub and GHCR are the ones that need the strip resolved. The acceptance test uses quay for exactly this reason.

Resolving it for Docker Hub means picking one of:

1. Per-route opt-in to preserve client Authorization. Smallest change. Note the compounding effect on exactly these routes: the DLP scoping above already removed token_patterns there, so a preserved-auth registry route is one where the agent may send bearer tokens that neither the strip nor the pattern detector inspects. known_secrets still applies, so the bottle's real credentials are still caught.
2. A registry-aware gateway that performs the token dance itself and injects the result. Preserves the invariant fully; materially more work, and it makes the gateway speak a specific registry protocol.
3. Pre-seed images at provision time (host-side container image save into podman storage), so bottles never pull at runtime. Preserves the invariant, and limits nested containers to pre-approved images — which fits the custody positioning, at the cost of no ad-hoc docker pull.
4. Stop. Nested containers are not supported on this backend.

Podman 4.3.1 silently swallows container exit codes

Debian bookworm — which the current agent base image is built on — ships podman 4.3.1. Through its Docker-compatible API, docker run returns 0 no matter what the container did:

Command	podman 4.3.1	podman 5.4.2
docker run … sh -c 'exit 7' (compat API)	0	7
docker run … sh -c 'exit 0' (compat API)	0	0
podman run … sh -c 'exit 7' (native)	7	7

This is worse than a broken feature: every failing command an agent runs via docker run reports success. A test suite, a build step, or a CI script inside a bottle would pass while failing. It also silently defeated the acceptance test's egress-containment assertion, which is why that assertion now checks an in-band marker rather than an exit code.

Podman 5.4.2 (Debian trixie) fixes it, but needs two packages that bookworm's podman does not: passt (podman 5's default network tool) and nftables (netavark shells out to nft; without it every run fails with unable to upgrade to tcp, received 500). With both installed, exit codes propagate correctly and the compat API behaves.

The open question this leaves is where podman 5 comes from, since the agent base is bookworm-based:

1. Move the agent images to Debian trixie. Trixie is current stable. Correct, and the blast radius is every bottle, not just this feature.
2. Drop the compat socket and use podman natively (podman-docker provides a docker shim; compose comes from podman-compose). Native podman propagates exit codes correctly even on 4.3.1. Contained to this feature, at the cost of docker compose becoming docker-compose/podman-compose.
3. Ship bookworm's podman 4.3.1 with the compat socket — not viable. Silent false success is a correctness bug agents cannot see.

Recommendation

1. Do not revive rootless Docker on this backend. This document is the record of why.
2. Nested containers, if wanted, come from podman under the single-mapping constraint — with the explicit understanding that the nested-container boundary carries no security weight. root in a nested container is the agent user outside it.
3. Nested containers are therefore a build/test convenience. The bottle remains the security boundary, exactly as it was.