Files

T

didericis 29c46356ef feat(macos): run containers inside a bottle via guest-local podman

Adds the `nested_containers` bottle flag. On the macOS backend it starts
a rootless podman service inside the bottle and exposes its
Docker-compatible API socket, so the agent still runs `docker` and
`docker compose`. No host daemon socket is mounted and the guest gains
no capabilities; backends that cannot run a guest-local engine reject
the flag in the shared prepare template rather than ignoring it.

Podman rather than rootless Docker because Apple Container's capability
bounding set omits CAP_SYS_ADMIN, which the kernel requires to write a
multi-range uid_map via newuidmap. With no subordinate UID range podman
falls back to a single-UID self-mapping an unprivileged process may
write itself, so the image build strips /etc/subuid and /etc/subgid
entries rather than adding them.

That mapping is also why nested containers are not an isolation layer:
root inside one is the agent user outside it. They are a build/test
convenience; the bottle remains the boundary.

Ports the spike branch onto main, renaming docker_access — it implied
Docker and granted access to nothing on the host — and drops podman
from the derived layer now that every built-in image ships it (#451).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

2026-07-21 21:25:27 -04:00

3.6 KiB

Raw Blame History

Egress proxy OOMs on large downloads

Found on 2026-07-21 while running the rootless-podman spike (docs/research/rootless-docker-in-apple-container-spike.md). Recorded rather than fixed — the fix is a security-relevant decision, not a mechanical patch.

Summary

A single large HTTPS download through the gateway kills the egress proxy. mitmdump buffers whole response bodies so the DLP detectors can scan them, grows past the gateway container's memory limit, and is OOM-killed by the cgroup. Nothing restarts it.

Two properties make this worse than a failed download:

The gateway is a per-host singleton. Every bottle shares it, so one bottle's download takes egress away from all of them.
There is no restart on death. The gateway supervisor is while : ; do wait ; done; a killed daemon stays dead until the infra container is recreated.

So ordinary agent activity — pulling a container image, downloading a model or dataset, fetching a large tarball — is a denial of service against every other bottle on the host. No malice required, though it is trivially reachable on purpose.

Evidence

Triggered by docker compose up pulling quay.io/fedora/python-312 (two layers, ~82MB and ~83MB) inside a bottle. The pull itself succeeded; the next request failed:

initializing source docker://quay.io/fedora/python-312:latest:
  pinging container registry quay.io: Get "https://quay.io/v2/":
  proxyconnect tcp: dial tcp 192.168.128.39:9099: connect: connection refused

From the gateway's dmesg:

python3 invoked oom-killer: gfp_mask=0x100cca(GFP_HIGHUSER_MOVABLE), order=0
oom-kill:constraint=CONSTRAINT_MEMCG,
  oom_memcg=/container/bot-bottle-mac-infra,
  task_memcg=/container/bot-bottle-mac-infra,task=mitmdump,pid=118
Memory cgroup out of memory: Killed process 118 (mitmdump)
  total-vm:1391936kB, anon-rss:997768kB

~1GB RSS against a 1024MB container. Note the amplification: ~165MB of layers produced ~1GB of resident memory, so the buffering is several copies deep (encoded body, decoded body, and the text conversion the regex detectors scan).

Afterwards the gateway container was still running and healthy-looking — orchestrator, supervise, and git-http all alive — with no mitmdump process at all, and it stayed that way until the container was recreated. A liveness check on the container would not have caught this.

Reproduction

Launch any bottle with an egress route to a host serving a large file.
Download >~150MB over HTTPS through the proxy.
dmesg | grep -i oom inside bot-bottle-mac-infra, and note that no mitmdump process remains.

Beware a false negative when checking: truncating the process listing (cut -c1-45) cuts before the binary name, because mitmdump runs as /usr/local/bin/python3.12 /usr/local/bin/mitmdump ….

Fix options, not yet chosen

Restart dead daemons. Smallest change and strictly an improvement: an OOM then degrades one download instead of removing egress for every bottle. Does not stop the OOM.
Cap the scanned body size. Above a threshold, stop buffering — either skip the scan or stream it. This is the root-cause fix and a security decision: a size threshold is exactly the hole an exfiltrator would aim for, so "skip above N" trades a DoS for a covert channel. Streaming with a bounded window keeps coverage, at more complexity.
Raise the gateway's memory limit. Moves the threshold; does not remove it.

Worth noting that (1) and (2) are complementary — the restart gap is worth closing regardless of how the memory behaviour is resolved.

3.6 KiB Raw Blame History