feat(macos): run containers inside a bottle via guest-local podman
Adds the `nested_containers` bottle flag. On the macOS backend it starts a rootless podman service inside the bottle and exposes its Docker-compatible API socket, so the agent still runs `docker` and `docker compose`. No host daemon socket is mounted and the guest gains no capabilities; backends that cannot run a guest-local engine reject the flag in the shared prepare template rather than ignoring it. Podman rather than rootless Docker because Apple Container's capability bounding set omits CAP_SYS_ADMIN, which the kernel requires to write a multi-range uid_map via newuidmap. With no subordinate UID range podman falls back to a single-UID self-mapping an unprivileged process may write itself, so the image build strips /etc/subuid and /etc/subgid entries rather than adding them. That mapping is also why nested containers are not an isolation layer: root inside one is the agent user outside it. They are a build/test convenience; the bottle remains the boundary. Ports the spike branch onto main, renaming docker_access — it implied Docker and granted access to nothing on the host — and drops podman from the derived layer now that every built-in image ships it (#451). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
@@ -0,0 +1,86 @@
|
||||
# Egress proxy OOMs on large downloads
|
||||
|
||||
Found on 2026-07-21 while running the rootless-podman spike
|
||||
(`docs/research/rootless-docker-in-apple-container-spike.md`). Recorded
|
||||
rather than fixed — the fix is a security-relevant decision, not a
|
||||
mechanical patch.
|
||||
|
||||
## Summary
|
||||
|
||||
A single large HTTPS download through the gateway kills the egress
|
||||
proxy. `mitmdump` buffers whole response bodies so the DLP detectors can
|
||||
scan them, grows past the gateway container's memory limit, and is
|
||||
OOM-killed by the cgroup. Nothing restarts it.
|
||||
|
||||
Two properties make this worse than a failed download:
|
||||
|
||||
- **The gateway is a per-host singleton.** Every bottle shares it, so
|
||||
one bottle's download takes egress away from all of them.
|
||||
- **There is no restart on death.** The gateway supervisor is
|
||||
`while : ; do wait ; done`; a killed daemon stays dead until the infra
|
||||
container is recreated.
|
||||
|
||||
So ordinary agent activity — pulling a container image, downloading a
|
||||
model or dataset, fetching a large tarball — is a denial of service
|
||||
against every other bottle on the host. No malice required, though it is
|
||||
trivially reachable on purpose.
|
||||
|
||||
## Evidence
|
||||
|
||||
Triggered by `docker compose up` pulling `quay.io/fedora/python-312`
|
||||
(two layers, ~82MB and ~83MB) inside a bottle. The pull itself
|
||||
succeeded; the *next* request failed:
|
||||
|
||||
```
|
||||
initializing source docker://quay.io/fedora/python-312:latest:
|
||||
pinging container registry quay.io: Get "https://quay.io/v2/":
|
||||
proxyconnect tcp: dial tcp 192.168.128.39:9099: connect: connection refused
|
||||
```
|
||||
|
||||
From the gateway's `dmesg`:
|
||||
|
||||
```
|
||||
python3 invoked oom-killer: gfp_mask=0x100cca(GFP_HIGHUSER_MOVABLE), order=0
|
||||
oom-kill:constraint=CONSTRAINT_MEMCG,
|
||||
oom_memcg=/container/bot-bottle-mac-infra,
|
||||
task_memcg=/container/bot-bottle-mac-infra,task=mitmdump,pid=118
|
||||
Memory cgroup out of memory: Killed process 118 (mitmdump)
|
||||
total-vm:1391936kB, anon-rss:997768kB
|
||||
```
|
||||
|
||||
~1GB RSS against a 1024MB container. Note the amplification: ~165MB of
|
||||
layers produced ~1GB of resident memory, so the buffering is several
|
||||
copies deep (encoded body, decoded body, and the text conversion the
|
||||
regex detectors scan).
|
||||
|
||||
Afterwards the gateway container was still running and healthy-looking —
|
||||
orchestrator, supervise, and git-http all alive — with no `mitmdump`
|
||||
process at all, and it stayed that way until the container was
|
||||
recreated. A liveness check on the container would not have caught this.
|
||||
|
||||
## Reproduction
|
||||
|
||||
1. Launch any bottle with an egress route to a host serving a large file.
|
||||
2. Download >~150MB over HTTPS through the proxy.
|
||||
3. `dmesg | grep -i oom` inside `bot-bottle-mac-infra`, and note that no
|
||||
`mitmdump` process remains.
|
||||
|
||||
Beware a false negative when checking: truncating the process listing
|
||||
(`cut -c1-45`) cuts before the binary name, because `mitmdump` runs as
|
||||
`/usr/local/bin/python3.12 /usr/local/bin/mitmdump …`.
|
||||
|
||||
## Fix options, not yet chosen
|
||||
|
||||
1. **Restart dead daemons.** Smallest change and strictly an
|
||||
improvement: an OOM then degrades one download instead of removing
|
||||
egress for every bottle. Does not stop the OOM.
|
||||
2. **Cap the scanned body size.** Above a threshold, stop buffering —
|
||||
either skip the scan or stream it. This is the root-cause fix and a
|
||||
security decision: a size threshold is exactly the hole an exfiltrator
|
||||
would aim for, so "skip above N" trades a DoS for a covert channel.
|
||||
Streaming with a bounded window keeps coverage, at more complexity.
|
||||
3. **Raise the gateway's memory limit.** Moves the threshold; does not
|
||||
remove it.
|
||||
|
||||
Worth noting that (1) and (2) are complementary — the restart gap is
|
||||
worth closing regardless of how the memory behaviour is resolved.
|
||||
Reference in New Issue
Block a user