didericis
ba25845886
tracker-policy-pr / check-pr (pull_request) Successful in 12s
lint / lint (push) Successful in 51s
test / integration-docker (pull_request) Successful in 40s
test / unit (pull_request) Successful in 46s
test / integration-firecracker (pull_request) Successful in 3m35s
test / coverage (pull_request) Successful in 43s
test / publish-infra (pull_request) Has been skipped
Adds the `nested_containers` bottle flag. On the macOS backend it starts a rootless podman service inside the bottle and exposes its Docker-compatible API socket, so the agent still runs `docker` and `docker compose`. No host daemon socket is mounted and the guest gains no capabilities; backends that cannot run a guest-local engine reject the flag in the shared prepare template rather than ignoring it. Podman rather than rootless Docker because Apple Container's capability bounding set omits CAP_SYS_ADMIN, which the kernel requires to write a multi-range uid_map via newuidmap. With no subordinate UID range podman falls back to a single-UID self-mapping an unprivileged process may write itself, so the image build strips /etc/subuid and /etc/subgid entries rather than adding them. That mapping is also why nested containers are not an isolation layer: root inside one is the agent user outside it. They are a build/test convenience; the bottle remains the boundary. Ports the spike branch onto main, renaming docker_access — it implied Docker and granted access to nothing on the host — and drops podman from the derived layer now that every built-in image ships it (#451). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
87 lines
3.6 KiB
Markdown
87 lines
3.6 KiB
Markdown
# Egress proxy OOMs on large downloads
|
|
|
|
Found on 2026-07-21 while running the rootless-podman spike
|
|
(`docs/research/rootless-docker-in-apple-container-spike.md`). Recorded
|
|
rather than fixed — the fix is a security-relevant decision, not a
|
|
mechanical patch.
|
|
|
|
## Summary
|
|
|
|
A single large HTTPS download through the gateway kills the egress
|
|
proxy. `mitmdump` buffers whole response bodies so the DLP detectors can
|
|
scan them, grows past the gateway container's memory limit, and is
|
|
OOM-killed by the cgroup. Nothing restarts it.
|
|
|
|
Two properties make this worse than a failed download:
|
|
|
|
- **The gateway is a per-host singleton.** Every bottle shares it, so
|
|
one bottle's download takes egress away from all of them.
|
|
- **There is no restart on death.** The gateway supervisor is
|
|
`while : ; do wait ; done`; a killed daemon stays dead until the infra
|
|
container is recreated.
|
|
|
|
So ordinary agent activity — pulling a container image, downloading a
|
|
model or dataset, fetching a large tarball — is a denial of service
|
|
against every other bottle on the host. No malice required, though it is
|
|
trivially reachable on purpose.
|
|
|
|
## Evidence
|
|
|
|
Triggered by `docker compose up` pulling `quay.io/fedora/python-312`
|
|
(two layers, ~82MB and ~83MB) inside a bottle. The pull itself
|
|
succeeded; the *next* request failed:
|
|
|
|
```
|
|
initializing source docker://quay.io/fedora/python-312:latest:
|
|
pinging container registry quay.io: Get "https://quay.io/v2/":
|
|
proxyconnect tcp: dial tcp 192.168.128.39:9099: connect: connection refused
|
|
```
|
|
|
|
From the gateway's `dmesg`:
|
|
|
|
```
|
|
python3 invoked oom-killer: gfp_mask=0x100cca(GFP_HIGHUSER_MOVABLE), order=0
|
|
oom-kill:constraint=CONSTRAINT_MEMCG,
|
|
oom_memcg=/container/bot-bottle-mac-infra,
|
|
task_memcg=/container/bot-bottle-mac-infra,task=mitmdump,pid=118
|
|
Memory cgroup out of memory: Killed process 118 (mitmdump)
|
|
total-vm:1391936kB, anon-rss:997768kB
|
|
```
|
|
|
|
~1GB RSS against a 1024MB container. Note the amplification: ~165MB of
|
|
layers produced ~1GB of resident memory, so the buffering is several
|
|
copies deep (encoded body, decoded body, and the text conversion the
|
|
regex detectors scan).
|
|
|
|
Afterwards the gateway container was still running and healthy-looking —
|
|
orchestrator, supervise, and git-http all alive — with no `mitmdump`
|
|
process at all, and it stayed that way until the container was
|
|
recreated. A liveness check on the container would not have caught this.
|
|
|
|
## Reproduction
|
|
|
|
1. Launch any bottle with an egress route to a host serving a large file.
|
|
2. Download >~150MB over HTTPS through the proxy.
|
|
3. `dmesg | grep -i oom` inside `bot-bottle-mac-infra`, and note that no
|
|
`mitmdump` process remains.
|
|
|
|
Beware a false negative when checking: truncating the process listing
|
|
(`cut -c1-45`) cuts before the binary name, because `mitmdump` runs as
|
|
`/usr/local/bin/python3.12 /usr/local/bin/mitmdump …`.
|
|
|
|
## Fix options, not yet chosen
|
|
|
|
1. **Restart dead daemons.** Smallest change and strictly an
|
|
improvement: an OOM then degrades one download instead of removing
|
|
egress for every bottle. Does not stop the OOM.
|
|
2. **Cap the scanned body size.** Above a threshold, stop buffering —
|
|
either skip the scan or stream it. This is the root-cause fix and a
|
|
security decision: a size threshold is exactly the hole an exfiltrator
|
|
would aim for, so "skip above N" trades a DoS for a covert channel.
|
|
Streaming with a bounded window keeps coverage, at more complexity.
|
|
3. **Raise the gateway's memory limit.** Moves the threshold; does not
|
|
remove it.
|
|
|
|
Worth noting that (1) and (2) are complementary — the restart gap is
|
|
worth closing regardless of how the memory behaviour is resolved.
|