docs(research): record rootless-Docker negative result and egress OOM
Rootless Docker cannot run in an Apple Container bottle: the runtime's capability bounding set omits CAP_SYS_ADMIN, which the kernel requires to write a multi-range uid_map via newuidmap. Every other prerequisite (setuid bit, subuid ranges, initial userns, no nosuid, no seccomp) was already correct, so this is not a packaging gap we can close. Also records two defects found while validating the podman alternative: - a large download OOM-kills the shared egress proxy, which buffers whole response bodies to scan them; the gateway is a per-host singleton with no restart-on-death, so one bottle's pull removes egress for every bottle - registry auth collides with the proxy's unconditional Authorization strip, measured: a valid bearer token behaves exactly like sending none Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01GnMp8SUGH57hZX7192Rv2F
This commit is contained in:
@@ -0,0 +1,86 @@
|
||||
# Egress proxy OOMs on large downloads
|
||||
|
||||
Found on 2026-07-21 while running the rootless-podman spike
|
||||
(`docs/research/rootless-docker-in-apple-container-spike.md`). Recorded
|
||||
rather than fixed — the fix is a security-relevant decision, not a
|
||||
mechanical patch.
|
||||
|
||||
## Summary
|
||||
|
||||
A single large HTTPS download through the gateway kills the egress
|
||||
proxy. `mitmdump` buffers whole response bodies so the DLP detectors can
|
||||
scan them, grows past the gateway container's memory limit, and is
|
||||
OOM-killed by the cgroup. Nothing restarts it.
|
||||
|
||||
Two properties make this worse than a failed download:
|
||||
|
||||
- **The gateway is a per-host singleton.** Every bottle shares it, so
|
||||
one bottle's download takes egress away from all of them.
|
||||
- **There is no restart on death.** The gateway supervisor is
|
||||
`while : ; do wait ; done`; a killed daemon stays dead until the infra
|
||||
container is recreated.
|
||||
|
||||
So ordinary agent activity — pulling a container image, downloading a
|
||||
model or dataset, fetching a large tarball — is a denial of service
|
||||
against every other bottle on the host. No malice required, though it is
|
||||
trivially reachable on purpose.
|
||||
|
||||
## Evidence
|
||||
|
||||
Triggered by `docker compose up` pulling `quay.io/fedora/python-312`
|
||||
(two layers, ~82MB and ~83MB) inside a bottle. The pull itself
|
||||
succeeded; the *next* request failed:
|
||||
|
||||
```
|
||||
initializing source docker://quay.io/fedora/python-312:latest:
|
||||
pinging container registry quay.io: Get "https://quay.io/v2/":
|
||||
proxyconnect tcp: dial tcp 192.168.128.39:9099: connect: connection refused
|
||||
```
|
||||
|
||||
From the gateway's `dmesg`:
|
||||
|
||||
```
|
||||
python3 invoked oom-killer: gfp_mask=0x100cca(GFP_HIGHUSER_MOVABLE), order=0
|
||||
oom-kill:constraint=CONSTRAINT_MEMCG,
|
||||
oom_memcg=/container/bot-bottle-mac-infra,
|
||||
task_memcg=/container/bot-bottle-mac-infra,task=mitmdump,pid=118
|
||||
Memory cgroup out of memory: Killed process 118 (mitmdump)
|
||||
total-vm:1391936kB, anon-rss:997768kB
|
||||
```
|
||||
|
||||
~1GB RSS against a 1024MB container. Note the amplification: ~165MB of
|
||||
layers produced ~1GB of resident memory, so the buffering is several
|
||||
copies deep (encoded body, decoded body, and the text conversion the
|
||||
regex detectors scan).
|
||||
|
||||
Afterwards the gateway container was still running and healthy-looking —
|
||||
orchestrator, supervise, and git-http all alive — with no `mitmdump`
|
||||
process at all, and it stayed that way until the container was
|
||||
recreated. A liveness check on the container would not have caught this.
|
||||
|
||||
## Reproduction
|
||||
|
||||
1. Launch any bottle with an egress route to a host serving a large file.
|
||||
2. Download >~150MB over HTTPS through the proxy.
|
||||
3. `dmesg | grep -i oom` inside `bot-bottle-mac-infra`, and note that no
|
||||
`mitmdump` process remains.
|
||||
|
||||
Beware a false negative when checking: truncating the process listing
|
||||
(`cut -c1-45`) cuts before the binary name, because `mitmdump` runs as
|
||||
`/usr/local/bin/python3.12 /usr/local/bin/mitmdump …`.
|
||||
|
||||
## Fix options, not yet chosen
|
||||
|
||||
1. **Restart dead daemons.** Smallest change and strictly an
|
||||
improvement: an OOM then degrades one download instead of removing
|
||||
egress for every bottle. Does not stop the OOM.
|
||||
2. **Cap the scanned body size.** Above a threshold, stop buffering —
|
||||
either skip the scan or stream it. This is the root-cause fix and a
|
||||
security decision: a size threshold is exactly the hole an exfiltrator
|
||||
would aim for, so "skip above N" trades a DoS for a covert channel.
|
||||
Streaming with a bounded window keeps coverage, at more complexity.
|
||||
3. **Raise the gateway's memory limit.** Moves the threshold; does not
|
||||
remove it.
|
||||
|
||||
Worth noting that (1) and (2) are complementary — the restart gap is
|
||||
worth closing regardless of how the memory behaviour is resolved.
|
||||
Reference in New Issue
Block a user