Files

T

didericis 6a22b9f654 docs(research): record rootless-Docker negative result and egress OOM

Rootless Docker cannot run in an Apple Container bottle: the runtime's
capability bounding set omits CAP_SYS_ADMIN, which the kernel requires
to write a multi-range uid_map via newuidmap. Every other prerequisite
(setuid bit, subuid ranges, initial userns, no nosuid, no seccomp) was
already correct, so this is not a packaging gap we can close.

Also records two defects found while validating the podman alternative:

- a large download OOM-kills the shared egress proxy, which buffers
  whole response bodies to scan them; the gateway is a per-host
  singleton with no restart-on-death, so one bottle's pull removes
  egress for every bottle
- registry auth collides with the proxy's unconditional Authorization
  strip, measured: a valid bearer token behaves exactly like sending
  none

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GnMp8SUGH57hZX7192Rv2F

2026-07-21 12:58:59 -04:00

3.6 KiB

Raw Permalink Blame History

Egress proxy OOMs on large downloads

Found on 2026-07-21 while running the rootless-podman spike (docs/research/rootless-docker-in-apple-container-spike.md). Recorded rather than fixed — the fix is a security-relevant decision, not a mechanical patch.

Summary

A single large HTTPS download through the gateway kills the egress proxy. mitmdump buffers whole response bodies so the DLP detectors can scan them, grows past the gateway container's memory limit, and is OOM-killed by the cgroup. Nothing restarts it.

Two properties make this worse than a failed download:

The gateway is a per-host singleton. Every bottle shares it, so one bottle's download takes egress away from all of them.
There is no restart on death. The gateway supervisor is while : ; do wait ; done; a killed daemon stays dead until the infra container is recreated.

So ordinary agent activity — pulling a container image, downloading a model or dataset, fetching a large tarball — is a denial of service against every other bottle on the host. No malice required, though it is trivially reachable on purpose.

Evidence

Triggered by docker compose up pulling quay.io/fedora/python-312 (two layers, ~82MB and ~83MB) inside a bottle. The pull itself succeeded; the next request failed:

initializing source docker://quay.io/fedora/python-312:latest:
  pinging container registry quay.io: Get "https://quay.io/v2/":
  proxyconnect tcp: dial tcp 192.168.128.39:9099: connect: connection refused

From the gateway's dmesg:

python3 invoked oom-killer: gfp_mask=0x100cca(GFP_HIGHUSER_MOVABLE), order=0
oom-kill:constraint=CONSTRAINT_MEMCG,
  oom_memcg=/container/bot-bottle-mac-infra,
  task_memcg=/container/bot-bottle-mac-infra,task=mitmdump,pid=118
Memory cgroup out of memory: Killed process 118 (mitmdump)
  total-vm:1391936kB, anon-rss:997768kB

~1GB RSS against a 1024MB container. Note the amplification: ~165MB of layers produced ~1GB of resident memory, so the buffering is several copies deep (encoded body, decoded body, and the text conversion the regex detectors scan).

Afterwards the gateway container was still running and healthy-looking — orchestrator, supervise, and git-http all alive — with no mitmdump process at all, and it stayed that way until the container was recreated. A liveness check on the container would not have caught this.

Reproduction

Launch any bottle with an egress route to a host serving a large file.
Download >~150MB over HTTPS through the proxy.
dmesg | grep -i oom inside bot-bottle-mac-infra, and note that no mitmdump process remains.

Beware a false negative when checking: truncating the process listing (cut -c1-45) cuts before the binary name, because mitmdump runs as /usr/local/bin/python3.12 /usr/local/bin/mitmdump ….

Fix options, not yet chosen

Restart dead daemons. Smallest change and strictly an improvement: an OOM then degrades one download instead of removing egress for every bottle. Does not stop the OOM.
Cap the scanned body size. Above a threshold, stop buffering — either skip the scan or stream it. This is the root-cause fix and a security decision: a size threshold is exactly the hole an exfiltrator would aim for, so "skip above N" trades a DoS for a covert channel. Streaming with a bounded window keeps coverage, at more complexity.
Raise the gateway's memory limit. Moves the threshold; does not remove it.

Worth noting that (1) and (2) are complementary — the restart gap is worth closing regardless of how the memory behaviour is resolved.

3.6 KiB Raw Permalink Blame History