# Egress proxy OOMs on large downloads

Found on 2026-07-21 while running the rootless-podman spike
(`docs/research/rootless-docker-in-apple-container-spike.md`). Recorded
rather than fixed — the fix is a security-relevant decision, not a
mechanical patch.

## Summary

A single large HTTPS download through the gateway kills the egress
proxy. `mitmdump` buffers whole response bodies so the DLP detectors can
scan them, grows past the gateway container's memory limit, and is
OOM-killed by the cgroup. Nothing restarts it.

Two properties make this worse than a failed download:

- **The gateway is a per-host singleton.** Every bottle shares it, so
  one bottle's download takes egress away from all of them.
- **There is no restart on death.** The gateway supervisor is
  `while : ; do wait ; done`; a killed daemon stays dead until the infra
  container is recreated.

So ordinary agent activity — pulling a container image, downloading a
model or dataset, fetching a large tarball — is a denial of service
against every other bottle on the host. No malice required, though it is
trivially reachable on purpose.

## Evidence

Triggered by `docker compose up` pulling `quay.io/fedora/python-312`
(two layers, ~82MB and ~83MB) inside a bottle. The pull itself
succeeded; the *next* request failed:

```
initializing source docker://quay.io/fedora/python-312:latest:
  pinging container registry quay.io: Get "https://quay.io/v2/":
  proxyconnect tcp: dial tcp 192.168.128.39:9099: connect: connection refused
```

From the gateway's `dmesg`:

```
python3 invoked oom-killer: gfp_mask=0x100cca(GFP_HIGHUSER_MOVABLE), order=0
oom-kill:constraint=CONSTRAINT_MEMCG,
  oom_memcg=/container/bot-bottle-mac-infra,
  task_memcg=/container/bot-bottle-mac-infra,task=mitmdump,pid=118
Memory cgroup out of memory: Killed process 118 (mitmdump)
  total-vm:1391936kB, anon-rss:997768kB
```

~1GB RSS against a 1024MB container. Note the amplification: ~165MB of
layers produced ~1GB of resident memory, so the buffering is several
copies deep (encoded body, decoded body, and the text conversion the
regex detectors scan).

Afterwards the gateway container was still running and healthy-looking —
orchestrator, supervise, and git-http all alive — with no `mitmdump`
process at all, and it stayed that way until the container was
recreated. A liveness check on the container would not have caught this.

## Reproduction

1. Launch any bottle with an egress route to a host serving a large file.
2. Download >~150MB over HTTPS through the proxy.
3. `dmesg | grep -i oom` inside `bot-bottle-mac-infra`, and note that no
   `mitmdump` process remains.

Beware a false negative when checking: truncating the process listing
(`cut -c1-45`) cuts before the binary name, because `mitmdump` runs as
`/usr/local/bin/python3.12 /usr/local/bin/mitmdump …`.

## Fix options, not yet chosen

1. **Restart dead daemons.** Smallest change and strictly an
   improvement: an OOM then degrades one download instead of removing
   egress for every bottle. Does not stop the OOM.
2. **Cap the scanned body size.** Above a threshold, stop buffering —
   either skip the scan or stream it. This is the root-cause fix and a
   security decision: a size threshold is exactly the hole an exfiltrator
   would aim for, so "skip above N" trades a DoS for a covert channel.
   Streaming with a bounded window keeps coverage, at more complexity.
3. **Raise the gateway's memory limit.** Moves the threshold; does not
   remove it.

Worth noting that (1) and (2) are complementary — the restart gap is
worth closing regardless of how the memory behaviour is resolved.