Egress gateway OOMs on large downloads and never restarts, dropping egress for every bottle #455

New Issue

2026-07-21T15:13:24-04:00

didericis-claude commented

2026-07-21 15:13:24 -04:00

Summary

A single large HTTPS download through the shared egress gateway kills the proxy. mitmdump buffers whole response bodies so the DLP detectors can scan them, grows past the gateway container's memory limit, and is OOM-killed by the cgroup. Nothing restarts it.

Two properties make this worse than a failed download:

The gateway is a per-host singleton. Every bottle shares it, so one bottle's download takes egress away from all of them.
No restart on death. The supervisor is while : ; do wait ; done; a killed mitmdump stays dead until the infra container is recreated. The container still looks healthy (orchestrator, supervise, git-http all alive), so a container liveness check does not catch it.

So ordinary agent activity — pulling a container image, downloading a model/dataset, fetching a large tarball — is a denial of service against every other bottle on the host. No malice required, though it is trivially reachable on purpose.

Evidence

Found 2026-07-21 during the rootless-podman spike (#392). Triggered by docker compose up pulling quay.io/fedora/python-312 (~82MB + ~83MB layers) inside a bottle. The pull succeeded; the next request failed with connection refused to the proxy. From the gateway dmesg:

oom-kill:constraint=CONSTRAINT_MEMCG, oom_memcg=/container/bot-bottle-mac-infra, task=mitmdump
Memory cgroup out of memory: Killed process 118 (mitmdump) total-vm:1391936kB, anon-rss:997768kB

~1GB RSS against a 1024MB container. Note the amplification: ~165MB of layers produced ~1GB resident — several buffered copies deep (encoded body, decoded body, and the text conversion the regex detectors scan).

Full write-up: docs/research/egress-proxy-oom-on-large-downloads.md (branch spike/rootless-docker-macos).

Fix options (not yet chosen)

Restart dead daemons. Smallest change, strictly an improvement: an OOM then degrades one download instead of removing egress for every bottle. Does not stop the OOM.
Cap the scanned body size. Root-cause fix, but a security decision: above a threshold, stop buffering (skip or stream). A "skip above N" threshold is exactly the hole an exfiltrator would aim for, trading a DoS for a covert channel.

Notes

Pre-existing defect, not specific to containers — pulling images just makes it easy to hit.
Does not block #392. For the nested-container work we disable DLP on the image-registry routes, which sidesteps the buffering for those hosts; this issue tracks the general fix.

Split out from #392 (comment referencing the OOM finding).

## Summary A single large HTTPS download through the shared egress gateway kills the proxy. `mitmdump` buffers whole response bodies so the DLP detectors can scan them, grows past the gateway container's memory limit, and is OOM-killed by the cgroup. Nothing restarts it. Two properties make this worse than a failed download: - **The gateway is a per-host singleton.** Every bottle shares it, so one bottle's download takes egress away from *all* of them. - **No restart on death.** The supervisor is `while : ; do wait ; done`; a killed `mitmdump` stays dead until the infra container is recreated. The container still looks healthy (orchestrator, supervise, git-http all alive), so a container liveness check does not catch it. So ordinary agent activity — pulling a container image, downloading a model/dataset, fetching a large tarball — is a denial of service against every other bottle on the host. No malice required, though it is trivially reachable on purpose. ## Evidence Found 2026-07-21 during the rootless-podman spike (#392). Triggered by `docker compose up` pulling `quay.io/fedora/python-312` (~82MB + ~83MB layers) inside a bottle. The pull *succeeded*; the next request failed with `connection refused` to the proxy. From the gateway `dmesg`: ``` oom-kill:constraint=CONSTRAINT_MEMCG, oom_memcg=/container/bot-bottle-mac-infra, task=mitmdump Memory cgroup out of memory: Killed process 118 (mitmdump) total-vm:1391936kB, anon-rss:997768kB ``` ~1GB RSS against a 1024MB container. Note the amplification: ~165MB of layers produced ~1GB resident — several buffered copies deep (encoded body, decoded body, and the text conversion the regex detectors scan). Full write-up: `docs/research/egress-proxy-oom-on-large-downloads.md` (branch `spike/rootless-docker-macos`). ## Fix options (not yet chosen) 1. **Restart dead daemons.** Smallest change, strictly an improvement: an OOM then degrades one download instead of removing egress for every bottle. Does not stop the OOM. 2. **Cap the scanned body size.** Root-cause fix, but a security decision: above a threshold, stop buffering (skip or stream). A "skip above N" threshold is exactly the hole an exfiltrator would aim for, trading a DoS for a covert channel. ## Notes - Pre-existing defect, not specific to containers — pulling images just makes it easy to hit. - **Does not block #392.** For the nested-container work we disable DLP on the image-registry routes, which sidesteps the buffering for those hosts; this issue tracks the general fix. Split out from #392 (comment referencing the OOM finding).

didericis-claude added the Kind/Bug

Priority

High

2

labels 2026-07-21 15:13:24 -04:00

didericis-claude referenced this issue

2026-07-21 15:13:55 -04:00

Run Docker/Compose inside bottles without bypassing isolation #392

didericis-claude referenced this issue

2026-07-21 15:39:35 -04:00

Run containers inside a bottle (nested_containers) #456

didericis-claude referenced this issue