Run containers inside a bottle (nested_containers) #456

Open
didericis-claude wants to merge 13 commits from prd-nested-containers into main
2 changed files with 64 additions and 5 deletions
Showing only changes of commit 0ba25352b9 - Show all commits
+12
View File
@@ -116,8 +116,20 @@ egress:
dlp: { outbound_detectors: false, inbound_detectors: false }
- host: quay.io
dlp: { outbound_detectors: false, inbound_detectors: false }
# Registries redirect layer blobs to a separate CDN host, which needs its
# own route or the pull 403s partway through.
- host: production.cloudfront.docker.com
- host: pkg-containers.githubusercontent.com
- host: cdn01.quay.io
```
Inside a nested container the same allowlist applies — an allowlisted host
returns 200 and anything else gets a 403 from the proxy. The gateway's CA and
proxy settings are wired in automatically, so `docker run … curl https://…`
works with no extra flags. One caveat: Alpine's BusyBox `wget` drops the
connection after TLS interception and reports "error getting response"; `curl`
against the same host works, so reach for `curl` when testing egress.
### Firecracker on Linux
On Linux, a KVM-capable host defaults to the Firecracker backend. It needs:
+52 -5
View File
@@ -78,8 +78,15 @@ describe the one thing the design refuses to do.
prerequisite it needs (podman, the storage/network helpers, writable device
nodes, an *empty* subordinate range) rather than degrading.
- The derived image `…-nested-containers` layers the Docker CLI, the compose
plugin, `fuse-overlayfs`, `slirp4netns`, and `uidmap` onto the agent image.
Podman itself already ships in every built-in image (#451).
plugin, and podman 5's networking stack — `passt` (pasta), `netavark`,
`nftables`, `aardvark-dns` — plus `fuse-overlayfs`, `slirp4netns`, and
`uidmap` onto the agent image. Podman itself already ships in every built-in
image (#451), but with `--no-install-recommends`, so none of the networking
pieces arrive with it. Each absence fails at a different and misleading
layer: no pasta and nothing starts; no `nft` and containers are created but
never start; no aardvark-dns and DNS inside a container fails while
everything else looks healthy. Image pulls keep working throughout, which is
what makes these read as compat-API bugs.
- `BottleBackend.supports_nested_containers` — false by default, so a backend
that cannot honor the flag fails in the shared `prepare` template.
@@ -91,16 +98,56 @@ reaches the guest), `events_logger="file"` (no journald socket).
### Registry reach
Image pulls egress through the bottle's proxy like everything else, so each
registry needs a route. Docker Hub and GHCR need `preserve_auth: true` — their
registry needs a route**and so does the CDN it redirects blobs to**, which
is a separate host: `production.cloudfront.docker.com` for Docker Hub,
`pkg-containers.githubusercontent.com` for GHCR, `cdn0*.quay.io` for quay.
Without those the pull authenticates, fetches the manifest, and then 403s
partway through. Docker Hub and GHCR need `preserve_auth: true` — their
token dance uses a client-fetched per-scope bearer token that the proxy would
otherwise strip. Layer pulls should also set `dlp.outbound_detectors: false`
and `dlp.inbound_detectors: false`: body scanning on a multi-hundred-MB layer
is what triggers #455, and a registry route's scanned bodies are compressed
layer blobs rather than anything a detector can read.
### Reaching the network from a nested container
Public DNS inside a nested container fails by design: everything egresses
through the proxy. What was actually broken was reaching the proxy at all, and
it took four attempts to fix because podman applies proxy settings at several
layers and the last writer wins:
| Layer | Applies to |
|---|---|
| `~/.docker/config.json` proxies block | **every container the Docker CLI starts** — client-side, beats everything below |
| service environment | podman's own pulls, non-CLI API clients |
| `containers.conf` `env` | native `podman run` |
| `containers.conf` `hosts_file`, `http_proxy` | native `podman run` only — the compat API ignores both |
The gateway is named `bot-bottle-gateway`, which resolves only through the
bottle's `/etc/hosts`; a nested container gets its own. Since no config key
reaches the compat path, the name is resolved in the bottle and the *address*
is substituted into the proxy URL at each layer above. `NO_PROXY` keeps the
name, which is matched against what a client asks for.
The gateway TLS-intercepts, so the CA bundle is mounted read-only and
`SSL_CERT_FILE`, `CURL_CA_BUNDLE`, `REQUESTS_CA_BUNDLE`, and
`NODE_EXTRA_CA_CERTS` point at it — no distro-specific trust commands.
## Verified on macOS 26 / Apple Container 1.0.0 / podman 5.4.2
With plain `docker`, no extra flags: image pulls from Docker Hub, GHCR, and
quay; `docker run` with correct exit-code propagation; `docker compose` up,
logs, down, and a published port curl'd from the bottle; `curl https://quay.io`
and `https://pypi.org/simple/` returning 200 from inside a nested container;
and `https://example.com` returning **403** — the egress allowlist applies to
nested containers, not just the bottle.
Alpine's BusyBox `wget` drops the connection after TLS interception and
reports "error getting response". The proxy logs the decrypted request, and
`curl` on the same host succeeds, so this is a BusyBox client limitation
rather than anything in the egress path.
## Open questions
- Whether the `docker` and `firecracker` backends want an equivalent, or
whether guest-local containers stay macOS-only.
- Outbound networking *from* a nested container (DNS, proxy env, CA) beyond
what the compat socket gives the bottle itself.