docs(nested-containers): record what live verification established
test / unit (pull_request) Failing after 33s
tracker-policy-pr / check-pr (pull_request) Failing after 34s
test / integration-docker (pull_request) Successful in 42s
test / integration-firecracker (pull_request) Successful in 4m4s
test / coverage (pull_request) Has been skipped
test / publish-infra (pull_request) Has been skipped

Documents the proxy-layer precedence that took four attempts to get
right, the blob-CDN routes every registry needs alongside its own, the
podman 5 networking packages and how each one fails when absent, and
the BusyBox wget caveat. Closes the DNS open question: public
resolution inside a nested container fails by design; reaching the
proxy was the real problem.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
2026-07-21 21:01:35 -04:00
parent 0bf7f9ece6
commit 913bcab6a0
2 changed files with 64 additions and 5 deletions
+52 -5
View File
@@ -78,8 +78,15 @@ describe the one thing the design refuses to do.
prerequisite it needs (podman, the storage/network helpers, writable device
nodes, an *empty* subordinate range) rather than degrading.
- The derived image `…-nested-containers` layers the Docker CLI, the compose
plugin, `fuse-overlayfs`, `slirp4netns`, and `uidmap` onto the agent image.
Podman itself already ships in every built-in image (#451).
plugin, and podman 5's networking stack — `passt` (pasta), `netavark`,
`nftables`, `aardvark-dns` — plus `fuse-overlayfs`, `slirp4netns`, and
`uidmap` onto the agent image. Podman itself already ships in every built-in
image (#451), but with `--no-install-recommends`, so none of the networking
pieces arrive with it. Each absence fails at a different and misleading
layer: no pasta and nothing starts; no `nft` and containers are created but
never start; no aardvark-dns and DNS inside a container fails while
everything else looks healthy. Image pulls keep working throughout, which is
what makes these read as compat-API bugs.
- `BottleBackend.supports_nested_containers` — false by default, so a backend
that cannot honor the flag fails in the shared `prepare` template.
@@ -91,16 +98,56 @@ reaches the guest), `events_logger="file"` (no journald socket).
### Registry reach
Image pulls egress through the bottle's proxy like everything else, so each
registry needs a route. Docker Hub and GHCR need `preserve_auth: true` — their
registry needs a route**and so does the CDN it redirects blobs to**, which
is a separate host: `production.cloudfront.docker.com` for Docker Hub,
`pkg-containers.githubusercontent.com` for GHCR, `cdn0*.quay.io` for quay.
Without those the pull authenticates, fetches the manifest, and then 403s
partway through. Docker Hub and GHCR need `preserve_auth: true` — their
token dance uses a client-fetched per-scope bearer token that the proxy would
otherwise strip. Layer pulls should also set `dlp.outbound_detectors: false`
and `dlp.inbound_detectors: false`: body scanning on a multi-hundred-MB layer
is what triggers #455, and a registry route's scanned bodies are compressed
layer blobs rather than anything a detector can read.
### Reaching the network from a nested container
Public DNS inside a nested container fails by design: everything egresses
through the proxy. What was actually broken was reaching the proxy at all, and
it took four attempts to fix because podman applies proxy settings at several
layers and the last writer wins:
| Layer | Applies to |
|---|---|
| `~/.docker/config.json` proxies block | **every container the Docker CLI starts** — client-side, beats everything below |
| service environment | podman's own pulls, non-CLI API clients |
| `containers.conf` `env` | native `podman run` |
| `containers.conf` `hosts_file`, `http_proxy` | native `podman run` only — the compat API ignores both |
The gateway is named `bot-bottle-gateway`, which resolves only through the
bottle's `/etc/hosts`; a nested container gets its own. Since no config key
reaches the compat path, the name is resolved in the bottle and the *address*
is substituted into the proxy URL at each layer above. `NO_PROXY` keeps the
name, which is matched against what a client asks for.
The gateway TLS-intercepts, so the CA bundle is mounted read-only and
`SSL_CERT_FILE`, `CURL_CA_BUNDLE`, `REQUESTS_CA_BUNDLE`, and
`NODE_EXTRA_CA_CERTS` point at it — no distro-specific trust commands.
## Verified on macOS 26 / Apple Container 1.0.0 / podman 5.4.2
With plain `docker`, no extra flags: image pulls from Docker Hub, GHCR, and
quay; `docker run` with correct exit-code propagation; `docker compose` up,
logs, down, and a published port curl'd from the bottle; `curl https://quay.io`
and `https://pypi.org/simple/` returning 200 from inside a nested container;
and `https://example.com` returning **403** — the egress allowlist applies to
nested containers, not just the bottle.
Alpine's BusyBox `wget` drops the connection after TLS interception and
reports "error getting response". The proxy logs the decrypted request, and
`curl` on the same host succeeds, so this is a BusyBox client limitation
rather than anything in the egress path.
## Open questions
- Whether the `docker` and `firecracker` backends want an equivalent, or
whether guest-local containers stay macOS-only.
- Outbound networking *from* a nested container (DNS, proxy env, CA) beyond
what the compat socket gives the bottle itself.