From 0ba25352b9e222296d9ecc0e8cd94c3655a4d45a Mon Sep 17 00:00:00 2001 From: didericis Date: Tue, 21 Jul 2026 21:01:35 -0400 Subject: [PATCH] docs(nested-containers): record what live verification established Documents the proxy-layer precedence that took four attempts to get right, the blob-CDN routes every registry needs alongside its own, the podman 5 networking packages and how each one fails when absent, and the BusyBox wget caveat. Closes the DNS open question: public resolution inside a nested container fails by design; reaching the proxy was the real problem. Co-Authored-By: Claude Opus 4.8 --- README.md | 12 ++++++ docs/prds/prd-new-nested-containers.md | 57 +++++++++++++++++++++++--- 2 files changed, 64 insertions(+), 5 deletions(-) diff --git a/README.md b/README.md index eb81c28..2ab0df2 100644 --- a/README.md +++ b/README.md @@ -116,8 +116,20 @@ egress: dlp: { outbound_detectors: false, inbound_detectors: false } - host: quay.io dlp: { outbound_detectors: false, inbound_detectors: false } + # Registries redirect layer blobs to a separate CDN host, which needs its + # own route or the pull 403s partway through. + - host: production.cloudfront.docker.com + - host: pkg-containers.githubusercontent.com + - host: cdn01.quay.io ``` +Inside a nested container the same allowlist applies — an allowlisted host +returns 200 and anything else gets a 403 from the proxy. The gateway's CA and +proxy settings are wired in automatically, so `docker run … curl https://…` +works with no extra flags. One caveat: Alpine's BusyBox `wget` drops the +connection after TLS interception and reports "error getting response"; `curl` +against the same host works, so reach for `curl` when testing egress. + ### Firecracker on Linux On Linux, a KVM-capable host defaults to the Firecracker backend. It needs: diff --git a/docs/prds/prd-new-nested-containers.md b/docs/prds/prd-new-nested-containers.md index 4c4caca..a91b658 100644 --- a/docs/prds/prd-new-nested-containers.md +++ b/docs/prds/prd-new-nested-containers.md @@ -78,8 +78,15 @@ describe the one thing the design refuses to do. prerequisite it needs (podman, the storage/network helpers, writable device nodes, an *empty* subordinate range) rather than degrading. - The derived image `…-nested-containers` layers the Docker CLI, the compose - plugin, `fuse-overlayfs`, `slirp4netns`, and `uidmap` onto the agent image. - Podman itself already ships in every built-in image (#451). + plugin, and podman 5's networking stack — `passt` (pasta), `netavark`, + `nftables`, `aardvark-dns` — plus `fuse-overlayfs`, `slirp4netns`, and + `uidmap` onto the agent image. Podman itself already ships in every built-in + image (#451), but with `--no-install-recommends`, so none of the networking + pieces arrive with it. Each absence fails at a different and misleading + layer: no pasta and nothing starts; no `nft` and containers are created but + never start; no aardvark-dns and DNS inside a container fails while + everything else looks healthy. Image pulls keep working throughout, which is + what makes these read as compat-API bugs. - `BottleBackend.supports_nested_containers` — false by default, so a backend that cannot honor the flag fails in the shared `prepare` template. @@ -91,16 +98,56 @@ reaches the guest), `events_logger="file"` (no journald socket). ### Registry reach Image pulls egress through the bottle's proxy like everything else, so each -registry needs a route. Docker Hub and GHCR need `preserve_auth: true` — their +registry needs a route — **and so does the CDN it redirects blobs to**, which +is a separate host: `production.cloudfront.docker.com` for Docker Hub, +`pkg-containers.githubusercontent.com` for GHCR, `cdn0*.quay.io` for quay. +Without those the pull authenticates, fetches the manifest, and then 403s +partway through. Docker Hub and GHCR need `preserve_auth: true` — their token dance uses a client-fetched per-scope bearer token that the proxy would otherwise strip. Layer pulls should also set `dlp.outbound_detectors: false` and `dlp.inbound_detectors: false`: body scanning on a multi-hundred-MB layer is what triggers #455, and a registry route's scanned bodies are compressed layer blobs rather than anything a detector can read. +### Reaching the network from a nested container + +Public DNS inside a nested container fails by design: everything egresses +through the proxy. What was actually broken was reaching the proxy at all, and +it took four attempts to fix because podman applies proxy settings at several +layers and the last writer wins: + +| Layer | Applies to | +|---|---| +| `~/.docker/config.json` proxies block | **every container the Docker CLI starts** — client-side, beats everything below | +| service environment | podman's own pulls, non-CLI API clients | +| `containers.conf` `env` | native `podman run` | +| `containers.conf` `hosts_file`, `http_proxy` | native `podman run` only — the compat API ignores both | + +The gateway is named `bot-bottle-gateway`, which resolves only through the +bottle's `/etc/hosts`; a nested container gets its own. Since no config key +reaches the compat path, the name is resolved in the bottle and the *address* +is substituted into the proxy URL at each layer above. `NO_PROXY` keeps the +name, which is matched against what a client asks for. + +The gateway TLS-intercepts, so the CA bundle is mounted read-only and +`SSL_CERT_FILE`, `CURL_CA_BUNDLE`, `REQUESTS_CA_BUNDLE`, and +`NODE_EXTRA_CA_CERTS` point at it — no distro-specific trust commands. + +## Verified on macOS 26 / Apple Container 1.0.0 / podman 5.4.2 + +With plain `docker`, no extra flags: image pulls from Docker Hub, GHCR, and +quay; `docker run` with correct exit-code propagation; `docker compose` up, +logs, down, and a published port curl'd from the bottle; `curl https://quay.io` +and `https://pypi.org/simple/` returning 200 from inside a nested container; +and `https://example.com` returning **403** — the egress allowlist applies to +nested containers, not just the bottle. + +Alpine's BusyBox `wget` drops the connection after TLS interception and +reports "error getting response". The proxy logs the decrypted request, and +`curl` on the same host succeeds, so this is a BusyBox client limitation +rather than anything in the egress path. + ## Open questions - Whether the `docker` and `firecracker` backends want an equivalent, or whether guest-local containers stay macOS-only. -- Outbound networking *from* a nested container (DNS, proxy env, CA) beyond - what the compat socket gives the bottle itself.