docs(nested-containers): record what live verification established
Documents the proxy-layer precedence that took four attempts to get right, the blob-CDN routes every registry needs alongside its own, the podman 5 networking packages and how each one fails when absent, and the BusyBox wget caveat. Closes the DNS open question: public resolution inside a nested container fails by design; reaching the proxy was the real problem. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
@@ -116,8 +116,20 @@ egress:
|
|||||||
dlp: { outbound_detectors: false, inbound_detectors: false }
|
dlp: { outbound_detectors: false, inbound_detectors: false }
|
||||||
- host: quay.io
|
- host: quay.io
|
||||||
dlp: { outbound_detectors: false, inbound_detectors: false }
|
dlp: { outbound_detectors: false, inbound_detectors: false }
|
||||||
|
# Registries redirect layer blobs to a separate CDN host, which needs its
|
||||||
|
# own route or the pull 403s partway through.
|
||||||
|
- host: production.cloudfront.docker.com
|
||||||
|
- host: pkg-containers.githubusercontent.com
|
||||||
|
- host: cdn01.quay.io
|
||||||
```
|
```
|
||||||
|
|
||||||
|
Inside a nested container the same allowlist applies — an allowlisted host
|
||||||
|
returns 200 and anything else gets a 403 from the proxy. The gateway's CA and
|
||||||
|
proxy settings are wired in automatically, so `docker run … curl https://…`
|
||||||
|
works with no extra flags. One caveat: Alpine's BusyBox `wget` drops the
|
||||||
|
connection after TLS interception and reports "error getting response"; `curl`
|
||||||
|
against the same host works, so reach for `curl` when testing egress.
|
||||||
|
|
||||||
### Firecracker on Linux
|
### Firecracker on Linux
|
||||||
|
|
||||||
On Linux, a KVM-capable host defaults to the Firecracker backend. It needs:
|
On Linux, a KVM-capable host defaults to the Firecracker backend. It needs:
|
||||||
|
|||||||
@@ -78,8 +78,15 @@ describe the one thing the design refuses to do.
|
|||||||
prerequisite it needs (podman, the storage/network helpers, writable device
|
prerequisite it needs (podman, the storage/network helpers, writable device
|
||||||
nodes, an *empty* subordinate range) rather than degrading.
|
nodes, an *empty* subordinate range) rather than degrading.
|
||||||
- The derived image `…-nested-containers` layers the Docker CLI, the compose
|
- The derived image `…-nested-containers` layers the Docker CLI, the compose
|
||||||
plugin, `fuse-overlayfs`, `slirp4netns`, and `uidmap` onto the agent image.
|
plugin, and podman 5's networking stack — `passt` (pasta), `netavark`,
|
||||||
Podman itself already ships in every built-in image (#451).
|
`nftables`, `aardvark-dns` — plus `fuse-overlayfs`, `slirp4netns`, and
|
||||||
|
`uidmap` onto the agent image. Podman itself already ships in every built-in
|
||||||
|
image (#451), but with `--no-install-recommends`, so none of the networking
|
||||||
|
pieces arrive with it. Each absence fails at a different and misleading
|
||||||
|
layer: no pasta and nothing starts; no `nft` and containers are created but
|
||||||
|
never start; no aardvark-dns and DNS inside a container fails while
|
||||||
|
everything else looks healthy. Image pulls keep working throughout, which is
|
||||||
|
what makes these read as compat-API bugs.
|
||||||
- `BottleBackend.supports_nested_containers` — false by default, so a backend
|
- `BottleBackend.supports_nested_containers` — false by default, so a backend
|
||||||
that cannot honor the flag fails in the shared `prepare` template.
|
that cannot honor the flag fails in the shared `prepare` template.
|
||||||
|
|
||||||
@@ -91,16 +98,56 @@ reaches the guest), `events_logger="file"` (no journald socket).
|
|||||||
### Registry reach
|
### Registry reach
|
||||||
|
|
||||||
Image pulls egress through the bottle's proxy like everything else, so each
|
Image pulls egress through the bottle's proxy like everything else, so each
|
||||||
registry needs a route. Docker Hub and GHCR need `preserve_auth: true` — their
|
registry needs a route — **and so does the CDN it redirects blobs to**, which
|
||||||
|
is a separate host: `production.cloudfront.docker.com` for Docker Hub,
|
||||||
|
`pkg-containers.githubusercontent.com` for GHCR, `cdn0*.quay.io` for quay.
|
||||||
|
Without those the pull authenticates, fetches the manifest, and then 403s
|
||||||
|
partway through. Docker Hub and GHCR need `preserve_auth: true` — their
|
||||||
token dance uses a client-fetched per-scope bearer token that the proxy would
|
token dance uses a client-fetched per-scope bearer token that the proxy would
|
||||||
otherwise strip. Layer pulls should also set `dlp.outbound_detectors: false`
|
otherwise strip. Layer pulls should also set `dlp.outbound_detectors: false`
|
||||||
and `dlp.inbound_detectors: false`: body scanning on a multi-hundred-MB layer
|
and `dlp.inbound_detectors: false`: body scanning on a multi-hundred-MB layer
|
||||||
is what triggers #455, and a registry route's scanned bodies are compressed
|
is what triggers #455, and a registry route's scanned bodies are compressed
|
||||||
layer blobs rather than anything a detector can read.
|
layer blobs rather than anything a detector can read.
|
||||||
|
|
||||||
|
### Reaching the network from a nested container
|
||||||
|
|
||||||
|
Public DNS inside a nested container fails by design: everything egresses
|
||||||
|
through the proxy. What was actually broken was reaching the proxy at all, and
|
||||||
|
it took four attempts to fix because podman applies proxy settings at several
|
||||||
|
layers and the last writer wins:
|
||||||
|
|
||||||
|
| Layer | Applies to |
|
||||||
|
|---|---|
|
||||||
|
| `~/.docker/config.json` proxies block | **every container the Docker CLI starts** — client-side, beats everything below |
|
||||||
|
| service environment | podman's own pulls, non-CLI API clients |
|
||||||
|
| `containers.conf` `env` | native `podman run` |
|
||||||
|
| `containers.conf` `hosts_file`, `http_proxy` | native `podman run` only — the compat API ignores both |
|
||||||
|
|
||||||
|
The gateway is named `bot-bottle-gateway`, which resolves only through the
|
||||||
|
bottle's `/etc/hosts`; a nested container gets its own. Since no config key
|
||||||
|
reaches the compat path, the name is resolved in the bottle and the *address*
|
||||||
|
is substituted into the proxy URL at each layer above. `NO_PROXY` keeps the
|
||||||
|
name, which is matched against what a client asks for.
|
||||||
|
|
||||||
|
The gateway TLS-intercepts, so the CA bundle is mounted read-only and
|
||||||
|
`SSL_CERT_FILE`, `CURL_CA_BUNDLE`, `REQUESTS_CA_BUNDLE`, and
|
||||||
|
`NODE_EXTRA_CA_CERTS` point at it — no distro-specific trust commands.
|
||||||
|
|
||||||
|
## Verified on macOS 26 / Apple Container 1.0.0 / podman 5.4.2
|
||||||
|
|
||||||
|
With plain `docker`, no extra flags: image pulls from Docker Hub, GHCR, and
|
||||||
|
quay; `docker run` with correct exit-code propagation; `docker compose` up,
|
||||||
|
logs, down, and a published port curl'd from the bottle; `curl https://quay.io`
|
||||||
|
and `https://pypi.org/simple/` returning 200 from inside a nested container;
|
||||||
|
and `https://example.com` returning **403** — the egress allowlist applies to
|
||||||
|
nested containers, not just the bottle.
|
||||||
|
|
||||||
|
Alpine's BusyBox `wget` drops the connection after TLS interception and
|
||||||
|
reports "error getting response". The proxy logs the decrypted request, and
|
||||||
|
`curl` on the same host succeeds, so this is a BusyBox client limitation
|
||||||
|
rather than anything in the egress path.
|
||||||
|
|
||||||
## Open questions
|
## Open questions
|
||||||
|
|
||||||
- Whether the `docker` and `firecracker` backends want an equivalent, or
|
- Whether the `docker` and `firecracker` backends want an equivalent, or
|
||||||
whether guest-local containers stay macOS-only.
|
whether guest-local containers stay macOS-only.
|
||||||
- Outbound networking *from* a nested container (DNS, proxy env, CA) beyond
|
|
||||||
what the compat socket gives the bottle itself.
|
|
||||||
|
|||||||
Reference in New Issue
Block a user