bot-bottle/docs/prds/prd-new-nested-containers.md

PRD prd-new: Containers inside a bottle

• Status: Draft
• Author: Claude
• Created: 2026-07-21
• Issue: #392

Summary

Let an agent run docker and docker compose inside its bottle by starting a guest-local rootless podman service that exposes a Docker-compatible API socket. Gated per bottle by nested_containers: true. No host daemon socket is mounted and no capability is added to the guest, on any backend.

Problem

Agent tasks routinely involve containers: docker compose up to verify a scaffolded stack, a throwaway database for a test run, building an image to check a Dockerfile works. Bottles have no way to do any of that, so those tasks either fail or get done outside the bottle — which defeats the point of the bottle.

Goals / success criteria

• A bottle with nested_containers: true on the macos-container backend can run docker compose up -d --wait against a stock compose file, pulling from a routed registry and serving from the workspace.
• The agent's habits do not change: docker and docker compose, not podman.
• No host Docker socket is mounted, on any backend.
• No capability is added to the Apple Container guest.
• Backends without a guest-local engine reject the flag with a clear error rather than ignoring it.
• Bottles that do not set the flag carry none of its cost.

Non-goals

• Nested containers as an isolation layer. The single-UID mapping means root inside a nested container is the agent user outside it. This is for build and test workloads; the bottle stays the security boundary.
• The docker and firecracker backends. Both reject the flag for now.
• Reaching registries that are not routed through the bottle's egress proxy.

Design

Why podman, not rootless Docker

Apple Container's capability bounding set omits CAP_SYS_ADMIN, which the kernel requires in order to write a multi-range uid_map via newuidmap. Rootless Docker has no path that avoids that write, so it cannot run in a bottle without granting CAP_SYS_ADMIN — which is close to root in practical terms and gives up most of what the bottle is for.

Podman does have a path: with no subordinate UID range configured it falls back to a single-UID self-mapping, which an unprivileged process may write itself. The image build therefore removes the agent user's /etc/subuid and /etc/subgid entries rather than adding them — their presence is exactly what would send podman down the newuidmap path. Full negative result in docs/research/rootless-docker-in-apple-container-spike.md.

The flag, and its name

The flag is kept because it gates real costs, not because podman needs a privilege grant: ~100MB of derived image, a resident service per bottle, and relaxed modes on /dev/fuse and /dev/net/tun that the majority of bottles should not get.

It is named nested_containers, not docker_access. It implies no Docker and grants access to nothing on the host — naming it after Docker access would describe the one thing the design refuses to do.

Shape

• bot_bottle/backend/macos_container/nested_containers.py — derived-image build, guest env, device preparation, service start/readiness.
• bot_bottle/backend/macos_container/nested-containers-init.sh — the unprivileged bootstrap that runs inside the bottle. Fails closed on every prerequisite it needs (podman, the storage/network helpers, writable device nodes, an empty subordinate range) rather than degrading.
• The derived image …-nested-containers layers the Docker CLI, the compose plugin, and podman 5's networking stack — passt (pasta), netavark, nftables, aardvark-dns — plus fuse-overlayfs, slirp4netns, and uidmap onto the agent image. Podman itself already ships in every built-in image (#451), but with --no-install-recommends, so none of the networking pieces arrive with it. Each absence fails at a different and misleading layer: no pasta and nothing starts; no nft and containers are created but never start; no aardvark-dns and DNS inside a container fails while everything else looks healthy. Image pulls keep working throughout, which is what makes these read as compat-API bugs.
• BottleBackend.supports_nested_containers — false by default, so a backend that cannot honor the flag fails in the shared prepare template.

Guest configuration the single-UID mapping forces: ignore_chown_errors (no second UID for layers to be chowned to), cgroups="disabled" and cgroup_manager="cgroupfs" (no cgroup delegation reaches the guest), events_logger="file" (no journald socket).

Registry reach

Image pulls egress through the bottle's proxy like everything else, so each registry needs a route — and so does the CDN it redirects blobs to, which is a separate host: production.cloudfront.docker.com for Docker Hub, pkg-containers.githubusercontent.com for GHCR, cdn0*.quay.io for quay. Without those the pull authenticates, fetches the manifest, and then 403s partway through. Docker Hub and GHCR need preserve_auth: true — their token dance uses a client-fetched per-scope bearer token that the proxy would otherwise strip. Layer pulls should also set dlp.outbound_detectors: false and dlp.inbound_detectors: false: body scanning on a multi-hundred-MB layer is what triggers #455, and a registry route's scanned bodies are compressed layer blobs rather than anything a detector can read.

Reaching the network from a nested container

Public DNS inside a nested container fails by design: everything egresses through the proxy. What was actually broken was reaching the proxy at all, and it took four attempts to fix because podman applies proxy settings at several layers and the last writer wins:

Layer	Applies to
~/.docker/config.json proxies block	every container the Docker CLI starts — client-side, beats everything below
service environment	podman's own pulls, non-CLI API clients
containers.conf env	native podman run
containers.conf hosts_file, http_proxy	native podman run only — the compat API ignores both

The gateway is named bot-bottle-gateway, which resolves only through the bottle's /etc/hosts; a nested container gets its own. Since no config key reaches the compat path, the name is resolved in the bottle and the address is substituted into the proxy URL at each layer above. NO_PROXY keeps the name, which is matched against what a client asks for.

The gateway TLS-intercepts, so the CA bundle is mounted read-only and SSL_CERT_FILE, CURL_CA_BUNDLE, REQUESTS_CA_BUNDLE, and NODE_EXTRA_CA_CERTS point at it — no distro-specific trust commands.

Verified on macOS 26 / Apple Container 1.0.0 / podman 5.4.2

With plain docker, no extra flags: image pulls from Docker Hub, GHCR, and quay; docker run with correct exit-code propagation; docker compose up, logs, down, and a published port curl'd from the bottle; curl https://quay.io and https://pypi.org/simple/ returning 200 from inside a nested container; and https://example.com returning 403 — the egress allowlist applies to nested containers, not just the bottle.

Alpine's BusyBox wget drops the connection after TLS interception and reports "error getting response". The proxy logs the decrypted request, and curl on the same host succeeds, so this is a BusyBox client limitation rather than anything in the egress path.

Open questions

• Whether the docker and firecracker backends want an equivalent, or whether guest-local containers stay macOS-only.