bot-bottle/docs/prds/prd-new-nested-containers.md

# PRD prd-new: Containers inside a bottle

- **Status:** Draft
- **Author:** Claude
- **Created:** 2026-07-21
- **Issue:** #392

## Summary

Let an agent run `docker` and `docker compose` *inside* its bottle by starting
a guest-local rootless podman service that exposes a Docker-compatible API
socket. Gated per bottle by `nested_containers: true`. No host daemon socket is
mounted and no capability is added to the guest, on any backend.

## Problem

Agent tasks routinely involve containers: `docker compose up` to verify a
scaffolded stack, a throwaway database for a test run, building an image to
check a Dockerfile works. Bottles have no way to do any of that, so those tasks
either fail or get done outside the bottle — which defeats the point of the
bottle.

## Goals / success criteria

- A bottle with `nested_containers: true` on the `macos-container` backend can
  run `docker compose up -d --wait` against a stock compose file, pulling from
  a routed registry and serving from the workspace.
- The agent's habits do not change: `docker` and `docker compose`, not
  `podman`.
- No host Docker socket is mounted, on any backend.
- No capability is added to the Apple Container guest.
- Backends without a guest-local engine reject the flag with a clear error
  rather than ignoring it.
- Bottles that do not set the flag carry none of its cost.

## Non-goals

- Nested containers as an isolation layer. The single-UID mapping means `root`
  inside a nested container is the agent user outside it. This is for build and
  test workloads; the bottle stays the security boundary.
- The `docker` and `firecracker` backends. Both reject the flag for now.
- Reaching registries that are not routed through the bottle's egress proxy.

## Design

### Why podman, not rootless Docker

Apple Container's capability bounding set omits `CAP_SYS_ADMIN`, which the
kernel requires in order to write a multi-range `uid_map` via `newuidmap`.
Rootless Docker has no path that avoids that write, so it cannot run in a
bottle without granting `CAP_SYS_ADMIN` — which is close to root in practical
terms and gives up most of what the bottle is for.

Podman does have a path: with **no** subordinate UID range configured it falls
back to a single-UID self-mapping, which an unprivileged process may write
itself. The image build therefore *removes* the agent user's `/etc/subuid` and
`/etc/subgid` entries rather than adding them — their presence is exactly what
would send podman down the `newuidmap` path. Full negative result in
[`docs/research/rootless-docker-in-apple-container-spike.md`](../research/rootless-docker-in-apple-container-spike.md).

### The flag, and its name

The flag is kept because it gates real costs, not because podman needs a
privilege grant: ~100MB of derived image, a resident service per bottle, and
relaxed modes on `/dev/fuse` and `/dev/net/tun` that the majority of bottles
should not get.

It is named `nested_containers`, not `docker_access`. It implies no Docker and
grants access to nothing on the host — naming it after Docker access would
describe the one thing the design refuses to do.

### Shape

- `bot_bottle/backend/macos_container/nested_containers.py` — derived-image
  build, guest env, device preparation, service start/readiness.
- `bot_bottle/backend/macos_container/nested-containers-init.sh` — the
  unprivileged bootstrap that runs inside the bottle. Fails closed on every
  prerequisite it needs (podman, the storage/network helpers, writable device
  nodes, an *empty* subordinate range) rather than degrading.
- The derived image `…-nested-containers` layers the Docker CLI, the compose
  plugin, and podman 5's networking stack — `passt` (pasta), `netavark`,
  `nftables`, `aardvark-dns` — plus `fuse-overlayfs`, `slirp4netns`, and
  `uidmap` onto the agent image. Podman itself already ships in every built-in
  image (#451), but with `--no-install-recommends`, so none of the networking
  pieces arrive with it. Each absence fails at a different and misleading
  layer: no pasta and nothing starts; no `nft` and containers are created but
  never start; no aardvark-dns and DNS inside a container fails while
  everything else looks healthy. Image pulls keep working throughout, which is
  what makes these read as compat-API bugs.
- `BottleBackend.supports_nested_containers` — false by default, so a backend
  that cannot honor the flag fails in the shared `prepare` template.

Guest configuration the single-UID mapping forces:
`ignore_chown_errors` (no second UID for layers to be chowned to),
`cgroups="disabled"` and `cgroup_manager="cgroupfs"` (no cgroup delegation
reaches the guest), `events_logger="file"` (no journald socket).

### Registry reach

Image pulls egress through the bottle's proxy like everything else, so each
registry needs a route — **and so does the CDN it redirects blobs to**, which
is a separate host: `production.cloudfront.docker.com` for Docker Hub,
`pkg-containers.githubusercontent.com` for GHCR, `cdn0*.quay.io` for quay.
Without those the pull authenticates, fetches the manifest, and then 403s
partway through. Docker Hub and GHCR need `preserve_auth: true` — their
token dance uses a client-fetched per-scope bearer token that the proxy would
otherwise strip. Layer pulls should also set `dlp.outbound_detectors: false`
and `dlp.inbound_detectors: false`: body scanning on a multi-hundred-MB layer
is what triggers #455, and a registry route's scanned bodies are compressed
layer blobs rather than anything a detector can read.

### Reaching the network from a nested container

Public DNS inside a nested container fails by design: everything egresses
through the proxy. What was actually broken was reaching the proxy at all, and
it took four attempts to fix because podman applies proxy settings at several
layers and the last writer wins:

| Layer | Applies to |
|---|---|
| `~/.docker/config.json` proxies block | **every container the Docker CLI starts** — client-side, beats everything below |
| service environment | podman's own pulls, non-CLI API clients |
| `containers.conf` `env` | native `podman run` |
| `containers.conf` `hosts_file`, `http_proxy` | native `podman run` only — the compat API ignores both |

The gateway is named `bot-bottle-gateway`, which resolves only through the
bottle's `/etc/hosts`; a nested container gets its own. Since no config key
reaches the compat path, the name is resolved in the bottle and the *address*
is substituted into the proxy URL at each layer above. `NO_PROXY` keeps the
name, which is matched against what a client asks for.

The gateway TLS-intercepts, so the CA bundle is mounted read-only and
`SSL_CERT_FILE`, `CURL_CA_BUNDLE`, `REQUESTS_CA_BUNDLE`, and
`NODE_EXTRA_CA_CERTS` point at it — no distro-specific trust commands.

## Verified on macOS 26 / Apple Container 1.0.0 / podman 5.4.2

With plain `docker`, no extra flags: image pulls from Docker Hub, GHCR, and
quay; `docker run` with correct exit-code propagation; `docker compose` up,
logs, down, and a published port curl'd from the bottle; `curl https://quay.io`
and `https://pypi.org/simple/` returning 200 from inside a nested container;
and `https://example.com` returning **403** — the egress allowlist applies to
nested containers, not just the bottle.

Alpine's BusyBox `wget` drops the connection after TLS interception and
reports "error getting response". The proxy logs the decrypted request, and
`curl` on the same host succeeds, so this is a BusyBox client limitation
rather than anything in the egress path.

## Open questions

- Whether the `docker` and `firecracker` backends want an equivalent, or
  whether guest-local containers stay macOS-only.