docs(prd): 0069 firecracker-native, docker-free backend

Capture the vision: drop the host Docker dependency (rootfs export +
sidecar containers) so the Firecracker backend needs only firecracker +
KVM. Staged plan (persistent per-host sidecar → nix-built fixed images →
in-VM Dockerfile builds → sidecar/builder as VMs), the docker inventory,
the secret-concentration tradeoff, and the link to rootless launch /
the paused minimal-runner work (#348).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WBMWTEtQdJ4W5UrWuLHCck

docs/prds/0069-firecracker-native-docker-free.md

@@ -0,0 +1,158 @@
+# PRD 0069: Firecracker-native, Docker-free backend
+
+- **Status:** Draft
+- **Author:** Claude
+- **Created:** 2026-07-12
+- **Issue:** #348
+
+## Summary
+
+Make the Firecracker backend depend on **firecracker + KVM only**, removing
+Docker from the host. Two moves get us there: run the **sidecar bundle as a
+persistent, per-host service** (eventually a Firecracker VM) instead of a
+per-bottle container, and **build agent rootfs images without a host Docker
+daemon** (nix for the fixed images; an in-VM builder for user Dockerfiles).
+
+## Motivation
+
+Two operator-facing costs and one security constraint:
+
+- **Resource cost.** Every bottle spins up its own sidecar bundle (egress
+  mitmproxy + git-gate + supervise). N bottles → N heavy bundles.
+- **Operational simplicity.** Per-launch `docker run` churn, a Docker daemon
+  to keep healthy, and Docker's iptables to coexist with.
+- **Privilege / minimal-runner.** The backend needs the Docker socket
+  (rootfs export + sidecar containers), and membership in the `docker`
+  group is root-equivalent. This blocks a genuinely unprivileged run: a
+  dedicated CI-runner user (or any confined caller) is effectively root as
+  long as Docker is required. See the paused work in
+  `nix/gitea-runner.nix` / the coverage CI gate — it can't be "minimal"
+  until Docker is gone.
+
+Removing Docker collapses all three.
+
+## Where Docker is used today (inventory)
+
+1. **Rootfs source** — `docker build` (agent image from a Dockerfile) then
+   `docker create` + `docker export | tar` → `mke2fs -d` (see
+   `bot_bottle/backend/firecracker/util.py:build_base_rootfs_dir` and
+   `docker_image_id`).
+2. **Sidecar bundle** — `docker run bot-bottle-sidecars-<slug>` per bottle
+   (egress / git-gate / supervise); the VM reaches it at the host TAP IP via
+   DNAT (`launch.py`, `resolve_common`).
+3. **Digests / cache keys** — `docker image inspect` for the rootfs cache
+   key.
+
+## Goals
+
+- Host prerequisites for the Firecracker backend become **firecracker,
+  `/dev/kvm`, iproute2, nftables** — no Docker daemon, no `docker` group.
+- Launch is **rootless** (open the pre-created TAP pool + KVM; no socket).
+- One sidecar per host instead of one per bottle (resource + ops win).
+
+## Non-goals
+
+- Changing the macOS (Apple Container) or legacy Docker backends.
+- Removing OCI/Dockerfile support for agent images — users keep writing
+  Dockerfiles; only the *host* stops needing a Docker daemon.
+
+## Design (staged)
+
+Ordered so each stage is independently valuable and de-risks the next.
+
+### Stage 1 — Sidecar: persistent + per-host, source-IP-keyed
+
+One sidecar bundle per host, shared by all bottles, with per-bottle policy
+keyed on the **source IP** of incoming traffic. This is safe here because
+the point-to-point `/31` TAP + the `bot_bottle_fc` nft table make the source
+IP of anything from `bbfcN` *provably* that bottle's guest IP (no spoofing,
+no cross-bottle traffic) — so the sidecar can attribute a request to a
+bottle with confidence a shared bridge could not offer.
+
+Per-service, sharing differs:
+
+- **supervise** — host-level is a clean win (unified approval queue, ~no
+  secrets). Do first.
+- **egress (mitmproxy)** — shareable via a client-IP addon that selects the
+  per-bottle allowlist / DLP / token-injection. Higher blast radius: one
+  process now holds every bottle's upstream tokens (see Security).
+- **git-gate** — most secret-dense (per-repo deploy keys) and least
+  naturally shareable (git carries no source-IP-scoped auth). Keep
+  per-bottle unless there's a strong reason.
+
+Needs a small **control plane**: add/remove a bottle's routes/keys/proposals
+on launch/teardown with live reload, replacing "config baked at launch,
+torn down at exit."
+
+Can ship as a container first (quick resource/ops win) and become a VM in
+Stage 4.
+
+### Stage 2 — Fixed images built with nix (no Docker)
+
+The images bot-bottle *ships* — the sidecar, the agent base, and the builder
+(Stage 3) — are built declaratively with nix (`nixos-generators` /
+`make-ext4-fs` / `pkgs.dockerTools` for the rootfs), producing an ext4 or
+tar with correct ownership. Removes Docker for everything we own and gives
+the rootless-rootfs correctness (#347) for free on these images.
+
+### Stage 3 — User Dockerfiles built in a builder VM (the unlock)
+
+The variable part — a user's own agent Dockerfile — builds **inside a
+throwaway (or persistent) Firecracker builder VM** running `buildah`/`podman`
+(rootless, daemonless) or a full in-guest dockerd. The host runs no Docker.
+Bonus: an untrusted Dockerfile executes in a disposable VM, which is *more*
+isolated than `docker build` on the host.
+
+With this, launch touches no host Docker → the backend is rootless → the
+dedicated CI-runner user needs no `docker` group. **This is the stage that
+unblocks the minimal-runner / coverage-CI work.**
+
+Open problems to solve here (prototype first):
+- **Build cache.** No Docker layer cache; a persistent cache disk on the
+  builder VM (or content-addressed rootfs cache) so rebuilds aren't full
+  re-`apt`.
+- **Build-time egress.** `apt`/pulls need network → through the sidecar,
+  which is itself built earlier → nix-built fixed images break the
+  chicken-and-egg (nothing needs Docker to come up).
+
+### Stage 4 — Sidecar (and builder) as Firecracker VMs
+
+Full firecracker-native: the sidecar is a VM on its own TAP; agent VM →
+sidecar VM is VM-to-VM, so the host forwards `bbfcN` → sidecar TAP and the
+nft table grows forward rules (today it *drops* all non-DNAT'd egress). A
+**per-host** sidecar VM makes the boot/memory overhead amortized and gives a
+stable IP every agent points at.
+
+## Security considerations
+
+- **Secret concentration.** Per-bottle sidecars isolate secrets at the
+  process boundary — each holds only its bottle's tokens/keys. A host
+  sidecar concentrates *all* bottles' secrets in one long-lived process and
+  shifts isolation to **application-level** (correct source-IP keying). A
+  single attribution bug leaks bottle A's token into bottle B's request — a
+  class of bug that can't exist per-bottle. Mitigation: lean on the
+  unspoofable TAP+nft attribution; consider keeping the most secret-dense
+  service (git-gate) per-bottle.
+- **Build isolation improves.** Running untrusted Dockerfiles in a
+  disposable VM is stronger than host `docker build`.
+- **Shared fate.** A host sidecar crash/compromise now affects every bottle.
+
+## Open questions
+
+- Build cache design (per-builder-VM disk vs content-addressed host cache).
+- VM-to-VM routing + the nft forward rules for a sidecar VM.
+- Control-plane shape for dynamic per-bottle sidecar config + live reload.
+- Whether egress is worth sharing given the secret-concentration tradeoff,
+  or only supervise (+ keep egress/git-gate per-bottle for now).
+
+## Rollout / relation to other work
+
+- Stages 1–2 are high-value and comparatively cheap; **Stage 3 is the
+  unlock** for rootless launch and the paused dedicated-runner work; Stage 4
+  is the pure finish and the most networking effort.
+- Prototype Stage 3 first ("Dockerfile → agent rootfs, inside a Firecracker
+  VM, cached, with build-time egress"): if that's ergonomic and fast, the
+  rest follows.
+- Related: #347 (rootless rootfs ownership — subsumed by nix-built fixed
+  images + the in-VM builder), and the deferred coverage CI gate / gitea
+  runner (blocked on Stage 3).