diff --git a/docs/prds/0069-firecracker-native-docker-free.md b/docs/prds/0069-firecracker-native-docker-free.md new file mode 100644 index 0000000..0a24167 --- /dev/null +++ b/docs/prds/0069-firecracker-native-docker-free.md @@ -0,0 +1,158 @@ +# PRD 0069: Firecracker-native, Docker-free backend + +- **Status:** Draft +- **Author:** Claude +- **Created:** 2026-07-12 +- **Issue:** #348 + +## Summary + +Make the Firecracker backend depend on **firecracker + KVM only**, removing +Docker from the host. Two moves get us there: run the **sidecar bundle as a +persistent, per-host service** (eventually a Firecracker VM) instead of a +per-bottle container, and **build agent rootfs images without a host Docker +daemon** (nix for the fixed images; an in-VM builder for user Dockerfiles). + +## Motivation + +Two operator-facing costs and one security constraint: + +- **Resource cost.** Every bottle spins up its own sidecar bundle (egress + mitmproxy + git-gate + supervise). N bottles → N heavy bundles. +- **Operational simplicity.** Per-launch `docker run` churn, a Docker daemon + to keep healthy, and Docker's iptables to coexist with. +- **Privilege / minimal-runner.** The backend needs the Docker socket + (rootfs export + sidecar containers), and membership in the `docker` + group is root-equivalent. This blocks a genuinely unprivileged run: a + dedicated CI-runner user (or any confined caller) is effectively root as + long as Docker is required. See the paused work in + `nix/gitea-runner.nix` / the coverage CI gate — it can't be "minimal" + until Docker is gone. + +Removing Docker collapses all three. + +## Where Docker is used today (inventory) + +1. **Rootfs source** — `docker build` (agent image from a Dockerfile) then + `docker create` + `docker export | tar` → `mke2fs -d` (see + `bot_bottle/backend/firecracker/util.py:build_base_rootfs_dir` and + `docker_image_id`). +2. **Sidecar bundle** — `docker run bot-bottle-sidecars-` per bottle + (egress / git-gate / supervise); the VM reaches it at the host TAP IP via + DNAT (`launch.py`, `resolve_common`). +3. **Digests / cache keys** — `docker image inspect` for the rootfs cache + key. + +## Goals + +- Host prerequisites for the Firecracker backend become **firecracker, + `/dev/kvm`, iproute2, nftables** — no Docker daemon, no `docker` group. +- Launch is **rootless** (open the pre-created TAP pool + KVM; no socket). +- One sidecar per host instead of one per bottle (resource + ops win). + +## Non-goals + +- Changing the macOS (Apple Container) or legacy Docker backends. +- Removing OCI/Dockerfile support for agent images — users keep writing + Dockerfiles; only the *host* stops needing a Docker daemon. + +## Design (staged) + +Ordered so each stage is independently valuable and de-risks the next. + +### Stage 1 — Sidecar: persistent + per-host, source-IP-keyed + +One sidecar bundle per host, shared by all bottles, with per-bottle policy +keyed on the **source IP** of incoming traffic. This is safe here because +the point-to-point `/31` TAP + the `bot_bottle_fc` nft table make the source +IP of anything from `bbfcN` *provably* that bottle's guest IP (no spoofing, +no cross-bottle traffic) — so the sidecar can attribute a request to a +bottle with confidence a shared bridge could not offer. + +Per-service, sharing differs: + +- **supervise** — host-level is a clean win (unified approval queue, ~no + secrets). Do first. +- **egress (mitmproxy)** — shareable via a client-IP addon that selects the + per-bottle allowlist / DLP / token-injection. Higher blast radius: one + process now holds every bottle's upstream tokens (see Security). +- **git-gate** — most secret-dense (per-repo deploy keys) and least + naturally shareable (git carries no source-IP-scoped auth). Keep + per-bottle unless there's a strong reason. + +Needs a small **control plane**: add/remove a bottle's routes/keys/proposals +on launch/teardown with live reload, replacing "config baked at launch, +torn down at exit." + +Can ship as a container first (quick resource/ops win) and become a VM in +Stage 4. + +### Stage 2 — Fixed images built with nix (no Docker) + +The images bot-bottle *ships* — the sidecar, the agent base, and the builder +(Stage 3) — are built declaratively with nix (`nixos-generators` / +`make-ext4-fs` / `pkgs.dockerTools` for the rootfs), producing an ext4 or +tar with correct ownership. Removes Docker for everything we own and gives +the rootless-rootfs correctness (#347) for free on these images. + +### Stage 3 — User Dockerfiles built in a builder VM (the unlock) + +The variable part — a user's own agent Dockerfile — builds **inside a +throwaway (or persistent) Firecracker builder VM** running `buildah`/`podman` +(rootless, daemonless) or a full in-guest dockerd. The host runs no Docker. +Bonus: an untrusted Dockerfile executes in a disposable VM, which is *more* +isolated than `docker build` on the host. + +With this, launch touches no host Docker → the backend is rootless → the +dedicated CI-runner user needs no `docker` group. **This is the stage that +unblocks the minimal-runner / coverage-CI work.** + +Open problems to solve here (prototype first): +- **Build cache.** No Docker layer cache; a persistent cache disk on the + builder VM (or content-addressed rootfs cache) so rebuilds aren't full + re-`apt`. +- **Build-time egress.** `apt`/pulls need network → through the sidecar, + which is itself built earlier → nix-built fixed images break the + chicken-and-egg (nothing needs Docker to come up). + +### Stage 4 — Sidecar (and builder) as Firecracker VMs + +Full firecracker-native: the sidecar is a VM on its own TAP; agent VM → +sidecar VM is VM-to-VM, so the host forwards `bbfcN` → sidecar TAP and the +nft table grows forward rules (today it *drops* all non-DNAT'd egress). A +**per-host** sidecar VM makes the boot/memory overhead amortized and gives a +stable IP every agent points at. + +## Security considerations + +- **Secret concentration.** Per-bottle sidecars isolate secrets at the + process boundary — each holds only its bottle's tokens/keys. A host + sidecar concentrates *all* bottles' secrets in one long-lived process and + shifts isolation to **application-level** (correct source-IP keying). A + single attribution bug leaks bottle A's token into bottle B's request — a + class of bug that can't exist per-bottle. Mitigation: lean on the + unspoofable TAP+nft attribution; consider keeping the most secret-dense + service (git-gate) per-bottle. +- **Build isolation improves.** Running untrusted Dockerfiles in a + disposable VM is stronger than host `docker build`. +- **Shared fate.** A host sidecar crash/compromise now affects every bottle. + +## Open questions + +- Build cache design (per-builder-VM disk vs content-addressed host cache). +- VM-to-VM routing + the nft forward rules for a sidecar VM. +- Control-plane shape for dynamic per-bottle sidecar config + live reload. +- Whether egress is worth sharing given the secret-concentration tradeoff, + or only supervise (+ keep egress/git-gate per-bottle for now). + +## Rollout / relation to other work + +- Stages 1–2 are high-value and comparatively cheap; **Stage 3 is the + unlock** for rootless launch and the paused dedicated-runner work; Stage 4 + is the pure finish and the most networking effort. +- Prototype Stage 3 first ("Dockerfile → agent rootfs, inside a Firecracker + VM, cached, with build-time egress"): if that's ergonomic and fast, the + rest follows. +- Related: #347 (rootless rootfs ownership — subsumed by nix-built fixed + images + the in-VM builder), and the deferred coverage CI gate / gitea + runner (blocked on Stage 3).