bot-bottle/docs/prds/0069-firecracker-native-docker-free.md

PRD 0069: Firecracker-native, Docker-free backend

• Status: Draft (partially superseded)
• Author: Claude
• Created: 2026-07-12
• Issue: #348

Superseded in part by PRD 0070 (#351): the sidecar-consolidation framing here (Stage 1, per-host sidecar; Stage 4, sidecar-as-VM) is taken over by 0070's per-host orchestrator. This PRD still owns the docker-free image-building work — Stage 2 (nix-built fixed images, a dependency of 0070) and Stage 3 (in-VM Dockerfile builder).

Summary

Make the Firecracker backend depend on firecracker + KVM only, removing Docker from the host. Two moves get us there: run the sidecar bundle as a persistent, per-host service (eventually a Firecracker VM) instead of a per-bottle container, and build agent rootfs images without a host Docker daemon (nix for the fixed images; an in-VM builder for user Dockerfiles).

Motivation

Two operator-facing costs and one security constraint:

• Resource cost. Every bottle spins up its own sidecar bundle (egress mitmproxy + git-gate + supervise). N bottles → N heavy bundles.
• Operational simplicity. Per-launch docker run churn, a Docker daemon to keep healthy, and Docker's iptables to coexist with.
• Privilege / minimal-runner. The backend needs the Docker socket (rootfs export + sidecar containers), and membership in the docker group is root-equivalent. This blocks a genuinely unprivileged run: a dedicated CI-runner user (or any confined caller) is effectively root as long as Docker is required. See the paused work in nix/gitea-runner.nix / the coverage CI gate — it can't be "minimal" until Docker is gone.

Removing Docker collapses all three.

Where Docker is used today (inventory)

1. Rootfs source — docker build (agent image from a Dockerfile) then docker create + docker export | tar → mke2fs -d (see bot_bottle/backend/firecracker/util.py:build_base_rootfs_dir and docker_image_id).
2. Sidecar bundle — docker run bot-bottle-sidecars-<slug> per bottle (egress / git-gate / supervise); the VM reaches it at the host TAP IP via DNAT (launch.py, resolve_common).
3. Digests / cache keys — docker image inspect for the rootfs cache key.

Goals

• Host prerequisites for the Firecracker backend become firecracker, /dev/kvm, iproute2, nftables — no Docker daemon, no docker group.
• Launch is rootless (open the pre-created TAP pool + KVM; no socket).
• One sidecar per host instead of one per bottle (resource + ops win).

Non-goals

• Changing the macOS (Apple Container) or legacy Docker backends.
• Removing OCI/Dockerfile support for agent images — users keep writing Dockerfiles; only the host stops needing a Docker daemon.

Design (staged)

Ordered so each stage is independently valuable and de-risks the next.

Stage 1 — Sidecar: persistent + per-host, source-IP-keyed

One sidecar bundle per host, shared by all bottles, with per-bottle policy keyed on the source IP of incoming traffic. This is safe here because the point-to-point /31 TAP + the bot_bottle_fc nft table make the source IP of anything from bbfcN provably that bottle's guest IP (no spoofing, no cross-bottle traffic) — so the sidecar can attribute a request to a bottle with confidence a shared bridge could not offer.

Per-service, sharing differs:

• supervise — host-level is a clean win (unified approval queue, ~no secrets). Do first.
• egress (mitmproxy) — shareable via a client-IP addon that selects the per-bottle allowlist / DLP / token-injection. Higher blast radius: one process now holds every bottle's upstream tokens (see Security).
• git-gate — most secret-dense (per-repo deploy keys) and least naturally shareable (git carries no source-IP-scoped auth). Keep per-bottle unless there's a strong reason.

Needs a small control plane: add/remove a bottle's routes/keys/proposals on launch/teardown with live reload, replacing "config baked at launch, torn down at exit."

Can ship as a container first (quick resource/ops win) and become a VM in Stage 4.

Stage 2 — Fixed images built with nix (no Docker)

The images bot-bottle ships — the sidecar, the agent base, and the builder (Stage 3) — are built declaratively with nix (nixos-generators / make-ext4-fs / pkgs.dockerTools for the rootfs), producing an ext4 or tar with correct ownership. Removes Docker for everything we own and gives the rootless-rootfs correctness (#347) for free on these images.

Stage 3 — User Dockerfiles built in a builder VM (the unlock)

The variable part — a user's own agent Dockerfile — builds inside a throwaway (or persistent) Firecracker builder VM running buildah/podman (rootless, daemonless) or a full in-guest dockerd. The host runs no Docker. Bonus: an untrusted Dockerfile executes in a disposable VM, which is more isolated than docker build on the host.

With this, launch touches no host Docker → the backend is rootless → the dedicated CI-runner user needs no docker group. This is the stage that unblocks the minimal-runner / coverage-CI work.

Open problems to solve here (prototype first):

• Build cache. No Docker layer cache; a persistent cache disk on the builder VM (or content-addressed rootfs cache) so rebuilds aren't full re-apt.
• Build-time egress. apt/pulls need network → through the sidecar, which is itself built earlier → nix-built fixed images break the chicken-and-egg (nothing needs Docker to come up).

Stage 4 — Sidecar (and builder) as Firecracker VMs

Full firecracker-native: the sidecar is a VM on its own TAP; agent VM → sidecar VM is VM-to-VM, so the host forwards bbfcN → sidecar TAP and the nft table grows forward rules (today it drops all non-DNAT'd egress). A per-host sidecar VM makes the boot/memory overhead amortized and gives a stable IP every agent points at.

Security considerations

• Secret concentration. Per-bottle sidecars isolate secrets at the process boundary — each holds only its bottle's tokens/keys. A host sidecar concentrates all bottles' secrets in one long-lived process and shifts isolation to application-level (correct source-IP keying). A single attribution bug leaks bottle A's token into bottle B's request — a class of bug that can't exist per-bottle. Mitigation: lean on the unspoofable TAP+nft attribution; consider keeping the most secret-dense service (git-gate) per-bottle.
• Build isolation improves. Running untrusted Dockerfiles in a disposable VM is stronger than host docker build.
• Shared fate. A host sidecar crash/compromise now affects every bottle.

Open questions

• Build cache design (per-builder-VM disk vs content-addressed host cache).
• VM-to-VM routing + the nft forward rules for a sidecar VM.
• Control-plane shape for dynamic per-bottle sidecar config + live reload.
• Whether egress is worth sharing given the secret-concentration tradeoff, or only supervise (+ keep egress/git-gate per-bottle for now).

Rollout / relation to other work

• Stages 1–2 are high-value and comparatively cheap; Stage 3 is the unlock for rootless launch and the paused dedicated-runner work; Stage 4 is the pure finish and the most networking effort.
• Prototype Stage 3 first ("Dockerfile → agent rootfs, inside a Firecracker VM, cached, with build-time egress"): if that's ergonomic and fast, the rest follows.
• Related: #347 (rootless rootfs ownership — subsumed by nix-built fixed images + the in-VM builder), and the deferred coverage CI gate / gitea runner (blocked on Stage 3).