bot-bottle/docs/prds/0069-firecracker-native-docker-free.md

# PRD 0069: Firecracker-native, Docker-free backend

- **Status:** Draft (partially superseded)
- **Author:** Claude
- **Created:** 2026-07-12
- **Issue:** #348

> **Superseded in part by [PRD 0070](0070-per-host-orchestrator.md) (#351):**
> the sidecar-consolidation framing here (Stage 1, per-host sidecar; Stage 4,
> sidecar-as-VM) is taken over by 0070's per-host orchestrator. This PRD still
> owns the docker-free **image-building** work — Stage 2 (nix-built fixed
> images, a dependency of 0070) and Stage 3 (in-VM Dockerfile builder).

## Summary

Make the Firecracker backend depend on **firecracker + KVM only**, removing
Docker from the host. Two moves get us there: run the **sidecar bundle as a
persistent, per-host service** (eventually a Firecracker VM) instead of a
per-bottle container, and **build agent rootfs images without a host Docker
daemon** (nix for the fixed images; an in-VM builder for user Dockerfiles).

## Motivation

Two operator-facing costs and one security constraint:

- **Resource cost.** Every bottle spins up its own sidecar bundle (egress
  mitmproxy + git-gate + supervise). N bottles → N heavy bundles.
- **Operational simplicity.** Per-launch `docker run` churn, a Docker daemon
  to keep healthy, and Docker's iptables to coexist with.
- **Privilege / minimal-runner.** The backend needs the Docker socket
  (rootfs export + sidecar containers), and membership in the `docker`
  group is root-equivalent. This blocks a genuinely unprivileged run: a
  dedicated CI-runner user (or any confined caller) is effectively root as
  long as Docker is required. See the paused work in
  `nix/gitea-runner.nix` / the coverage CI gate — it can't be "minimal"
  until Docker is gone.

Removing Docker collapses all three.

## Where Docker is used today (inventory)

1. **Rootfs source** — `docker build` (agent image from a Dockerfile) then
   `docker create` + `docker export | tar` → `mke2fs -d` (see
   `bot_bottle/backend/firecracker/util.py:build_base_rootfs_dir` and
   `docker_image_id`).
2. **Sidecar bundle** — `docker run bot-bottle-sidecars-<slug>` per bottle
   (egress / git-gate / supervise); the VM reaches it at the host TAP IP via
   DNAT (`launch.py`, `resolve_common`).
3. **Digests / cache keys** — `docker image inspect` for the rootfs cache
   key.

## Goals

- Host prerequisites for the Firecracker backend become **firecracker,
  `/dev/kvm`, iproute2, nftables** — no Docker daemon, no `docker` group.
- Launch is **rootless** (open the pre-created TAP pool + KVM; no socket).
- One sidecar per host instead of one per bottle (resource + ops win).

## Non-goals

- Changing the macOS (Apple Container) or legacy Docker backends.
- Removing OCI/Dockerfile support for agent images — users keep writing
  Dockerfiles; only the *host* stops needing a Docker daemon.

## Design (staged)

Ordered so each stage is independently valuable and de-risks the next.

### Stage 1 — Sidecar: persistent + per-host, source-IP-keyed

One sidecar bundle per host, shared by all bottles, with per-bottle policy
keyed on the **source IP** of incoming traffic. This is safe here because
the point-to-point `/31` TAP + the `bot_bottle_fc` nft table make the source
IP of anything from `bbfcN` *provably* that bottle's guest IP (no spoofing,
no cross-bottle traffic) — so the sidecar can attribute a request to a
bottle with confidence a shared bridge could not offer.

Per-service, sharing differs:

- **supervise** — host-level is a clean win (unified approval queue, ~no
  secrets). Do first.
- **egress (mitmproxy)** — shareable via a client-IP addon that selects the
  per-bottle allowlist / DLP / token-injection. Higher blast radius: one
  process now holds every bottle's upstream tokens (see Security).
- **git-gate** — most secret-dense (per-repo deploy keys) and least
  naturally shareable (git carries no source-IP-scoped auth). Keep
  per-bottle unless there's a strong reason.

Needs a small **control plane**: add/remove a bottle's routes/keys/proposals
on launch/teardown with live reload, replacing "config baked at launch,
torn down at exit."

Can ship as a container first (quick resource/ops win) and become a VM in
Stage 4.

### Stage 2 — Fixed images built with nix (no Docker)

The images bot-bottle *ships* — the sidecar, the agent base, and the builder
(Stage 3) — are built declaratively with nix (`nixos-generators` /
`make-ext4-fs` / `pkgs.dockerTools` for the rootfs), producing an ext4 or
tar with correct ownership. Removes Docker for everything we own and gives
the rootless-rootfs correctness (#347) for free on these images.

### Stage 3 — User Dockerfiles built in a builder VM (the unlock)

The variable part — a user's own agent Dockerfile — builds **inside a
throwaway (or persistent) Firecracker builder VM** running `buildah`/`podman`
(rootless, daemonless) or a full in-guest dockerd. The host runs no Docker.
Bonus: an untrusted Dockerfile executes in a disposable VM, which is *more*
isolated than `docker build` on the host.

With this, launch touches no host Docker → the backend is rootless → the
dedicated CI-runner user needs no `docker` group. **This is the stage that
unblocks the minimal-runner / coverage-CI work.**

Open problems to solve here (prototype first):
- **Build cache.** No Docker layer cache; a persistent cache disk on the
  builder VM (or content-addressed rootfs cache) so rebuilds aren't full
  re-`apt`.
- **Build-time egress.** `apt`/pulls need network → through the sidecar,
  which is itself built earlier → nix-built fixed images break the
  chicken-and-egg (nothing needs Docker to come up).

### Stage 4 — Sidecar (and builder) as Firecracker VMs

Full firecracker-native: the sidecar is a VM on its own TAP; agent VM →
sidecar VM is VM-to-VM, so the host forwards `bbfcN` → sidecar TAP and the
nft table grows forward rules (today it *drops* all non-DNAT'd egress). A
**per-host** sidecar VM makes the boot/memory overhead amortized and gives a
stable IP every agent points at.

## Security considerations

- **Secret concentration.** Per-bottle sidecars isolate secrets at the
  process boundary — each holds only its bottle's tokens/keys. A host
  sidecar concentrates *all* bottles' secrets in one long-lived process and
  shifts isolation to **application-level** (correct source-IP keying). A
  single attribution bug leaks bottle A's token into bottle B's request — a
  class of bug that can't exist per-bottle. Mitigation: lean on the
  unspoofable TAP+nft attribution; consider keeping the most secret-dense
  service (git-gate) per-bottle.
- **Build isolation improves.** Running untrusted Dockerfiles in a
  disposable VM is stronger than host `docker build`.
- **Shared fate.** A host sidecar crash/compromise now affects every bottle.

## Open questions

- Build cache design (per-builder-VM disk vs content-addressed host cache).
- VM-to-VM routing + the nft forward rules for a sidecar VM.
- Control-plane shape for dynamic per-bottle sidecar config + live reload.
- Whether egress is worth sharing given the secret-concentration tradeoff,
  or only supervise (+ keep egress/git-gate per-bottle for now).

## Rollout / relation to other work

- Stages 1–2 are high-value and comparatively cheap; **Stage 3 is the
  unlock** for rootless launch and the paused dedicated-runner work; Stage 4
  is the pure finish and the most networking effort.
- Prototype Stage 3 first ("Dockerfile → agent rootfs, inside a Firecracker
  VM, cached, with build-time egress"): if that's ergonomic and fast, the
  rest follows.
- Related: #347 (rootless rootfs ownership — subsumed by nix-built fixed
  images + the in-VM builder), and the deferred coverage CI gate / gitea
  runner (blocked on Stage 3).