bot-bottle/docs/prds/0069-firecracker-native-docker-free.md

# PRD 0069: Firecracker-native, Docker-free backend

- **Status:** Draft (partially superseded)
- **Author:** Claude
- **Created:** 2026-07-12
- **Issue:** #348

> **Superseded in part by [PRD 0070](0070-per-host-orchestrator.md) (#351):**
> the sidecar-consolidation framing here (Stage 1, per-host sidecar; Stage 4,
> sidecar-as-VM) is taken over by 0070's per-host orchestrator. This PRD still
> owns the docker-free **image-provisioning** work — Stage 2 (pull the fixed
> images from an OCI registry instead of building them with host Docker, a
> dependency of 0070) and Stage 3 (in-VM Dockerfile builder).

## Summary

Make the Firecracker backend depend on **firecracker + KVM only**, removing
Docker from the host. Two moves get us there: run the **sidecar bundle as a
persistent, per-host service** (eventually a Firecracker VM) instead of a
per-bottle container, and **provision rootfs images without a host Docker
daemon** — pull the fixed images (orchestrator/gateway/infra) from an OCI
registry and unpack them daemonlessly, and build user Dockerfiles in an in-VM
builder. The images are still *built* with Docker, but off the launch host
(CI / a publish step) and pushed to the registry; the launch host only pulls.

## Motivation

Two operator-facing costs and one security constraint:

- **Resource cost.** Every bottle spins up its own sidecar bundle (egress
  mitmproxy + git-gate + supervise). N bottles → N heavy bundles.
- **Operational simplicity.** Per-launch `docker run` churn, a Docker daemon
  to keep healthy, and Docker's iptables to coexist with.
- **Privilege / minimal-runner.** The backend needs the Docker socket
  (rootfs export + sidecar containers), and membership in the `docker`
  group is root-equivalent. This blocks a genuinely unprivileged run: a
  dedicated CI-runner user (or any confined caller) is effectively root as
  long as Docker is required. See the paused work in
  `nix/gitea-runner.nix` / the coverage CI gate — it can't be "minimal"
  until Docker is gone.

Removing Docker collapses all three.

## Where Docker is used today (inventory)

1. **Rootfs source** — `docker build` (agent image from a Dockerfile) then
   `docker create` + `docker export | tar` → `mke2fs -d` (see
   `bot_bottle/backend/firecracker/util.py:build_base_rootfs_dir` and
   `docker_image_id`).
2. **Sidecar bundle** — `docker run bot-bottle-sidecars-<slug>` per bottle
   (egress / git-gate / supervise); the VM reaches it at the host TAP IP via
   DNAT (`launch.py`, `resolve_common`).
3. **Digests / cache keys** — `docker image inspect` for the rootfs cache
   key.

## Goals

- Host prerequisites for the Firecracker backend become **firecracker,
  `/dev/kvm`, iproute2, nftables** — no Docker daemon, no `docker` group.
- Launch is **rootless** (open the pre-created TAP pool + KVM; no socket).
- One sidecar per host instead of one per bottle (resource + ops win).

## Non-goals

- Changing the macOS (Apple Container) or legacy Docker backends.
- Removing OCI/Dockerfile support for agent images — users keep writing
  Dockerfiles; only the *host* stops needing a Docker daemon.

## Design (staged)

Ordered so each stage is independently valuable and de-risks the next.

### Stage 1 — Sidecar: persistent + per-host, source-IP-keyed

One sidecar bundle per host, shared by all bottles, with per-bottle policy
keyed on the **source IP** of incoming traffic. This is safe here because
the point-to-point `/31` TAP + the `bot_bottle_fc` nft table make the source
IP of anything from `bbfcN` *provably* that bottle's guest IP (no spoofing,
no cross-bottle traffic) — so the sidecar can attribute a request to a
bottle with confidence a shared bridge could not offer.

Per-service, sharing differs:

- **supervise** — host-level is a clean win (unified approval queue, ~no
  secrets). Do first.
- **egress (mitmproxy)** — shareable via a client-IP addon that selects the
  per-bottle allowlist / DLP / token-injection. Higher blast radius: one
  process now holds every bottle's upstream tokens (see Security).
- **git-gate** — most secret-dense (per-repo deploy keys) and least
  naturally shareable (git carries no source-IP-scoped auth). Keep
  per-bottle unless there's a strong reason.

Needs a small **control plane**: add/remove a bottle's routes/keys/proposals
on launch/teardown with live reload, replacing "config baked at launch,
torn down at exit."

Can ship as a container first (quick resource/ops win) and become a VM in
Stage 4.

### Stage 2 — Fixed rootfs prebuilt + pulled as an artifact (no host Docker)

The one fixed image the Firecracker backend needs at launch — the combined
**infra** rootfs the infra VM boots (orchestrator control plane + gateway +
buildah, with the control-plane init as PID 1) — is **prebuilt end-to-end off
the launch host and published as a versioned, ready-to-boot ext4 artifact**.
The launch host **downloads the `.ext4` and boots it directly** — no
`docker build`, no `docker export`, no `mke2fs`, no image tooling at all.

This is possible because the infra rootfs is already **host- and
bottle-agnostic**: the per-boot bits (authorized_keys, guest IP) arrive on the
**kernel cmdline**, not in the rootfs (see `build_base_rootfs_dir`). So one
published ext4 boots on any launch host.

- **Artifact.** `rootfs.ext4` + a `rootfs.ext4.sha256`, published as a Gitea
  **generic package** (`bot-bottle-infra/<tag>`) — generic packages take
  arbitrary large binaries (no attachment size cap / file-type allowlist that
  release attachments impose). The matching `vmlinux` kernel can ship the same
  way, so the whole VM is fetchable.
- **Pull.** The launch host `GET`s
  `…/api/packages/<owner>/generic/bot-bottle-infra/<tag>/rootfs.ext4` (+
  `.sha256`) for its pinned tag, verifies the checksum, caches it under the
  tag, and attaches it as the infra VM's root disk. Host prerequisite is an
  HTTP client — nothing else. Public packages need no auth to pull; a token
  with `read:package` covers a private instance.
- **Registry.** The artifact base URL + owner are configurable, defaulting to
  this deployment's Gitea (`https://gitea.dideric.is` / `didericis`);
  overridable via env for other deployments / air-gapped mirrors.
- **Versioning.** A pinned tag bumped when the infra rootfs contents change
  (bot_bottle's shipped files, the base deps, or the init), so a launch host
  pulls the artifact matching its code and a content change can't silently
  boot a stale rootfs. A checksum mismatch fails closed.
- **Publish.** A `publish` step (CLI subcommand / CI job) runs the full
  pipeline **on a build/CI host** — `docker build` the three Dockerfiles →
  export → inject guest boot → `mke2fs` → upload the `.ext4` + `.sha256`.
  Building still uses Docker, but never on the launch/runner host, which is
  the one #348 needs unprivileged.
- **Dev escape hatch.** An explicit opt-in still builds the rootfs locally
  with Docker (for iterating on the Dockerfiles without a publish
  round-trip); it is never the default path.

Removes Docker from the launch host entirely for the fixed image, and the
launch host needs no OCI/rootfs tooling — just fetch + boot. The build-time
cache / build-time-egress open problems a from-scratch build would face don't
arise: the launch host never builds, it downloads a finished disk.

### Stage 3 — User Dockerfiles built in a builder VM (the unlock)

The variable part — a user's own agent Dockerfile — builds **inside a
throwaway (or persistent) Firecracker builder VM** running `buildah`/`podman`
(rootless, daemonless) or a full in-guest dockerd. The host runs no Docker.
Bonus: an untrusted Dockerfile executes in a disposable VM, which is *more*
isolated than `docker build` on the host.

With this, launch touches no host Docker → the backend is rootless → the
dedicated CI-runner user needs no `docker` group. **This is the stage that
unblocks the minimal-runner / coverage-CI work.**

Open problems to solve here (prototype first):
- **Build cache.** No Docker layer cache; a persistent cache disk on the
  builder VM (or content-addressed rootfs cache) so rebuilds aren't full
  re-`apt`.
- **Build-time egress.** `apt`/pulls need network → through the sidecar,
  which is itself built earlier → nix-built fixed images break the
  chicken-and-egg (nothing needs Docker to come up).

### Stage 4 — Sidecar (and builder) as Firecracker VMs

Full firecracker-native: the sidecar is a VM on its own TAP; agent VM →
sidecar VM is VM-to-VM, so the host forwards `bbfcN` → sidecar TAP and the
nft table grows forward rules (today it *drops* all non-DNAT'd egress). A
**per-host** sidecar VM makes the boot/memory overhead amortized and gives a
stable IP every agent points at.

## Security considerations

- **Secret concentration.** Per-bottle sidecars isolate secrets at the
  process boundary — each holds only its bottle's tokens/keys. A host
  sidecar concentrates *all* bottles' secrets in one long-lived process and
  shifts isolation to **application-level** (correct source-IP keying). A
  single attribution bug leaks bottle A's token into bottle B's request — a
  class of bug that can't exist per-bottle. Mitigation: lean on the
  unspoofable TAP+nft attribution; consider keeping the most secret-dense
  service (git-gate) per-bottle.
- **Build isolation improves.** Running untrusted Dockerfiles in a
  disposable VM is stronger than host `docker build`.
- **Shared fate.** A host sidecar crash/compromise now affects every bottle.

## Open questions

- Build cache design (per-builder-VM disk vs content-addressed host cache).
- VM-to-VM routing + the nft forward rules for a sidecar VM.
- Control-plane shape for dynamic per-bottle sidecar config + live reload.
- Whether egress is worth sharing given the secret-concentration tradeoff,
  or only supervise (+ keep egress/git-gate per-bottle for now).

## Rollout / relation to other work

- Stages 1–2 are high-value and comparatively cheap; **Stage 3 is the
  unlock** for rootless launch and the paused dedicated-runner work; Stage 4
  is the pure finish and the most networking effort.
- Prototype Stage 3 first ("Dockerfile → agent rootfs, inside a Firecracker
  VM, cached, with build-time egress"): if that's ergonomic and fast, the
  rest follows.
- Related: #347 (rootless rootfs ownership — subsumed by nix-built fixed
  images + the in-VM builder), and the deferred coverage CI gate / gitea
  runner (blocked on Stage 3).