bot-bottle/docs/prds/0069-firecracker-native-docker-free.md

PRD 0069: Firecracker-native, Docker-free backend

• Status: Draft (partially superseded)
• Author: Claude
• Created: 2026-07-12
• Issue: #348

Superseded in part by PRD 0070 (#351): the sidecar-consolidation framing here (Stage 1, per-host sidecar; Stage 4, sidecar-as-VM) is taken over by 0070's per-host orchestrator. This PRD still owns the docker-free image-provisioning work — Stage 2 (pull the fixed images from an OCI registry instead of building them with host Docker, a dependency of 0070) and Stage 3 (in-VM Dockerfile builder).

Summary

Make the Firecracker backend depend on firecracker + KVM only, removing Docker from the host. Two moves get us there: run the sidecar bundle as a persistent, per-host service (eventually a Firecracker VM) instead of a per-bottle container, and provision rootfs images without a host Docker daemon — pull the fixed images (orchestrator/gateway/infra) from an OCI registry and unpack them daemonlessly, and build user Dockerfiles in an in-VM builder. The images are still built with Docker, but off the launch host (CI / a publish step) and pushed to the registry; the launch host only pulls.

Motivation

Two operator-facing costs and one security constraint:

• Resource cost. Every bottle spins up its own sidecar bundle (egress mitmproxy + git-gate + supervise). N bottles → N heavy bundles.
• Operational simplicity. Per-launch docker run churn, a Docker daemon to keep healthy, and Docker's iptables to coexist with.
• Privilege / minimal-runner. The backend needs the Docker socket (rootfs export + sidecar containers), and membership in the docker group is root-equivalent. This blocks a genuinely unprivileged run: a dedicated CI-runner user (or any confined caller) is effectively root as long as Docker is required. See the paused work in nix/gitea-runner.nix / the coverage CI gate — it can't be "minimal" until Docker is gone.

Removing Docker collapses all three.

Where Docker is used today (inventory)

1. Rootfs source — docker build (agent image from a Dockerfile) then docker create + docker export | tar → mke2fs -d (see bot_bottle/backend/firecracker/util.py:build_base_rootfs_dir and docker_image_id).
2. Sidecar bundle — docker run bot-bottle-sidecars-<slug> per bottle (egress / git-gate / supervise); the VM reaches it at the host TAP IP via DNAT (launch.py, resolve_common).
3. Digests / cache keys — docker image inspect for the rootfs cache key.

Goals

• Host prerequisites for the Firecracker backend become firecracker, /dev/kvm, iproute2, nftables — no Docker daemon, no docker group.
• Launch is rootless (open the pre-created TAP pool + KVM; no socket).
• One sidecar per host instead of one per bottle (resource + ops win).

Non-goals

• Changing the macOS (Apple Container) or legacy Docker backends.
• Removing OCI/Dockerfile support for agent images — users keep writing Dockerfiles; only the host stops needing a Docker daemon.

Design (staged)

Ordered so each stage is independently valuable and de-risks the next.

Stage 1 — Sidecar: persistent + per-host, source-IP-keyed

One sidecar bundle per host, shared by all bottles, with per-bottle policy keyed on the source IP of incoming traffic. This is safe here because the point-to-point /31 TAP + the bot_bottle_fc nft table make the source IP of anything from bbfcN provably that bottle's guest IP (no spoofing, no cross-bottle traffic) — so the sidecar can attribute a request to a bottle with confidence a shared bridge could not offer.

Per-service, sharing differs:

• supervise — host-level is a clean win (unified approval queue, ~no secrets). Do first.
• egress (mitmproxy) — shareable via a client-IP addon that selects the per-bottle allowlist / DLP / token-injection. Higher blast radius: one process now holds every bottle's upstream tokens (see Security).
• git-gate — most secret-dense (per-repo deploy keys) and least naturally shareable (git carries no source-IP-scoped auth). Keep per-bottle unless there's a strong reason.

Needs a small control plane: add/remove a bottle's routes/keys/proposals on launch/teardown with live reload, replacing "config baked at launch, torn down at exit."

Can ship as a container first (quick resource/ops win) and become a VM in Stage 4.

Stage 2 — Fixed rootfs prebuilt + pulled as an artifact (no host Docker)

The one fixed image the Firecracker backend needs at launch — the combined infra rootfs the infra VM boots (orchestrator control plane + gateway + buildah, with the control-plane init as PID 1) — is prebuilt end-to-end off the launch host and published as a versioned, ready-to-boot ext4 artifact. The launch host downloads the .ext4 and boots it directly — no docker build, no docker export, no mke2fs, no image tooling at all.

This is possible because the infra rootfs is already host- and bottle-agnostic: the per-boot bits (authorized_keys, guest IP) arrive on the kernel cmdline, not in the rootfs (see build_base_rootfs_dir). So one published ext4 boots on any launch host.

• Artifact. rootfs.ext4 + a rootfs.ext4.sha256, published as a Gitea generic package (bot-bottle-firecracker-infra/<tag>) — generic packages take arbitrary large binaries (no attachment size cap / file-type allowlist that release attachments impose). The matching vmlinux kernel can ship the same way, so the whole VM is fetchable.
• Pull. The launch host GETs …/api/packages/<owner>/generic/bot-bottle-firecracker-infra/<tag>/rootfs.ext4 (+ .sha256) for its pinned tag, verifies the checksum, caches it under the tag, and attaches it as the infra VM's root disk. Host prerequisite is an HTTP client — nothing else. Public packages need no auth to pull; a token with read:package covers a private instance.
• Registry. The artifact base URL + owner are configurable, defaulting to this deployment's Gitea (https://gitea.dideric.is / didericis); overridable via env for other deployments / air-gapped mirrors.
• Versioning. A pinned tag bumped when the infra rootfs contents change (bot_bottle's shipped files, the base deps, or the init), so a launch host pulls the artifact matching its code and a content change can't silently boot a stale rootfs. A checksum mismatch fails closed.
• Publish. A publish step (CLI subcommand / CI job) runs the full pipeline on a build/CI host — docker build the three Dockerfiles → export → inject guest boot → mke2fs → upload the .ext4 + .sha256. Building still uses Docker, but never on the launch/runner host, which is the one #348 needs unprivileged.
• Dev escape hatch. An explicit opt-in still builds the rootfs locally with Docker (for iterating on the Dockerfiles without a publish round-trip); it is never the default path.

Removes Docker from the launch host entirely for the fixed image, and the launch host needs no OCI/rootfs tooling — just fetch + boot. The build-time cache / build-time-egress open problems a from-scratch build would face don't arise: the launch host never builds, it downloads a finished disk.

Stage 3 — User Dockerfiles built in a builder VM (the unlock)

The variable part — a user's own agent Dockerfile — builds inside a throwaway (or persistent) Firecracker builder VM running buildah/podman (rootless, daemonless) or a full in-guest dockerd. The host runs no Docker. Bonus: an untrusted Dockerfile executes in a disposable VM, which is more isolated than docker build on the host.

With this, launch touches no host Docker → the backend is rootless → the dedicated CI-runner user needs no docker group. This is the stage that unblocks the minimal-runner / coverage-CI work.

Open problems to solve here (prototype first):

• Build cache. No Docker layer cache; a persistent cache disk on the builder VM (or content-addressed rootfs cache) so rebuilds aren't full re-apt.
• Build-time egress. apt/pulls need network → through the sidecar, which is itself built earlier → nix-built fixed images break the chicken-and-egg (nothing needs Docker to come up).

Stage 4 — Sidecar (and builder) as Firecracker VMs

Full firecracker-native: the sidecar is a VM on its own TAP; agent VM → sidecar VM is VM-to-VM, so the host forwards bbfcN → sidecar TAP and the nft table grows forward rules (today it drops all non-DNAT'd egress). A per-host sidecar VM makes the boot/memory overhead amortized and gives a stable IP every agent points at.

Security considerations

• Secret concentration. Per-bottle sidecars isolate secrets at the process boundary — each holds only its bottle's tokens/keys. A host sidecar concentrates all bottles' secrets in one long-lived process and shifts isolation to application-level (correct source-IP keying). A single attribution bug leaks bottle A's token into bottle B's request — a class of bug that can't exist per-bottle. Mitigation: lean on the unspoofable TAP+nft attribution; consider keeping the most secret-dense service (git-gate) per-bottle.
• Build isolation improves. Running untrusted Dockerfiles in a disposable VM is stronger than host docker build.
• Shared fate. A host sidecar crash/compromise now affects every bottle.

Open questions

• Build cache design (per-builder-VM disk vs content-addressed host cache).
• VM-to-VM routing + the nft forward rules for a sidecar VM.
• Control-plane shape for dynamic per-bottle sidecar config + live reload.
• Whether egress is worth sharing given the secret-concentration tradeoff, or only supervise (+ keep egress/git-gate per-bottle for now).

Rollout / relation to other work

• Stages 1–2 are high-value and comparatively cheap; Stage 3 is the unlock for rootless launch and the paused dedicated-runner work; Stage 4 is the pure finish and the most networking effort.
• Prototype Stage 3 first ("Dockerfile → agent rootfs, inside a Firecracker VM, cached, with build-time egress"): if that's ergonomic and fast, the rest follows.
• Related: #347 (rootless rootfs ownership — subsumed by nix-built fixed images + the in-VM builder), and the deferred coverage CI gate / gitea runner (blocked on Stage 3).