diff --git a/docs/prds/0069-firecracker-native-docker-free.md b/docs/prds/0069-firecracker-native-docker-free.md index 0a24167..c771c6c 100644 --- a/docs/prds/0069-firecracker-native-docker-free.md +++ b/docs/prds/0069-firecracker-native-docker-free.md @@ -1,10 +1,16 @@ # PRD 0069: Firecracker-native, Docker-free backend -- **Status:** Draft +- **Status:** Draft (partially superseded) - **Author:** Claude - **Created:** 2026-07-12 - **Issue:** #348 +> **Superseded in part by [PRD 0070](0070-per-host-orchestrator.md) (#351):** +> the sidecar-consolidation framing here (Stage 1, per-host sidecar; Stage 4, +> sidecar-as-VM) is taken over by 0070's per-host orchestrator. This PRD still +> owns the docker-free **image-building** work — Stage 2 (nix-built fixed +> images, a dependency of 0070) and Stage 3 (in-VM Dockerfile builder). + ## Summary Make the Firecracker backend depend on **firecracker + KVM only**, removing diff --git a/docs/prds/0070-per-host-orchestrator.md b/docs/prds/0070-per-host-orchestrator.md new file mode 100644 index 0000000..e10fdf1 --- /dev/null +++ b/docs/prds/0070-per-host-orchestrator.md @@ -0,0 +1,266 @@ +# PRD 0070: Per-host orchestrator service + +- **Status:** Draft +- **Author:** Claude +- **Created:** 2026-07-12 +- **Issue:** #351 +- **Supersedes:** the Stage-1 / Stage-4 sidecar-consolidation framing of + PRD 0069 (#348). Depends on 0069's nix-built fixed images (Stage 2) for + bootstrapping; 0069 still owns the docker-free image-building work. + +## Summary + +Replace the **per-bottle sidecar bundle** with a single **persistent, +per-host orchestrator**: one long-lived service that runs the sidecar +functions (egress / git-gate / supervise), coordinates with the console, +and brokers agent launches and teardown. It is **virtualized from the +start** using each backend's native isolation primitive — a Firecracker +microVM on the Firecracker backend, an Apple container on macOS, a Docker +container on the legacy backend — and is fronted by a single +**backend-agnostic contract**. Per-backend variation lives on +`BottleBackend`, not in the orchestrator. + +## Motivation + +Today each bottle spins up its own sidecar bundle (egress mitmproxy + +git-gate + supervise). That costs: + +- **Resources.** N bottles → N heavy bundles booting and idling. +- **Operational churn.** Per-launch container/VM lifecycle for the + sidecars, a control path baked at launch and torn down at exit. +- **A blurry contract.** "How a bottle talks to its sidecar" is + re-implemented per backend instead of being one agreed interface. + +A per-host orchestrator collapses the first two and forces the third to +be made explicit. It's also the component that will own per-host runtime +**state** (slot leases, the approval queue, the bottle registry) — today +that's ad-hoc `fcntl`-locked files. + +## Security review (read this first) + +Consolidation is a real change to the trust model. The goal is to **not +significantly weaken** the posture; some properties strengthen, some +weaken, and the weakened ones must be mitigated by design, not hand-waved. + +### What gets stronger + +- **Build/host isolation of untrusted inputs** (with 0069 Stage 3): user + Dockerfiles build in a disposable VM instead of on the host. +- **One audited privileged surface.** Today the launcher runs as the full + host user and needs the Docker socket (root-equivalent). The orchestrator + model replaces that with a **thin launch broker** (below) — a small, + structured, auditable privileged core instead of a fat socket. +- **Attribution is enforced, not assumed.** Making source-IP identity a + first-class contract invariant (below) means each backend must *prove* + it, rather than the sidecar implicitly trusting network position. + +### What gets weaker, and the mitigation + +1. **Secret concentration.** Per-bottle sidecars isolate secrets at the + process boundary — each holds only its bottle's tokens/keys. A host + orchestrator concentrates **every bottle's** egress tokens, git deploy + keys, and the console credential in one long-lived process. A single + attribution bug leaks bottle A's token into bottle B's request — a class + of bug that *cannot exist* per-bottle. + - *Mitigation:* lean on the enforced source-IP invariant for + attribution; keep the most secret-dense, least-shareable service + (**git-gate**, per-repo deploy keys, no natural source-IP scoping) + **per-bottle** unless there's a compelling reason; scope each secret + to the bottle in the state DB so a lookup can't return the wrong + bottle's secret by construction (key every secret access by the + verified source identity, never by ambient state). + +2. **Shared fate.** Orchestrator down = no new launches, and running + agents lose egress / git / supervise. Compromise = the whole host's + fleet, plus launch authority, plus the console token. + - *Mitigation:* the orchestrator is itself confined (its own VM/container + with its own fail-closed egress); make it **restartable without killing + running agent VMs** (agents keep running; they briefly lose sidecar + connectivity until it's back); persist state to a host volume so a + restart re-adopts live bottles rather than losing them. + +3. **The launch broker is the new privileged core.** We don't eliminate + host privilege — we shrink and relocate it. If the broker accepts + arbitrary paths/commands, the orchestrator VM can escape through it. + - *Mitigation:* the broker takes **structured requests only** — "launch + bottle from *this* content-addressed, nix-built rootfs on TAP slot + *k*", never "run this argv". It validates against a fixed image set, + not caller-supplied paths. It is small enough to audit line-by-line. + +4. **The egress proxy now parses every bottle's traffic in one process.** + Higher blast radius for a mitmproxy/TLS-bump bug. + - *Mitigation:* this is the argument for virtualizing the orchestrator + from the start (Stage B, not a host daemon) — the code that TLS-bumps + and parses agent traffic and holds every token runs **inside its own + confined VM**, not as a host process. If egress sharing's blast radius + feels too high, egress can stay per-bottle while supervise (near-zero + secrets) goes host-level first. + +### The attribution invariant + +Source-IP attribution is what makes a shared orchestrator safe: one +process serves every bottle and tells them apart by source address. The +*mechanism* is identical everywhere (read source IP → look up bottle); the +**guarantee that the address can't be forged is a per-backend +responsibility** and part of the contract: + +> **Invariant:** a packet's source address, as seen by the orchestrator, +> *provably* identifies the originating bottle. + +- **Firecracker** — enforced by the `/31` point-to-point TAP + the + `bot_bottle_fc` nft table (strongest; already built). +- **Docker** — the per-bottle `--internal` network + anti-spoof; weaker, + must be made explicit. +- **Apple** — the host-only network. + +If a backend can't honor the invariant, source-IP consolidation is not +safe there and that backend keeps per-bottle sidecars. The invariant is a +hard precondition, not an aspiration. + +## Design + +### The contract (backend-agnostic) + +Three surfaces; only one is per-backend. + +1. **Control plane (CLI / console → orchestrator)** — an RPC: + `launch_bottle`, `teardown_bottle`, `register_policy`, + `deregister_bottle`, `supervise_queue`. Fully backend-agnostic. Both the + local `cli.py` and the remote console funnel through it, so policy is + uniform and `cli.py` becomes a thin client rather than a parallel + launcher. +2. **Data plane (agent → orchestrator)** — the egress / git / supervise + endpoints. Already agnostic today (agents dial `http://sidecar:9099`); + only the *address* and *how packets get there* are per-backend. +3. **Launch / wire (orchestrator → backend)** — the irreducibly + backend-specific part; lives on `BottleBackend`. + +### One `Orchestrator`, no subclass tree + +The orchestrator is a **single concrete class** holding all the +backend-neutral logic — egress addon, git-gate, supervise, source-IP +attribution, live-reload control plane, console client. It never branches +on backend; it *composes* a `BottleBackend`. That composition is what makes +the contract agnostic: there is nothing backend-specific left in the +orchestrator to leak. + +Rejected alternative: an `Orchestrator` ABC with per-backend +implementations. The interesting logic (proxies, attribution, control +plane) is backend-neutral, so three subclasses would triplicate the hard +part; and a second hierarchy paralleling `BottleBackend` reintroduces the +same hand-maintained lockstep coupling we just removed from the netpool +constants (PR #350). Composition over a parallel tree. + +### `BottleBackend` absorbs the per-backend variation + +A small, cohesive surface — reused for launching agent bottles *and* the +orchestrator's own unit (the orchestrator is just another native unit): + +``` +launch_unit(spec) -> Handle # agent bottle OR the orchestrator itself + # (fc microVM / apple ctr / docker ctr) +wire(unit, endpoint) -> None # DNAT+forward (fc) | attach shared net (docker/apple) +endpoint_of(unit) -> Endpoint # address resolution +health(unit) -> Status +``` + +Plus the **launch broker** — the answer to "a VM/container can't spawn its +own host-network siblings." The orchestrator can't directly open host +`/dev/kvm` + a host TAP fd (Firecracker), and a container can't spawn +siblings without a root-equivalent socket (Docker). So every backend +exposes a broker the orchestrator calls to launch an agent: + +- **Firecracker** — a thin, structured host shim (see security #3). This + replaces today's implicit "launcher runs as host user." +- **Docker** — the socket today (fat, root-equivalent — the thing 0069's + Stage 3 removes); a narrower broker later. +- **Apple** — the `container` CLI/daemon. + +If `BottleBackend` bloats, the pressure valve is composition one level +down: vend a `backend.network()` / `Wiring` collaborator rather than +piling methods on — the same discipline, recursed. + +### State: one SQLite DB, owned by the orchestrator + +The orchestrator is the natural owner of per-host **runtime state**: + +- pool **slot leases** (which bottle holds slot *i*) — replaces today's + `fcntl`-locked files with WAL-mode transactions; +- the **supervise approval queue** + remembered approvals; +- the **live bottle registry** (source IP → bottle → policy/secrets refs), + the lookup table the attribution invariant reads. + +This is deliberately **not** a "single source of truth for all config." +Config splits into three tiers with different homes: + +| Tier | Example | Home | +|---|---|---| +| Build-time constants | pool size, IP base, nft table | flat `.env` (PR #350) — must be readable by Nix eval + root bash, zero runtime | +| User-authored config | bottle manifests, egress routes, secret refs | declarative files under `~/.bot-bottle/` — trust boundary at `$HOME`, git-trackable, "unknown keys die at load" | +| Runtime state | slot leases, approvals, registry | **SQLite**, owned by the orchestrator | + +SQLite is right for the runtime tier (mutable, concurrent, queried) and +wrong for the other two (Nix can't read it at eval time; it fights the +declarative manifest trust model). Keep the tiers separate. + +## Sequencing + +Jump straight to the **virtualized** end state (not a host-daemon stepping +stone): a host daemon's agent→`localhost` transport is throwaway once the +orchestrator becomes a VM. Decouple the two risks instead: + +- **Consolidation risk** (one process, all secrets, attribution, reload) + and **packaging/transport risk** (VM-to-VM wiring, the shim) are + independent. Develop the orchestrator **service as a plain process + dev-harness** first, so the consolidation logic (attribution, reload, + secret handling) is proven with fast iteration — *then* wrap that exact + service in the VM and solve wiring separately. + +Backend order (cheapest proof → hardest → last): + +1. **Docker orchestrator** — nearly free (the sidecar bundle is already + containers; collapse N bundles into one persistent container). Proves + consolidation + the `BottleBackend` seam with the least moving parts. +2. **Firecracker orchestrator** — the real work: the shim + VM-to-VM + routing (host forwards `bbfcN` → orchestrator TAP; the nft table grows + forward rules where today it drops all non-DNAT egress). Built against + the dev-harness so the app logic is already proven. +3. **macOS (Apple container)** — last (container-to-container networking). + +Keep the sidecar **service one shared thing** throughout. + +## Non-goals + +- Removing OCI/Dockerfile support for agent images (0069's concern). +- A single database for *all* config (see the three-tier table). +- Changing the per-bottle isolation of agent workloads — only the sidecar + is consolidated; agents stay one-VM/container-each. + +## Relationship to other work + +- **PRD 0069 (#348):** 0070 subsumes its Stage 1 (per-host sidecar) and + Stage 4 (sidecar-as-VM). 0069 retains Stage 2 (nix-built fixed images — + a **dependency** here: the orchestrator and agent base must be + nix-built so the broker launches from a fixed image set and bootstrapping + has no chicken-and-egg) and Stage 3 (in-VM Dockerfile builder). +- **Minimal CI runner (paused):** the Firecracker broker + no host Docker + is what lets a dedicated `gitea` runner user drop the root-equivalent + `docker` group — it only needs broker-socket access + `kvm`/pool group + membership. This work unblocks it. +- **PR #350 (netpool single-source):** the same "one source per fact, + composition over parallel hierarchies" discipline the contract follows. + +## Open questions + +- **Egress sharing tradeoff:** is the secret-concentration blast radius of + one shared mitmproxy worth the resource win, or share only supervise + (near-zero secrets) and keep egress + git-gate per-bottle initially? +- **Control-plane shape:** RPC transport (unix socket / vsock / HTTP over + the TAP) and the live-reload protocol for per-bottle policy. +- **State re-adoption:** exact scheme for an orchestrator restart to + re-adopt running agent VMs from the SQLite registry without racing + in-flight launches. +- **VM-to-VM routing:** the nft forward rules + addressing for a + per-host orchestrator VM on its own TAP. +- **Broker request schema:** the exact structured contract that stays + auditable and can't be coerced into launching arbitrary payloads.