docs/prds/0069-firecracker-native-docker-free.md

@@ -1,10 +1,16 @@
 # PRD 0069: Firecracker-native, Docker-free backend
 
-- **Status:** Draft
+- **Status:** Draft (partially superseded)
 - **Author:** Claude
 - **Created:** 2026-07-12
 - **Issue:** #348
 
+> **Superseded in part by [PRD 0070](0070-per-host-orchestrator.md) (#351):**
+> the sidecar-consolidation framing here (Stage 1, per-host sidecar; Stage 4,
+> sidecar-as-VM) is taken over by 0070's per-host orchestrator. This PRD still
+> owns the docker-free **image-building** work — Stage 2 (nix-built fixed
+> images, a dependency of 0070) and Stage 3 (in-VM Dockerfile builder).
+
 ## Summary
 
 Make the Firecracker backend depend on **firecracker + KVM only**, removing

docs/prds/0070-per-host-orchestrator.md

@@ -0,0 +1,266 @@
+# PRD 0070: Per-host orchestrator service
+
+- **Status:** Draft
+- **Author:** Claude
+- **Created:** 2026-07-12
+- **Issue:** #351
+- **Supersedes:** the Stage-1 / Stage-4 sidecar-consolidation framing of
+  PRD 0069 (#348). Depends on 0069's nix-built fixed images (Stage 2) for
+  bootstrapping; 0069 still owns the docker-free image-building work.
+
+## Summary
+
+Replace the **per-bottle sidecar bundle** with a single **persistent,
+per-host orchestrator**: one long-lived service that runs the sidecar
+functions (egress / git-gate / supervise), coordinates with the console,
+and brokers agent launches and teardown. It is **virtualized from the
+start** using each backend's native isolation primitive — a Firecracker
+microVM on the Firecracker backend, an Apple container on macOS, a Docker
+container on the legacy backend — and is fronted by a single
+**backend-agnostic contract**. Per-backend variation lives on
+`BottleBackend`, not in the orchestrator.
+
+## Motivation
+
+Today each bottle spins up its own sidecar bundle (egress mitmproxy +
+git-gate + supervise). That costs:
+
+- **Resources.** N bottles → N heavy bundles booting and idling.
+- **Operational churn.** Per-launch container/VM lifecycle for the
+  sidecars, a control path baked at launch and torn down at exit.
+- **A blurry contract.** "How a bottle talks to its sidecar" is
+  re-implemented per backend instead of being one agreed interface.
+
+A per-host orchestrator collapses the first two and forces the third to
+be made explicit. It's also the component that will own per-host runtime
+**state** (slot leases, the approval queue, the bottle registry) — today
+that's ad-hoc `fcntl`-locked files.
+
+## Security review (read this first)
+
+Consolidation is a real change to the trust model. The goal is to **not
+significantly weaken** the posture; some properties strengthen, some
+weaken, and the weakened ones must be mitigated by design, not hand-waved.
+
+### What gets stronger
+
+- **Build/host isolation of untrusted inputs** (with 0069 Stage 3): user
+  Dockerfiles build in a disposable VM instead of on the host.
+- **One audited privileged surface.** Today the launcher runs as the full
+  host user and needs the Docker socket (root-equivalent). The orchestrator
+  model replaces that with a **thin launch broker** (below) — a small,
+  structured, auditable privileged core instead of a fat socket.
+- **Attribution is enforced, not assumed.** Making source-IP identity a
+  first-class contract invariant (below) means each backend must *prove*
+  it, rather than the sidecar implicitly trusting network position.
+
+### What gets weaker, and the mitigation
+
+1. **Secret concentration.** Per-bottle sidecars isolate secrets at the
+   process boundary — each holds only its bottle's tokens/keys. A host
+   orchestrator concentrates **every bottle's** egress tokens, git deploy
+   keys, and the console credential in one long-lived process. A single
+   attribution bug leaks bottle A's token into bottle B's request — a class
+   of bug that *cannot exist* per-bottle.
+   - *Mitigation:* lean on the enforced source-IP invariant for
+     attribution; keep the most secret-dense, least-shareable service
+     (**git-gate**, per-repo deploy keys, no natural source-IP scoping)
+     **per-bottle** unless there's a compelling reason; scope each secret
+     to the bottle in the state DB so a lookup can't return the wrong
+     bottle's secret by construction (key every secret access by the
+     verified source identity, never by ambient state).
+
+2. **Shared fate.** Orchestrator down = no new launches, and running
+   agents lose egress / git / supervise. Compromise = the whole host's
+   fleet, plus launch authority, plus the console token.
+   - *Mitigation:* the orchestrator is itself confined (its own VM/container
+     with its own fail-closed egress); make it **restartable without killing
+     running agent VMs** (agents keep running; they briefly lose sidecar
+     connectivity until it's back); persist state to a host volume so a
+     restart re-adopts live bottles rather than losing them.
+
+3. **The launch broker is the new privileged core.** We don't eliminate
+   host privilege — we shrink and relocate it. If the broker accepts
+   arbitrary paths/commands, the orchestrator VM can escape through it.
+   - *Mitigation:* the broker takes **structured requests only** — "launch
+     bottle from *this* content-addressed, nix-built rootfs on TAP slot
+     *k*", never "run this argv". It validates against a fixed image set,
+     not caller-supplied paths. It is small enough to audit line-by-line.
+
+4. **The egress proxy now parses every bottle's traffic in one process.**
+   Higher blast radius for a mitmproxy/TLS-bump bug.
+   - *Mitigation:* this is the argument for virtualizing the orchestrator
+     from the start (Stage B, not a host daemon) — the code that TLS-bumps
+     and parses agent traffic and holds every token runs **inside its own
+     confined VM**, not as a host process. If egress sharing's blast radius
+     feels too high, egress can stay per-bottle while supervise (near-zero
+     secrets) goes host-level first.
+
+### The attribution invariant
+
+Source-IP attribution is what makes a shared orchestrator safe: one
+process serves every bottle and tells them apart by source address. The
+*mechanism* is identical everywhere (read source IP → look up bottle); the
+**guarantee that the address can't be forged is a per-backend
+responsibility** and part of the contract:
+
+> **Invariant:** a packet's source address, as seen by the orchestrator,
+> *provably* identifies the originating bottle.
+
+- **Firecracker** — enforced by the `/31` point-to-point TAP + the
+  `bot_bottle_fc` nft table (strongest; already built).
+- **Docker** — the per-bottle `--internal` network + anti-spoof; weaker,
+  must be made explicit.
+- **Apple** — the host-only network.
+
+If a backend can't honor the invariant, source-IP consolidation is not
+safe there and that backend keeps per-bottle sidecars. The invariant is a
+hard precondition, not an aspiration.
+
+## Design
+
+### The contract (backend-agnostic)
+
+Three surfaces; only one is per-backend.
+
+1. **Control plane (CLI / console → orchestrator)** — an RPC:
+   `launch_bottle`, `teardown_bottle`, `register_policy`,
+   `deregister_bottle`, `supervise_queue`. Fully backend-agnostic. Both the
+   local `cli.py` and the remote console funnel through it, so policy is
+   uniform and `cli.py` becomes a thin client rather than a parallel
+   launcher.
+2. **Data plane (agent → orchestrator)** — the egress / git / supervise
+   endpoints. Already agnostic today (agents dial `http://sidecar:9099`);
+   only the *address* and *how packets get there* are per-backend.
+3. **Launch / wire (orchestrator → backend)** — the irreducibly
+   backend-specific part; lives on `BottleBackend`.
+
+### One `Orchestrator`, no subclass tree
+
+The orchestrator is a **single concrete class** holding all the
+backend-neutral logic — egress addon, git-gate, supervise, source-IP
+attribution, live-reload control plane, console client. It never branches
+on backend; it *composes* a `BottleBackend`. That composition is what makes
+the contract agnostic: there is nothing backend-specific left in the
+orchestrator to leak.
+
+Rejected alternative: an `Orchestrator` ABC with per-backend
+implementations. The interesting logic (proxies, attribution, control
+plane) is backend-neutral, so three subclasses would triplicate the hard
+part; and a second hierarchy paralleling `BottleBackend` reintroduces the
+same hand-maintained lockstep coupling we just removed from the netpool
+constants (PR #350). Composition over a parallel tree.
+
+### `BottleBackend` absorbs the per-backend variation
+
+A small, cohesive surface — reused for launching agent bottles *and* the
+orchestrator's own unit (the orchestrator is just another native unit):
+
+```
+launch_unit(spec) -> Handle      # agent bottle OR the orchestrator itself
+                                 # (fc microVM / apple ctr / docker ctr)
+wire(unit, endpoint) -> None     # DNAT+forward (fc) | attach shared net (docker/apple)
+endpoint_of(unit) -> Endpoint    # address resolution
+health(unit) -> Status
+```
+
+Plus the **launch broker** — the answer to "a VM/container can't spawn its
+own host-network siblings." The orchestrator can't directly open host
+`/dev/kvm` + a host TAP fd (Firecracker), and a container can't spawn
+siblings without a root-equivalent socket (Docker). So every backend
+exposes a broker the orchestrator calls to launch an agent:
+
+- **Firecracker** — a thin, structured host shim (see security #3). This
+  replaces today's implicit "launcher runs as host user."
+- **Docker** — the socket today (fat, root-equivalent — the thing 0069's
+  Stage 3 removes); a narrower broker later.
+- **Apple** — the `container` CLI/daemon.
+
+If `BottleBackend` bloats, the pressure valve is composition one level
+down: vend a `backend.network()` / `Wiring` collaborator rather than
+piling methods on — the same discipline, recursed.
+
+### State: one SQLite DB, owned by the orchestrator
+
+The orchestrator is the natural owner of per-host **runtime state**:
+
+- pool **slot leases** (which bottle holds slot *i*) — replaces today's
+  `fcntl`-locked files with WAL-mode transactions;
+- the **supervise approval queue** + remembered approvals;
+- the **live bottle registry** (source IP → bottle → policy/secrets refs),
+  the lookup table the attribution invariant reads.
+
+This is deliberately **not** a "single source of truth for all config."
+Config splits into three tiers with different homes:
+
+| Tier | Example | Home |
+|---|---|---|
+| Build-time constants | pool size, IP base, nft table | flat `.env` (PR #350) — must be readable by Nix eval + root bash, zero runtime |
+| User-authored config | bottle manifests, egress routes, secret refs | declarative files under `~/.bot-bottle/` — trust boundary at `$HOME`, git-trackable, "unknown keys die at load" |
+| Runtime state | slot leases, approvals, registry | **SQLite**, owned by the orchestrator |
+
+SQLite is right for the runtime tier (mutable, concurrent, queried) and
+wrong for the other two (Nix can't read it at eval time; it fights the
+declarative manifest trust model). Keep the tiers separate.
+
+## Sequencing
+
+Jump straight to the **virtualized** end state (not a host-daemon stepping
+stone): a host daemon's agent→`localhost` transport is throwaway once the
+orchestrator becomes a VM. Decouple the two risks instead:
+
+- **Consolidation risk** (one process, all secrets, attribution, reload)
+  and **packaging/transport risk** (VM-to-VM wiring, the shim) are
+  independent. Develop the orchestrator **service as a plain process
+  dev-harness** first, so the consolidation logic (attribution, reload,
+  secret handling) is proven with fast iteration — *then* wrap that exact
+  service in the VM and solve wiring separately.
+
+Backend order (cheapest proof → hardest → last):
+
+1. **Docker orchestrator** — nearly free (the sidecar bundle is already
+   containers; collapse N bundles into one persistent container). Proves
+   consolidation + the `BottleBackend` seam with the least moving parts.
+2. **Firecracker orchestrator** — the real work: the shim + VM-to-VM
+   routing (host forwards `bbfcN` → orchestrator TAP; the nft table grows
+   forward rules where today it drops all non-DNAT egress). Built against
+   the dev-harness so the app logic is already proven.
+3. **macOS (Apple container)** — last (container-to-container networking).
+
+Keep the sidecar **service one shared thing** throughout.
+
+## Non-goals
+
+- Removing OCI/Dockerfile support for agent images (0069's concern).
+- A single database for *all* config (see the three-tier table).
+- Changing the per-bottle isolation of agent workloads — only the sidecar
+  is consolidated; agents stay one-VM/container-each.
+
+## Relationship to other work
+
+- **PRD 0069 (#348):** 0070 subsumes its Stage 1 (per-host sidecar) and
+  Stage 4 (sidecar-as-VM). 0069 retains Stage 2 (nix-built fixed images —
+  a **dependency** here: the orchestrator and agent base must be
+  nix-built so the broker launches from a fixed image set and bootstrapping
+  has no chicken-and-egg) and Stage 3 (in-VM Dockerfile builder).
+- **Minimal CI runner (paused):** the Firecracker broker + no host Docker
+  is what lets a dedicated `gitea` runner user drop the root-equivalent
+  `docker` group — it only needs broker-socket access + `kvm`/pool group
+  membership. This work unblocks it.
+- **PR #350 (netpool single-source):** the same "one source per fact,
+  composition over parallel hierarchies" discipline the contract follows.
+
+## Open questions
+
+- **Egress sharing tradeoff:** is the secret-concentration blast radius of
+  one shared mitmproxy worth the resource win, or share only supervise
+  (near-zero secrets) and keep egress + git-gate per-bottle initially?
+- **Control-plane shape:** RPC transport (unix socket / vsock / HTTP over
+  the TAP) and the live-reload protocol for per-bottle policy.
+- **State re-adoption:** exact scheme for an orchestrator restart to
+  re-adopt running agent VMs from the SQLite registry without racing
+  in-flight launches.
+- **VM-to-VM routing:** the nft forward rules + addressing for a
+  per-host orchestrator VM on its own TAP.
+- **Broker request schema:** the exact structured contract that stays
+  auditable and can't be coerced into launching arbitrary payloads.