docs(prd): 0070 per-host orchestrator service

Fold the per-bottle sidecar bundle into a single persistent per-host
orchestrator: runs the sidecar functions (egress/git-gate/supervise),
coordinates with the console, and brokers agent launches. Virtualized
from the start with backend-native isolation (fc VM / apple ctr / docker
ctr), fronted by a single backend-agnostic contract; per-backend
variation lives on BottleBackend, not a parallel Orchestrator hierarchy.

Leads with the security review (secret concentration, shared fate, the
launch-broker-as-new-privileged-core, and the source-IP attribution
invariant each backend must enforce). Proposes one SQLite DB owned by the
orchestrator for runtime state (slot leases, approvals, registry) —
distinct from build-time constants (flat .env) and user config
(declarative ~/.bot-bottle). Sequences docker -> firecracker -> macos,
developing the service as a plain-process dev-harness before the VM.

Supersedes 0069's Stage-1/4 sidecar framing; depends on 0069's nix-built
fixed images. Tracking issue #351.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WBMWTEtQdJ4W5UrWuLHCck

.gitea/workflows/test.yml

@@ -71,45 +71,30 @@ jobs:
       - name: Run integration tests
         run: python3 -m unittest discover -t . -s tests/integration -v
 
-  # Combined unit+integration coverage + the diff-coverage gate (the hard
+  # Combined unit+integration coverage report (informational). See
-  # gate: new/changed lines >= 90%). See docs/decisions/0004-coverage-policy.md.
+  # docs/decisions/0004-coverage-policy.md.
   #
-  # This runs on a self-hosted KVM runner (label `kvm`), NOT ubuntu-latest,
+  # The hard diff-coverage gate (changed lines >= 90%) is DEFERRED: the
-  # because the Firecracker backend's subprocess/VM orchestration
+  # Firecracker backend's VM/SSH orchestration is covered by the integration
-  # (launch/boot/SSH/isolation-probe) is covered by the integration suite,
+  # suite, which needs /dev/kvm + the provisioned TAP/nft pool — a
-  # and that suite needs `/dev/kvm` + the provisioned TAP/nft pool — which a
+  # container-based runner skips it and those lines read uncovered, so the
-  # container-based runner doesn't have. On such a runner the firecracker
+  # gate can't pass here. Re-enabling it on a self-hosted KVM runner is
-  # integration test skips and its ~230 orchestration lines read as
+  # tracked separately (see PRD 0069 / #348 and the ci-runner branch).
-  # uncovered, so the gate can't pass there.
-  #
-  # Runner prerequisites (provision once on the host; see the README
-  # "Firecracker on Linux" section): the `firecracker` binary on PATH,
-  # `/dev/kvm` accessible to the runner user, Docker, the cached guest
-  # kernel + static dropbear (BOT_BOTTLE_FC_KERNEL / BOT_BOTTLE_FC_DROPBEAR),
-  # and the network pool installed as the persistent systemd unit
-  # (`./cli.py backend setup --backend=firecracker`). The preflight step
-  # below fails fast with instructions if anything is missing.
   coverage:
-    runs-on: [self-hosted, kvm]
+    runs-on: ubuntu-latest
     steps:
       - name: Checkout
         uses: actions/checkout@v4
         with:
           fetch-depth: 0
 
-      - name: Preflight — Firecracker host is ready
+      - name: Set up Python
-        run: |
+        uses: actions/setup-python@v5
-          command -v firecracker >/dev/null || {
+        with:
-            echo "firecracker not on PATH — provision the runner (README: Firecracker on Linux)"; exit 1; }
+          python-version: "3.12"
-          test -e /dev/kvm || { echo "/dev/kvm missing — KVM not available on this runner"; exit 1; }
-          # `backend status` exits non-zero unless the TAP pool is up + no
-          # range overlap; it prints the exact `backend setup` fix.
-          python3 cli.py backend status --backend=firecracker
 
-      - name: Combined coverage (unit + integration, incl. firecracker)
+      - name: Install dev requirements
+        run: python3 -m pip install -r requirements-dev.txt
+
+      - name: Combined coverage report (unit + integration)
         run: PYTHON=python3 bash scripts/coverage.sh critical
-
-      - name: Diff-coverage gate (changed lines >= 90%)
-        run: |
-          git fetch --no-tags origin main:refs/remotes/origin/main
-          python3 scripts/diff_coverage.py --base origin/main --min 90

README.md

@@ -90,8 +90,6 @@ BOT_BOTTLE_BACKEND=firecracker ./cli.py start <agent>
 
 > **NixOS:** enable `virtualisation.docker`, ensure the KVM module is loaded (`boot.kernelModules = [ "kvm-intel" ];` or `kvm-amd`), and add your user to the `kvm` and `docker` groups. For the network pool, consume the flake module — `imports = [ inputs.bot-bottle.nixosModules.firecracker-netpool ]; services.bot-bottle-firecracker = { enable = true; owner = "you"; };` — then `nixos-rebuild switch` (imperative nft/TAP rules don't survive a rebuild; channel users can `imports = [ <bot-bottle>/nix/firecracker-netpool.nix ]`). `firecracker` isn't in nixpkgs by default as a user binary — install the release binary (pin the version) and put it on `PATH`.
 
-> **CI:** the coverage gate (`.gitea/workflows/test.yml` → `coverage` job) runs on a self-hosted runner labelled `kvm`, because the Firecracker backend's VM/SSH orchestration is exercised only by the integration suite, which needs `/dev/kvm` + the provisioned pool (a container runner would skip it and read as uncovered). Provision that runner exactly like a normal Firecracker host — `firecracker` on `PATH`, `/dev/kvm`, Docker, the cached guest kernel + static dropbear, and the pool installed as the persistent systemd unit — then register it with the `kvm` label. The unit/lint jobs still run on `ubuntu-latest`.
-
 ```sh
 ./cli.py start <agent>   # builds the image on first run, drops you into claude
 ```

docs/prds/0069-firecracker-native-docker-free.md

@@ -1,10 +1,16 @@
 # PRD 0069: Firecracker-native, Docker-free backend
 
-- **Status:** Draft
+- **Status:** Draft (partially superseded)
 - **Author:** Claude
 - **Created:** 2026-07-12
 - **Issue:** #348
 
+> **Superseded in part by [PRD 0070](0070-per-host-orchestrator.md) (#351):**
+> the sidecar-consolidation framing here (Stage 1, per-host sidecar; Stage 4,
+> sidecar-as-VM) is taken over by 0070's per-host orchestrator. This PRD still
+> owns the docker-free **image-building** work — Stage 2 (nix-built fixed
+> images, a dependency of 0070) and Stage 3 (in-VM Dockerfile builder).
+
 ## Summary
 
 Make the Firecracker backend depend on **firecracker + KVM only**, removing

docs/prds/0070-per-host-orchestrator.md

@@ -0,0 +1,266 @@
+# PRD 0070: Per-host orchestrator service
+
+- **Status:** Draft
+- **Author:** Claude
+- **Created:** 2026-07-12
+- **Issue:** #351
+- **Supersedes:** the Stage-1 / Stage-4 sidecar-consolidation framing of
+  PRD 0069 (#348). Depends on 0069's nix-built fixed images (Stage 2) for
+  bootstrapping; 0069 still owns the docker-free image-building work.
+
+## Summary
+
+Replace the **per-bottle sidecar bundle** with a single **persistent,
+per-host orchestrator**: one long-lived service that runs the sidecar
+functions (egress / git-gate / supervise), coordinates with the console,
+and brokers agent launches and teardown. It is **virtualized from the
+start** using each backend's native isolation primitive — a Firecracker
+microVM on the Firecracker backend, an Apple container on macOS, a Docker
+container on the legacy backend — and is fronted by a single
+**backend-agnostic contract**. Per-backend variation lives on
+`BottleBackend`, not in the orchestrator.
+
+## Motivation
+
+Today each bottle spins up its own sidecar bundle (egress mitmproxy +
+git-gate + supervise). That costs:
+
+- **Resources.** N bottles → N heavy bundles booting and idling.
+- **Operational churn.** Per-launch container/VM lifecycle for the
+  sidecars, a control path baked at launch and torn down at exit.
+- **A blurry contract.** "How a bottle talks to its sidecar" is
+  re-implemented per backend instead of being one agreed interface.
+
+A per-host orchestrator collapses the first two and forces the third to
+be made explicit. It's also the component that will own per-host runtime
+**state** (slot leases, the approval queue, the bottle registry) — today
+that's ad-hoc `fcntl`-locked files.
+
+## Security review (read this first)
+
+Consolidation is a real change to the trust model. The goal is to **not
+significantly weaken** the posture; some properties strengthen, some
+weaken, and the weakened ones must be mitigated by design, not hand-waved.
+
+### What gets stronger
+
+- **Build/host isolation of untrusted inputs** (with 0069 Stage 3): user
+  Dockerfiles build in a disposable VM instead of on the host.
+- **One audited privileged surface.** Today the launcher runs as the full
+  host user and needs the Docker socket (root-equivalent). The orchestrator
+  model replaces that with a **thin launch broker** (below) — a small,
+  structured, auditable privileged core instead of a fat socket.
+- **Attribution is enforced, not assumed.** Making source-IP identity a
+  first-class contract invariant (below) means each backend must *prove*
+  it, rather than the sidecar implicitly trusting network position.
+
+### What gets weaker, and the mitigation
+
+1. **Secret concentration.** Per-bottle sidecars isolate secrets at the
+   process boundary — each holds only its bottle's tokens/keys. A host
+   orchestrator concentrates **every bottle's** egress tokens, git deploy
+   keys, and the console credential in one long-lived process. A single
+   attribution bug leaks bottle A's token into bottle B's request — a class
+   of bug that *cannot exist* per-bottle.
+   - *Mitigation:* lean on the enforced source-IP invariant for
+     attribution; keep the most secret-dense, least-shareable service
+     (**git-gate**, per-repo deploy keys, no natural source-IP scoping)
+     **per-bottle** unless there's a compelling reason; scope each secret
+     to the bottle in the state DB so a lookup can't return the wrong
+     bottle's secret by construction (key every secret access by the
+     verified source identity, never by ambient state).
+
+2. **Shared fate.** Orchestrator down = no new launches, and running
+   agents lose egress / git / supervise. Compromise = the whole host's
+   fleet, plus launch authority, plus the console token.
+   - *Mitigation:* the orchestrator is itself confined (its own VM/container
+     with its own fail-closed egress); make it **restartable without killing
+     running agent VMs** (agents keep running; they briefly lose sidecar
+     connectivity until it's back); persist state to a host volume so a
+     restart re-adopts live bottles rather than losing them.
+
+3. **The launch broker is the new privileged core.** We don't eliminate
+   host privilege — we shrink and relocate it. If the broker accepts
+   arbitrary paths/commands, the orchestrator VM can escape through it.
+   - *Mitigation:* the broker takes **structured requests only** — "launch
+     bottle from *this* content-addressed, nix-built rootfs on TAP slot
+     *k*", never "run this argv". It validates against a fixed image set,
+     not caller-supplied paths. It is small enough to audit line-by-line.
+
+4. **The egress proxy now parses every bottle's traffic in one process.**
+   Higher blast radius for a mitmproxy/TLS-bump bug.
+   - *Mitigation:* this is the argument for virtualizing the orchestrator
+     from the start (Stage B, not a host daemon) — the code that TLS-bumps
+     and parses agent traffic and holds every token runs **inside its own
+     confined VM**, not as a host process. If egress sharing's blast radius
+     feels too high, egress can stay per-bottle while supervise (near-zero
+     secrets) goes host-level first.
+
+### The attribution invariant
+
+Source-IP attribution is what makes a shared orchestrator safe: one
+process serves every bottle and tells them apart by source address. The
+*mechanism* is identical everywhere (read source IP → look up bottle); the
+**guarantee that the address can't be forged is a per-backend
+responsibility** and part of the contract:
+
+> **Invariant:** a packet's source address, as seen by the orchestrator,
+> *provably* identifies the originating bottle.
+
+- **Firecracker** — enforced by the `/31` point-to-point TAP + the
+  `bot_bottle_fc` nft table (strongest; already built).
+- **Docker** — the per-bottle `--internal` network + anti-spoof; weaker,
+  must be made explicit.
+- **Apple** — the host-only network.
+
+If a backend can't honor the invariant, source-IP consolidation is not
+safe there and that backend keeps per-bottle sidecars. The invariant is a
+hard precondition, not an aspiration.
+
+## Design
+
+### The contract (backend-agnostic)
+
+Three surfaces; only one is per-backend.
+
+1. **Control plane (CLI / console → orchestrator)** — an RPC:
+   `launch_bottle`, `teardown_bottle`, `register_policy`,
+   `deregister_bottle`, `supervise_queue`. Fully backend-agnostic. Both the
+   local `cli.py` and the remote console funnel through it, so policy is
+   uniform and `cli.py` becomes a thin client rather than a parallel
+   launcher.
+2. **Data plane (agent → orchestrator)** — the egress / git / supervise
+   endpoints. Already agnostic today (agents dial `http://sidecar:9099`);
+   only the *address* and *how packets get there* are per-backend.
+3. **Launch / wire (orchestrator → backend)** — the irreducibly
+   backend-specific part; lives on `BottleBackend`.
+
+### One `Orchestrator`, no subclass tree
+
+The orchestrator is a **single concrete class** holding all the
+backend-neutral logic — egress addon, git-gate, supervise, source-IP
+attribution, live-reload control plane, console client. It never branches
+on backend; it *composes* a `BottleBackend`. That composition is what makes
+the contract agnostic: there is nothing backend-specific left in the
+orchestrator to leak.
+
+Rejected alternative: an `Orchestrator` ABC with per-backend
+implementations. The interesting logic (proxies, attribution, control
+plane) is backend-neutral, so three subclasses would triplicate the hard
+part; and a second hierarchy paralleling `BottleBackend` reintroduces the
+same hand-maintained lockstep coupling we just removed from the netpool
+constants (PR #350). Composition over a parallel tree.
+
+### `BottleBackend` absorbs the per-backend variation
+
+A small, cohesive surface — reused for launching agent bottles *and* the
+orchestrator's own unit (the orchestrator is just another native unit):
+
+```
+launch_unit(spec) -> Handle      # agent bottle OR the orchestrator itself
+                                 # (fc microVM / apple ctr / docker ctr)
+wire(unit, endpoint) -> None     # DNAT+forward (fc) | attach shared net (docker/apple)
+endpoint_of(unit) -> Endpoint    # address resolution
+health(unit) -> Status
+```
+
+Plus the **launch broker** — the answer to "a VM/container can't spawn its
+own host-network siblings." The orchestrator can't directly open host
+`/dev/kvm` + a host TAP fd (Firecracker), and a container can't spawn
+siblings without a root-equivalent socket (Docker). So every backend
+exposes a broker the orchestrator calls to launch an agent:
+
+- **Firecracker** — a thin, structured host shim (see security #3). This
+  replaces today's implicit "launcher runs as host user."
+- **Docker** — the socket today (fat, root-equivalent — the thing 0069's
+  Stage 3 removes); a narrower broker later.
+- **Apple** — the `container` CLI/daemon.
+
+If `BottleBackend` bloats, the pressure valve is composition one level
+down: vend a `backend.network()` / `Wiring` collaborator rather than
+piling methods on — the same discipline, recursed.
+
+### State: one SQLite DB, owned by the orchestrator
+
+The orchestrator is the natural owner of per-host **runtime state**:
+
+- pool **slot leases** (which bottle holds slot *i*) — replaces today's
+  `fcntl`-locked files with WAL-mode transactions;
+- the **supervise approval queue** + remembered approvals;
+- the **live bottle registry** (source IP → bottle → policy/secrets refs),
+  the lookup table the attribution invariant reads.
+
+This is deliberately **not** a "single source of truth for all config."
+Config splits into three tiers with different homes:
+
+| Tier | Example | Home |
+|---|---|---|
+| Build-time constants | pool size, IP base, nft table | flat `.env` (PR #350) — must be readable by Nix eval + root bash, zero runtime |
+| User-authored config | bottle manifests, egress routes, secret refs | declarative files under `~/.bot-bottle/` — trust boundary at `$HOME`, git-trackable, "unknown keys die at load" |
+| Runtime state | slot leases, approvals, registry | **SQLite**, owned by the orchestrator |
+
+SQLite is right for the runtime tier (mutable, concurrent, queried) and
+wrong for the other two (Nix can't read it at eval time; it fights the
+declarative manifest trust model). Keep the tiers separate.
+
+## Sequencing
+
+Jump straight to the **virtualized** end state (not a host-daemon stepping
+stone): a host daemon's agent→`localhost` transport is throwaway once the
+orchestrator becomes a VM. Decouple the two risks instead:
+
+- **Consolidation risk** (one process, all secrets, attribution, reload)
+  and **packaging/transport risk** (VM-to-VM wiring, the shim) are
+  independent. Develop the orchestrator **service as a plain process
+  dev-harness** first, so the consolidation logic (attribution, reload,
+  secret handling) is proven with fast iteration — *then* wrap that exact
+  service in the VM and solve wiring separately.
+
+Backend order (cheapest proof → hardest → last):
+
+1. **Docker orchestrator** — nearly free (the sidecar bundle is already
+   containers; collapse N bundles into one persistent container). Proves
+   consolidation + the `BottleBackend` seam with the least moving parts.
+2. **Firecracker orchestrator** — the real work: the shim + VM-to-VM
+   routing (host forwards `bbfcN` → orchestrator TAP; the nft table grows
+   forward rules where today it drops all non-DNAT egress). Built against
+   the dev-harness so the app logic is already proven.
+3. **macOS (Apple container)** — last (container-to-container networking).
+
+Keep the sidecar **service one shared thing** throughout.
+
+## Non-goals
+
+- Removing OCI/Dockerfile support for agent images (0069's concern).
+- A single database for *all* config (see the three-tier table).
+- Changing the per-bottle isolation of agent workloads — only the sidecar
+  is consolidated; agents stay one-VM/container-each.
+
+## Relationship to other work
+
+- **PRD 0069 (#348):** 0070 subsumes its Stage 1 (per-host sidecar) and
+  Stage 4 (sidecar-as-VM). 0069 retains Stage 2 (nix-built fixed images —
+  a **dependency** here: the orchestrator and agent base must be
+  nix-built so the broker launches from a fixed image set and bootstrapping
+  has no chicken-and-egg) and Stage 3 (in-VM Dockerfile builder).
+- **Minimal CI runner (paused):** the Firecracker broker + no host Docker
+  is what lets a dedicated `gitea` runner user drop the root-equivalent
+  `docker` group — it only needs broker-socket access + `kvm`/pool group
+  membership. This work unblocks it.
+- **PR #350 (netpool single-source):** the same "one source per fact,
+  composition over parallel hierarchies" discipline the contract follows.
+
+## Open questions
+
+- **Egress sharing tradeoff:** is the secret-concentration blast radius of
+  one shared mitmproxy worth the resource win, or share only supervise
+  (near-zero secrets) and keep egress + git-gate per-bottle initially?
+- **Control-plane shape:** RPC transport (unix socket / vsock / HTTP over
+  the TAP) and the live-reload protocol for per-bottle policy.
+- **State re-adoption:** exact scheme for an orchestrator restart to
+  re-adopt running agent VMs from the SQLite registry without racing
+  in-flight launches.
+- **VM-to-VM routing:** the nft forward rules + addressing for a
+  per-host orchestrator VM on its own TAP.
+- **Broker request schema:** the exact structured contract that stays
+  auditable and can't be coerced into launching arbitrary payloads.