# PRD 0070: Per-host orchestrator service

- **Status:** Draft
- **Author:** Claude
- **Created:** 2026-07-12
- **Issue:** #351
- **Supersedes:** the Stage-1 / Stage-4 sidecar-consolidation framing of
  PRD 0069 (#348). Depends on 0069's nix-built fixed images (Stage 2) for
  bootstrapping; 0069 still owns the docker-free image-building work.

## Summary

Replace the **per-bottle sidecar bundle** with a single **persistent,
per-host orchestrator**: one long-lived service that runs the sidecar
functions (egress / git-gate / supervise), coordinates with the console,
and brokers agent launches and teardown. It is **virtualized from the
start** using each backend's native isolation primitive — a Firecracker
microVM on the Firecracker backend, an Apple container on macOS, a Docker
container on the legacy backend — and is fronted by a single
**backend-agnostic contract**. Per-backend variation lives on
`BottleBackend`, not in the orchestrator.

## Motivation

Today each bottle spins up its own sidecar bundle (egress mitmproxy +
git-gate + supervise). That costs:

- **Resources.** N bottles → N heavy bundles booting and idling.
- **Operational churn.** Per-launch container/VM lifecycle for the
  sidecars, a control path baked at launch and torn down at exit.
- **A blurry contract.** "How a bottle talks to its sidecar" is
  re-implemented per backend instead of being one agreed interface.

A per-host orchestrator collapses the first two and forces the third to
be made explicit. It's also the component that will own per-host runtime
**state** (slot leases, the approval queue, the bottle registry) — today
that's ad-hoc `fcntl`-locked files.

## Security review (read this first)

Consolidation is a real change to the trust model. The goal is to **not
significantly weaken** the posture; some properties strengthen, some
weaken, and the weakened ones must be mitigated by design, not hand-waved.

### What gets stronger

- **Build/host isolation of untrusted inputs** (with 0069 Stage 3): user
  Dockerfiles build in a disposable VM instead of on the host.
- **One audited privileged surface.** Today the launcher runs as the full
  host user and needs the Docker socket (root-equivalent). The orchestrator
  model replaces that with a **thin launch broker** (below) — a small,
  structured, auditable privileged core instead of a fat socket.
- **Attribution is enforced, not assumed.** Making source-IP identity a
  first-class contract invariant (below) means each backend must *prove*
  it, rather than the sidecar implicitly trusting network position.

### What gets weaker, and the mitigation

1. **Secret concentration.** Per-bottle sidecars isolate secrets at the
   process boundary — each holds only its bottle's tokens/keys. A host
   orchestrator concentrates **every bottle's** egress tokens, git deploy
   keys, and the console credential in one long-lived process. A single
   attribution bug leaks bottle A's token into bottle B's request — a class
   of bug that *cannot exist* per-bottle.
   - *Mitigation:* lean on the enforced source-IP invariant for
     attribution; keep the most secret-dense, least-shareable service
     (**git-gate**, per-repo deploy keys, no natural source-IP scoping)
     **per-bottle** unless there's a compelling reason; scope each secret
     to the bottle in the state DB so a lookup can't return the wrong
     bottle's secret by construction (key every secret access by the
     verified source identity, never by ambient state).

2. **Shared fate.** Orchestrator down = no new launches, and running
   agents lose egress / git / supervise. Compromise = the whole host's
   fleet, plus launch authority, plus the console token.
   - *Mitigation:* the orchestrator is itself confined (its own VM/container
     with its own fail-closed egress); make it **restartable without killing
     running agent VMs** (agents keep running; they briefly lose sidecar
     connectivity until it's back); persist state to a host volume so a
     restart re-adopts live bottles rather than losing them.

3. **The launch broker is the new privileged core.** We don't eliminate
   host privilege — we shrink and relocate it. If the broker accepts
   arbitrary paths/commands, the orchestrator VM can escape through it.
   - *Mitigation:* the broker takes **structured requests only** — "launch
     bottle from *this* content-addressed, nix-built rootfs on TAP slot
     *k*", never "run this argv". It validates against a fixed image set,
     not caller-supplied paths. It is small enough to audit line-by-line.

4. **The egress proxy now parses every bottle's traffic in one process.**
   Higher blast radius for a mitmproxy/TLS-bump bug.
   - *Mitigation:* this is the argument for virtualizing the orchestrator
     from the start (Stage B, not a host daemon) — the code that TLS-bumps
     and parses agent traffic and holds every token runs **inside its own
     confined VM**, not as a host process. If egress sharing's blast radius
     feels too high, egress can stay per-bottle while supervise (near-zero
     secrets) goes host-level first.

### The attribution invariant

Source-IP attribution is what makes a shared orchestrator safe: one
process serves every bottle and tells them apart by source address. The
*mechanism* is identical everywhere (read source IP → look up bottle); the
**guarantee that the address can't be forged is a per-backend
responsibility** and part of the contract:

> **Invariant:** a packet's source address, as seen by the orchestrator,
> *provably* identifies the originating bottle.

- **Firecracker** — enforced by the `/31` point-to-point TAP + the
  `bot_bottle_fc` nft table (strongest; already built).
- **Docker** — the per-bottle `--internal` network + anti-spoof; weaker,
  must be made explicit.
- **Apple** — the host-only network.

If a backend can't honor the invariant, source-IP consolidation is not
safe there and that backend keeps per-bottle sidecars. The invariant is a
hard precondition, not an aspiration.

## Design

### The contract (backend-agnostic)

Three surfaces; only one is per-backend.

1. **Control plane (CLI / console → orchestrator)** — an RPC:
   `launch_bottle`, `teardown_bottle`, `register_policy`,
   `deregister_bottle`, `supervise_queue`. Fully backend-agnostic. Both the
   local `cli.py` and the remote console funnel through it, so policy is
   uniform and `cli.py` becomes a thin client rather than a parallel
   launcher.
2. **Data plane (agent → orchestrator)** — the egress / git / supervise
   endpoints. Already agnostic today (agents dial `http://sidecar:9099`);
   only the *address* and *how packets get there* are per-backend.
3. **Launch / wire (orchestrator → backend)** — the irreducibly
   backend-specific part; lives on `BottleBackend`.

### One `Orchestrator`, no subclass tree

The orchestrator is a **single concrete class** holding all the
backend-neutral logic — egress addon, git-gate, supervise, source-IP
attribution, live-reload control plane, console client. It never branches
on backend; it *composes* a `BottleBackend`. That composition is what makes
the contract agnostic: there is nothing backend-specific left in the
orchestrator to leak.

Rejected alternative: an `Orchestrator` ABC with per-backend
implementations. The interesting logic (proxies, attribution, control
plane) is backend-neutral, so three subclasses would triplicate the hard
part; and a second hierarchy paralleling `BottleBackend` reintroduces the
same hand-maintained lockstep coupling we just removed from the netpool
constants (PR #350). Composition over a parallel tree.

### `BottleBackend` absorbs the per-backend variation

A small, cohesive surface — reused for launching agent bottles *and* the
orchestrator's own unit (the orchestrator is just another native unit):

```
launch_unit(spec) -> Handle      # agent bottle OR the orchestrator itself
                                 # (fc microVM / apple ctr / docker ctr)
wire(unit, endpoint) -> None     # DNAT+forward (fc) | attach shared net (docker/apple)
endpoint_of(unit) -> Endpoint    # address resolution
health(unit) -> Status
```

Plus the **launch broker** — the answer to "a VM/container can't spawn its
own host-network siblings." The orchestrator can't directly open host
`/dev/kvm` + a host TAP fd (Firecracker), and a container can't spawn
siblings without a root-equivalent socket (Docker). So every backend
exposes a broker the orchestrator calls to launch an agent:

- **Firecracker** — a thin, structured host shim (see security #3). This
  replaces today's implicit "launcher runs as host user."
- **Docker** — the socket today (fat, root-equivalent — the thing 0069's
  Stage 3 removes); a narrower broker later.
- **Apple** — the `container` CLI/daemon.

If `BottleBackend` bloats, the pressure valve is composition one level
down: vend a `backend.network()` / `Wiring` collaborator rather than
piling methods on — the same discipline, recursed.

### State: one SQLite DB, owned by the orchestrator

The orchestrator is the natural owner of per-host **runtime state**:

- pool **slot leases** (which bottle holds slot *i*) — replaces today's
  `fcntl`-locked files with WAL-mode transactions;
- the **supervise approval queue** + remembered approvals;
- the **live bottle registry** (source IP → bottle → policy/secrets refs),
  the lookup table the attribution invariant reads.

This is deliberately **not** a "single source of truth for all config."
Config splits into three tiers with different homes:

| Tier | Example | Home |
|---|---|---|
| Build-time constants | pool size, IP base, nft table | flat `.env` (PR #350) — must be readable by Nix eval + root bash, zero runtime |
| User-authored config | bottle manifests, egress routes, secret refs | declarative files under `~/.bot-bottle/` — trust boundary at `$HOME`, git-trackable, "unknown keys die at load" |
| Runtime state | slot leases, approvals, registry | **SQLite**, owned by the orchestrator |

SQLite is right for the runtime tier (mutable, concurrent, queried) and
wrong for the other two (Nix can't read it at eval time; it fights the
declarative manifest trust model). Keep the tiers separate.

## Sequencing

Jump straight to the **virtualized** end state (not a host-daemon stepping
stone): a host daemon's agent→`localhost` transport is throwaway once the
orchestrator becomes a VM. Decouple the two risks instead:

- **Consolidation risk** (one process, all secrets, attribution, reload)
  and **packaging/transport risk** (VM-to-VM wiring, the shim) are
  independent. Develop the orchestrator **service as a plain process
  dev-harness** first, so the consolidation logic (attribution, reload,
  secret handling) is proven with fast iteration — *then* wrap that exact
  service in the VM and solve wiring separately.

Backend order (cheapest proof → hardest → last):

1. **Docker orchestrator** — nearly free (the sidecar bundle is already
   containers; collapse N bundles into one persistent container). Proves
   consolidation + the `BottleBackend` seam with the least moving parts.
2. **Firecracker orchestrator** — the real work: the shim + VM-to-VM
   routing (host forwards `bbfcN` → orchestrator TAP; the nft table grows
   forward rules where today it drops all non-DNAT egress). Built against
   the dev-harness so the app logic is already proven.
3. **macOS (Apple container)** — last (container-to-container networking).

Keep the sidecar **service one shared thing** throughout.

## Non-goals

- Removing OCI/Dockerfile support for agent images (0069's concern).
- A single database for *all* config (see the three-tier table).
- Changing the per-bottle isolation of agent workloads — only the sidecar
  is consolidated; agents stay one-VM/container-each.

## Relationship to other work

- **PRD 0069 (#348):** 0070 subsumes its Stage 1 (per-host sidecar) and
  Stage 4 (sidecar-as-VM). 0069 retains Stage 2 (nix-built fixed images —
  a **dependency** here: the orchestrator and agent base must be
  nix-built so the broker launches from a fixed image set and bootstrapping
  has no chicken-and-egg) and Stage 3 (in-VM Dockerfile builder).
- **Minimal CI runner (paused):** the Firecracker broker + no host Docker
  is what lets a dedicated `gitea` runner user drop the root-equivalent
  `docker` group — it only needs broker-socket access + `kvm`/pool group
  membership. This work unblocks it.
- **PR #350 (netpool single-source):** the same "one source per fact,
  composition over parallel hierarchies" discipline the contract follows.

## Open questions

- **Egress sharing tradeoff:** is the secret-concentration blast radius of
  one shared mitmproxy worth the resource win, or share only supervise
  (near-zero secrets) and keep egress + git-gate per-bottle initially?
- **Control-plane shape:** RPC transport (unix socket / vsock / HTTP over
  the TAP) and the live-reload protocol for per-bottle policy.
- **State re-adoption:** exact scheme for an orchestrator restart to
  re-adopt running agent VMs from the SQLite registry without racing
  in-flight launches.
- **VM-to-VM routing:** the nft forward rules + addressing for a
  per-host orchestrator VM on its own TAP.
- **Broker request schema:** the exact structured contract that stays
  auditable and can't be coerced into launching arbitrary payloads.