bot-bottle/docs/prds/0070-per-host-orchestrator.md

# PRD 0070: Per-host orchestrator service

- **Status:** Draft
- **Author:** Claude
- **Created:** 2026-07-12
- **Issue:** #351
- **Supersedes:** the Stage-1 / Stage-4 sidecar-consolidation framing of
  PRD 0069 (#348). Depends on 0069's nix-built fixed images (Stage 2) for
  bootstrapping; 0069 still owns the docker-free image-building work.

## Summary

Replace the **per-bottle sidecar bundle** with a single **persistent,
per-host orchestrator**: one long-lived service that runs the sidecar
functions (egress / git-gate / supervise), coordinates with the console,
and brokers agent launches and teardown. It is **virtualized from the
start** using each backend's native isolation primitive — a Firecracker
microVM on the Firecracker backend, an Apple container on macOS, a Docker
container on the legacy backend — and is fronted by a single
**backend-agnostic contract**. Per-backend variation lives on
`BottleBackend`, not in the orchestrator.

## Motivation

Today each bottle spins up its own sidecar bundle (egress mitmproxy +
git-gate + supervise). That costs:

- **Resources.** N bottles → N heavy bundles booting and idling.
- **Operational churn.** Per-launch container/VM lifecycle for the
  sidecars, a control path baked at launch and torn down at exit.
- **A blurry contract.** "How a bottle talks to its sidecar" is
  re-implemented per backend instead of being one agreed interface.

A per-host orchestrator collapses the first two and forces the third to
be made explicit. It's also the component that will own per-host runtime
**state** (slot leases, the approval queue, the bottle registry) — today
that's ad-hoc `fcntl`-locked files.

## Security review (read this first)

Consolidation is a real change to the trust model. The goal is to **not
significantly weaken** the posture; some properties strengthen, some
weaken, and the weakened ones must be mitigated by design, not hand-waved.

### What gets stronger

- **Build/host isolation of untrusted inputs** (with 0069 Stage 3): user
  Dockerfiles build in a disposable VM instead of on the host.
- **One audited privileged surface.** Today the launcher runs as the full
  host user and needs the Docker socket (root-equivalent). The orchestrator
  model replaces that with a **thin launch broker** (below) — a small,
  structured, auditable privileged core instead of a fat socket.
- **Attribution is enforced, not assumed.** Making source-IP identity a
  first-class contract invariant (below) means each backend must *prove*
  it, rather than the sidecar implicitly trusting network position.

### What gets weaker, and the mitigation

1. **Secret concentration.** Per-bottle sidecars isolate secrets at the
   process boundary — each holds only its bottle's tokens/keys. A host
   orchestrator concentrates **every bottle's** egress tokens, git deploy
   keys, and the console credential in one long-lived process. A single
   attribution bug leaks bottle A's token into bottle B's request — a class
   of bug that *cannot exist* per-bottle.
   - *Mitigation:* lean on the enforced source-IP invariant for
     attribution; keep the most secret-dense, least-shareable service
     (**git-gate**, per-repo deploy keys, no natural source-IP scoping)
     **per-bottle** unless there's a compelling reason; scope each secret
     to the bottle in the state DB so a lookup can't return the wrong
     bottle's secret by construction (key every secret access by the
     verified source identity, never by ambient state).

2. **Shared fate.** Orchestrator down = no new launches, and running
   agents lose egress / git / supervise. Compromise = the whole host's
   fleet, plus launch authority, plus the console token.
   - *Mitigation:* the orchestrator is itself confined (its own VM/container
     with its own fail-closed egress); make it **restartable without killing
     running agent VMs** (agents keep running; they briefly lose sidecar
     connectivity until it's back); persist state to a host volume so a
     restart re-adopts live bottles rather than losing them.

3. **The launch broker is the new privileged core.** We don't eliminate
   host privilege — we shrink and relocate it. If the broker accepts
   arbitrary paths/commands, the orchestrator VM can escape through it.
   - *Mitigation:* the broker takes **structured requests only** — "launch
     bottle from *this* content-addressed, nix-built rootfs on TAP slot
     *k*", never "run this argv". It validates against a fixed image set,
     not caller-supplied paths. It is small enough to audit line-by-line.

4. **The egress proxy now parses every bottle's traffic in one process.**
   Higher blast radius for a mitmproxy/TLS-bump bug.
   - *Mitigation:* this is the argument for virtualizing the orchestrator
     from the start (Stage B, not a host daemon) — the code that TLS-bumps
     and parses agent traffic and holds every token runs **inside its own
     confined VM**, not as a host process. If egress sharing's blast radius
     feels too high, egress can stay per-bottle while supervise (near-zero
     secrets) goes host-level first.

### The attribution invariant

Source-IP attribution is what makes a shared orchestrator safe: one
process serves every bottle and tells them apart by source address. The
*mechanism* is identical everywhere (read source IP → look up bottle); the
**guarantee that the address can't be forged is a per-backend
responsibility** and part of the contract:

> **Invariant:** a packet's source address, as seen by the orchestrator,
> *provably* identifies the originating bottle.

- **Firecracker** — enforced by the `/31` point-to-point TAP + the
  `bot_bottle_fc` nft table (strongest; already built).
- **Docker** — the per-bottle `--internal` network + anti-spoof; weaker,
  must be made explicit.
- **Apple** — the host-only network.

If a backend can't honor the invariant, source-IP consolidation is not
safe there and that backend keeps per-bottle sidecars. The invariant is a
hard precondition, not an aspiration.

**Defense-in-depth — a per-bottle identity token.** On top of the
network-layer invariant, inject a per-bottle secret token into every
request the agent makes to the orchestrator (the agent already egresses
through the sidecar proxy, so this is cheap to add). It gives an
**application-layer** proof of identity independent of the network layer:

- On **Firecracker** the `/31` + nft already make source IP unspoofable
  *by construction*, so the token is belt-and-suspenders there — but cheap
  insurance against a misconfigured invariant.
- On **weaker backends** (Docker) it is load-bearing, providing attribution
  that doesn't lean on network anti-spoof.

Requirements: the token is **per-bottle, unguessable, and
non-cross-leakable** — a bottle can only ever prove it is *itself* (it
can't learn another bottle's token, given bottle isolation), so a hostile
agent gains nothing by presenting it. The orchestrator provisions the
token at launch and the sidecar requires it to attribute + authorize.
Note this hardens *attribution*, not *secret exposure*: it's app-layer, so
a compromised orchestrator still sees every token (that's the
concentration problem, addressed separately under Secret handling).

### Secret handling — a FUTURE pattern (not v1)

> **Status: future / not required for the initial orchestrator.** The
> initial cut can inject secrets as today; this section records the
> direction so v1 doesn't paint itself into a corner. Tracked as its own
> work in **#355** (generic `SecretProvider`).

The residual weakness after all of the above is **long-lived credential
concentration** — the data-plane proxies must hold every bottle's upstream
tokens because the agent must never see them. You can't policy-gate the
component whose job is to *use* all the secrets, so a full RCE of a proxy
drains its authorized set regardless. Two moves bound this without
pretending to prevent it:

1. **Vault as a separate trust domain.** The long-lived *roots* live in a
   distinct process (ideally its own VM) that the byte-parsing data plane
   never shares memory with. The proxies request secrets from it; the
   crown jewels are not in the process an agent-facing parser can pop.
2. **Derive short-lived, scoped creds where the upstream allows it.** The
   vault holds the root and mints expiring, narrowly-scoped credentials
   per bottle-start (or per request) — GitHub App installation tokens,
   OAuth/STS token exchange, forge deploy tokens. A compromise then leaks
   short-lived material, not permanent keys. For upstreams stuck on static
   keys the vault passes the value through (only the at-rest / audit /
   revocation benefits apply, not lifetime reduction) — this residue is
   accepted and documented, not solved.

Even a plain per-request fetch (no derivation) still buys **at-rest**
reduction (process memory holds only in-flight secrets), a **detection /
rate-limit / revocation chokepoint**, and clean **cross-component
scoping** (an egress RCE can't request git-gate's creds). It does *not*
prevent abuse of currently-authorized access during the compromise window.

**Mechanism (see #355):** generalize the existing `DeployKeyProvisioner`
(PRD 0048) into a user-extensible **`SecretProvider`** droppable into the
manifest anywhere a raw token value is accepted, discovered from
`~/.bot-bottle/contrib/<name>/secret_provider.py` exactly as user
`AgentProvider`s are — because maintaining every forge/cloud/OAuth
provider in-tree is untenable. The orchestrator's vault mints via these
providers; but the abstraction is shippable independently and today's
per-bottle sidecars can use it too.

## Design

### The contract (backend-agnostic)

Three surfaces; only one is per-backend.

1. **Control plane (CLI / console → orchestrator)** — an RPC:
   `launch_bottle`, `teardown_bottle`, `register_policy`,
   `deregister_bottle`, `supervise_queue`. Fully backend-agnostic. Both the
   local `cli.py` and the remote console funnel through it, so policy is
   uniform and `cli.py` becomes a thin client rather than a parallel
   launcher. **Transport: HTTP** — the most universal/reliable choice on
   every host (no vsock/unix-socket portability caveats); a local unix
   socket is a fine optimization, but HTTP is the wire contract.
2. **Data plane (agent → orchestrator)** — the egress / git / supervise
   endpoints. Already agnostic today (agents dial `http://sidecar:9099`);
   only the *address* and *how packets get there* are per-backend.
3. **Launch / wire (orchestrator → backend)** — the irreducibly
   backend-specific part; lives on `BottleBackend`.

### One `Orchestrator`, no subclass tree

The orchestrator is a **single concrete class** holding all the
backend-neutral logic — egress addon, git-gate, supervise, source-IP
attribution, live-reload control plane, console client. It never branches
on backend; it *composes* a `BottleBackend`. That composition is what makes
the contract agnostic: there is nothing backend-specific left in the
orchestrator to leak.

Rejected alternative: an `Orchestrator` ABC with per-backend
implementations. The interesting logic (proxies, attribution, control
plane) is backend-neutral, so three subclasses would triplicate the hard
part; and a second hierarchy paralleling `BottleBackend` reintroduces the
same hand-maintained lockstep coupling we just removed from the netpool
constants (PR #350). Composition over a parallel tree.

### `BottleBackend` absorbs the per-backend variation

A small, cohesive surface — reused for launching agent bottles *and* the
orchestrator's own unit (the orchestrator is just another native unit):

```
launch_unit(spec) -> Handle      # agent bottle OR the orchestrator itself
                                 # (fc microVM / apple ctr / docker ctr)
wire(unit, endpoint) -> None     # DNAT+forward (fc) | attach shared net (docker/apple)
endpoint_of(unit) -> Endpoint    # address resolution
health(unit) -> Status
```

Plus the **launch broker** — the answer to "a VM/container can't spawn its
own host-network siblings." The orchestrator can't directly open host
`/dev/kvm` + a host TAP fd (Firecracker), and a container can't spawn
siblings without a root-equivalent socket (Docker). So every backend
exposes a broker the orchestrator calls to launch an agent:

- **Firecracker** — a thin, structured host shim (see security #3). This
  replaces today's implicit "launcher runs as host user."
- **Docker** — the socket today (fat, root-equivalent — the thing 0069's
  Stage 3 removes); a narrower broker later.
- **Apple** — the `container` CLI/daemon.

**Broker request schema:** human-readable JSON of **static flags + ids
only** (never free-form paths/argv — that's what keeps it un-coercible
into arbitrary launches), wrapped as a **signed JWT** so the broker
verifies *provenance*: the request came from the real orchestrator, not a
forged one from a compromised co-located component. The orchestrator signs;
the broker verifies with the orchestrator's public key (provisioned at
broker install). This is the concrete form of security #3's "structured
requests only." Same JSON-with-ids + JWT shape for all
orchestrator↔broker/sidecar communication; cases that don't fit get
handled as they arise, with this as the default.

If `BottleBackend` bloats, the pressure valve is composition one level
down: vend a `backend.network()` / `Wiring` collaborator rather than
piling methods on — the same discipline, recursed.

### State: one SQLite DB, owned by the orchestrator

The orchestrator is the natural owner of per-host **runtime state**:

- pool **slot leases** (which bottle holds slot *i*) — replaces today's
  `fcntl`-locked files with SQLite transactions;
- the **supervise approval queue** + remembered approvals;
- the **live bottle registry** (source IP → bottle → policy/secrets refs),
  the lookup table the attribution invariant reads.

This is deliberately **not** a "single source of truth for all config."
Config splits into three tiers with different homes:

| Tier | Example | Home |
|---|---|---|
| Build-time constants | pool size, IP base, nft table | flat `.env` (PR #350) — must be readable by Nix eval + root bash, zero runtime |
| User-authored config | bottle manifests, egress routes, secret refs | declarative files under `~/.bot-bottle/` — trust boundary at `$HOME`, git-trackable, "unknown keys die at load" |
| Runtime state | slot leases, approvals, registry | one shared **`bot-bottle.db`**, solely owned by the orchestrator |

SQLite is right for the runtime tier (mutable, concurrent, queried) and
wrong for the other two (Nix can't read it at eval time; it fights the
declarative manifest trust model). Keep the tiers separate.

**One shared `bot-bottle.db` for all runtime state** (decided in review).
The registry co-tenants the existing host `bot-bottle.db` — the `DbStore`
framework already namespaces each store by `schema_key`, so slot
leases / approvals / registry share one file. One place to query, back up,
and integrate a console against.

- **Host-resident, for durability.** Re-adoption sweeps the registry after
  an orchestrator restart, so state *must* outlive the orchestrator
  instance. The file lives on the host (`bot_bottle_root()/db/bot-bottle.db`);
  the orchestrator unit reaches it, it doesn't carry it.
- **Integrity by sole ownership, not mount permissions.** Agents can't
  touch the DB directly wherever it lives (network-isolated in their
  bottles). The risk is a *compromised agent-facing data-plane service*
  (egress/git-gate, which parse hostile bytes) writing the registry and
  forging attribution. Because it's now one shared file, coarse `ro`/`rw`
  mount-splitting no longer isolates the registry — so the rule is stronger
  and simpler: **only the orchestrator (control plane) opens `bot-bottle.db`;
  the data plane and the console reach state through the control-plane RPC,
  never a direct file handle.** No agent-facing component gets the file, so
  none can forge attribution. (This supersedes the earlier `ro`-mount idea.)
  - *Transitional caveat:* today the per-bottle **supervise sidecar
    rw-bind-mounts `bot-bottle.db`** to write proposals — exactly the
    pattern the orchestrator removes (supervise consolidates into the
    orchestrator; sidecar writes become RPC calls). Until that lands, don't
    put the attribution registry behind a data-plane-writable mount.

Implementation note for the VM slices: SQLite **WAL** over a guest share
(virtiofs/9p) is finicky (the `-shm`/`-wal` files need real mmap/locking),
which is a second reason the DB wants a **host-side owner** the orchestrator
reaches over the RPC rather than a shared mount into the VM. WAL on the
shared DB is therefore a deliberate, tested future change — not enabled ad
hoc. `sqlite3` itself is stdlib, so "the host needs SQLite" is a non-cost.

## Sequencing

Jump straight to the **virtualized** end state (not a host-daemon stepping
stone): a host daemon's agent→`localhost` transport is throwaway once the
orchestrator becomes a VM. Decouple the two risks instead:

- **Consolidation risk** (one process, all secrets, attribution, reload)
  and **packaging/transport risk** (VM-to-VM wiring, the shim) are
  independent. Develop the orchestrator **service as a plain process
  dev-harness** first, so the consolidation logic (attribution, reload,
  secret handling) is proven with fast iteration — *then* wrap that exact
  service in the VM and solve wiring separately.

### Slices

Built bottom-up as a stack of small PRs off this one (status: **1–4
implemented** in #352 → #356 → #357 → #358):

1. **Dev-harness core** ✅ — SQLite registry (runtime state) + fail-closed
   attribution (source IP + identity token) + the HTTP control plane.
2. **Launch lifecycle + broker contract** ✅ — `launch_bottle` /
   `teardown_bottle` + the signed, structured `LaunchBroker` request
   (HS256 JWT, static ids/flags only, provenance-verified, fail-closed),
   with a stub broker for the harness and registry rollback on failure.
3. **Docker launch broker** ✅ — the first real broker: `docker run` / `rm`
   per bottle, built only from the request's static fields; proves the
   `BottleBackend` seam on the cheapest backend.
4. **Consolidated per-host sidecar** ✅ — one persistent sidecar shared by
   all bottles (idempotent singleton) instead of one per bottle — the core
   consolidation win, source-IP-keyed via the attribution invariant.
5. **Firecracker broker + sidecar** — the real work: the shim + VM-to-VM
   routing (host forwards `bbfcN` → orchestrator TAP; the nft table grows
   forward rules where today it drops all non-DNAT egress), against the
   already-proven dev-harness.
6. **macOS (Apple container)** — last (container-to-container networking).

Backends land cheapest-first (docker → firecracker → macOS); keep the
sidecar **service one shared thing** throughout.

## Non-goals

- Removing OCI/Dockerfile support for agent images (0069's concern).
- A single database for *all* config (see the three-tier table).
- Changing the per-bottle isolation of agent workloads — only the sidecar
  is consolidated; agents stay one-VM/container-each.

## Relationship to other work

- **PRD 0069 (#348):** 0070 subsumes its Stage 1 (per-host sidecar) and
  Stage 4 (sidecar-as-VM). 0069 retains Stage 2 (nix-built fixed images —
  a **dependency** here: the orchestrator and agent base must be
  nix-built so the broker launches from a fixed image set and bootstrapping
  has no chicken-and-egg) and Stage 3 (in-VM Dockerfile builder).
- **Minimal CI runner (paused):** the Firecracker broker + no host Docker
  is what lets a dedicated `gitea` runner user drop the root-equivalent
  `docker` group — it only needs broker-socket access + `kvm`/pool group
  membership. This work unblocks it.
- **Generic `SecretProvider` (#355):** the future secret-handling
  mechanism (see "Secret handling") — generalizes PRD 0048's
  `DeployKeyProvisioner` into a user-extensible provider that mints
  short-lived creds. Shippable independently; the orchestrator's vault
  mints through it.
- **PR #350 (netpool single-source):** the same "one source per fact,
  composition over parallel hierarchies" discipline the contract follows.

## Decisions (review 2026-07-13)

- **Egress sharing:** **consolidate egress** (worth it). Treat it as a
  minimal, hardened attack surface for a malicious agent rather than
  keeping it per-bottle; pair it with the identity token above and the
  short-lived-vault-token mitigations (Secret handling / #355).
- **Control-plane transport:** **HTTP** — most universal/reliable on every
  host; unix socket is an optional local optimization (see the contract).
- **Broker request schema:** **signed-JWT JSON, static flags + ids only**
  (see the launch broker) — provenance + un-coercible by construction.
- **State re-adoption:** the restart procedure is:
  1. **Singleton:** a new orchestrator launch requires no pre-existing
     orchestrator; any found (healthy or not) is fully shut down first.
  2. Re-adoption **waits for the new orchestrator to be healthy**.
  3. Once healthy, it discovers all agents needing adoption via **both the
     SQLite registry and live process/VM inspection** *before serving any
     other request*. The two-source sweep is what closes the *in-flight
     launch* race — a launch that started (VM booting / slot claimed) but
     hadn't committed to SQLite when the old orchestrator died would be
     invisible to a SQLite-only sweep; process inspection catches it.
     Launches should also write an **intent record ahead of committing
     resources** so the sweep can reconcile intent vs. actual.

## Open questions

- **VM-to-VM routing:** per-backend, and in the design it *is*
  `BottleBackend.wire()` (DNAT+forward for fc, shared-net for
  docker/apple), not orchestrator logic. **Not a blocker** — resolvable at
  implementation time (may require a bit of host modification, as the pool
  setup already does on NixOS); an earlier "sidecars in VMs" spike showed
  it's feasible.
- **Live-reload protocol** for per-bottle policy over the HTTP control
  plane (add/remove routes/keys/proposals without a restart).
- **Identity-token delivery:** exactly how the per-bottle token is placed
  where the agent can present it but not swap in another bottle's.