# PRD 0070: Per-host orchestrator service - **Status:** Draft - **Author:** Claude - **Created:** 2026-07-12 - **Issue:** #351 - **Supersedes:** the Stage-1 / Stage-4 sidecar-consolidation framing of PRD 0069 (#348). Depends on 0069's nix-built fixed images (Stage 2) for bootstrapping; 0069 still owns the docker-free image-building work. ## Summary Replace the **per-bottle gateway bundle** with a single **persistent, per-host orchestrator**: one long-lived service that runs the gateway functions (egress / git-gate / supervise), coordinates with the console, and brokers agent launches and teardown. It is **virtualized from the start** using each backend's native isolation primitive — a Firecracker microVM on the Firecracker backend, an Apple container on macOS, a Docker container on the legacy backend — and is fronted by a single **backend-agnostic contract**. Per-backend variation lives on `BottleBackend`, not in the orchestrator. ## Motivation Today each bottle spins up its own gateway bundle (egress mitmproxy + git-gate + supervise). That costs: - **Resources.** N bottles → N heavy bundles booting and idling. - **Operational churn.** Per-launch container/VM lifecycle for the gateways, a control path baked at launch and torn down at exit. - **A blurry contract.** "How a bottle talks to its gateway" is re-implemented per backend instead of being one agreed interface. A per-host orchestrator collapses the first two and forces the third to be made explicit. It's also the component that will own per-host runtime **state** (slot leases, the approval queue, the bottle registry) — today that's ad-hoc `fcntl`-locked files. ## Security review (read this first) Consolidation is a real change to the trust model. The goal is to **not significantly weaken** the posture; some properties strengthen, some weaken, and the weakened ones must be mitigated by design, not hand-waved. ### What gets stronger - **Build/host isolation of untrusted inputs** (with 0069 Stage 3): user Dockerfiles build in a disposable VM instead of on the host. - **One audited privileged surface.** Today the launcher runs as the full host user and needs the Docker socket (root-equivalent). The orchestrator model replaces that with a **thin launch broker** (below) — a small, structured, auditable privileged core instead of a fat socket. - **Attribution is enforced, not assumed.** Making source-IP identity a first-class contract invariant (below) means each backend must *prove* it, rather than the gateway implicitly trusting network position. ### What gets weaker, and the mitigation 1. **Secret concentration.** Per-bottle gateways isolate secrets at the process boundary — each holds only its bottle's tokens/keys. A host orchestrator concentrates **every bottle's** egress tokens, git deploy keys, and the console credential in one long-lived process. A single attribution bug leaks bottle A's token into bottle B's request — a class of bug that *cannot exist* per-bottle. - *Mitigation:* lean on the enforced source-IP invariant for attribution; keep the most secret-dense, least-shareable service (**git-gate**, per-repo deploy keys, no natural source-IP scoping) **per-bottle** unless there's a compelling reason; scope each secret to the bottle in the state DB so a lookup can't return the wrong bottle's secret by construction (key every secret access by the verified source identity, never by ambient state). 2. **Shared fate.** Orchestrator down = no new launches, and running agents lose egress / git / supervise. Compromise = the whole host's fleet, plus launch authority, plus the console token. - *Mitigation:* the orchestrator is itself confined (its own VM/container with its own fail-closed egress); make it **restartable without killing running agent VMs** (agents keep running; they briefly lose gateway connectivity until it's back); persist state to a host volume so a restart re-adopts live bottles rather than losing them. 3. **The launch broker is the new privileged core.** We don't eliminate host privilege — we shrink and relocate it. If the broker accepts arbitrary paths/commands, the orchestrator VM can escape through it. - *Mitigation:* the broker takes **structured requests only** — "launch bottle from *this* content-addressed, nix-built rootfs on TAP slot *k*", never "run this argv". It validates against a fixed image set, not caller-supplied paths. It is small enough to audit line-by-line. 4. **The egress proxy now parses every bottle's traffic in one process.** Higher blast radius for a mitmproxy/TLS-bump bug. - *Mitigation:* this is the argument for virtualizing the orchestrator from the start (Stage B, not a host daemon) — the code that TLS-bumps and parses agent traffic and holds every token runs **inside its own confined VM**, not as a host process. If egress sharing's blast radius feels too high, egress can stay per-bottle while supervise (near-zero secrets) goes host-level first. ### The attribution invariant Source-IP attribution is what makes a shared orchestrator safe: one process serves every bottle and tells them apart by source address. The *mechanism* is identical everywhere (read source IP → look up bottle); the **guarantee that the address can't be forged is a per-backend responsibility** and part of the contract: > **Invariant:** a packet's source address, as seen by the orchestrator, > *provably* identifies the originating bottle. - **Firecracker** — enforced by the `/31` point-to-point TAP + the `bot_bottle_fc` nft table (strongest; already built). - **Docker** — the per-bottle `--internal` network + anti-spoof; weaker, must be made explicit. - **Apple** — the host-only network. If a backend can't honor the invariant, source-IP consolidation is not safe there and that backend keeps per-bottle gateways. The invariant is a hard precondition, not an aspiration. **Defense-in-depth — a per-bottle identity token.** On top of the network-layer invariant, inject a per-bottle secret token into every request the agent makes to the orchestrator (the agent already egresses through the gateway proxy, so this is cheap to add). It gives an **application-layer** proof of identity independent of the network layer: - On **Firecracker** the `/31` + nft already make source IP unspoofable *by construction*, so the token is belt-and-suspenders there — but cheap insurance against a misconfigured invariant. - On **weaker backends** (Docker) it is load-bearing, providing attribution that doesn't lean on network anti-spoof. Requirements: the token is **per-bottle, unguessable, and non-cross-leakable** — a bottle can only ever prove it is *itself* (it can't learn another bottle's token, given bottle isolation), so a hostile agent gains nothing by presenting it. The orchestrator provisions the token at launch and the gateway requires it to attribute + authorize. Note this hardens *attribution*, not *secret exposure*: it's app-layer, so a compromised orchestrator still sees every token (that's the concentration problem, addressed separately under Secret handling). ### Secret handling — a FUTURE pattern (not v1) > **Status: future / not required for the initial orchestrator.** The > initial cut can inject secrets as today; this section records the > direction so v1 doesn't paint itself into a corner. Tracked as its own > work in **#355** (generic `SecretProvider`). The residual weakness after all of the above is **long-lived credential concentration** — the data-plane proxies must hold every bottle's upstream tokens because the agent must never see them. You can't policy-gate the component whose job is to *use* all the secrets, so a full RCE of a proxy drains its authorized set regardless. Two moves bound this without pretending to prevent it: 1. **Vault as a separate trust domain.** The long-lived *roots* live in a distinct process (ideally its own VM) that the byte-parsing data plane never shares memory with. The proxies request secrets from it; the crown jewels are not in the process an agent-facing parser can pop. 2. **Derive short-lived, scoped creds where the upstream allows it.** The vault holds the root and mints expiring, narrowly-scoped credentials per bottle-start (or per request) — GitHub App installation tokens, OAuth/STS token exchange, forge deploy tokens. A compromise then leaks short-lived material, not permanent keys. For upstreams stuck on static keys the vault passes the value through (only the at-rest / audit / revocation benefits apply, not lifetime reduction) — this residue is accepted and documented, not solved. Even a plain per-request fetch (no derivation) still buys **at-rest** reduction (process memory holds only in-flight secrets), a **detection / rate-limit / revocation chokepoint**, and clean **cross-component scoping** (an egress RCE can't request git-gate's creds). It does *not* prevent abuse of currently-authorized access during the compromise window. **Mechanism (see #355):** generalize the existing `DeployKeyProvisioner` (PRD 0048) into a user-extensible **`SecretProvider`** droppable into the manifest anywhere a raw token value is accepted, discovered from `~/.bot-bottle/contrib//secret_provider.py` exactly as user `AgentProvider`s are — because maintaining every forge/cloud/OAuth provider in-tree is untenable. The orchestrator's vault mints via these providers; but the abstraction is shippable independently and today's per-bottle gateways can use it too. ## Design ### The contract (backend-agnostic) Three surfaces; only one is per-backend. 1. **Control plane (CLI / console → orchestrator)** — an RPC: `launch_bottle`, `teardown_bottle`, `register_policy`, `deregister_bottle`, `supervise_queue`. Fully backend-agnostic. Both the local `cli.py` and the remote console funnel through it, so policy is uniform and `cli.py` becomes a thin client rather than a parallel launcher. **Transport: HTTP** — the most universal/reliable choice on every host (no vsock/unix-socket portability caveats); a local unix socket is a fine optimization, but HTTP is the wire contract. 2. **Data plane (agent → orchestrator)** — the egress / git / supervise endpoints. Already agnostic today (agents dial `http://gateway:9099`); only the *address* and *how packets get there* are per-backend. 3. **Launch / wire (orchestrator → backend)** — the irreducibly backend-specific part; lives on `BottleBackend`. ### One `Orchestrator`, no subclass tree The orchestrator is a **single concrete class** holding all the backend-neutral logic — egress addon, git-gate, supervise, source-IP attribution, live-reload control plane, console client. It never branches on backend; it *composes* a `BottleBackend`. That composition is what makes the contract agnostic: there is nothing backend-specific left in the orchestrator to leak. Rejected alternative: an `Orchestrator` ABC with per-backend implementations. The interesting logic (proxies, attribution, control plane) is backend-neutral, so three subclasses would triplicate the hard part; and a second hierarchy paralleling `BottleBackend` reintroduces the same hand-maintained lockstep coupling we just removed from the netpool constants (PR #350). Composition over a parallel tree. ### `BottleBackend` absorbs the per-backend variation A small, cohesive surface — reused for launching agent bottles *and* the orchestrator's own unit (the orchestrator is just another native unit): ``` launch_unit(spec) -> Handle # agent bottle OR the orchestrator itself # (fc microVM / apple ctr / docker ctr) wire(unit, endpoint) -> None # DNAT+forward (fc) | attach shared net (docker/apple) endpoint_of(unit) -> Endpoint # address resolution health(unit) -> Status ``` Plus the **launch broker** — the answer to "a VM/container can't spawn its own host-network siblings." The orchestrator can't directly open host `/dev/kvm` + a host TAP fd (Firecracker), and a container can't spawn siblings without a root-equivalent socket (Docker). So every backend exposes a broker the orchestrator calls to launch an agent: - **Firecracker** — a thin, structured host shim (see security #3). This replaces today's implicit "launcher runs as host user." - **Docker** — the socket today (fat, root-equivalent — the thing 0069's Stage 3 removes); a narrower broker later. - **Apple** — the `container` CLI/daemon. **Broker request schema:** human-readable JSON of **static flags + ids only** (never free-form paths/argv — that's what keeps it un-coercible into arbitrary launches), wrapped as a **signed JWT** so the broker verifies *provenance*: the request came from the real orchestrator, not a forged one from a compromised co-located component. The orchestrator signs; the broker verifies with the orchestrator's public key (provisioned at broker install). This is the concrete form of security #3's "structured requests only." Same JSON-with-ids + JWT shape for all orchestrator↔broker/gateway communication; cases that don't fit get handled as they arise, with this as the default. If `BottleBackend` bloats, the pressure valve is composition one level down: vend a `backend.network()` / `Wiring` collaborator rather than piling methods on — the same discipline, recursed. ### State: one SQLite DB, owned by the orchestrator The orchestrator is the natural owner of per-host **runtime state**: - pool **slot leases** (which bottle holds slot *i*) — replaces today's `fcntl`-locked files with SQLite transactions; - the **supervise approval queue** + remembered approvals; - the **live bottle registry** (source IP → bottle → policy/secrets refs), the lookup table the attribution invariant reads. This is deliberately **not** a "single source of truth for all config." Config splits into three tiers with different homes: | Tier | Example | Home | |---|---|---| | Build-time constants | pool size, IP base, nft table | flat `.env` (PR #350) — must be readable by Nix eval + root bash, zero runtime | | User-authored config | bottle manifests, egress routes, secret refs | declarative files under `~/.bot-bottle/` — trust boundary at `$HOME`, git-trackable, "unknown keys die at load" | | Runtime state | slot leases, approvals, registry | one shared **`bot-bottle.db`**, solely owned by the orchestrator | SQLite is right for the runtime tier (mutable, concurrent, queried) and wrong for the other two (Nix can't read it at eval time; it fights the declarative manifest trust model). Keep the tiers separate. **One shared `bot-bottle.db` for all runtime state** (decided in review). The registry co-tenants the existing host `bot-bottle.db` — the `DbStore` framework already namespaces each store by `schema_key`, so slot leases / approvals / registry share one file. One place to query, back up, and integrate a console against. - **Host-resident, for durability.** Re-adoption sweeps the registry after an orchestrator restart, so state *must* outlive the orchestrator instance. The file lives on the host (`bot_bottle_root()/db/bot-bottle.db`); the orchestrator unit reaches it, it doesn't carry it. - **Integrity by sole ownership, not mount permissions.** Agents can't touch the DB directly wherever it lives (network-isolated in their bottles). The risk is a *compromised agent-facing data-plane service* (egress/git-gate, which parse hostile bytes) writing the registry and forging attribution. Because it's now one shared file, coarse `ro`/`rw` mount-splitting no longer isolates the registry — so the rule is stronger and simpler: **only the orchestrator (control plane) opens `bot-bottle.db`; the data plane and the console reach state through the control-plane RPC, never a direct file handle.** No agent-facing component gets the file, so none can forge attribution. (This supersedes the earlier `ro`-mount idea.) - *Transitional caveat:* today the per-bottle **supervise gateway rw-bind-mounts `bot-bottle.db`** to write proposals — exactly the pattern the orchestrator removes (supervise consolidates into the orchestrator; gateway writes become RPC calls). Until that lands, don't put the attribution registry behind a data-plane-writable mount. Implementation note for the VM slices: SQLite **WAL** over a guest share (virtiofs/9p) is finicky (the `-shm`/`-wal` files need real mmap/locking), which is a second reason the DB wants a **host-side owner** the orchestrator reaches over the RPC rather than a shared mount into the VM. WAL on the shared DB is therefore a deliberate, tested future change — not enabled ad hoc. `sqlite3` itself is stdlib, so "the host needs SQLite" is a non-cost. ### Gateway CA: host-resident, like the DB The shared gateway bumps TLS with a self-generated mitmproxy CA, and **every bottle installs that CA** into its trust store to accept the bumped leaves. So the CA is durable per-host state with the same rule as the DB: it must outlive any single gateway container, or a restart mints a fresh CA that every already-running bottle distrusts — the TLS handshake then fails even after the bottle re-resolves and reconnects to the moved gateway (issue #450, a re-attachment blocker distinct from #443/#445). The CA lives on the **host filesystem** at `bot_bottle_root()/gateway-ca` (`host_gateway_ca_dir()`), bind-mounted into the container at mitmproxy's confdir. This is deliberately a host bind-mount, **not a Docker named volume**: a named volume survives `docker rm` but is silently wiped by `docker volume prune` / `docker system prune --volumes` during routine host maintenance, which is exactly how the ephemeral-CA symptom shows up in practice. A path under the app-data root is never pruned by docker, and stays directly inspectable and rotatable from the host. mitmproxy reuses an existing CA and generates one only on first run, so the bind-mount alone gives "adopt-existing, generate-on-first-run" for free. **Deliberate rollover** is the explicit inverse: `rotate_gateway_ca()` removes the persisted CA material so the next start remints it, and the `python -m bot_bottle.orchestrator.rotate_ca` one-shot wires that together with dropping the running gateway container (whose mitmproxy still holds the old CA in memory). Rotation does not auto-re-provision the new CA into running bottles — those re-attach to install the new anchor — so it is an operator action with a brief egress interruption, never an implicit one. ## Sequencing Jump straight to the **virtualized** end state (not a host-daemon stepping stone): a host daemon's agent→`localhost` transport is throwaway once the orchestrator becomes a VM. Decouple the two risks instead: - **Consolidation risk** (one process, all secrets, attribution, reload) and **packaging/transport risk** (VM-to-VM wiring, the shim) are independent. Develop the orchestrator **service as a plain process dev-harness** first, so the consolidation logic (attribution, reload, secret handling) is proven with fast iteration — *then* wrap that exact service in the VM and solve wiring separately. Backend order (cheapest proof → hardest → last): 1. **Docker orchestrator** — nearly free (the gateway bundle is already containers; collapse N bundles into one persistent container). Proves consolidation + the `BottleBackend` seam with the least moving parts. 2. **Firecracker orchestrator** — the real work: the shim + VM-to-VM routing (host forwards `bbfcN` → orchestrator TAP; the nft table grows forward rules where today it drops all non-DNAT egress). Built against the dev-harness so the app logic is already proven. 3. **macOS (Apple container)** — last (container-to-container networking). Keep the gateway **service one shared thing** throughout. ## Non-goals - Removing OCI/Dockerfile support for agent images (0069's concern). - A single database for *all* config (see the three-tier table). - Changing the per-bottle isolation of agent workloads — only the gateway is consolidated; agents stay one-VM/container-each. ## Relationship to other work - **PRD 0069 (#348):** 0070 subsumes its Stage 1 (per-host sidecar) and Stage 4 (sidecar-as-VM). 0069 retains Stage 2 (nix-built fixed images — a **dependency** here: the orchestrator and agent base must be nix-built so the broker launches from a fixed image set and bootstrapping has no chicken-and-egg) and Stage 3 (in-VM Dockerfile builder). - **Minimal CI runner (paused):** the Firecracker broker + no host Docker is what lets a dedicated `gitea` runner user drop the root-equivalent `docker` group — it only needs broker-socket access + `kvm`/pool group membership. This work unblocks it. - **Generic `SecretProvider` (#355):** the future secret-handling mechanism (see "Secret handling") — generalizes PRD 0048's `DeployKeyProvisioner` into a user-extensible provider that mints short-lived creds. Shippable independently; the orchestrator's vault mints through it. - **PR #350 (netpool single-source):** the same "one source per fact, composition over parallel hierarchies" discipline the contract follows. ## Decisions (review 2026-07-13) - **Egress sharing:** **consolidate egress** (worth it). Treat it as a minimal, hardened attack surface for a malicious agent rather than keeping it per-bottle; pair it with the identity token above and the short-lived-vault-token mitigations (Secret handling / #355). - **Control-plane transport:** **HTTP** — most universal/reliable on every host; unix socket is an optional local optimization (see the contract). - **Broker request schema:** **signed-JWT JSON, static flags + ids only** (see the launch broker) — provenance + un-coercible by construction. - **State re-adoption:** the restart procedure is: 1. **Singleton:** a new orchestrator launch requires no pre-existing orchestrator; any found (healthy or not) is fully shut down first. 2. Re-adoption **waits for the new orchestrator to be healthy**. 3. Once healthy, it discovers all agents needing adoption via **both the SQLite registry and live process/VM inspection** *before serving any other request*. The two-source sweep is what closes the *in-flight launch* race — a launch that started (VM booting / slot claimed) but hadn't committed to SQLite when the old orchestrator died would be invisible to a SQLite-only sweep; process inspection catches it. Launches should also write an **intent record ahead of committing resources** so the sweep can reconcile intent vs. actual. ## Open questions - **VM-to-VM routing:** per-backend, and in the design it *is* `BottleBackend.wire()` (DNAT+forward for fc, shared-net for docker/apple), not orchestrator logic. **Not a blocker** — resolvable at implementation time (may require a bit of host modification, as the pool setup already does on NixOS); an earlier "gateways in VMs" spike showed it's feasible. - **Live-reload protocol** for per-bottle policy over the HTTP control plane (add/remove routes/keys/proposals without a restart). - **Identity-token delivery:** exactly how the per-bottle token is placed where the agent can present it but not swap in another bottle's.