bot-bottle/docs/prds/0070-per-host-orchestrator.md

PRD 0070: Per-host orchestrator service

• Status: Draft
• Author: Claude
• Created: 2026-07-12
• Issue: #351
• Supersedes: the Stage-1 / Stage-4 sidecar-consolidation framing of PRD 0069 (#348). Depends on 0069's nix-built fixed images (Stage 2) for bootstrapping; 0069 still owns the docker-free image-building work.

Summary

Replace the per-bottle sidecar bundle with a single persistent, per-host orchestrator: one long-lived service that runs the sidecar functions (egress / git-gate / supervise), coordinates with the console, and brokers agent launches and teardown. It is virtualized from the start using each backend's native isolation primitive — a Firecracker microVM on the Firecracker backend, an Apple container on macOS, a Docker container on the legacy backend — and is fronted by a single backend-agnostic contract. Per-backend variation lives on BottleBackend, not in the orchestrator.

Motivation

Today each bottle spins up its own sidecar bundle (egress mitmproxy + git-gate + supervise). That costs:

• Resources. N bottles → N heavy bundles booting and idling.
• Operational churn. Per-launch container/VM lifecycle for the sidecars, a control path baked at launch and torn down at exit.
• A blurry contract. "How a bottle talks to its sidecar" is re-implemented per backend instead of being one agreed interface.

A per-host orchestrator collapses the first two and forces the third to be made explicit. It's also the component that will own per-host runtime state (slot leases, the approval queue, the bottle registry) — today that's ad-hoc fcntl-locked files.

Security review (read this first)

Consolidation is a real change to the trust model. The goal is to not significantly weaken the posture; some properties strengthen, some weaken, and the weakened ones must be mitigated by design, not hand-waved.

What gets stronger

• Build/host isolation of untrusted inputs (with 0069 Stage 3): user Dockerfiles build in a disposable VM instead of on the host.
• One audited privileged surface. Today the launcher runs as the full host user and needs the Docker socket (root-equivalent). The orchestrator model replaces that with a thin launch broker (below) — a small, structured, auditable privileged core instead of a fat socket.
• Attribution is enforced, not assumed. Making source-IP identity a first-class contract invariant (below) means each backend must prove it, rather than the sidecar implicitly trusting network position.

What gets weaker, and the mitigation

1. Secret concentration. Per-bottle sidecars isolate secrets at the process boundary — each holds only its bottle's tokens/keys. A host orchestrator concentrates every bottle's egress tokens, git deploy keys, and the console credential in one long-lived process. A single attribution bug leaks bottle A's token into bottle B's request — a class of bug that cannot exist per-bottle.

   • Mitigation: lean on the enforced source-IP invariant for attribution; keep the most secret-dense, least-shareable service (git-gate, per-repo deploy keys, no natural source-IP scoping) per-bottle unless there's a compelling reason; scope each secret to the bottle in the state DB so a lookup can't return the wrong bottle's secret by construction (key every secret access by the verified source identity, never by ambient state).

2. Shared fate. Orchestrator down = no new launches, and running agents lose egress / git / supervise. Compromise = the whole host's fleet, plus launch authority, plus the console token.

   • Mitigation: the orchestrator is itself confined (its own VM/container with its own fail-closed egress); make it restartable without killing running agent VMs (agents keep running; they briefly lose sidecar connectivity until it's back); persist state to a host volume so a restart re-adopts live bottles rather than losing them.

3. The launch broker is the new privileged core. We don't eliminate host privilege — we shrink and relocate it. If the broker accepts arbitrary paths/commands, the orchestrator VM can escape through it.

   • Mitigation: the broker takes structured requests only — "launch bottle from this content-addressed, nix-built rootfs on TAP slot k", never "run this argv". It validates against a fixed image set, not caller-supplied paths. It is small enough to audit line-by-line.

4. The egress proxy now parses every bottle's traffic in one process. Higher blast radius for a mitmproxy/TLS-bump bug.

   • Mitigation: this is the argument for virtualizing the orchestrator from the start (Stage B, not a host daemon) — the code that TLS-bumps and parses agent traffic and holds every token runs inside its own confined VM, not as a host process. If egress sharing's blast radius feels too high, egress can stay per-bottle while supervise (near-zero secrets) goes host-level first.

The attribution invariant

Source-IP attribution is what makes a shared orchestrator safe: one process serves every bottle and tells them apart by source address. The mechanism is identical everywhere (read source IP → look up bottle); the guarantee that the address can't be forged is a per-backend responsibility and part of the contract:

Invariant: a packet's source address, as seen by the orchestrator, provably identifies the originating bottle.

• Firecracker — enforced by the /31 point-to-point TAP + the bot_bottle_fc nft table (strongest; already built).
• Docker — the per-bottle --internal network + anti-spoof; weaker, must be made explicit.
• Apple — the host-only network.

If a backend can't honor the invariant, source-IP consolidation is not safe there and that backend keeps per-bottle sidecars. The invariant is a hard precondition, not an aspiration.

Defense-in-depth — a per-bottle identity token. On top of the network-layer invariant, inject a per-bottle secret token into every request the agent makes to the orchestrator (the agent already egresses through the sidecar proxy, so this is cheap to add). It gives an application-layer proof of identity independent of the network layer:

• On Firecracker the /31 + nft already make source IP unspoofable by construction, so the token is belt-and-suspenders there — but cheap insurance against a misconfigured invariant.
• On weaker backends (Docker) it is load-bearing, providing attribution that doesn't lean on network anti-spoof.

Requirements: the token is per-bottle, unguessable, and non-cross-leakable — a bottle can only ever prove it is itself (it can't learn another bottle's token, given bottle isolation), so a hostile agent gains nothing by presenting it. The orchestrator provisions the token at launch and the sidecar requires it to attribute + authorize. Note this hardens attribution, not secret exposure: it's app-layer, so a compromised orchestrator still sees every token (that's the concentration problem, addressed separately under Secret handling).

Secret handling — a FUTURE pattern (not v1)

Status: future / not required for the initial orchestrator. The initial cut can inject secrets as today; this section records the direction so v1 doesn't paint itself into a corner. Tracked as its own work in #355 (generic SecretProvider).

The residual weakness after all of the above is long-lived credential concentration — the data-plane proxies must hold every bottle's upstream tokens because the agent must never see them. You can't policy-gate the component whose job is to use all the secrets, so a full RCE of a proxy drains its authorized set regardless. Two moves bound this without pretending to prevent it:

1. Vault as a separate trust domain. The long-lived roots live in a distinct process (ideally its own VM) that the byte-parsing data plane never shares memory with. The proxies request secrets from it; the crown jewels are not in the process an agent-facing parser can pop.
2. Derive short-lived, scoped creds where the upstream allows it. The vault holds the root and mints expiring, narrowly-scoped credentials per bottle-start (or per request) — GitHub App installation tokens, OAuth/STS token exchange, forge deploy tokens. A compromise then leaks short-lived material, not permanent keys. For upstreams stuck on static keys the vault passes the value through (only the at-rest / audit / revocation benefits apply, not lifetime reduction) — this residue is accepted and documented, not solved.

Even a plain per-request fetch (no derivation) still buys at-rest reduction (process memory holds only in-flight secrets), a detection / rate-limit / revocation chokepoint, and clean cross-component scoping (an egress RCE can't request git-gate's creds). It does not prevent abuse of currently-authorized access during the compromise window.

Mechanism (see #355): generalize the existing DeployKeyProvisioner (PRD 0048) into a user-extensible SecretProvider droppable into the manifest anywhere a raw token value is accepted, discovered from ~/.bot-bottle/contrib/<name>/secret_provider.py exactly as user AgentProviders are — because maintaining every forge/cloud/OAuth provider in-tree is untenable. The orchestrator's vault mints via these providers; but the abstraction is shippable independently and today's per-bottle sidecars can use it too.

Design

The contract (backend-agnostic)

Three surfaces; only one is per-backend.

1. Control plane (CLI / console → orchestrator) — an RPC: launch_bottle, teardown_bottle, register_policy, deregister_bottle, supervise_queue. Fully backend-agnostic. Both the local cli.py and the remote console funnel through it, so policy is uniform and cli.py becomes a thin client rather than a parallel launcher. Transport: HTTP — the most universal/reliable choice on every host (no vsock/unix-socket portability caveats); a local unix socket is a fine optimization, but HTTP is the wire contract.
2. Data plane (agent → orchestrator) — the egress / git / supervise endpoints. Already agnostic today (agents dial http://sidecar:9099); only the address and how packets get there are per-backend.
3. Launch / wire (orchestrator → backend) — the irreducibly backend-specific part; lives on BottleBackend.

One Orchestrator, no subclass tree

The orchestrator is a single concrete class holding all the backend-neutral logic — egress addon, git-gate, supervise, source-IP attribution, live-reload control plane, console client. It never branches on backend; it composes a BottleBackend. That composition is what makes the contract agnostic: there is nothing backend-specific left in the orchestrator to leak.

Rejected alternative: an Orchestrator ABC with per-backend implementations. The interesting logic (proxies, attribution, control plane) is backend-neutral, so three subclasses would triplicate the hard part; and a second hierarchy paralleling BottleBackend reintroduces the same hand-maintained lockstep coupling we just removed from the netpool constants (PR #350). Composition over a parallel tree.

BottleBackend absorbs the per-backend variation

A small, cohesive surface — reused for launching agent bottles and the orchestrator's own unit (the orchestrator is just another native unit):

launch_unit(spec) -> Handle      # agent bottle OR the orchestrator itself
                                 # (fc microVM / apple ctr / docker ctr)
wire(unit, endpoint) -> None     # DNAT+forward (fc) | attach shared net (docker/apple)
endpoint_of(unit) -> Endpoint    # address resolution
health(unit) -> Status

Plus the launch broker — the answer to "a VM/container can't spawn its own host-network siblings." The orchestrator can't directly open host /dev/kvm + a host TAP fd (Firecracker), and a container can't spawn siblings without a root-equivalent socket (Docker). So every backend exposes a broker the orchestrator calls to launch an agent:

• Firecracker — a thin, structured host shim (see security #3). This replaces today's implicit "launcher runs as host user."
• Docker — the socket today (fat, root-equivalent — the thing 0069's Stage 3 removes); a narrower broker later.
• Apple — the container CLI/daemon.

Broker request schema: human-readable JSON of static flags + ids only (never free-form paths/argv — that's what keeps it un-coercible into arbitrary launches), wrapped as a signed JWT so the broker verifies provenance: the request came from the real orchestrator, not a forged one from a compromised co-located component. The orchestrator signs; the broker verifies with the orchestrator's public key (provisioned at broker install). This is the concrete form of security #3's "structured requests only." Same JSON-with-ids + JWT shape for all orchestrator↔broker/sidecar communication; cases that don't fit get handled as they arise, with this as the default.

If BottleBackend bloats, the pressure valve is composition one level down: vend a backend.network() / Wiring collaborator rather than piling methods on — the same discipline, recursed.

State: one SQLite DB, owned by the orchestrator

The orchestrator is the natural owner of per-host runtime state:

• pool slot leases (which bottle holds slot i) — replaces today's fcntl-locked files with SQLite transactions;
• the supervise approval queue + remembered approvals;
• the live bottle registry (source IP → bottle → policy/secrets refs), the lookup table the attribution invariant reads.

This is deliberately not a "single source of truth for all config." Config splits into three tiers with different homes:

Tier	Example	Home
Build-time constants	pool size, IP base, nft table	flat .env (PR #350) — must be readable by Nix eval + root bash, zero runtime
User-authored config	bottle manifests, egress routes, secret refs	declarative files under ~/.bot-bottle/ — trust boundary at $HOME, git-trackable, "unknown keys die at load"
Runtime state	slot leases, approvals, registry	one shared bot-bottle.db, solely owned by the orchestrator

SQLite is right for the runtime tier (mutable, concurrent, queried) and wrong for the other two (Nix can't read it at eval time; it fights the declarative manifest trust model). Keep the tiers separate.

One shared bot-bottle.db for all runtime state (decided in review). The registry co-tenants the existing host bot-bottle.db — the DbStore framework already namespaces each store by schema_key, so slot leases / approvals / registry share one file. One place to query, back up, and integrate a console against.

• Host-resident, for durability. Re-adoption sweeps the registry after an orchestrator restart, so state must outlive the orchestrator instance. The file lives on the host (bot_bottle_root()/db/bot-bottle.db); the orchestrator unit reaches it, it doesn't carry it.
• Integrity by sole ownership, not mount permissions. Agents can't touch the DB directly wherever it lives (network-isolated in their bottles). The risk is a compromised agent-facing data-plane service (egress/git-gate, which parse hostile bytes) writing the registry and forging attribution. Because it's now one shared file, coarse ro/rw mount-splitting no longer isolates the registry — so the rule is stronger and simpler: only the orchestrator (control plane) opens bot-bottle.db; the data plane and the console reach state through the control-plane RPC, never a direct file handle. No agent-facing component gets the file, so none can forge attribution. (This supersedes the earlier ro-mount idea.)
  • Transitional caveat: today the per-bottle supervise sidecar rw-bind-mounts bot-bottle.db to write proposals — exactly the pattern the orchestrator removes (supervise consolidates into the orchestrator; sidecar writes become RPC calls). Until that lands, don't put the attribution registry behind a data-plane-writable mount.

Implementation note for the VM slices: SQLite WAL over a guest share (virtiofs/9p) is finicky (the -shm/-wal files need real mmap/locking), which is a second reason the DB wants a host-side owner the orchestrator reaches over the RPC rather than a shared mount into the VM. WAL on the shared DB is therefore a deliberate, tested future change — not enabled ad hoc. sqlite3 itself is stdlib, so "the host needs SQLite" is a non-cost.

Sequencing

Jump straight to the virtualized end state (not a host-daemon stepping stone): a host daemon's agent→localhost transport is throwaway once the orchestrator becomes a VM. Decouple the two risks instead:

• Consolidation risk (one process, all secrets, attribution, reload) and packaging/transport risk (VM-to-VM wiring, the shim) are independent. Develop the orchestrator service as a plain process dev-harness first, so the consolidation logic (attribution, reload, secret handling) is proven with fast iteration — then wrap that exact service in the VM and solve wiring separately.

Backend order (cheapest proof → hardest → last):

1. Docker orchestrator — nearly free (the sidecar bundle is already containers; collapse N bundles into one persistent container). Proves consolidation + the BottleBackend seam with the least moving parts.
2. Firecracker orchestrator — the real work: the shim + VM-to-VM routing (host forwards bbfcN → orchestrator TAP; the nft table grows forward rules where today it drops all non-DNAT egress). Built against the dev-harness so the app logic is already proven.
3. macOS (Apple container) — last (container-to-container networking).

Keep the sidecar service one shared thing throughout.

Non-goals

• Removing OCI/Dockerfile support for agent images (0069's concern).
• A single database for all config (see the three-tier table).
• Changing the per-bottle isolation of agent workloads — only the sidecar is consolidated; agents stay one-VM/container-each.

Relationship to other work

• PRD 0069 (#348): 0070 subsumes its Stage 1 (per-host sidecar) and Stage 4 (sidecar-as-VM). 0069 retains Stage 2 (nix-built fixed images — a dependency here: the orchestrator and agent base must be nix-built so the broker launches from a fixed image set and bootstrapping has no chicken-and-egg) and Stage 3 (in-VM Dockerfile builder).
• Minimal CI runner (paused): the Firecracker broker + no host Docker is what lets a dedicated gitea runner user drop the root-equivalent docker group — it only needs broker-socket access + kvm/pool group membership. This work unblocks it.
• Generic SecretProvider (#355): the future secret-handling mechanism (see "Secret handling") — generalizes PRD 0048's DeployKeyProvisioner into a user-extensible provider that mints short-lived creds. Shippable independently; the orchestrator's vault mints through it.
• PR #350 (netpool single-source): the same "one source per fact, composition over parallel hierarchies" discipline the contract follows.

Decisions (review 2026-07-13)

• Egress sharing: consolidate egress (worth it). Treat it as a minimal, hardened attack surface for a malicious agent rather than keeping it per-bottle; pair it with the identity token above and the short-lived-vault-token mitigations (Secret handling / #355).
• Control-plane transport: HTTP — most universal/reliable on every host; unix socket is an optional local optimization (see the contract).
• Broker request schema: signed-JWT JSON, static flags + ids only (see the launch broker) — provenance + un-coercible by construction.
• State re-adoption: the restart procedure is:
  1. Singleton: a new orchestrator launch requires no pre-existing orchestrator; any found (healthy or not) is fully shut down first.
  2. Re-adoption waits for the new orchestrator to be healthy.
  3. Once healthy, it discovers all agents needing adoption via both the SQLite registry and live process/VM inspection before serving any other request. The two-source sweep is what closes the in-flight launch race — a launch that started (VM booting / slot claimed) but hadn't committed to SQLite when the old orchestrator died would be invisible to a SQLite-only sweep; process inspection catches it. Launches should also write an intent record ahead of committing resources so the sweep can reconcile intent vs. actual.

Open questions

• VM-to-VM routing: per-backend, and in the design it is BottleBackend.wire() (DNAT+forward for fc, shared-net for docker/apple), not orchestrator logic. Not a blocker — resolvable at implementation time (may require a bit of host modification, as the pool setup already does on NixOS); an earlier "sidecars in VMs" spike showed it's feasible.
• Live-reload protocol for per-bottle policy over the HTTP control plane (add/remove routes/keys/proposals without a restart).
• Identity-token delivery: exactly how the per-bottle token is placed where the agent can present it but not swap in another bottle's.