bot-bottle/docs/prds/0070-per-host-orchestrator.md

PRD 0070: Per-host orchestrator service

• Status: Draft
• Author: Claude
• Created: 2026-07-12
• Issue: #351
• Supersedes: the Stage-1 / Stage-4 sidecar-consolidation framing of PRD 0069 (#348). Depends on 0069's nix-built fixed images (Stage 2) for bootstrapping; 0069 still owns the docker-free image-building work.

Summary

Replace the per-bottle gateway bundle with a single persistent, per-host orchestrator: one long-lived service that runs the gateway functions (egress / git-gate / supervise), coordinates with the console, and brokers agent launches and teardown. It is virtualized from the start using each backend's native isolation primitive — a Firecracker microVM on the Firecracker backend, an Apple container on macOS, a Docker container on the legacy backend — and is fronted by a single backend-agnostic contract. Per-backend variation lives on BottleBackend, not in the orchestrator.

Motivation

Today each bottle spins up its own gateway bundle (egress mitmproxy + git-gate + supervise). That costs:

• Resources. N bottles → N heavy bundles booting and idling.
• Operational churn. Per-launch container/VM lifecycle for the gateways, a control path baked at launch and torn down at exit.
• A blurry contract. "How a bottle talks to its gateway" is re-implemented per backend instead of being one agreed interface.

A per-host orchestrator collapses the first two and forces the third to be made explicit. It's also the component that will own per-host runtime state (slot leases, the approval queue, the bottle registry) — today that's ad-hoc fcntl-locked files.

Security review (read this first)

Consolidation is a real change to the trust model. The goal is to not significantly weaken the posture; some properties strengthen, some weaken, and the weakened ones must be mitigated by design, not hand-waved.

What gets stronger

• Build/host isolation of untrusted inputs (with 0069 Stage 3): user Dockerfiles build in a disposable VM instead of on the host.
• One audited privileged surface. Today the launcher runs as the full host user and needs the Docker socket (root-equivalent). The orchestrator model replaces that with a thin launch broker (below) — a small, structured, auditable privileged core instead of a fat socket.
• Attribution is enforced, not assumed. Making source-IP identity a first-class contract invariant (below) means each backend must prove it, rather than the gateway implicitly trusting network position.

What gets weaker, and the mitigation

1. Secret concentration. Per-bottle gateways isolate secrets at the process boundary — each holds only its bottle's tokens/keys. A host orchestrator concentrates every bottle's egress tokens, git deploy keys, and the console credential in one long-lived process. A single attribution bug leaks bottle A's token into bottle B's request — a class of bug that cannot exist per-bottle.

   • Mitigation: lean on the enforced source-IP invariant for attribution; keep the most secret-dense, least-shareable service (git-gate, per-repo deploy keys, no natural source-IP scoping) per-bottle unless there's a compelling reason; scope each secret to the bottle in the state DB so a lookup can't return the wrong bottle's secret by construction (key every secret access by the verified source identity, never by ambient state).

2. Shared fate. Orchestrator down = no new launches, and running agents lose egress / git / supervise. Compromise = the whole host's fleet, plus launch authority, plus the console token.

   • Mitigation: the orchestrator is itself confined (its own VM/container with its own fail-closed egress); make it restartable without killing running agent VMs (agents keep running; they briefly lose gateway connectivity until it's back); persist state to a host volume so a restart re-adopts live bottles rather than losing them.

3. The launch broker is the new privileged core. We don't eliminate host privilege — we shrink and relocate it. If the broker accepts arbitrary paths/commands, the orchestrator VM can escape through it.

   • Mitigation: the broker takes structured requests only — "launch bottle from this content-addressed, nix-built rootfs on TAP slot k", never "run this argv". It validates against a fixed image set, not caller-supplied paths. It is small enough to audit line-by-line.

4. The egress proxy now parses every bottle's traffic in one process. Higher blast radius for a mitmproxy/TLS-bump bug.

   • Mitigation: this is the argument for virtualizing the orchestrator from the start (Stage B, not a host daemon) — the code that TLS-bumps and parses agent traffic and holds every token runs inside its own confined VM, not as a host process. If egress sharing's blast radius feels too high, egress can stay per-bottle while supervise (near-zero secrets) goes host-level first.

The attribution invariant

Source-IP attribution is what makes a shared orchestrator safe: one process serves every bottle and tells them apart by source address. The mechanism is identical everywhere (read source IP → look up bottle); the guarantee that the address can't be forged is a per-backend responsibility and part of the contract:

Invariant: a packet's source address, as seen by the orchestrator, provably identifies the originating bottle.

• Firecracker — enforced by the /31 point-to-point TAP + the bot_bottle_fc nft table (strongest; already built).
• Docker — the per-bottle --internal network + anti-spoof; weaker, must be made explicit.
• Apple — the host-only network.

If a backend can't honor the invariant, source-IP consolidation is not safe there and that backend keeps per-bottle gateways. The invariant is a hard precondition, not an aspiration.

Defense-in-depth — a per-bottle identity token. On top of the network-layer invariant, inject a per-bottle secret token into every request the agent makes to the orchestrator (the agent already egresses through the gateway proxy, so this is cheap to add). It gives an application-layer proof of identity independent of the network layer:

• On Firecracker the /31 + nft already make source IP unspoofable by construction, so the token is belt-and-suspenders there — but cheap insurance against a misconfigured invariant.
• On weaker backends (Docker) it is load-bearing, providing attribution that doesn't lean on network anti-spoof.

Requirements: the token is per-bottle, unguessable, and non-cross-leakable — a bottle can only ever prove it is itself (it can't learn another bottle's token, given bottle isolation), so a hostile agent gains nothing by presenting it. The orchestrator provisions the token at launch and the gateway requires it to attribute + authorize. Note this hardens attribution, not secret exposure: it's app-layer, so a compromised orchestrator still sees every token (that's the concentration problem, addressed separately under Secret handling).

Secret handling — a FUTURE pattern (not v1)

Status: future / not required for the initial orchestrator. The initial cut can inject secrets as today; this section records the direction so v1 doesn't paint itself into a corner. Tracked as its own work in #355 (generic SecretProvider).

The residual weakness after all of the above is long-lived credential concentration — the data-plane proxies must hold every bottle's upstream tokens because the agent must never see them. You can't policy-gate the component whose job is to use all the secrets, so a full RCE of a proxy drains its authorized set regardless. Two moves bound this without pretending to prevent it:

1. Vault as a separate trust domain. The long-lived roots live in a distinct process (ideally its own VM) that the byte-parsing data plane never shares memory with. The proxies request secrets from it; the crown jewels are not in the process an agent-facing parser can pop.
2. Derive short-lived, scoped creds where the upstream allows it. The vault holds the root and mints expiring, narrowly-scoped credentials per bottle-start (or per request) — GitHub App installation tokens, OAuth/STS token exchange, forge deploy tokens. A compromise then leaks short-lived material, not permanent keys. For upstreams stuck on static keys the vault passes the value through (only the at-rest / audit / revocation benefits apply, not lifetime reduction) — this residue is accepted and documented, not solved.

Even a plain per-request fetch (no derivation) still buys at-rest reduction (process memory holds only in-flight secrets), a detection / rate-limit / revocation chokepoint, and clean cross-component scoping (an egress RCE can't request git-gate's creds). It does not prevent abuse of currently-authorized access during the compromise window.

Mechanism (see #355): generalize the existing DeployKeyProvisioner (PRD 0048) into a user-extensible SecretProvider droppable into the manifest anywhere a raw token value is accepted, discovered from ~/.bot-bottle/contrib/<name>/secret_provider.py exactly as user AgentProviders are — because maintaining every forge/cloud/OAuth provider in-tree is untenable. The orchestrator's vault mints via these providers; but the abstraction is shippable independently and today's per-bottle gateways can use it too.

Design

The contract (backend-agnostic)

Three surfaces; only one is per-backend.

1. Control plane (CLI / console → orchestrator) — an RPC: launch_bottle, teardown_bottle, register_policy, deregister_bottle, supervise_queue. Fully backend-agnostic. Both the local cli.py and the remote console funnel through it, so policy is uniform and cli.py becomes a thin client rather than a parallel launcher. Transport: HTTP — the most universal/reliable choice on every host (no vsock/unix-socket portability caveats); a local unix socket is a fine optimization, but HTTP is the wire contract.
2. Data plane (agent → orchestrator) — the egress / git / supervise endpoints. Already agnostic today (agents dial http://gateway:9099); only the address and how packets get there are per-backend.
3. Launch / wire (orchestrator → backend) — the irreducibly backend-specific part; lives on BottleBackend.

One Orchestrator, no subclass tree

The orchestrator is a single concrete class holding all the backend-neutral logic — egress addon, git-gate, supervise, source-IP attribution, live-reload control plane, console client. It never branches on backend; it composes a BottleBackend. That composition is what makes the contract agnostic: there is nothing backend-specific left in the orchestrator to leak.

Rejected alternative: an Orchestrator ABC with per-backend implementations. The interesting logic (proxies, attribution, control plane) is backend-neutral, so three subclasses would triplicate the hard part; and a second hierarchy paralleling BottleBackend reintroduces the same hand-maintained lockstep coupling we just removed from the netpool constants (PR #350). Composition over a parallel tree.

BottleBackend absorbs the per-backend variation

A small, cohesive surface — reused for launching agent bottles and the orchestrator's own unit (the orchestrator is just another native unit):

launch_unit(spec) -> Handle      # agent bottle OR the orchestrator itself
                                 # (fc microVM / apple ctr / docker ctr)
wire(unit, endpoint) -> None     # DNAT+forward (fc) | attach shared net (docker/apple)
endpoint_of(unit) -> Endpoint    # address resolution
health(unit) -> Status

Plus the launch broker — the answer to "a VM/container can't spawn its own host-network siblings." The orchestrator can't directly open host /dev/kvm + a host TAP fd (Firecracker), and a container can't spawn siblings without a root-equivalent socket (Docker). So every backend exposes a broker the orchestrator calls to launch an agent:

• Firecracker — a thin, structured host shim (see security #3). This replaces today's implicit "launcher runs as host user."
• Docker — the socket today (fat, root-equivalent — the thing 0069's Stage 3 removes); a narrower broker later.
• Apple — the container CLI/daemon.

Broker request schema: human-readable JSON of static flags + ids only (never free-form paths/argv — that's what keeps it un-coercible into arbitrary launches), wrapped as a signed JWT so the broker verifies provenance: the request came from the real orchestrator, not a forged one from a compromised co-located component. The orchestrator signs; the broker verifies with the orchestrator's public key (provisioned at broker install). This is the concrete form of security #3's "structured requests only." Same JSON-with-ids + JWT shape for all orchestrator↔broker/gateway communication; cases that don't fit get handled as they arise, with this as the default.

If BottleBackend bloats, the pressure valve is composition one level down: vend a backend.network() / Wiring collaborator rather than piling methods on — the same discipline, recursed.

State: one SQLite DB, owned by the orchestrator

The orchestrator is the natural owner of per-host runtime state:

• pool slot leases (which bottle holds slot i) — replaces today's fcntl-locked files with SQLite transactions;
• the supervise approval queue + remembered approvals;
• the live bottle registry (source IP → bottle → policy/secrets refs), the lookup table the attribution invariant reads.

This is deliberately not a "single source of truth for all config." Config splits into three tiers with different homes:

Tier	Example	Home
Build-time constants	pool size, IP base, nft table	flat .env (PR #350) — must be readable by Nix eval + root bash, zero runtime
User-authored config	bottle manifests, egress routes, secret refs	declarative files under ~/.bot-bottle/ — trust boundary at $HOME, git-trackable, "unknown keys die at load"
Runtime state	slot leases, approvals, registry	one shared bot-bottle.db, solely owned by the orchestrator

SQLite is right for the runtime tier (mutable, concurrent, queried) and wrong for the other two (Nix can't read it at eval time; it fights the declarative manifest trust model). Keep the tiers separate.

One shared bot-bottle.db for all runtime state (decided in review). The registry co-tenants the existing host bot-bottle.db — the DbStore framework already namespaces each store by schema_key, so slot leases / approvals / registry share one file. One place to query, back up, and integrate a console against.

• Host-resident, for durability. Re-adoption sweeps the registry after an orchestrator restart, so state must outlive the orchestrator instance. The file lives on the host (bot_bottle_root()/db/bot-bottle.db); the orchestrator unit reaches it, it doesn't carry it.
• Integrity by sole ownership, not mount permissions. Agents can't touch the DB directly wherever it lives (network-isolated in their bottles). The risk is a compromised agent-facing data-plane service (egress/git-gate, which parse hostile bytes) writing the registry and forging attribution. Because it's now one shared file, coarse ro/rw mount-splitting no longer isolates the registry — so the rule is stronger and simpler: only the orchestrator (control plane) opens bot-bottle.db; the data plane and the console reach state through the control-plane RPC, never a direct file handle. No agent-facing component gets the file, so none can forge attribution. (This supersedes the earlier ro-mount idea.)
  • Transitional caveat: today the per-bottle supervise gateway rw-bind-mounts bot-bottle.db to write proposals — exactly the pattern the orchestrator removes (supervise consolidates into the orchestrator; gateway writes become RPC calls). Until that lands, don't put the attribution registry behind a data-plane-writable mount.

Implementation note for the VM slices: SQLite WAL over a guest share (virtiofs/9p) is finicky (the -shm/-wal files need real mmap/locking), which is a second reason the DB wants a host-side owner the orchestrator reaches over the RPC rather than a shared mount into the VM. WAL on the shared DB is therefore a deliberate, tested future change — not enabled ad hoc. sqlite3 itself is stdlib, so "the host needs SQLite" is a non-cost.

Gateway CA: host-resident, like the DB

The shared gateway bumps TLS with a self-generated mitmproxy CA, and every bottle installs that CA into its trust store to accept the bumped leaves. So the CA is durable per-host state with the same rule as the DB: it must outlive any single gateway container, or a restart mints a fresh CA that every already-running bottle distrusts — the TLS handshake then fails even after the bottle re-resolves and reconnects to the moved gateway (issue #450, a re-attachment blocker distinct from #443/#445).

The CA lives on the host filesystem at bot_bottle_root()/gateway-ca (host_gateway_ca_dir()), bind-mounted into the container at mitmproxy's confdir. This is deliberately a host bind-mount, not a container-runtime named volume: a named volume survives ordinary container removal but can be silently wiped by Docker's or Apple Container's volume-prune commands during routine host maintenance, which is exactly how the ephemeral-CA symptom shows up in practice. A path under the app-data root is not managed or pruned by the container runtime, and stays directly inspectable and rotatable from the host. mitmproxy reuses an existing CA and generates one only on first run, so the bind-mount alone gives "adopt-existing, generate-on-first-run" for free.

The macOS backend uses the same host-resident CA directory and bind-mounts it into the consolidated Apple infra container. Its bot-bottle-mac-db named volume remains container-only because that prevents incoherent cross-kernel SQLite locking, but the CA is deliberately not stored there: Apple Container also has a container volume prune operation, and the named volume is temporarily unreferenced while the infra container is recreated. Keeping the CA on the host makes both ordinary recreation and volume pruning safe.

Deliberate rollover is the explicit inverse: rotate_gateway_ca() removes the persisted CA material so the next start remints it, and the python -m bot_bottle.orchestrator.rotate_ca one-shot wires that together with dropping the running gateway container (whose mitmproxy still holds the old CA in memory). Rotation does not auto-re-provision the new CA into running bottles — those re-attach to install the new anchor — so it is an operator action with a brief egress interruption, never an implicit one.

Sequencing

Jump straight to the virtualized end state (not a host-daemon stepping stone): a host daemon's agent→localhost transport is throwaway once the orchestrator becomes a VM. Decouple the two risks instead:

• Consolidation risk (one process, all secrets, attribution, reload) and packaging/transport risk (VM-to-VM wiring, the shim) are independent. Develop the orchestrator service as a plain process dev-harness first, so the consolidation logic (attribution, reload, secret handling) is proven with fast iteration — then wrap that exact service in the VM and solve wiring separately.

Backend order (cheapest proof → hardest → last):

1. Docker orchestrator — nearly free (the gateway bundle is already containers; collapse N bundles into one persistent container). Proves consolidation + the BottleBackend seam with the least moving parts.
2. Firecracker orchestrator — the real work: the shim + VM-to-VM routing (host forwards bbfcN → orchestrator TAP; the nft table grows forward rules where today it drops all non-DNAT egress). Built against the dev-harness so the app logic is already proven.
3. macOS (Apple container) — last (container-to-container networking).

Keep the gateway service one shared thing throughout.

Non-goals

• Removing OCI/Dockerfile support for agent images (0069's concern).
• A single database for all config (see the three-tier table).
• Changing the per-bottle isolation of agent workloads — only the gateway is consolidated; agents stay one-VM/container-each.

Relationship to other work

• PRD 0069 (#348): 0070 subsumes its Stage 1 (per-host sidecar) and Stage 4 (sidecar-as-VM). 0069 retains Stage 2 (nix-built fixed images — a dependency here: the orchestrator and agent base must be nix-built so the broker launches from a fixed image set and bootstrapping has no chicken-and-egg) and Stage 3 (in-VM Dockerfile builder).
• Minimal CI runner (paused): the Firecracker broker + no host Docker is what lets a dedicated gitea runner user drop the root-equivalent docker group — it only needs broker-socket access + kvm/pool group membership. This work unblocks it.
• Generic SecretProvider (#355): the future secret-handling mechanism (see "Secret handling") — generalizes PRD 0048's DeployKeyProvisioner into a user-extensible provider that mints short-lived creds. Shippable independently; the orchestrator's vault mints through it.
• PR #350 (netpool single-source): the same "one source per fact, composition over parallel hierarchies" discipline the contract follows.

Decisions (review 2026-07-13)

• Egress sharing: consolidate egress (worth it). Treat it as a minimal, hardened attack surface for a malicious agent rather than keeping it per-bottle; pair it with the identity token above and the short-lived-vault-token mitigations (Secret handling / #355).
• Control-plane transport: HTTP — most universal/reliable on every host; unix socket is an optional local optimization (see the contract).
• Broker request schema: signed-JWT JSON, static flags + ids only (see the launch broker) — provenance + un-coercible by construction.
• State re-adoption: the restart procedure is:
  1. Singleton: a new orchestrator launch requires no pre-existing orchestrator; any found (healthy or not) is fully shut down first.
  2. Re-adoption waits for the new orchestrator to be healthy.
  3. Once healthy, it discovers all agents needing adoption via both the SQLite registry and live process/VM inspection before serving any other request. The two-source sweep is what closes the in-flight launch race — a launch that started (VM booting / slot claimed) but hadn't committed to SQLite when the old orchestrator died would be invisible to a SQLite-only sweep; process inspection catches it. Launches should also write an intent record ahead of committing resources so the sweep can reconcile intent vs. actual.

Open questions

• VM-to-VM routing: per-backend, and in the design it is BottleBackend.wire() (DNAT+forward for fc, shared-net for docker/apple), not orchestrator logic. Not a blocker — resolvable at implementation time (may require a bit of host modification, as the pool setup already does on NixOS); an earlier "gateways in VMs" spike showed it's feasible.
• Live-reload protocol for per-bottle policy over the HTTP control plane (add/remove routes/keys/proposals without a restart).
• Identity-token delivery: exactly how the per-bottle token is placed where the agent can present it but not swap in another bottle's.