bot-bottle/docs/prds/0070-per-host-orchestrator.md

PRD 0070: Per-host orchestrator service

• Status: Draft
• Author: Claude
• Created: 2026-07-12
• Issue: #351
• Supersedes: the Stage-1 / Stage-4 sidecar-consolidation framing of PRD 0069 (#348). Depends on 0069's nix-built fixed images (Stage 2) for bootstrapping; 0069 still owns the docker-free image-building work.

Summary

Replace the per-bottle sidecar bundle with a single persistent, per-host orchestrator: one long-lived service that runs the sidecar functions (egress / git-gate / supervise), coordinates with the console, and brokers agent launches and teardown. It is virtualized from the start using each backend's native isolation primitive — a Firecracker microVM on the Firecracker backend, an Apple container on macOS, a Docker container on the legacy backend — and is fronted by a single backend-agnostic contract. Per-backend variation lives on BottleBackend, not in the orchestrator.

Motivation

Today each bottle spins up its own sidecar bundle (egress mitmproxy + git-gate + supervise). That costs:

• Resources. N bottles → N heavy bundles booting and idling.
• Operational churn. Per-launch container/VM lifecycle for the sidecars, a control path baked at launch and torn down at exit.
• A blurry contract. "How a bottle talks to its sidecar" is re-implemented per backend instead of being one agreed interface.

A per-host orchestrator collapses the first two and forces the third to be made explicit. It's also the component that will own per-host runtime state (slot leases, the approval queue, the bottle registry) — today that's ad-hoc fcntl-locked files.

Security review (read this first)

Consolidation is a real change to the trust model. The goal is to not significantly weaken the posture; some properties strengthen, some weaken, and the weakened ones must be mitigated by design, not hand-waved.

What gets stronger

• Build/host isolation of untrusted inputs (with 0069 Stage 3): user Dockerfiles build in a disposable VM instead of on the host.
• One audited privileged surface. Today the launcher runs as the full host user and needs the Docker socket (root-equivalent). The orchestrator model replaces that with a thin launch broker (below) — a small, structured, auditable privileged core instead of a fat socket.
• Attribution is enforced, not assumed. Making source-IP identity a first-class contract invariant (below) means each backend must prove it, rather than the sidecar implicitly trusting network position.

What gets weaker, and the mitigation

1. Secret concentration. Per-bottle sidecars isolate secrets at the process boundary — each holds only its bottle's tokens/keys. A host orchestrator concentrates every bottle's egress tokens, git deploy keys, and the console credential in one long-lived process. A single attribution bug leaks bottle A's token into bottle B's request — a class of bug that cannot exist per-bottle.

   • Mitigation: lean on the enforced source-IP invariant for attribution; keep the most secret-dense, least-shareable service (git-gate, per-repo deploy keys, no natural source-IP scoping) per-bottle unless there's a compelling reason; scope each secret to the bottle in the state DB so a lookup can't return the wrong bottle's secret by construction (key every secret access by the verified source identity, never by ambient state).

2. Shared fate. Orchestrator down = no new launches, and running agents lose egress / git / supervise. Compromise = the whole host's fleet, plus launch authority, plus the console token.

   • Mitigation: the orchestrator is itself confined (its own VM/container with its own fail-closed egress); make it restartable without killing running agent VMs (agents keep running; they briefly lose sidecar connectivity until it's back); persist state to a host volume so a restart re-adopts live bottles rather than losing them.

3. The launch broker is the new privileged core. We don't eliminate host privilege — we shrink and relocate it. If the broker accepts arbitrary paths/commands, the orchestrator VM can escape through it.

   • Mitigation: the broker takes structured requests only — "launch bottle from this content-addressed, nix-built rootfs on TAP slot k", never "run this argv". It validates against a fixed image set, not caller-supplied paths. It is small enough to audit line-by-line.

4. The egress proxy now parses every bottle's traffic in one process. Higher blast radius for a mitmproxy/TLS-bump bug.

   • Mitigation: this is the argument for virtualizing the orchestrator from the start (Stage B, not a host daemon) — the code that TLS-bumps and parses agent traffic and holds every token runs inside its own confined VM, not as a host process. If egress sharing's blast radius feels too high, egress can stay per-bottle while supervise (near-zero secrets) goes host-level first.

The attribution invariant

Source-IP attribution is what makes a shared orchestrator safe: one process serves every bottle and tells them apart by source address. The mechanism is identical everywhere (read source IP → look up bottle); the guarantee that the address can't be forged is a per-backend responsibility and part of the contract:

Invariant: a packet's source address, as seen by the orchestrator, provably identifies the originating bottle.

• Firecracker — enforced by the /31 point-to-point TAP + the bot_bottle_fc nft table (strongest; already built).
• Docker — the per-bottle --internal network + anti-spoof; weaker, must be made explicit.
• Apple — the host-only network.

If a backend can't honor the invariant, source-IP consolidation is not safe there and that backend keeps per-bottle sidecars. The invariant is a hard precondition, not an aspiration.

Design

The contract (backend-agnostic)

Three surfaces; only one is per-backend.

1. Control plane (CLI / console → orchestrator) — an RPC: launch_bottle, teardown_bottle, register_policy, deregister_bottle, supervise_queue. Fully backend-agnostic. Both the local cli.py and the remote console funnel through it, so policy is uniform and cli.py becomes a thin client rather than a parallel launcher.
2. Data plane (agent → orchestrator) — the egress / git / supervise endpoints. Already agnostic today (agents dial http://sidecar:9099); only the address and how packets get there are per-backend.
3. Launch / wire (orchestrator → backend) — the irreducibly backend-specific part; lives on BottleBackend.

One Orchestrator, no subclass tree

The orchestrator is a single concrete class holding all the backend-neutral logic — egress addon, git-gate, supervise, source-IP attribution, live-reload control plane, console client. It never branches on backend; it composes a BottleBackend. That composition is what makes the contract agnostic: there is nothing backend-specific left in the orchestrator to leak.

Rejected alternative: an Orchestrator ABC with per-backend implementations. The interesting logic (proxies, attribution, control plane) is backend-neutral, so three subclasses would triplicate the hard part; and a second hierarchy paralleling BottleBackend reintroduces the same hand-maintained lockstep coupling we just removed from the netpool constants (PR #350). Composition over a parallel tree.

BottleBackend absorbs the per-backend variation

A small, cohesive surface — reused for launching agent bottles and the orchestrator's own unit (the orchestrator is just another native unit):

launch_unit(spec) -> Handle      # agent bottle OR the orchestrator itself
                                 # (fc microVM / apple ctr / docker ctr)
wire(unit, endpoint) -> None     # DNAT+forward (fc) | attach shared net (docker/apple)
endpoint_of(unit) -> Endpoint    # address resolution
health(unit) -> Status

Plus the launch broker — the answer to "a VM/container can't spawn its own host-network siblings." The orchestrator can't directly open host /dev/kvm + a host TAP fd (Firecracker), and a container can't spawn siblings without a root-equivalent socket (Docker). So every backend exposes a broker the orchestrator calls to launch an agent:

• Firecracker — a thin, structured host shim (see security #3). This replaces today's implicit "launcher runs as host user."
• Docker — the socket today (fat, root-equivalent — the thing 0069's Stage 3 removes); a narrower broker later.
• Apple — the container CLI/daemon.

If BottleBackend bloats, the pressure valve is composition one level down: vend a backend.network() / Wiring collaborator rather than piling methods on — the same discipline, recursed.

State: one SQLite DB, owned by the orchestrator

The orchestrator is the natural owner of per-host runtime state:

• pool slot leases (which bottle holds slot i) — replaces today's fcntl-locked files with WAL-mode transactions;
• the supervise approval queue + remembered approvals;
• the live bottle registry (source IP → bottle → policy/secrets refs), the lookup table the attribution invariant reads.

This is deliberately not a "single source of truth for all config." Config splits into three tiers with different homes:

Tier	Example	Home
Build-time constants	pool size, IP base, nft table	flat .env (PR #350) — must be readable by Nix eval + root bash, zero runtime
User-authored config	bottle manifests, egress routes, secret refs	declarative files under ~/.bot-bottle/ — trust boundary at $HOME, git-trackable, "unknown keys die at load"
Runtime state	slot leases, approvals, registry	SQLite, owned by the orchestrator

SQLite is right for the runtime tier (mutable, concurrent, queried) and wrong for the other two (Nix can't read it at eval time; it fights the declarative manifest trust model). Keep the tiers separate.

Sequencing

Jump straight to the virtualized end state (not a host-daemon stepping stone): a host daemon's agent→localhost transport is throwaway once the orchestrator becomes a VM. Decouple the two risks instead:

• Consolidation risk (one process, all secrets, attribution, reload) and packaging/transport risk (VM-to-VM wiring, the shim) are independent. Develop the orchestrator service as a plain process dev-harness first, so the consolidation logic (attribution, reload, secret handling) is proven with fast iteration — then wrap that exact service in the VM and solve wiring separately.

Backend order (cheapest proof → hardest → last):

1. Docker orchestrator — nearly free (the sidecar bundle is already containers; collapse N bundles into one persistent container). Proves consolidation + the BottleBackend seam with the least moving parts.
2. Firecracker orchestrator — the real work: the shim + VM-to-VM routing (host forwards bbfcN → orchestrator TAP; the nft table grows forward rules where today it drops all non-DNAT egress). Built against the dev-harness so the app logic is already proven.
3. macOS (Apple container) — last (container-to-container networking).

Keep the sidecar service one shared thing throughout.

Non-goals

• Removing OCI/Dockerfile support for agent images (0069's concern).
• A single database for all config (see the three-tier table).
• Changing the per-bottle isolation of agent workloads — only the sidecar is consolidated; agents stay one-VM/container-each.

Relationship to other work

• PRD 0069 (#348): 0070 subsumes its Stage 1 (per-host sidecar) and Stage 4 (sidecar-as-VM). 0069 retains Stage 2 (nix-built fixed images — a dependency here: the orchestrator and agent base must be nix-built so the broker launches from a fixed image set and bootstrapping has no chicken-and-egg) and Stage 3 (in-VM Dockerfile builder).
• Minimal CI runner (paused): the Firecracker broker + no host Docker is what lets a dedicated gitea runner user drop the root-equivalent docker group — it only needs broker-socket access + kvm/pool group membership. This work unblocks it.
• PR #350 (netpool single-source): the same "one source per fact, composition over parallel hierarchies" discipline the contract follows.

Open questions

• Egress sharing tradeoff: is the secret-concentration blast radius of one shared mitmproxy worth the resource win, or share only supervise (near-zero secrets) and keep egress + git-gate per-bottle initially?
• Control-plane shape: RPC transport (unix socket / vsock / HTTP over the TAP) and the live-reload protocol for per-bottle policy.
• State re-adoption: exact scheme for an orchestrator restart to re-adopt running agent VMs from the SQLite registry without racing in-flight launches.
• VM-to-VM routing: the nft forward rules + addressing for a per-host orchestrator VM on its own TAP.
• Broker request schema: the exact structured contract that stays auditable and can't be coerced into launching arbitrary payloads.