bot-bottle/docs/prds/0070-per-host-orchestrator.md

PRD 0070: Per-host orchestrator service

• Status: Draft
• Author: Claude
• Created: 2026-07-12
• Issue: #351
• Supersedes: the Stage-1 / Stage-4 sidecar-consolidation framing of PRD 0069 (#348). Depends on 0069's nix-built fixed images (Stage 2) for bootstrapping; 0069 still owns the docker-free image-building work.

Summary

Replace the per-bottle gateway bundle with a single persistent, per-host orchestrator: one long-lived service that runs the gateway functions (egress / git-gate / supervise), coordinates with the console, and brokers agent launches and teardown. It is virtualized from the start using each backend's native isolation primitive — a Firecracker microVM on the Firecracker backend, an Apple container on macOS, a Docker container on the legacy backend — and is fronted by a single backend-agnostic contract. Per-backend variation lives on BottleBackend, not in the orchestrator.

Motivation

Today each bottle spins up its own gateway bundle (egress mitmproxy + git-gate + supervise). That costs:

• Resources. N bottles → N heavy bundles booting and idling.
• Operational churn. Per-launch container/VM lifecycle for the gateways, a control path baked at launch and torn down at exit.
• A blurry contract. "How a bottle talks to its gateway" is re-implemented per backend instead of being one agreed interface.

A per-host orchestrator collapses the first two and forces the third to be made explicit. It's also the component that will own per-host runtime state (slot leases, the approval queue, the bottle registry) — today that's ad-hoc fcntl-locked files.

Security review (read this first)

Consolidation is a real change to the trust model. The goal is to not significantly weaken the posture; some properties strengthen, some weaken, and the weakened ones must be mitigated by design, not hand-waved.

What gets stronger

• Build/host isolation of untrusted inputs (with 0069 Stage 3): user Dockerfiles build in a disposable VM instead of on the host.
• One audited privileged surface. Today the launcher runs as the full host user and needs the Docker socket (root-equivalent). The orchestrator model replaces that with a thin launch broker (below) — a small, structured, auditable privileged core instead of a fat socket.
• Attribution is enforced, not assumed. Making source-IP identity a first-class contract invariant (below) means each backend must prove it, rather than the gateway implicitly trusting network position.

What gets weaker, and the mitigation

1. Secret concentration. Per-bottle gateways isolate secrets at the process boundary — each holds only its bottle's tokens/keys. A host orchestrator concentrates every bottle's egress tokens, git deploy keys, and the console credential in one long-lived process. A single attribution bug leaks bottle A's token into bottle B's request — a class of bug that cannot exist per-bottle.

   • Mitigation: lean on the enforced source-IP invariant for attribution; keep the most secret-dense, least-shareable service (git-gate, per-repo deploy keys, no natural source-IP scoping) per-bottle unless there's a compelling reason; scope each secret to the bottle in the state DB so a lookup can't return the wrong bottle's secret by construction (key every secret access by the verified source identity, never by ambient state).

2. Shared fate. Orchestrator down = no new launches, and running agents lose egress / git / supervise. Compromise = the whole host's fleet, plus launch authority, plus the console token.

   • Mitigation: the orchestrator is itself confined (its own VM/container with its own fail-closed egress); make it restartable without killing running agent VMs (agents keep running; they briefly lose gateway connectivity until it's back); persist state to a host volume so a restart re-adopts live bottles rather than losing them.

3. The launch broker is the new privileged core. We don't eliminate host privilege — we shrink and relocate it. If the broker accepts arbitrary paths/commands, the orchestrator VM can escape through it.

   • Mitigation: the broker takes structured requests only — "launch bottle from this content-addressed, nix-built rootfs on TAP slot k", never "run this argv". It validates against a fixed image set, not caller-supplied paths. It is small enough to audit line-by-line.

4. The egress proxy now parses every bottle's traffic in one process. Higher blast radius for a mitmproxy/TLS-bump bug.

   • Mitigation: this is the argument for virtualizing the orchestrator from the start (Stage B, not a host daemon) — the code that TLS-bumps and parses agent traffic and holds every token runs inside its own confined VM, not as a host process. If egress sharing's blast radius feels too high, egress can stay per-bottle while supervise (near-zero secrets) goes host-level first.

The attribution invariant

Source-IP attribution is what makes a shared orchestrator safe: one process serves every bottle and tells them apart by source address. The mechanism is identical everywhere (read source IP → look up bottle); the guarantee that the address can't be forged is a per-backend responsibility and part of the contract:

Invariant: a packet's source address, as seen by the orchestrator, provably identifies the originating bottle.

• Firecracker — enforced by the /31 point-to-point TAP + the bot_bottle_fc nft table (strongest; already built).
• Docker — the per-bottle --internal network + anti-spoof; weaker, must be made explicit.
• Apple — the host-only network.

If a backend can't honor the invariant, source-IP consolidation is not safe there and that backend keeps per-bottle gateways. The invariant is a hard precondition, not an aspiration.

Defense-in-depth — a per-bottle identity token. On top of the network-layer invariant, inject a per-bottle secret token into every request the agent makes to the orchestrator (the agent already egresses through the gateway proxy, so this is cheap to add). It gives an application-layer proof of identity independent of the network layer:

• On Firecracker the /31 + nft already make source IP unspoofable by construction, so the token is belt-and-suspenders there — but cheap insurance against a misconfigured invariant.
• On weaker backends (Docker) it is load-bearing, providing attribution that doesn't lean on network anti-spoof.

Requirements: the token is per-bottle, unguessable, and non-cross-leakable — a bottle can only ever prove it is itself (it can't learn another bottle's token, given bottle isolation), so a hostile agent gains nothing by presenting it. The orchestrator provisions the token at launch and the gateway requires it to attribute + authorize. Note this hardens attribution, not secret exposure: it's app-layer, so a compromised orchestrator still sees every token (that's the concentration problem, addressed separately under Secret handling).

Secret handling — a FUTURE pattern (not v1)

Status: future / not required for the initial orchestrator. The initial cut can inject secrets as today; this section records the direction so v1 doesn't paint itself into a corner. Tracked as its own work in #355 (generic SecretProvider).

The residual weakness after all of the above is long-lived credential concentration — the data-plane proxies must hold every bottle's upstream tokens because the agent must never see them. You can't policy-gate the component whose job is to use all the secrets, so a full RCE of a proxy drains its authorized set regardless. Two moves bound this without pretending to prevent it:

1. Vault as a separate trust domain. The long-lived roots live in a distinct process (ideally its own VM) that the byte-parsing data plane never shares memory with. The proxies request secrets from it; the crown jewels are not in the process an agent-facing parser can pop.
2. Derive short-lived, scoped creds where the upstream allows it. The vault holds the root and mints expiring, narrowly-scoped credentials per bottle-start (or per request) — GitHub App installation tokens, OAuth/STS token exchange, forge deploy tokens. A compromise then leaks short-lived material, not permanent keys. For upstreams stuck on static keys the vault passes the value through (only the at-rest / audit / revocation benefits apply, not lifetime reduction) — this residue is accepted and documented, not solved.

Even a plain per-request fetch (no derivation) still buys at-rest reduction (process memory holds only in-flight secrets), a detection / rate-limit / revocation chokepoint, and clean cross-component scoping (an egress RCE can't request git-gate's creds). It does not prevent abuse of currently-authorized access during the compromise window.

Mechanism (see #355): generalize the existing DeployKeyProvisioner (PRD 0048) into a user-extensible SecretProvider droppable into the manifest anywhere a raw token value is accepted, discovered from ~/.bot-bottle/contrib/<name>/secret_provider.py exactly as user AgentProviders are — because maintaining every forge/cloud/OAuth provider in-tree is untenable. The orchestrator's vault mints via these providers; but the abstraction is shippable independently and today's per-bottle gateways can use it too.

Design

The contract (backend-agnostic)

Three surfaces; only one is per-backend.

1. Control plane (CLI / console → orchestrator) — an RPC: launch_bottle, teardown_bottle, register_policy, deregister_bottle, supervise_queue. Fully backend-agnostic. Both the local cli.py and the remote console funnel through it, so policy is uniform and cli.py becomes a thin client rather than a parallel launcher. Transport: HTTP — the most universal/reliable choice on every host (no vsock/unix-socket portability caveats); a local unix socket is a fine optimization, but HTTP is the wire contract.
2. Data plane (agent → orchestrator) — the egress / git / supervise endpoints. Already agnostic today (agents dial http://gateway:9099); only the address and how packets get there are per-backend.
3. Launch / wire (orchestrator → backend) — the irreducibly backend-specific part; lives on BottleBackend.

One Orchestrator, no subclass tree

The orchestrator is a single concrete class holding all the backend-neutral logic — egress addon, git-gate, supervise, source-IP attribution, live-reload control plane, console client. It never branches on backend; it composes a BottleBackend. That composition is what makes the contract agnostic: there is nothing backend-specific left in the orchestrator to leak.

Rejected alternative: an Orchestrator ABC with per-backend implementations. The interesting logic (proxies, attribution, control plane) is backend-neutral, so three subclasses would triplicate the hard part; and a second hierarchy paralleling BottleBackend reintroduces the same hand-maintained lockstep coupling we just removed from the netpool constants (PR #350). Composition over a parallel tree.

BottleBackend absorbs the per-backend variation

A small, cohesive surface — reused for launching agent bottles and the orchestrator's own unit (the orchestrator is just another native unit):

launch_unit(spec) -> Handle      # agent bottle OR the orchestrator itself
                                 # (fc microVM / apple ctr / docker ctr)
wire(unit, endpoint) -> None     # DNAT+forward (fc) | attach shared net (docker/apple)
endpoint_of(unit) -> Endpoint    # address resolution
health(unit) -> Status

Plus the launch broker — the answer to "a VM/container can't spawn its own host-network siblings." The orchestrator can't directly open host /dev/kvm + a host TAP fd (Firecracker), and a container can't spawn siblings without a root-equivalent socket (Docker). So every backend exposes a broker the orchestrator calls to launch an agent:

• Firecracker — a thin, structured host shim (see security #3). This replaces today's implicit "launcher runs as host user."
• Docker — the socket today (fat, root-equivalent — the thing 0069's Stage 3 removes); a narrower broker later.
• Apple — the container CLI/daemon.

Broker request schema: human-readable JSON of static flags + ids only (never free-form paths/argv — that's what keeps it un-coercible into arbitrary launches), wrapped as a signed JWT so the broker verifies provenance: the request came from the real orchestrator, not a forged one from a compromised co-located component. The orchestrator signs; the broker verifies with the orchestrator's public key (provisioned at broker install). This is the concrete form of security #3's "structured requests only." Same JSON-with-ids + JWT shape for all orchestrator↔broker/gateway communication; cases that don't fit get handled as they arise, with this as the default.

If BottleBackend bloats, the pressure valve is composition one level down: vend a backend.network() / Wiring collaborator rather than piling methods on — the same discipline, recursed.

State: one SQLite DB, owned by the orchestrator

The orchestrator is the natural owner of per-host runtime state:

• pool slot leases (which bottle holds slot i) — replaces today's fcntl-locked files with SQLite transactions;
• the supervise approval queue + remembered approvals;
• the live bottle registry (source IP → bottle → policy/secrets refs), the lookup table the attribution invariant reads.

This is deliberately not a "single source of truth for all config." Config splits into three tiers with different homes:

Tier	Example	Home
Build-time constants	pool size, IP base, nft table	flat .env (PR #350) — must be readable by Nix eval + root bash, zero runtime
User-authored config	bottle manifests, egress routes, secret refs	declarative files under ~/.bot-bottle/ — trust boundary at $HOME, git-trackable, "unknown keys die at load"
Runtime state	slot leases, approvals, registry	one shared bot-bottle.db, solely owned by the orchestrator

SQLite is right for the runtime tier (mutable, concurrent, queried) and wrong for the other two (Nix can't read it at eval time; it fights the declarative manifest trust model). Keep the tiers separate.

One shared bot-bottle.db for all runtime state (decided in review). The registry co-tenants the existing host bot-bottle.db — the DbStore framework already namespaces each store by schema_key, so slot leases / approvals / registry share one file. One place to query, back up, and integrate a console against.

• Host-resident, for durability. Re-adoption sweeps the registry after an orchestrator restart, so state must outlive the orchestrator instance. The file lives on the host (bot_bottle_root()/db/bot-bottle.db); the orchestrator unit reaches it, it doesn't carry it.
• Integrity by sole ownership, not mount permissions. Agents can't touch the DB directly wherever it lives (network-isolated in their bottles). The risk is a compromised agent-facing data-plane service (egress/git-gate, which parse hostile bytes) writing the registry and forging attribution. Because it's now one shared file, coarse ro/rw mount-splitting no longer isolates the registry — so the rule is stronger and simpler: only the orchestrator (control plane) opens bot-bottle.db; the data plane and the console reach state through the control-plane RPC, never a direct file handle. No agent-facing component gets the file, so none can forge attribution. (This supersedes the earlier ro-mount idea.) This rule is now in force across the data plane (issue #469): the supervise daemon, the egress addon, and the git-gate pre-receive hook queue proposals and poll for their responses over the agent-side supervise RPC (POST /supervise/propose + POST /supervise/poll, attributed by (source_ip, identity_token) exactly like /resolve), so a bottle can only ever queue or read its own proposals. The gateway/data-plane containers no longer bind-mount the DB directory or carry SUPERVISE_DB_PATH; the orchestrator is the sole opener of the one file.

Implementation note for the VM slices: SQLite WAL over a guest share (virtiofs/9p) is finicky (the -shm/-wal files need real mmap/locking), which is a second reason the DB wants a host-side owner the orchestrator reaches over the RPC rather than a shared mount into the VM. WAL on the shared DB is therefore a deliberate, tested future change — not enabled ad hoc. sqlite3 itself is stdlib, so "the host needs SQLite" is a non-cost.

Gateway CA: host-resident, like the DB

The shared gateway bumps TLS with a self-generated mitmproxy CA, and every bottle installs that CA into its trust store to accept the bumped leaves. So the CA is durable per-host state with the same rule as the DB: it must outlive any single gateway container, or a restart mints a fresh CA that every already-running bottle distrusts — the TLS handshake then fails even after the bottle re-resolves and reconnects to the moved gateway (issue #450, a re-attachment blocker distinct from #443/#445).

The CA lives on the host filesystem at bot_bottle_root()/gateway-ca (host_gateway_ca_dir()), bind-mounted into the container at mitmproxy's confdir. This is deliberately a host bind-mount, not a container-runtime named volume: a named volume survives ordinary container removal but can be silently wiped by Docker's or Apple Container's volume-prune commands during routine host maintenance, which is exactly how the ephemeral-CA symptom shows up in practice. A path under the app-data root is not managed or pruned by the container runtime, and stays directly inspectable and rotatable from the host. mitmproxy reuses an existing CA and generates one only on first run, so the bind-mount alone gives "adopt-existing, generate-on-first-run" for free.

The macOS backend uses the same host-resident CA directory and bind-mounts it into the consolidated Apple infra container. Its bot-bottle-mac-db named volume remains container-only because that prevents incoherent cross-kernel SQLite locking, but the CA is deliberately not stored there: Apple Container also has a container volume prune operation, and the named volume is temporarily unreferenced while the infra container is recreated. Keeping the CA on the host makes both ordinary recreation and volume pruning safe.

Deliberate rollover is the explicit inverse: rotate_gateway_ca() removes the persisted CA material so the next start remints it, and the python -m bot_bottle.orchestrator.rotate_ca one-shot wires that together with dropping the running gateway container (whose mitmproxy still holds the old CA in memory). Rotation does not auto-re-provision the new CA into running bottles — those re-attach to install the new anchor — so it is an operator action with a brief egress interruption, never an implicit one.

Separating the planes: distinct orchestrator and gateway runtimes

The combined per-host infra unit above (one VM on Firecracker/macOS, one container on Docker running both planes) was forced by DB coherence, not preference: two guest kernels sharing a virtiofs-mounted bot-bottle.db don't get coherent fcntl locks, so both planes had to live in the one guest that owned the file. Issue #469 removed that constraint — the data plane no longer opens the DB at all; it reaches the queue/registry over the control-plane RPC (see §State). With the coupling gone, the orchestrator (control plane) and the gateway (data plane) can run as separate runtimes: separate VMs on Firecracker, separate containers on macOS and Docker. (This only re-splits the orchestrator↔gateway boundary; agents stay one unit-each, unchanged.)

Why do it: less surface to compromise (a breached gateway shares no runtime with the control plane), independent lifecycle/placement (scale or relocate the gateway; put the orchestrator elsewhere), and separation of concerns the topology enforces rather than the process boundary merely implying it. The cost is lifecycle — two units to build/start/adopt/ health-check/stop per backend — not access (below) and not, right-sized, memory (below).

Access: keep the agent's isolation, gain it where it's missing

The only relationship that changes is gateway → orchestrator: loopback (127.0.0.1:8099) inside one unit today, a network hop when split. Everything else is unchanged — CLI/console reach the orchestrator as before, and the agent reaches only the gateway's data ports and never needs the orchestrator (it flows agent → gateway supervise MCP → orchestrator, never agent → orchestrator directly).

The invariant to preserve is agent cannot reach the orchestrator — and today it holds unevenly:

Backend	agent → orchestrator today	enforced by
Docker	reachable at gateway_ip:8099	token only — shared plain bridge, orchestrator binds 0.0.0.0
macOS	reachable at infra_ip:8099	token only — no pfctl rule
Firecracker	unreachable	nft: /31 TAP, forward chain drops all but the DNAT'd gateway ports

So on Docker/macOS the agent is one curl from the control plane, gated only by the JWT (one open-mode bug — exactly the class of the #469 Firecracker regression — or one token leak from a bypass). Firecracker already blocks it at L3. The split is the moment to fix that, not a risk to it:

• Docker / macOS — a dedicated bot-bottle-orchestrator control network. A second --internal network that only the orchestrator and gateway join. The gateway is dual-homed (data network bot-bottle-gateway + control network bot-bottle-orchestrator); the orchestrator joins the control network only (plus the host-loopback publish the CLI already uses). The agent is never attached to the control network, so it has no route to the orchestrator — the same L3 block Firecracker already has. The one wiring change: gateway daemons point BOT_BOTTLE_ORCHESTRATOR_URL at http://bot-bottle-orchestrator:8099 (control-net address) instead of loopback. DockerGateway already takes an orchestrator_url and owns its network — it needs the second --network and the URL retargeted; the orchestrator side is Dockerfile.orchestrator run as its own container on the control net. macOS mirrors this with a second container network and the orchestrator's pinned control-net IP (no Apple DNS — reuse the gateway_hosts /etc/hosts mechanism).
• Firecracker — the isolation is nearly free; the work is a second VM. The nft forward chain already drops everything an agent VM sends except the DNAT'd gateway ports, so re-pointing that single DNAT rule at the gateway VM (dnat to $(gw_guest) instead of $(orch_guest) in scripts/firecracker-netpool.sh) isolates agents from a separate orchestrator VM with zero new agent rules — the existing drop handles the rest. The only added nft is a second infra link that mirrors today's orch /31 block (a masquerade for its upstream egress) plus one iifname "$GW_IFACE" oifname "$ORCH_IFACE" accept so the gateway VM reaches orch_guest:8099. The actual effort is the lifecycle — a second infra VM in the pool (boot / adopt / health / teardown, split init scripts) — mechanical because it mirrors the existing single-VM code. All of it is validatable on a Firecracker host (inspect the nft table; confirm an agent VM cannot reach orch_guest:8099), so despite the PRD's earlier framing this is not the hard or risky part of the split.

Role-scoped JWT auth (#469) stays as defense-in-depth on all three, unchanged. A compromised gateway can already reach the orchestrator today (loopback, with its gateway JWT) and role scoping is what contains that — so the split adds no new surface for the gateway-compromise case; it only closes the agent path on Docker/macOS.

Memory: fixed ceilings, and where builds run

Firecracker guest memory is a fixed ceiling set at boot (mem_size_mib); it's demand-paged by the host (RSS grows with what the guest touches) but there is no reclaim without a balloon device, and none is configured — so plan for the ceiling. The combined infra VM is 4096 MiB, but that headroom is for buildah (in-guest agent-image builds), not the daemons, which are light. Steady-state daemon footprints are small: the orchestrator (stdlib-only Python HTTP + SQLite) is a few hundred MiB; the gateway (mitmproxy TLS bump + DLP body buffering, git-http, supervise, gitleaks) is 512 MiB–1 GB.

The real ceiling driver is where agent-image builds run (buildah's 2–4 GB working set). v1 decision: builds stay in the orchestrator — it is the control-plane/management side that owns launches (and already hosts builds in today's combined unit), and the data-plane gateway must not build. So the orchestrator keeps a ~4 GB ceiling (mostly idle, spiking during a build) and the gateway is the slim unit (~1 GB). Net for v1 the split is therefore roughly memory-neutral to ~+1 GB — a slim gateway added beside a still-build-capable orchestrator — not the ~1.5 GB total; that win is deferred to a later move of builds onto a dedicated slim build unit (#468-adjacent). Co-locating buildah with the control plane also keeps a network-heavy, image-defined workload next to the signing key + registry — a known tension the dedicated build unit resolves later. A per-unit balloon device to reclaim the orchestrator's build-vs-idle swing is the other future lever. The access and surface-separation gains land now regardless.

iroh belongs in the orchestrator

The remote-access transport (iroh — web console -(iroh)-> orchestrator, #468) terminates at the orchestrator: everything the console/phone drives — launch, teardown, list, supervise approvals, policy — is a control-plane operation, so the trusted control plane is the single remote entry point. Keep it out of the gateway: the data plane is the more-exposed, agent-facing unit, and a goal of the split is to shrink its surface and keep its image slim (no iroh/Rust dependency). Remote terminal access rides the same door and is likewise an orchestrator operation, not a data-plane one: the agent PTY is exec_agent-sourced (ssh -t / container exec --tty / docker exec), which the orchestrator owns — the gateway never holds a PTY. A --remote flag publishes a running agent's session; local and remote clients share it through a session multiplexer (one PTY, many attached clients), operator-role-gated, so the split's "one authenticated remote door" holds for terminals too. Detailed design — multiplexer location, input arbitration, per-client rendering, always-on vs. --remote — is tracked in #478.

Sequencing

Jump straight to the virtualized end state (not a host-daemon stepping stone): a host daemon's agent→localhost transport is throwaway once the orchestrator becomes a VM. Decouple the two risks instead:

• Consolidation risk (one process, all secrets, attribution, reload) and packaging/transport risk (VM-to-VM wiring, the shim) are independent. Develop the orchestrator service as a plain process dev-harness first, so the consolidation logic (attribution, reload, secret handling) is proven with fast iteration — then wrap that exact service in the VM and solve wiring separately.

Backend order (cheapest proof → hardest → last):

1. Docker orchestrator — nearly free (the gateway bundle is already containers; collapse N bundles into one persistent container). Proves consolidation + the BottleBackend seam with the least moving parts.
2. Firecracker orchestrator — the real work: the shim + VM-to-VM routing (host forwards bbfcN → orchestrator TAP; the nft table grows forward rules where today it drops all non-DNAT egress). Built against the dev-harness so the app logic is already proven.
3. macOS (Apple container) — last (container-to-container networking).

That ordering — and "the real work" — is about the original consolidation: the launch-broker shim and wiring the orchestrator into a VM. The later orchestrator/gateway split (§"Separating the planes") is a smaller, separate change, and its per-backend difficulty runs the other way: docker/macOS carry the real work (a new control network, a dual-homed gateway, the token-only→L3 isolation upgrade), while on Firecracker the isolation is nearly free (re-point one DNAT rule) and the only effort is a mechanical second infra VM. Firecracker is not the hard part of the split.

Keep the gateway service one shared thing throughout.

Non-goals

• Removing OCI/Dockerfile support for agent images (0069's concern).
• A single database for all config (see the three-tier table).
• Changing the per-bottle isolation of agent workloads — only the gateway is consolidated; agents stay one-VM/container-each.

Relationship to other work

• PRD 0069 (#348): 0070 subsumes its Stage 1 (per-host sidecar) and Stage 4 (sidecar-as-VM). 0069 retains Stage 2 (nix-built fixed images — a dependency here: the orchestrator and agent base must be nix-built so the broker launches from a fixed image set and bootstrapping has no chicken-and-egg) and Stage 3 (in-VM Dockerfile builder).
• Minimal CI runner (paused): the Firecracker broker + no host Docker is what lets a dedicated gitea runner user drop the root-equivalent docker group — it only needs broker-socket access + kvm/pool group membership. This work unblocks it.
• Generic SecretProvider (#355): the future secret-handling mechanism (see "Secret handling") — generalizes PRD 0048's DeployKeyProvisioner into a user-extensible provider that mints short-lived creds. Shippable independently; the orchestrator's vault mints through it.
• PR #350 (netpool single-source): the same "one source per fact, composition over parallel hierarchies" discipline the contract follows.

Decisions (review 2026-07-13)

• Egress sharing: consolidate egress (worth it). Treat it as a minimal, hardened attack surface for a malicious agent rather than keeping it per-bottle; pair it with the identity token above and the short-lived-vault-token mitigations (Secret handling / #355).
• Control-plane transport: HTTP — most universal/reliable on every host; unix socket is an optional local optimization (see the contract).
• Broker request schema: signed-JWT JSON, static flags + ids only (see the launch broker) — provenance + un-coercible by construction.
• State re-adoption: the restart procedure is:
  1. Singleton: a new orchestrator launch requires no pre-existing orchestrator; any found (healthy or not) is fully shut down first.
  2. Re-adoption waits for the new orchestrator to be healthy.
  3. Once healthy, it discovers all agents needing adoption via both the SQLite registry and live process/VM inspection before serving any other request. The two-source sweep is what closes the in-flight launch race — a launch that started (VM booting / slot claimed) but hadn't committed to SQLite when the old orchestrator died would be invisible to a SQLite-only sweep; process inspection catches it. Launches should also write an intent record ahead of committing resources so the sweep can reconcile intent vs. actual.

Open questions

• VM-to-VM routing: per-backend, and in the design it is BottleBackend.wire() (DNAT+forward for fc, shared-net for docker/apple), not orchestrator logic. Not a blocker — resolvable at implementation time (may require a bit of host modification, as the pool setup already does on NixOS); an earlier "gateways in VMs" spike showed it's feasible.
• Live-reload protocol for per-bottle policy over the HTTP control plane (add/remove routes/keys/proposals without a restart).
• Identity-token delivery: exactly how the per-bottle token is placed where the agent can present it but not swap in another bottle's.
• Remote terminal design (multiplexer location, input arbitration, per-client rendering, always-on vs. --remote) is tracked in #478.