bot-bottle/docs/prds/0070-per-host-orchestrator.md

# PRD 0070: Per-host orchestrator service

- **Status:** Draft
- **Author:** Claude
- **Created:** 2026-07-12
- **Issue:** #351
- **Supersedes:** the Stage-1 / Stage-4 sidecar-consolidation framing of
  PRD 0069 (#348). Depends on 0069's nix-built fixed images (Stage 2) for
  bootstrapping; 0069 still owns the docker-free image-building work.

## Summary

Replace the **per-bottle gateway bundle** with a single **persistent,
per-host orchestrator**: one long-lived service that runs the gateway
functions (egress / git-gate / supervise), coordinates with the console,
and brokers agent launches and teardown. It is **virtualized from the
start** using each backend's native isolation primitive — a Firecracker
microVM on the Firecracker backend, an Apple container on macOS, a Docker
container on the legacy backend — and is fronted by a single
**backend-agnostic contract**. Per-backend variation lives on
`BottleBackend`, not in the orchestrator.

## Motivation

Today each bottle spins up its own gateway bundle (egress mitmproxy +
git-gate + supervise). That costs:

- **Resources.** N bottles → N heavy bundles booting and idling.
- **Operational churn.** Per-launch container/VM lifecycle for the
  gateways, a control path baked at launch and torn down at exit.
- **A blurry contract.** "How a bottle talks to its gateway" is
  re-implemented per backend instead of being one agreed interface.

A per-host orchestrator collapses the first two and forces the third to
be made explicit. It's also the component that will own per-host runtime
**state** (slot leases, the approval queue, the bottle registry) — today
that's ad-hoc `fcntl`-locked files.

## Security review (read this first)

Consolidation is a real change to the trust model. The goal is to **not
significantly weaken** the posture; some properties strengthen, some
weaken, and the weakened ones must be mitigated by design, not hand-waved.

### What gets stronger

- **Build/host isolation of untrusted inputs** (with 0069 Stage 3): user
  Dockerfiles build in a disposable VM instead of on the host.
- **One audited privileged surface.** Today the launcher runs as the full
  host user and needs the Docker socket (root-equivalent). The orchestrator
  model replaces that with a **thin launch broker** (below) — a small,
  structured, auditable privileged core instead of a fat socket.
- **Attribution is enforced, not assumed.** Making source-IP identity a
  first-class contract invariant (below) means each backend must *prove*
  it, rather than the gateway implicitly trusting network position.

### What gets weaker, and the mitigation

1. **Secret concentration.** Per-bottle gateways isolate secrets at the
   process boundary — each holds only its bottle's tokens/keys. A host
   orchestrator concentrates **every bottle's** egress tokens, git deploy
   keys, and the console credential in one long-lived process. A single
   attribution bug leaks bottle A's token into bottle B's request — a class
   of bug that *cannot exist* per-bottle.
   - *Mitigation:* lean on the enforced source-IP invariant for
     attribution; keep the most secret-dense, least-shareable service
     (**git-gate**, per-repo deploy keys, no natural source-IP scoping)
     **per-bottle** unless there's a compelling reason; scope each secret
     to the bottle in the state DB so a lookup can't return the wrong
     bottle's secret by construction (key every secret access by the
     verified source identity, never by ambient state).

2. **Shared fate.** Orchestrator down = no new launches, and running
   agents lose egress / git / supervise. Compromise = the whole host's
   fleet, plus launch authority, plus the console token.
   - *Mitigation:* the orchestrator is itself confined (its own VM/container
     with its own fail-closed egress); make it **restartable without killing
     running agent VMs** (agents keep running; they briefly lose gateway
     connectivity until it's back); persist state to a host volume so a
     restart re-adopts live bottles rather than losing them.

3. **The launch broker is the new privileged core.** We don't eliminate
   host privilege — we shrink and relocate it. If the broker accepts
   arbitrary paths/commands, the orchestrator VM can escape through it.
   - *Mitigation:* the broker takes **structured requests only** — "launch
     bottle from *this* content-addressed, nix-built rootfs on TAP slot
     *k*", never "run this argv". It validates against a fixed image set,
     not caller-supplied paths. It is small enough to audit line-by-line.

4. **The egress proxy now parses every bottle's traffic in one process.**
   Higher blast radius for a mitmproxy/TLS-bump bug.
   - *Mitigation:* this is the argument for virtualizing the orchestrator
     from the start (Stage B, not a host daemon) — the code that TLS-bumps
     and parses agent traffic and holds every token runs **inside its own
     confined VM**, not as a host process. If egress sharing's blast radius
     feels too high, egress can stay per-bottle while supervise (near-zero
     secrets) goes host-level first.

### The attribution invariant

Source-IP attribution is what makes a shared orchestrator safe: one
process serves every bottle and tells them apart by source address. The
*mechanism* is identical everywhere (read source IP → look up bottle); the
**guarantee that the address can't be forged is a per-backend
responsibility** and part of the contract:

> **Invariant:** a packet's source address, as seen by the orchestrator,
> *provably* identifies the originating bottle.

- **Firecracker** — enforced by the `/31` point-to-point TAP + the
  `bot_bottle_fc` nft table (strongest; already built).
- **Docker** — the per-bottle `--internal` network + anti-spoof; weaker,
  must be made explicit.
- **Apple** — the host-only network.

If a backend can't honor the invariant, source-IP consolidation is not
safe there and that backend keeps per-bottle gateways. The invariant is a
hard precondition, not an aspiration.

**Defense-in-depth — a per-bottle identity token.** On top of the
network-layer invariant, inject a per-bottle secret token into every
request the agent makes to the orchestrator (the agent already egresses
through the gateway proxy, so this is cheap to add). It gives an
**application-layer** proof of identity independent of the network layer:

- On **Firecracker** the `/31` + nft already make source IP unspoofable
  *by construction*, so the token is belt-and-suspenders there — but cheap
  insurance against a misconfigured invariant.
- On **weaker backends** (Docker) it is load-bearing, providing attribution
  that doesn't lean on network anti-spoof.

Requirements: the token is **per-bottle, unguessable, and
non-cross-leakable** — a bottle can only ever prove it is *itself* (it
can't learn another bottle's token, given bottle isolation), so a hostile
agent gains nothing by presenting it. The orchestrator provisions the
token at launch and the gateway requires it to attribute + authorize.
Note this hardens *attribution*, not *secret exposure*: it's app-layer, so
a compromised orchestrator still sees every token (that's the
concentration problem, addressed separately under Secret handling).

### Secret handling — a FUTURE pattern (not v1)

> **Status: future / not required for the initial orchestrator.** The
> initial cut can inject secrets as today; this section records the
> direction so v1 doesn't paint itself into a corner. Tracked as its own
> work in **#355** (generic `SecretProvider`).

The residual weakness after all of the above is **long-lived credential
concentration** — the data-plane proxies must hold every bottle's upstream
tokens because the agent must never see them. You can't policy-gate the
component whose job is to *use* all the secrets, so a full RCE of a proxy
drains its authorized set regardless. Two moves bound this without
pretending to prevent it:

1. **Vault as a separate trust domain.** The long-lived *roots* live in a
   distinct process (ideally its own VM) that the byte-parsing data plane
   never shares memory with. The proxies request secrets from it; the
   crown jewels are not in the process an agent-facing parser can pop.
2. **Derive short-lived, scoped creds where the upstream allows it.** The
   vault holds the root and mints expiring, narrowly-scoped credentials
   per bottle-start (or per request) — GitHub App installation tokens,
   OAuth/STS token exchange, forge deploy tokens. A compromise then leaks
   short-lived material, not permanent keys. For upstreams stuck on static
   keys the vault passes the value through (only the at-rest / audit /
   revocation benefits apply, not lifetime reduction) — this residue is
   accepted and documented, not solved.

Even a plain per-request fetch (no derivation) still buys **at-rest**
reduction (process memory holds only in-flight secrets), a **detection /
rate-limit / revocation chokepoint**, and clean **cross-component
scoping** (an egress RCE can't request git-gate's creds). It does *not*
prevent abuse of currently-authorized access during the compromise window.

**Mechanism (see #355):** generalize the existing `DeployKeyProvisioner`
(PRD 0048) into a user-extensible **`SecretProvider`** droppable into the
manifest anywhere a raw token value is accepted, discovered from
`~/.bot-bottle/contrib/<name>/secret_provider.py` exactly as user
`AgentProvider`s are — because maintaining every forge/cloud/OAuth
provider in-tree is untenable. The orchestrator's vault mints via these
providers; but the abstraction is shippable independently and today's
per-bottle gateways can use it too.

## Design

### The contract (backend-agnostic)

Three surfaces; only one is per-backend.

1. **Control plane (CLI / console → orchestrator)** — an RPC:
   `launch_bottle`, `teardown_bottle`, `register_policy`,
   `deregister_bottle`, `supervise_queue`. Fully backend-agnostic. Both the
   local `cli.py` and the remote console funnel through it, so policy is
   uniform and `cli.py` becomes a thin client rather than a parallel
   launcher. **Transport: HTTP** — the most universal/reliable choice on
   every host (no vsock/unix-socket portability caveats); a local unix
   socket is a fine optimization, but HTTP is the wire contract.
2. **Data plane (agent → orchestrator)** — the egress / git / supervise
   endpoints. Already agnostic today (agents dial `http://gateway:9099`);
   only the *address* and *how packets get there* are per-backend.
3. **Launch / wire (orchestrator → backend)** — the irreducibly
   backend-specific part; lives on `BottleBackend`.

### One `Orchestrator`, no subclass tree

The orchestrator is a **single concrete class** holding all the
backend-neutral logic — egress addon, git-gate, supervise, source-IP
attribution, live-reload control plane, console client. It never branches
on backend; it *composes* a `BottleBackend`. That composition is what makes
the contract agnostic: there is nothing backend-specific left in the
orchestrator to leak.

Rejected alternative: an `Orchestrator` ABC with per-backend
implementations. The interesting logic (proxies, attribution, control
plane) is backend-neutral, so three subclasses would triplicate the hard
part; and a second hierarchy paralleling `BottleBackend` reintroduces the
same hand-maintained lockstep coupling we just removed from the netpool
constants (PR #350). Composition over a parallel tree.

### `BottleBackend` absorbs the per-backend variation

A small, cohesive surface — reused for launching agent bottles *and* the
orchestrator's own unit (the orchestrator is just another native unit):

```
launch_unit(spec) -> Handle      # agent bottle OR the orchestrator itself
                                 # (fc microVM / apple ctr / docker ctr)
wire(unit, endpoint) -> None     # DNAT+forward (fc) | attach shared net (docker/apple)
endpoint_of(unit) -> Endpoint    # address resolution
health(unit) -> Status
```

Plus the **launch broker** — the answer to "a VM/container can't spawn its
own host-network siblings." The orchestrator can't directly open host
`/dev/kvm` + a host TAP fd (Firecracker), and a container can't spawn
siblings without a root-equivalent socket (Docker). So every backend
exposes a broker the orchestrator calls to launch an agent:

- **Firecracker** — a thin, structured host shim (see security #3). This
  replaces today's implicit "launcher runs as host user."
- **Docker** — the socket today (fat, root-equivalent — the thing 0069's
  Stage 3 removes); a narrower broker later.
- **Apple** — the `container` CLI/daemon.

**Broker request schema:** human-readable JSON of **static flags + ids
only** (never free-form paths/argv — that's what keeps it un-coercible
into arbitrary launches), wrapped as a **signed JWT** so the broker
verifies *provenance*: the request came from the real orchestrator, not a
forged one from a compromised co-located component. The orchestrator signs;
the broker verifies with the orchestrator's public key (provisioned at
broker install). This is the concrete form of security #3's "structured
requests only." Same JSON-with-ids + JWT shape for all
orchestrator↔broker/gateway communication; cases that don't fit get
handled as they arise, with this as the default.

If `BottleBackend` bloats, the pressure valve is composition one level
down: vend a `backend.network()` / `Wiring` collaborator rather than
piling methods on — the same discipline, recursed.

### State: one SQLite DB, owned by the orchestrator

The orchestrator is the natural owner of per-host **runtime state**:

- pool **slot leases** (which bottle holds slot *i*) — replaces today's
  `fcntl`-locked files with SQLite transactions;
- the **supervise approval queue** + remembered approvals;
- the **live bottle registry** (source IP → bottle → policy/secrets refs),
  the lookup table the attribution invariant reads.

This is deliberately **not** a "single source of truth for all config."
Config splits into three tiers with different homes:

| Tier | Example | Home |
|---|---|---|
| Build-time constants | pool size, IP base, nft table | flat `.env` (PR #350) — must be readable by Nix eval + root bash, zero runtime |
| User-authored config | bottle manifests, egress routes, secret refs | declarative files under `~/.bot-bottle/` — trust boundary at `$HOME`, git-trackable, "unknown keys die at load" |
| Runtime state | slot leases, approvals, registry | one shared **`bot-bottle.db`**, solely owned by the orchestrator |

SQLite is right for the runtime tier (mutable, concurrent, queried) and
wrong for the other two (Nix can't read it at eval time; it fights the
declarative manifest trust model). Keep the tiers separate.

**One shared `bot-bottle.db` for all runtime state** (decided in review).
The registry co-tenants the existing host `bot-bottle.db` — the `DbStore`
framework already namespaces each store by `schema_key`, so slot
leases / approvals / registry share one file. One place to query, back up,
and integrate a console against.

- **Host-resident, for durability.** Re-adoption sweeps the registry after
  an orchestrator restart, so state *must* outlive the orchestrator
  instance. The file lives on the host (`bot_bottle_root()/db/bot-bottle.db`);
  the orchestrator unit reaches it, it doesn't carry it.
- **Integrity by sole ownership, not mount permissions.** Agents can't
  touch the DB directly wherever it lives (network-isolated in their
  bottles). The risk is a *compromised agent-facing data-plane service*
  (egress/git-gate, which parse hostile bytes) writing the registry and
  forging attribution. Because it's now one shared file, coarse `ro`/`rw`
  mount-splitting no longer isolates the registry — so the rule is stronger
  and simpler: **only the orchestrator (control plane) opens `bot-bottle.db`;
  the data plane and the console reach state through the control-plane RPC,
  never a direct file handle.** No agent-facing component gets the file, so
  none can forge attribution. (This supersedes the earlier `ro`-mount idea.)
  This rule is now **in force** across the data plane (issue #469): the
  supervise daemon, the egress addon, and the git-gate pre-receive hook queue
  proposals and poll for their responses over the agent-side supervise RPC
  (`POST /supervise/propose` + `POST /supervise/poll`, attributed by
  `(source_ip, identity_token)` exactly like `/resolve`), so a bottle can only
  ever queue or read its own proposals. The gateway/data-plane containers no
  longer bind-mount the DB directory or carry `SUPERVISE_DB_PATH`; the
  orchestrator is the sole opener of the one file.

Implementation note for the VM slices: SQLite **WAL** over a guest share
(virtiofs/9p) is finicky (the `-shm`/`-wal` files need real mmap/locking),
which is a second reason the DB wants a **host-side owner** the orchestrator
reaches over the RPC rather than a shared mount into the VM. WAL on the
shared DB is therefore a deliberate, tested future change — not enabled ad
hoc. `sqlite3` itself is stdlib, so "the host needs SQLite" is a non-cost.

### Gateway CA: host-resident, like the DB

The shared gateway bumps TLS with a self-generated mitmproxy CA, and **every
bottle installs that CA** into its trust store to accept the bumped leaves. So
the CA is durable per-host state with the same rule as the DB: it must outlive
any single gateway container, or a restart mints a fresh CA that every
already-running bottle distrusts — the TLS handshake then fails even after the
bottle re-resolves and reconnects to the moved gateway (issue #450, a
re-attachment blocker distinct from #443/#445).

The CA lives on the **host filesystem** at `bot_bottle_root()/gateway-ca`
(`host_gateway_ca_dir()`), bind-mounted into the container at mitmproxy's
confdir. This is deliberately a host bind-mount, **not a container-runtime
named volume**: a named volume survives ordinary container removal but can be
silently wiped by Docker's or Apple Container's volume-prune commands during
routine host maintenance, which is exactly how the ephemeral-CA symptom shows
up in practice. A path under the app-data root is not managed or pruned by the
container runtime, and stays directly inspectable and rotatable from the host.
mitmproxy reuses an existing CA and generates one only on first run, so the
bind-mount alone gives
"adopt-existing, generate-on-first-run" for free.

The macOS backend uses the same host-resident CA directory and bind-mounts it
into the consolidated Apple infra container. Its `bot-bottle-mac-db` named
volume remains container-only because that prevents incoherent cross-kernel
SQLite locking, but the CA is deliberately not stored there: Apple Container
also has a `container volume prune` operation, and the named volume is
temporarily unreferenced while the infra container is recreated. Keeping the
CA on the host makes both ordinary recreation and volume pruning safe.

**Deliberate rollover** is the explicit inverse: `rotate_gateway_ca()` removes
the persisted CA material so the next start remints it, and the
`python -m bot_bottle.orchestrator.rotate_ca` one-shot wires that together with
dropping the running gateway container (whose mitmproxy still holds the old CA
in memory). Rotation does not auto-re-provision the new CA into running bottles
— those re-attach to install the new anchor — so it is an operator action with
a brief egress interruption, never an implicit one.

### Separating the planes: distinct orchestrator and gateway runtimes

The combined per-host infra unit above (one VM on Firecracker/macOS, one
container on Docker running both planes) was forced by **DB coherence**, not
preference: two guest kernels sharing a virtiofs-mounted `bot-bottle.db` don't
get coherent `fcntl` locks, so both planes had to live in the one guest that
owned the file. **Issue #469 removed that constraint** — the data plane no
longer opens the DB at all; it reaches the queue/registry over the
control-plane RPC (see §State). With the coupling gone, the orchestrator
(control plane) and the gateway (data plane) can run as **separate runtimes**:
separate VMs on Firecracker, separate containers on macOS and Docker. (This
only re-splits the *orchestrator↔gateway* boundary; agents stay one
unit-each, unchanged.)

**Why do it:** less surface to compromise (a breached gateway shares no
runtime with the control plane), independent lifecycle/placement (scale or
relocate the gateway; put the orchestrator elsewhere), and separation of
concerns the topology *enforces* rather than the process boundary merely
implying it. **The cost is lifecycle** — two units to build/start/adopt/
health-check/stop per backend — not access (below) and not, right-sized,
memory (below).

#### Access: keep the agent's isolation, gain it where it's missing

The only relationship that changes is **gateway → orchestrator**: loopback
(`127.0.0.1:8099`) inside one unit today, a network hop when split. Everything
else is unchanged — CLI/console reach the orchestrator as before, and the
agent reaches only the gateway's data ports and **never** needs the
orchestrator (it flows agent → gateway supervise MCP → orchestrator, never
agent → orchestrator directly).

The invariant to preserve is **agent cannot reach the orchestrator** — and
today it holds *unevenly*:

| Backend | agent → orchestrator today | enforced by |
|---|---|---|
| Docker | reachable at `gateway_ip:8099` | **token only** — shared plain bridge, orchestrator binds `0.0.0.0` |
| macOS | reachable at `infra_ip:8099` | **token only** — no `pfctl` rule |
| Firecracker | **unreachable** | nft: /31 TAP, forward chain drops all but the DNAT'd gateway ports |

So on Docker/macOS the agent is one `curl` from the control plane, gated
*only* by the JWT (one open-mode bug — exactly the class of the #469
Firecracker regression — or one token leak from a bypass). Firecracker already
blocks it at L3. **The split is the moment to fix that, not a risk to it:**

- **Docker / macOS — a dedicated `bot-bottle-orchestrator` control network.**
  A second `--internal` network that **only** the orchestrator and gateway
  join. The gateway is **dual-homed** (data network `bot-bottle-gateway` +
  control network `bot-bottle-orchestrator`); the orchestrator joins the
  control network only (plus the host-loopback publish the CLI already uses).
  The agent is never attached to the control network, so it has **no route**
  to the orchestrator — the same L3 block Firecracker already has. The one
  wiring change: gateway daemons point `BOT_BOTTLE_ORCHESTRATOR_URL` at
  `http://bot-bottle-orchestrator:8099` (control-net address) instead of
  loopback. `DockerGateway` already takes an `orchestrator_url` and owns its
  network — it needs the second `--network` and the URL retargeted; the
  orchestrator side is `Dockerfile.orchestrator` run as its own container on
  the control net. macOS mirrors this with a second container network and the
  orchestrator's pinned control-net IP (no Apple DNS — reuse the
  `gateway_hosts` `/etc/hosts` mechanism).
- **Firecracker — nearly free.** The nft forward chain already drops
  everything a VM sends except the DNAT'd gateway ports, so a *separate*
  orchestrator VM is unreachable by agent VMs with **no new agent rules**. The
  new piece is a gateway-VM ↔ orchestrator-VM link — the same shape as today's
  orch `/31` link, but between the two infra VMs instead of loopback within
  one — with a forward-accept rule for that link alone. Agent egress DNATs to
  the **gateway** VM's data ports; nothing DNATs to the orchestrator VM.

Role-scoped JWT auth (#469) stays as defense-in-depth on all three,
unchanged. A *compromised gateway* can already reach the orchestrator today
(loopback, with its `gateway` JWT) and role scoping is what contains that — so
the split adds **no new surface** for the gateway-compromise case; it only
closes the *agent* path on Docker/macOS.

#### Memory: fixed ceilings, and where builds run

Firecracker guest memory is a **fixed ceiling** set at boot (`mem_size_mib`);
it's demand-paged by the host (RSS grows with what the guest touches) but there
is **no reclaim without a balloon device**, and none is configured — so plan
for the ceiling. The combined infra VM is **4096 MiB**, but that headroom is
for **buildah** (in-guest agent-image builds), not the daemons, which are
light. Steady-state daemon footprints are small: the **orchestrator**
(stdlib-only Python HTTP + SQLite) is a few hundred MiB; the **gateway**
(mitmproxy TLS bump + DLP body buffering, git-http, supervise, gitleaks) is
**512 MiB–1 GB**.

The real ceiling driver is **where agent-image builds run** (buildah's 2–4 GB
working set). **v1 decision: builds stay in the orchestrator** — it is the
control-plane/management side that owns launches (and already hosts builds in
today's combined unit), and the data-plane gateway must not build. So the
**orchestrator keeps a ~4 GB ceiling** (mostly idle, spiking during a build)
and the **gateway is the slim unit** (~1 GB). Net for v1 the split is therefore
roughly **memory-neutral to ~+1 GB** — a slim gateway added beside a
still-build-capable orchestrator — *not* the ~1.5 GB total; that win is
deferred to a later move of builds onto a **dedicated slim build unit**
(#468-adjacent). Co-locating buildah with the control plane also keeps a
network-heavy, image-defined workload next to the signing key + registry — a
known tension the dedicated build unit resolves later. A per-unit **balloon
device** to reclaim the orchestrator's build-vs-idle swing is the other future
lever. The access and surface-separation gains land now regardless.

#### iroh belongs in the orchestrator

The remote-access transport (iroh — `web console -(iroh)-> orchestrator`, #468)
terminates at the **orchestrator**: everything the console/phone drives —
launch, teardown, list, supervise approvals, policy — is a control-plane
operation, so the trusted control plane is the single remote entry point. Keep
it **out of the gateway**: the data plane is the more-exposed, agent-facing
unit, and a goal of the split is to shrink its surface and keep its image slim
(no iroh/Rust dependency). Agent-PTY-to-mobile streaming is a *data-plane*
stream and a separate concern; if it must reach the phone over the same
endpoint, **relay it through the orchestrator's iroh door** rather than opening
a second door in the gateway — one authenticated remote entry point, plane
split intact.

## Sequencing

Jump straight to the **virtualized** end state (not a host-daemon stepping
stone): a host daemon's agent→`localhost` transport is throwaway once the
orchestrator becomes a VM. Decouple the two risks instead:

- **Consolidation risk** (one process, all secrets, attribution, reload)
  and **packaging/transport risk** (VM-to-VM wiring, the shim) are
  independent. Develop the orchestrator **service as a plain process
  dev-harness** first, so the consolidation logic (attribution, reload,
  secret handling) is proven with fast iteration — *then* wrap that exact
  service in the VM and solve wiring separately.

Backend order (cheapest proof → hardest → last):

1. **Docker orchestrator** — nearly free (the gateway bundle is already
   containers; collapse N bundles into one persistent container). Proves
   consolidation + the `BottleBackend` seam with the least moving parts.
2. **Firecracker orchestrator** — the real work: the shim + VM-to-VM
   routing (host forwards `bbfcN` → orchestrator TAP; the nft table grows
   forward rules where today it drops all non-DNAT egress). Built against
   the dev-harness so the app logic is already proven.
3. **macOS (Apple container)** — last (container-to-container networking).

Keep the gateway **service one shared thing** throughout.

## Non-goals

- Removing OCI/Dockerfile support for agent images (0069's concern).
- A single database for *all* config (see the three-tier table).
- Changing the per-bottle isolation of agent workloads — only the gateway
  is consolidated; agents stay one-VM/container-each.

## Relationship to other work

- **PRD 0069 (#348):** 0070 subsumes its Stage 1 (per-host sidecar) and
  Stage 4 (sidecar-as-VM). 0069 retains Stage 2 (nix-built fixed images —
  a **dependency** here: the orchestrator and agent base must be
  nix-built so the broker launches from a fixed image set and bootstrapping
  has no chicken-and-egg) and Stage 3 (in-VM Dockerfile builder).
- **Minimal CI runner (paused):** the Firecracker broker + no host Docker
  is what lets a dedicated `gitea` runner user drop the root-equivalent
  `docker` group — it only needs broker-socket access + `kvm`/pool group
  membership. This work unblocks it.
- **Generic `SecretProvider` (#355):** the future secret-handling
  mechanism (see "Secret handling") — generalizes PRD 0048's
  `DeployKeyProvisioner` into a user-extensible provider that mints
  short-lived creds. Shippable independently; the orchestrator's vault
  mints through it.
- **PR #350 (netpool single-source):** the same "one source per fact,
  composition over parallel hierarchies" discipline the contract follows.

## Decisions (review 2026-07-13)

- **Egress sharing:** **consolidate egress** (worth it). Treat it as a
  minimal, hardened attack surface for a malicious agent rather than
  keeping it per-bottle; pair it with the identity token above and the
  short-lived-vault-token mitigations (Secret handling / #355).
- **Control-plane transport:** **HTTP** — most universal/reliable on every
  host; unix socket is an optional local optimization (see the contract).
- **Broker request schema:** **signed-JWT JSON, static flags + ids only**
  (see the launch broker) — provenance + un-coercible by construction.
- **State re-adoption:** the restart procedure is:
  1. **Singleton:** a new orchestrator launch requires no pre-existing
     orchestrator; any found (healthy or not) is fully shut down first.
  2. Re-adoption **waits for the new orchestrator to be healthy**.
  3. Once healthy, it discovers all agents needing adoption via **both the
     SQLite registry and live process/VM inspection** *before serving any
     other request*. The two-source sweep is what closes the *in-flight
     launch* race — a launch that started (VM booting / slot claimed) but
     hadn't committed to SQLite when the old orchestrator died would be
     invisible to a SQLite-only sweep; process inspection catches it.
     Launches should also write an **intent record ahead of committing
     resources** so the sweep can reconcile intent vs. actual.

## Open questions

- **VM-to-VM routing:** per-backend, and in the design it *is*
  `BottleBackend.wire()` (DNAT+forward for fc, shared-net for
  docker/apple), not orchestrator logic. **Not a blocker** — resolvable at
  implementation time (may require a bit of host modification, as the pool
  setup already does on NixOS); an earlier "gateways in VMs" spike showed
  it's feasible.
- **Live-reload protocol** for per-bottle policy over the HTTP control
  plane (add/remove routes/keys/proposals without a restart).
- **Identity-token delivery:** exactly how the per-bottle token is placed
  where the agent can present it but not swap in another bottle's.
- **Agent-PTY-to-mobile streaming path:** relay it through the orchestrator's
  iroh endpoint (default — one authenticated remote door, plane split intact),
  or give the data-plane stream its own transport. Only matters once remote
  terminal mirroring lands.