From ee26e9044f7afa2733458de15884149e9c07cdeb Mon Sep 17 00:00:00 2001 From: didericis Date: Fri, 24 Jul 2026 20:41:34 -0400 Subject: [PATCH] docs(prd-0070): fold in the orchestrator/gateway plane-separation design MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Now that #469 got the DB off the data plane (removing the cross-kernel SQLite-coherence reason the planes were consolidated), separating them into distinct runtimes is possible again. Adds a Design subsection to 0070 covering: * Access — the only hop that changes is gateway->orchestrator (loopback -> network). A dedicated `bot-bottle-orchestrator` --internal control network with a dual-homed gateway gives Docker/macOS the L3 agent isolation that only Firecracker's nft currently has (both are token-only today); Firecracker gets it nearly free (agents already can't reach non-DNAT'd ports). * Memory — firecracker's mem_size_mib is a fixed ceiling (no reclaim without a balloon); the 4 GB combined VM is really buildah headroom. Right-sized: ~512 MiB orchestrator + ~1 GB gateway < today's 4 GB, if agent-image builds move off the gateway. * iroh belongs in the orchestrator (control-plane remote entry point); keep the data-plane gateway slim and P2P-free. Plus open questions: where agent-image builds run post-split, and whether agent-PTY-to-mobile relays through the orchestrator's iroh endpoint. Co-Authored-By: Claude Opus 4.8 --- docs/prds/0070-per-host-orchestrator.md | 119 ++++++++++++++++++++++++ 1 file changed, 119 insertions(+) diff --git a/docs/prds/0070-per-host-orchestrator.md b/docs/prds/0070-per-host-orchestrator.md index 0b7c579..af61119 100644 --- a/docs/prds/0070-per-host-orchestrator.md +++ b/docs/prds/0070-per-host-orchestrator.md @@ -353,6 +353,116 @@ in memory). Rotation does not auto-re-provision the new CA into running bottles — those re-attach to install the new anchor — so it is an operator action with a brief egress interruption, never an implicit one. +### Separating the planes: distinct orchestrator and gateway runtimes + +The combined per-host infra unit above (one VM on Firecracker/macOS, one +container on Docker running both planes) was forced by **DB coherence**, not +preference: two guest kernels sharing a virtiofs-mounted `bot-bottle.db` don't +get coherent `fcntl` locks, so both planes had to live in the one guest that +owned the file. **Issue #469 removed that constraint** — the data plane no +longer opens the DB at all; it reaches the queue/registry over the +control-plane RPC (see §State). With the coupling gone, the orchestrator +(control plane) and the gateway (data plane) can run as **separate runtimes**: +separate VMs on Firecracker, separate containers on macOS and Docker. (This +only re-splits the *orchestrator↔gateway* boundary; agents stay one +unit-each, unchanged.) + +**Why do it:** less surface to compromise (a breached gateway shares no +runtime with the control plane), independent lifecycle/placement (scale or +relocate the gateway; put the orchestrator elsewhere), and separation of +concerns the topology *enforces* rather than the process boundary merely +implying it. **The cost is lifecycle** — two units to build/start/adopt/ +health-check/stop per backend — not access (below) and not, right-sized, +memory (below). + +#### Access: keep the agent's isolation, gain it where it's missing + +The only relationship that changes is **gateway → orchestrator**: loopback +(`127.0.0.1:8099`) inside one unit today, a network hop when split. Everything +else is unchanged — CLI/console reach the orchestrator as before, and the +agent reaches only the gateway's data ports and **never** needs the +orchestrator (it flows agent → gateway supervise MCP → orchestrator, never +agent → orchestrator directly). + +The invariant to preserve is **agent cannot reach the orchestrator** — and +today it holds *unevenly*: + +| Backend | agent → orchestrator today | enforced by | +|---|---|---| +| Docker | reachable at `gateway_ip:8099` | **token only** — shared plain bridge, orchestrator binds `0.0.0.0` | +| macOS | reachable at `infra_ip:8099` | **token only** — no `pfctl` rule | +| Firecracker | **unreachable** | nft: /31 TAP, forward chain drops all but the DNAT'd gateway ports | + +So on Docker/macOS the agent is one `curl` from the control plane, gated +*only* by the JWT (one open-mode bug — exactly the class of the #469 +Firecracker regression — or one token leak from a bypass). Firecracker already +blocks it at L3. **The split is the moment to fix that, not a risk to it:** + +- **Docker / macOS — a dedicated `bot-bottle-orchestrator` control network.** + A second `--internal` network that **only** the orchestrator and gateway + join. The gateway is **dual-homed** (data network `bot-bottle-gateway` + + control network `bot-bottle-orchestrator`); the orchestrator joins the + control network only (plus the host-loopback publish the CLI already uses). + The agent is never attached to the control network, so it has **no route** + to the orchestrator — the same L3 block Firecracker already has. The one + wiring change: gateway daemons point `BOT_BOTTLE_ORCHESTRATOR_URL` at + `http://bot-bottle-orchestrator:8099` (control-net address) instead of + loopback. `DockerGateway` already takes an `orchestrator_url` and owns its + network — it needs the second `--network` and the URL retargeted; the + orchestrator side is `Dockerfile.orchestrator` run as its own container on + the control net. macOS mirrors this with a second container network and the + orchestrator's pinned control-net IP (no Apple DNS — reuse the + `gateway_hosts` `/etc/hosts` mechanism). +- **Firecracker — nearly free.** The nft forward chain already drops + everything a VM sends except the DNAT'd gateway ports, so a *separate* + orchestrator VM is unreachable by agent VMs with **no new agent rules**. The + new piece is a gateway-VM ↔ orchestrator-VM link — the same shape as today's + orch `/31` link, but between the two infra VMs instead of loopback within + one — with a forward-accept rule for that link alone. Agent egress DNATs to + the **gateway** VM's data ports; nothing DNATs to the orchestrator VM. + +Role-scoped JWT auth (#469) stays as defense-in-depth on all three, +unchanged. A *compromised gateway* can already reach the orchestrator today +(loopback, with its `gateway` JWT) and role scoping is what contains that — so +the split adds **no new surface** for the gateway-compromise case; it only +closes the *agent* path on Docker/macOS. + +#### Memory: right-size, don't double + +Firecracker guest memory is a **fixed ceiling** set at boot (`mem_size_mib`); +it's demand-paged by the host (RSS grows with what the guest touches) but there +is **no reclaim without a balloon device**, and none is configured — so plan +for the ceiling. The combined infra VM is **4096 MiB**, but that headroom is +for **buildah** (in-guest agent-image builds), not the daemons, which are +light. + +Split naïvely that reads as 2×, but right-sized it is **less** than today: + +- **Orchestrator** — stdlib-only Python HTTP + SQLite, no builds → **256–512 + MiB**. +- **Gateway** — mitmproxy (TLS bump + DLP body buffering) + git-http + + supervise + gitleaks → **512 MiB–1 GB**, *if it does not build images*. + +The real lever is **where agent-image builds run** (buildah's 2–4 GB working +set). Move builds off the gateway — to a host-controller / dedicated build unit +(a #468-adjacent decision) — and a slim orchestrator (~512 MiB) + slim gateway +(~1 GB) ≈ **1.5 GB total vs. today's 4 GB**. A per-unit **balloon device** to +reclaim idle memory is a later option, not needed for v1. + +#### iroh belongs in the orchestrator + +The remote-access transport (iroh — `web console -(iroh)-> orchestrator`, #468) +terminates at the **orchestrator**: everything the console/phone drives — +launch, teardown, list, supervise approvals, policy — is a control-plane +operation, so the trusted control plane is the single remote entry point. Keep +it **out of the gateway**: the data plane is the more-exposed, agent-facing +unit, and a goal of the split is to shrink its surface and keep its image slim +(no iroh/Rust dependency). Agent-PTY-to-mobile streaming is a *data-plane* +stream and a separate concern; if it must reach the phone over the same +endpoint, **relay it through the orchestrator's iroh door** rather than opening +a second door in the gateway — one authenticated remote entry point, plane +split intact. + ## Sequencing Jump straight to the **virtualized** end state (not a host-daemon stepping @@ -440,3 +550,12 @@ Keep the gateway **service one shared thing** throughout. plane (add/remove routes/keys/proposals without a restart). - **Identity-token delivery:** exactly how the per-bottle token is placed where the agent can present it but not swap in another bottle's. +- **Where agent-image builds run once the planes split:** buildah's 2–4 GB + working set is what keeps the combined infra VM at 4 GB. Keeping the gateway + slim (~1 GB) means builds move off it — to a host-controller / dedicated + build unit (#468-adjacent) — or the gateway keeps the build headroom and the + memory win is smaller. Decides the gateway VM/container size. +- **Agent-PTY-to-mobile streaming path:** relay it through the orchestrator's + iroh endpoint (default — one authenticated remote door, plane split intact), + or give the data-plane stream its own transport. Only matters once remote + terminal mirroring lands.