docs(prd-0070): fold in the orchestrator/gateway plane-separation design
tracker-policy-pr / check-pr (pull_request) Successful in 16s
test / integration-docker (pull_request) Successful in 41s
test / unit (pull_request) Successful in 46s
test / integration-firecracker (pull_request) Successful in 3m40s
test / coverage (pull_request) Successful in 20s
test / publish-infra (pull_request) Has been skipped
tracker-policy-pr / check-pr (pull_request) Successful in 16s
test / integration-docker (pull_request) Successful in 41s
test / unit (pull_request) Successful in 46s
test / integration-firecracker (pull_request) Successful in 3m40s
test / coverage (pull_request) Successful in 20s
test / publish-infra (pull_request) Has been skipped
Now that #469 got the DB off the data plane (removing the cross-kernel SQLite-coherence reason the planes were consolidated), separating them into distinct runtimes is possible again. Adds a Design subsection to 0070 covering: * Access — the only hop that changes is gateway->orchestrator (loopback -> network). A dedicated `bot-bottle-orchestrator` --internal control network with a dual-homed gateway gives Docker/macOS the L3 agent isolation that only Firecracker's nft currently has (both are token-only today); Firecracker gets it nearly free (agents already can't reach non-DNAT'd ports). * Memory — firecracker's mem_size_mib is a fixed ceiling (no reclaim without a balloon); the 4 GB combined VM is really buildah headroom. Right-sized: ~512 MiB orchestrator + ~1 GB gateway < today's 4 GB, if agent-image builds move off the gateway. * iroh belongs in the orchestrator (control-plane remote entry point); keep the data-plane gateway slim and P2P-free. Plus open questions: where agent-image builds run post-split, and whether agent-PTY-to-mobile relays through the orchestrator's iroh endpoint. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
@@ -353,6 +353,116 @@ in memory). Rotation does not auto-re-provision the new CA into running bottles
|
|||||||
— those re-attach to install the new anchor — so it is an operator action with
|
— those re-attach to install the new anchor — so it is an operator action with
|
||||||
a brief egress interruption, never an implicit one.
|
a brief egress interruption, never an implicit one.
|
||||||
|
|
||||||
|
### Separating the planes: distinct orchestrator and gateway runtimes
|
||||||
|
|
||||||
|
The combined per-host infra unit above (one VM on Firecracker/macOS, one
|
||||||
|
container on Docker running both planes) was forced by **DB coherence**, not
|
||||||
|
preference: two guest kernels sharing a virtiofs-mounted `bot-bottle.db` don't
|
||||||
|
get coherent `fcntl` locks, so both planes had to live in the one guest that
|
||||||
|
owned the file. **Issue #469 removed that constraint** — the data plane no
|
||||||
|
longer opens the DB at all; it reaches the queue/registry over the
|
||||||
|
control-plane RPC (see §State). With the coupling gone, the orchestrator
|
||||||
|
(control plane) and the gateway (data plane) can run as **separate runtimes**:
|
||||||
|
separate VMs on Firecracker, separate containers on macOS and Docker. (This
|
||||||
|
only re-splits the *orchestrator↔gateway* boundary; agents stay one
|
||||||
|
unit-each, unchanged.)
|
||||||
|
|
||||||
|
**Why do it:** less surface to compromise (a breached gateway shares no
|
||||||
|
runtime with the control plane), independent lifecycle/placement (scale or
|
||||||
|
relocate the gateway; put the orchestrator elsewhere), and separation of
|
||||||
|
concerns the topology *enforces* rather than the process boundary merely
|
||||||
|
implying it. **The cost is lifecycle** — two units to build/start/adopt/
|
||||||
|
health-check/stop per backend — not access (below) and not, right-sized,
|
||||||
|
memory (below).
|
||||||
|
|
||||||
|
#### Access: keep the agent's isolation, gain it where it's missing
|
||||||
|
|
||||||
|
The only relationship that changes is **gateway → orchestrator**: loopback
|
||||||
|
(`127.0.0.1:8099`) inside one unit today, a network hop when split. Everything
|
||||||
|
else is unchanged — CLI/console reach the orchestrator as before, and the
|
||||||
|
agent reaches only the gateway's data ports and **never** needs the
|
||||||
|
orchestrator (it flows agent → gateway supervise MCP → orchestrator, never
|
||||||
|
agent → orchestrator directly).
|
||||||
|
|
||||||
|
The invariant to preserve is **agent cannot reach the orchestrator** — and
|
||||||
|
today it holds *unevenly*:
|
||||||
|
|
||||||
|
| Backend | agent → orchestrator today | enforced by |
|
||||||
|
|---|---|---|
|
||||||
|
| Docker | reachable at `gateway_ip:8099` | **token only** — shared plain bridge, orchestrator binds `0.0.0.0` |
|
||||||
|
| macOS | reachable at `infra_ip:8099` | **token only** — no `pfctl` rule |
|
||||||
|
| Firecracker | **unreachable** | nft: /31 TAP, forward chain drops all but the DNAT'd gateway ports |
|
||||||
|
|
||||||
|
So on Docker/macOS the agent is one `curl` from the control plane, gated
|
||||||
|
*only* by the JWT (one open-mode bug — exactly the class of the #469
|
||||||
|
Firecracker regression — or one token leak from a bypass). Firecracker already
|
||||||
|
blocks it at L3. **The split is the moment to fix that, not a risk to it:**
|
||||||
|
|
||||||
|
- **Docker / macOS — a dedicated `bot-bottle-orchestrator` control network.**
|
||||||
|
A second `--internal` network that **only** the orchestrator and gateway
|
||||||
|
join. The gateway is **dual-homed** (data network `bot-bottle-gateway` +
|
||||||
|
control network `bot-bottle-orchestrator`); the orchestrator joins the
|
||||||
|
control network only (plus the host-loopback publish the CLI already uses).
|
||||||
|
The agent is never attached to the control network, so it has **no route**
|
||||||
|
to the orchestrator — the same L3 block Firecracker already has. The one
|
||||||
|
wiring change: gateway daemons point `BOT_BOTTLE_ORCHESTRATOR_URL` at
|
||||||
|
`http://bot-bottle-orchestrator:8099` (control-net address) instead of
|
||||||
|
loopback. `DockerGateway` already takes an `orchestrator_url` and owns its
|
||||||
|
network — it needs the second `--network` and the URL retargeted; the
|
||||||
|
orchestrator side is `Dockerfile.orchestrator` run as its own container on
|
||||||
|
the control net. macOS mirrors this with a second container network and the
|
||||||
|
orchestrator's pinned control-net IP (no Apple DNS — reuse the
|
||||||
|
`gateway_hosts` `/etc/hosts` mechanism).
|
||||||
|
- **Firecracker — nearly free.** The nft forward chain already drops
|
||||||
|
everything a VM sends except the DNAT'd gateway ports, so a *separate*
|
||||||
|
orchestrator VM is unreachable by agent VMs with **no new agent rules**. The
|
||||||
|
new piece is a gateway-VM ↔ orchestrator-VM link — the same shape as today's
|
||||||
|
orch `/31` link, but between the two infra VMs instead of loopback within
|
||||||
|
one — with a forward-accept rule for that link alone. Agent egress DNATs to
|
||||||
|
the **gateway** VM's data ports; nothing DNATs to the orchestrator VM.
|
||||||
|
|
||||||
|
Role-scoped JWT auth (#469) stays as defense-in-depth on all three,
|
||||||
|
unchanged. A *compromised gateway* can already reach the orchestrator today
|
||||||
|
(loopback, with its `gateway` JWT) and role scoping is what contains that — so
|
||||||
|
the split adds **no new surface** for the gateway-compromise case; it only
|
||||||
|
closes the *agent* path on Docker/macOS.
|
||||||
|
|
||||||
|
#### Memory: right-size, don't double
|
||||||
|
|
||||||
|
Firecracker guest memory is a **fixed ceiling** set at boot (`mem_size_mib`);
|
||||||
|
it's demand-paged by the host (RSS grows with what the guest touches) but there
|
||||||
|
is **no reclaim without a balloon device**, and none is configured — so plan
|
||||||
|
for the ceiling. The combined infra VM is **4096 MiB**, but that headroom is
|
||||||
|
for **buildah** (in-guest agent-image builds), not the daemons, which are
|
||||||
|
light.
|
||||||
|
|
||||||
|
Split naïvely that reads as 2×, but right-sized it is **less** than today:
|
||||||
|
|
||||||
|
- **Orchestrator** — stdlib-only Python HTTP + SQLite, no builds → **256–512
|
||||||
|
MiB**.
|
||||||
|
- **Gateway** — mitmproxy (TLS bump + DLP body buffering) + git-http +
|
||||||
|
supervise + gitleaks → **512 MiB–1 GB**, *if it does not build images*.
|
||||||
|
|
||||||
|
The real lever is **where agent-image builds run** (buildah's 2–4 GB working
|
||||||
|
set). Move builds off the gateway — to a host-controller / dedicated build unit
|
||||||
|
(a #468-adjacent decision) — and a slim orchestrator (~512 MiB) + slim gateway
|
||||||
|
(~1 GB) ≈ **1.5 GB total vs. today's 4 GB**. A per-unit **balloon device** to
|
||||||
|
reclaim idle memory is a later option, not needed for v1.
|
||||||
|
|
||||||
|
#### iroh belongs in the orchestrator
|
||||||
|
|
||||||
|
The remote-access transport (iroh — `web console -(iroh)-> orchestrator`, #468)
|
||||||
|
terminates at the **orchestrator**: everything the console/phone drives —
|
||||||
|
launch, teardown, list, supervise approvals, policy — is a control-plane
|
||||||
|
operation, so the trusted control plane is the single remote entry point. Keep
|
||||||
|
it **out of the gateway**: the data plane is the more-exposed, agent-facing
|
||||||
|
unit, and a goal of the split is to shrink its surface and keep its image slim
|
||||||
|
(no iroh/Rust dependency). Agent-PTY-to-mobile streaming is a *data-plane*
|
||||||
|
stream and a separate concern; if it must reach the phone over the same
|
||||||
|
endpoint, **relay it through the orchestrator's iroh door** rather than opening
|
||||||
|
a second door in the gateway — one authenticated remote entry point, plane
|
||||||
|
split intact.
|
||||||
|
|
||||||
## Sequencing
|
## Sequencing
|
||||||
|
|
||||||
Jump straight to the **virtualized** end state (not a host-daemon stepping
|
Jump straight to the **virtualized** end state (not a host-daemon stepping
|
||||||
@@ -440,3 +550,12 @@ Keep the gateway **service one shared thing** throughout.
|
|||||||
plane (add/remove routes/keys/proposals without a restart).
|
plane (add/remove routes/keys/proposals without a restart).
|
||||||
- **Identity-token delivery:** exactly how the per-bottle token is placed
|
- **Identity-token delivery:** exactly how the per-bottle token is placed
|
||||||
where the agent can present it but not swap in another bottle's.
|
where the agent can present it but not swap in another bottle's.
|
||||||
|
- **Where agent-image builds run once the planes split:** buildah's 2–4 GB
|
||||||
|
working set is what keeps the combined infra VM at 4 GB. Keeping the gateway
|
||||||
|
slim (~1 GB) means builds move off it — to a host-controller / dedicated
|
||||||
|
build unit (#468-adjacent) — or the gateway keeps the build headroom and the
|
||||||
|
memory win is smaller. Decides the gateway VM/container size.
|
||||||
|
- **Agent-PTY-to-mobile streaming path:** relay it through the orchestrator's
|
||||||
|
iroh endpoint (default — one authenticated remote door, plane split intact),
|
||||||
|
or give the data-plane stream its own transport. Only matters once remote
|
||||||
|
terminal mirroring lands.
|
||||||
|
|||||||
Reference in New Issue
Block a user