From 8e459647fb79de8c9c5c35f3f8da915357209061 Mon Sep 17 00:00:00 2001 From: claude Date: Sun, 26 Jul 2026 04:42:56 +0000 Subject: [PATCH] docs: PRD for host-side control server (#468) Promote the in-process launch broker into a standalone host control server: the single privileged host component that brokers launches, owns orchestrator lifecycle, and is the sole writer of host-durable state. Closes the four broker gaps (transport, durable provisioned secret, replay protection, disciplined op vocabulary) and splits host state by owner and lifetime (orchestrator SQLite / host JSONL audit / gateway none). The payoff is dropping the Docker socket from the CLI. Co-Authored-By: Claude Opus 4.8 --- docs/prds/prd-new-host-control-server.md | 291 +++++++++++++++++++++++ 1 file changed, 291 insertions(+) create mode 100644 docs/prds/prd-new-host-control-server.md diff --git a/docs/prds/prd-new-host-control-server.md b/docs/prds/prd-new-host-control-server.md new file mode 100644 index 00000000..2a3f623a --- /dev/null +++ b/docs/prds/prd-new-host-control-server.md @@ -0,0 +1,291 @@ +# PRD prd-new: Host control server + +- **Status:** Draft +- **Author:** Claude +- **Created:** 2026-07-26 +- **Issue:** #468 + +## Summary + +Promote the in-process launch broker into a standalone **host control +server**: the single privileged component on the host. Both the CLI and the +orchestrator drive it over HTTP; it brokers agent launches, owns the +orchestrator's own lifecycle, and is the sole writer of host-durable state (the +tamper-evident audit record). This closes the four gaps between today's +well-formed broker *contract* ([`orchestrator/broker.py`](../../bot_bottle/orchestrator/broker.py)) +and a real out-of-process service — transport, durable provisioned secret, +replay protection, and a disciplined op vocabulary — and splits host state by +owner and lifetime. The prize: **the CLI no longer needs the Docker socket**, +which is what finally lets a dedicated Gitea runner user drop the +root-equivalent `docker` group (PRD 0070, "Relationship to other work"). + +## Problem + +Container launches run directly from a short-lived CLI process against the +Docker socket. That socket is root-equivalent, so every host that launches +bottles hands root to whoever invokes the CLI — including a CI runner user we +want to keep unprivileged. PRD 0070 already argues for replacing the fat socket +with a **thin, structured, auditable** launch broker, and the contract for that +broker exists and is tested in-process. But it is *only* in-process: +`LaunchBroker.submit(token)` is a method call from +`OrchestratorCore.launch_bottle` ([`service.py:116`](../../bot_bottle/orchestrator/service.py)), +and `DockerBroker` is on no production path — every backend starts the +orchestrator with `--broker stub` ([`__main__.py:54`](../../bot_bottle/orchestrator/__main__.py)). + +Four gaps stand between that scaffold and a host service: + +1. **No transport.** `submit` is an in-process call. A real service needs a + `BrokerClient` that POSTs the signed token and a host-side HTTP server that + verifies and acts. +2. **The signing secret is ephemeral and self-generated.** + [`__main__.py:53`](../../bot_bottle/orchestrator/__main__.py) does + `secrets.token_bytes(32)` and hands the *same value* to signer and verifier — + viable only because they share a process. A separate daemon needs the secret + provisioned out of band and durable across orchestrator restarts. +3. **Replay protection is claimed but not enforced.** `sign_request` emits `jti` + and `iat`; `verify_request` reads neither — no expiry window, no `jti` cache + ([`broker.py:88`](../../bot_bottle/orchestrator/broker.py)). Harmless + in-process; over a wire a captured launch token replays indefinitely. +4. **The op vocabulary is `launch` / `teardown` only.** Everything else + host-privileged still lives in the CLI, so the schema has to grow — carefully, + since PRD 0070's security argument rests on "structured requests only, static + flags + ids." + +Separately, host state has no clear owner. `OrchestratorCore.reconcile` takes +`live_source_ips` as a parameter *only because the orchestrator cannot see the +backend* ([`service.py:137`](../../bot_bottle/orchestrator/service.py)); the +egress traffic log is written to the container's stderr; and there is no durable, +tamper-evident home for the audit record that survives orchestrator destruction. + +## Goals / Success Criteria + +- A standalone host control server that the CLI and orchestrator reach over + **HTTP**, with three entry paths working end to end: + - `web console -(iroh)-> orchestrator -(http)-> host controller -> launch` + - `cli -(http)-> orchestrator -(http)-> host controller -> launch` + - `cli -(http)-> host controller` — start / restart / status of the + orchestrator **itself** (the bootstrap/recovery path #391 targets). +- The launch op is expressed as a **signed JWT of static flags + ids only** and + verified with **replay protection enforced**: `iat` expiry window + a `jti` + cache reject a captured token. +- The signing secret is **provisioned out of band and durable** across + orchestrator restarts (a `TrustDomain` per #476, with a key the orchestrator + never holds for the host controller's *own* endpoints). +- Host-privileged operations move off the CLI to the control server; **the CLI + no longer opens the Docker socket** for bottle operations. +- `Orchestrator.reconcile` no longer takes `live_source_ips` — live-bottle + enumeration becomes an internal control-server call. +- Host-durable state lands as an **append-only, hash-chained JSONL** audit log + owned solely by the host controller; operational state stays SQLite owned + solely by the orchestrator. + +## Non-goals + +- **Removing standing privilege.** This converts on-demand privilege (a CLI the + user invokes) into standing privilege (a daemon under launchd/systemd). The + win is that the privilege is *narrower* (structured requests vs. a raw socket), + not that it disappears. "Always running" is an accepted new property. +- **Asymmetric signing.** We stay HS256 — see Design / "Signing stays + symmetric." +- **Integrity against a live compromised orchestrator.** Host-location of the + audit log does not buy this: the orchestrator makes the decisions being audited + and can forge or omit entries wherever the file lives. An off-box copy is the + answer, tracked separately. +- **A single unified DB for all state.** Impossible over a guest-kernel share + (SQLite locking is not coherent); state is split by owner and lifetime instead. +- **The generic `SecretProvider` (#355)** and **remote terminal design (#478)** — + both ride the same door but are their own work. + +## Design + +### Topology + +The host controller is the sole privileged component. The orchestrator becomes a +client of it for launches, and the CLI becomes a client of it for *both* bottle +operations (indirectly, through the orchestrator) and orchestrator lifecycle +(directly, for bootstrap/recovery — startup can't route through the thing being +started). + +``` +web console ─(iroh)─▶ orchestrator ─┐ + ├─(http, signed JWT)─▶ host controller ─▶ launch +cli ────────(http)──▶ orchestrator ─┘ +cli ────────(http, bearer)──────────────────────────────▶ host controller (orchestrator lifecycle) +``` + +### Transport: `BrokerClient` + host server + +`LaunchBroker.submit(token)` keeps its exact signature and semantics; only the +*wire* changes. A new `BrokerClient` implements the same submit contract by +POSTing the signed token to the host controller (stdlib `urllib`, like the +existing [`orchestrator/client.py`](../../bot_bottle/orchestrator/client.py)), +and the host controller's launch handler is the existing `verify_request` + +`_launch`/`_teardown` path, now reached over HTTP instead of a method call. The +in-process `StubBroker` stays for the dev-harness and tests; `DockerBroker`'s +`_launch`/`_teardown` bodies move behind the server unchanged. Because the client +satisfies the same interface `OrchestratorCore` already depends on, the core does +not change to gain a real backend. + +### Signing stays symmetric (HS256) + +PRD 0070 nominally specifies asymmetric; the code is HS256 and we keep it. +Asymmetric matters when the verifier is *less* privileged than the signer — here +it is the reverse: the host controller (verifier) is strictly more privileged +than the orchestrator (signer), and a controller that could forge orchestrator +requests gains nothing, since it is already the component that launches. Staying +symmetric also honors the no-runtime-deps policy (stdlib has no Ed25519). This +matches the reasoning already inlined in `broker.py`'s module docstring. + +### Replay protection (gap 3) + +`sign_request` already emits `jti` and `iat`; `verify_request` must start reading +them: + +- **Expiry window.** Reject a token whose `iat` is outside a small window + (proposed ±60s, one constant) — bounds how long a captured token is useful and + how much clock skew is tolerated. +- **`jti` cache.** Keep a set of seen `jti`s for at least the expiry window; + reject a repeat. The cache only needs to remember tokens that could still be + within their window, so it self-trims by `iat` — no unbounded growth, no + persistence needed (a token older than the window is already rejected by the + expiry check, so a cache lost on restart cannot admit a replay). + +Both are fail-closed additions to the existing `BrokerAuthError` path: a missing +or malformed `jti`/`iat` is a rejection, not a pass. In-process callers are +unaffected in practice (fresh token per submit), so this can land ahead of the +transport. + +### Op vocabulary and the "ids + static flags" rule (gap 4) + +Each op moved off the CLI widens the privileged surface, so growth is governed by +one explicit rule, enforced in `verify_request`'s schema check: + +> A broker op carries **only ids and enumerated static flags** — a bottle id, a +> pool slot, a **content-addressed** image ref chosen from a fixed set, an op +> name from a closed vocabulary. Never a free-form path, argv, command, or +> caller-supplied filesystem location. If an operation cannot be expressed that +> way, it does not become a broker op. + +Operations that fit and move off the CLI (all today in +`backend/*/consolidated_launch.py`, driven by a short-lived CLI process): + +| Op | What it does | Fits the rule because | +|---|---|---| +| `launch` / `teardown` | existing | ids + slot + image ref | +| `orchestrator.ensure_running` | start the infra container | no arguments | +| `orchestrator.{start,restart,status}` | lifecycle (the #391 path) | no arguments | +| `list_live` | enumerate running bottles for reconcile | no arguments; returns ids/IPs | +| `allocate_ip` | `next_free_ip` over `_network_container_ips` | no arguments; returns an IP | +| `provision_git_gate` | `cp`/`exec` a per-bottle deploy key into the gateway | bottle id + key handle, no path | +| `reprovision` | `docker exec printenv ` on a live agent | bottle id + secret *name* | + +Image **builds** stay with the orchestrator for v1 (PRD 0070 §Memory: builds run +control-plane-side; a dedicated slim build unit is later, #468-adjacent), so no +`build` broker op is added here. + +With `list_live` as an internal control-server call, `Orchestrator.reconcile`'s +`live_source_ips` parameter goes away — the tell PRD 0070 called out that the +orchestrator couldn't see the backend disappears with it. + +### Secret provisioning (gap 2) + +The shared HS256 secret becomes a durable, out-of-band artifact via the +**`TrustDomain`** seam (#476, +[`trust_domain.py`](../../bot_bottle/trust_domain.py)): + +- The **launch-broker secret** is a `TrustDomain` whose key + (`host_signing_key()`, minted 0600 on first use, durable under + `bot_bottle_root()`) is provisioned to the orchestrator (signer) and the host + controller (verifier). Durability across orchestrator restarts is what makes + re-adoption work — a restart re-verifies against the same key. +- The **host controller's own lifecycle endpoints** (the direct `cli -> host + controller` path) get a **separate** `TrustDomain` key the orchestrator never + holds — exactly the second domain #476's PRD reserves. The orchestrator must + not be able to mint the credentials used to start and stop it. + +This reuses the seam #476 landed rather than re-deriving provisioning per +backend (the PR #471 bug class). + +### One daemon, structurally separate handlers (open decision 1) + +The audit writer and the broker live in **one daemon** for install simplicity, +but with **no shared parsing** and **different credentials per handler**: + +- the **launch** handler requires the signed launch **JWT** (provenance + + un-coercible schema); +- the **audit-append** handler takes a plain **bearer token** and writes to the + JSONL log. + +This does not defend against orchestrator compromise (it holds both creds) — it +stops a bug in the boring audit path from reaching the privileged launch path. +The launcher stays small enough to audit line-by-line, per PRD 0070. + +### State ownership: split by owner and lifetime + +A single mounted DB is impossible — SQLite locking is not coherent across guest +kernels over a share, which is why the macOS backend already uses a container-only +volume (`INFRA_DB_VOLUME`). So state splits three ways (depends on #469, which +gets `bot-bottle.db` off the data plane first): + +| Owner | State | Home | Shape | +|---|---|---|---| +| **Orchestrator** | `orchestrator_bottles` registry; `bottled_agent_secrets` (encrypted egress tokens); `supervise_proposals` / `supervise_responses` | volume nothing else mounts (generalizing the macOS design) | **SQLite** — mutable, transactional, queried | +| **Host controller** | supervise audit entries; egress traffic log (today → container stderr); host-side config | host filesystem, survives orchestrator/volume destruction | **JSONL** — append-only | +| **Gateway** | none | — | after #469 the data plane holds no DB state | + +The historical record is **JSONL, not SQLite**, because it is append-only, never +updated, never transactionally queried: `O_APPEND` writes are atomic, there is no +locking protocol to get wrong, hash-chaining for tamper-evidence is cheap, and it +survives container-runtime volume pruning (the #450 lesson) and stays readable +without the orchestrator running. Both halves of "the audit record" — supervise +decisions and the egress traffic log — land in the one place. + +The orchestrator is **sole mounter and sole writer** of its SQLite volume; the +host controller is **sole writer** of the JSONL log, over the authenticated +audit-append channel. + +## Implementation chunks + +Ordered, each independently mergeable: + +1. **Enforce replay protection** in `verify_request` (`iat` window + `jti` + cache). Pure in-process change; no transport needed. Closes gap 3. +2. **`BrokerClient` + host launch server** over HTTP, reusing `verify_request` + and the existing `DockerBroker` bodies. Wire `OrchestratorCore` to a + `BrokerClient` behind a flag; keep `StubBroker` for the dev-harness. Closes + gap 1. +3. **Durable secret via `TrustDomain`** — provision the launch-broker key to + signer + verifier; add the host controller's own lifecycle `TrustDomain`. + Closes gap 2. +4. **Grow the op vocabulary** one op at a time (`list_live` first — it also + removes `reconcile`'s `live_source_ips`), each behind the ids + static-flags + rule. Closes gap 4. +5. **JSONL audit log** — the host-controller-owned, hash-chained historical + record with the plain-bearer audit-append handler; redirect the egress traffic + log into it. +6. **Drop the Docker socket from the CLI** once every host-privileged op it used + is a broker op — the payoff that unblocks the unprivileged Gitea runner user. + +## Open questions + +1. **Schema-width rule enforcement.** The "ids + static flags" rule is stated; + should `verify_request` reject unknown claim keys outright (strict schema) to + keep the surface from drifting? Leaning yes. +2. **`jti` cache scope under a plane split.** With one host controller per host + the in-memory cache is sufficient; if lifecycle ever fans out, the cache and + the expiry window need to be co-located with the single verifier. +3. **Audit-append back-pressure.** What the audit handler does if the JSONL sink + is unavailable (fail-closed vs. buffer) — resolve before shipping chunk 5. + +## References + +- **PRD 0070** — the contract, the launch broker, and the state tiers this + implements. +- **#469** — get `bot-bottle.db` off the data plane (lands underneath this). +- **#476** ([`prd-new-control-plane-auth-provisioning`](prd-new-control-plane-auth-provisioning.md)) + — the `TrustDomain` seam this plugs the host controller's key into. +- **#391** — backend-agnostic orchestrator restart (the bootstrap path). +- **#386** — prebuilt images from the Gitea OCI registry (the fixed image set the + broker validates against). +- **#355** — generic `SecretProvider`. +- **#478** — remote terminal design.