docs: defer broker replay protection to its own issue (#494 )

Per PR review, replay protection is too heavy for the host control server MVP. Drop it from the four-gap framing (now three gaps), remove the enforcement design section and implementation chunk, and track the iat-window + jti-cache work in #494 as an independent in-process change. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
docs: PRD for host-side control server (#468 )
2026-07-26 13:32:41 -04:00 · 2026-07-26 13:32:41 -04:00
1 changed files with 273 additions and 0 deletions
@@ -0,0 +1,273 @@
 # PRD prd-new: Host control server
 - **Status:** Draft
 - **Author:** Claude
 - **Created:** 2026-07-26
 - **Issue:** #468
 ## Summary
 Promote the in-process launch broker into a standalone **host control
 server**: the single privileged component on the host. Both the CLI and the
 orchestrator drive it over HTTP; it brokers agent launches, owns the
 orchestrator's own lifecycle, and is the sole writer of host-durable state (the
 tamper-evident audit record). This closes the three gaps between today's
 well-formed broker *contract* ([`orchestrator/broker.py`](../../bot_bottle/orchestrator/broker.py))
 and a real out-of-process service — transport, durable provisioned secret,
 and a disciplined op vocabulary — and splits host state by
 owner and lifetime. The prize: **the CLI no longer needs the Docker socket**,
 which is what finally lets a dedicated Gitea runner user drop the
 root-equivalent `docker` group (PRD 0070, "Relationship to other work").
 ## Problem
 Container launches run directly from a short-lived CLI process against the
 Docker socket. That socket is root-equivalent, so every host that launches
 bottles hands root to whoever invokes the CLI — including a CI runner user we
 want to keep unprivileged. PRD 0070 already argues for replacing the fat socket
 with a **thin, structured, auditable** launch broker, and the contract for that
 broker exists and is tested in-process. But it is *only* in-process:
 `LaunchBroker.submit(token)` is a method call from
 `OrchestratorCore.launch_bottle` ([`service.py:116`](../../bot_bottle/orchestrator/service.py)),
 and `DockerBroker` is on no production path — every backend starts the
 orchestrator with `--broker stub` ([`__main__.py:54`](../../bot_bottle/orchestrator/__main__.py)).
 Three gaps stand between that scaffold and a host service:
 1. **No transport.** `submit` is an in-process call. A real service needs a
   `BrokerClient` that POSTs the signed token and a host-side HTTP server that
   verifies and acts.
 2. **The signing secret is ephemeral and self-generated.**
   [`__main__.py:53`](../../bot_bottle/orchestrator/__main__.py) does
   `secrets.token_bytes(32)` and hands the *same value* to signer and verifier —
   viable only because they share a process. A separate daemon needs the secret
   provisioned out of band and durable across orchestrator restarts.
 3. **The op vocabulary is `launch` / `teardown` only.** Everything else
   host-privileged still lives in the CLI, so the schema has to grow — carefully,
   since PRD 0070's security argument rests on "structured requests only, static
   flags + ids."
 Separately, host state has no clear owner. `OrchestratorCore.reconcile` takes
 `live_source_ips` as a parameter *only because the orchestrator cannot see the
 backend* ([`service.py:137`](../../bot_bottle/orchestrator/service.py)); the
 egress traffic log is written to the container's stderr; and there is no durable,
 tamper-evident home for the audit record that survives orchestrator destruction.
 ## Goals / Success Criteria
 - A standalone host control server that the CLI and orchestrator reach over
  **HTTP**, with three entry paths working end to end:
  - `web console -(iroh)-> orchestrator -(http)-> host controller -> launch`
  - `cli -(http)-> orchestrator -(http)-> host controller -> launch`
  - `cli -(http)-> host controller` — start / restart / status of the
    orchestrator **itself** (the bootstrap/recovery path #391 targets).
 - The launch op is expressed as a **signed JWT of static flags + ids only**,
  verified against a closed schema.
 - The signing secret is **provisioned out of band and durable** across
  orchestrator restarts (a `TrustDomain` per #476, with a key the orchestrator
  never holds for the host controller's *own* endpoints).
 - Host-privileged operations move off the CLI to the control server; **the CLI
  no longer opens the Docker socket** for bottle operations.
 - `Orchestrator.reconcile` no longer takes `live_source_ips` — live-bottle
  enumeration becomes an internal control-server call.
 - Host-durable state lands as an **append-only, hash-chained JSONL** audit log
  owned solely by the host controller; operational state stays SQLite owned
  solely by the orchestrator.
 ## Non-goals
 - **Removing standing privilege.** This converts on-demand privilege (a CLI the
  user invokes) into standing privilege (a daemon under launchd/systemd). The
  win is that the privilege is *narrower* (structured requests vs. a raw socket),
  not that it disappears. "Always running" is an accepted new property.
 - **Asymmetric signing.** We stay HS256 — see Design / "Signing stays
  symmetric."
 - **Integrity against a live compromised orchestrator.** Host-location of the
  audit log does not buy this: the orchestrator makes the decisions being audited
  and can forge or omit entries wherever the file lives. An off-box copy is the
  answer, tracked separately.
 - **A single unified DB for all state.** Impossible over a guest-kernel share
  (SQLite locking is not coherent); state is split by owner and lifetime instead.
 - **The generic `SecretProvider` (#355)** and **remote terminal design (#478)** —
  both ride the same door but are their own work.
 ## Design
 ### Topology
 The host controller is the sole privileged component. The orchestrator becomes a
 client of it for launches, and the CLI becomes a client of it for *both* bottle
 operations (indirectly, through the orchestrator) and orchestrator lifecycle
 (directly, for bootstrap/recovery — startup can't route through the thing being
 started).
 ```
 web console ─(iroh)─▶ orchestrator ─┐
                                    ├─(http, signed JWT)─▶ host controller ─▶ launch
 cli ────────(http)──▶ orchestrator ─┘
 cli ────────(http, bearer)──────────────────────────────▶ host controller   (orchestrator lifecycle)
 ```
 ### Transport: `BrokerClient` + host server
 `LaunchBroker.submit(token)` keeps its exact signature and semantics; only the
 *wire* changes. A new `BrokerClient` implements the same submit contract by
 POSTing the signed token to the host controller (stdlib `urllib`, like the
 existing [`orchestrator/client.py`](../../bot_bottle/orchestrator/client.py)),
 and the host controller's launch handler is the existing `verify_request` +
 `_launch`/`_teardown` path, now reached over HTTP instead of a method call. The
 in-process `StubBroker` stays for the dev-harness and tests; `DockerBroker`'s
 `_launch`/`_teardown` bodies move behind the server unchanged. Because the client
 satisfies the same interface `OrchestratorCore` already depends on, the core does
 not change to gain a real backend.
 ### Signing stays symmetric (HS256)
 PRD 0070 nominally specifies asymmetric; the code is HS256 and we keep it.
 Asymmetric matters when the verifier is *less* privileged than the signer — here
 it is the reverse: the host controller (verifier) is strictly more privileged
 than the orchestrator (signer), and a controller that could forge orchestrator
 requests gains nothing, since it is already the component that launches. Staying
 symmetric also honors the no-runtime-deps policy (stdlib has no Ed25519). This
 matches the reasoning already inlined in `broker.py`'s module docstring.
 ### Replay protection is out of scope (tracked in #494)
 Once the launch token travels over a wire, a captured token could be replayed —
 `sign_request` already emits `jti`/`iat` but `verify_request` reads neither, so
 there is no expiry window or `jti` cache today. Enforcing that (an `iat` window +
 a self-trimming `jti` cache) is a pure in-process change that lands independently
 of this work, and it is deferred to **#494** rather than gating the MVP of the
 host control server. Nothing here depends on it; it can merge before or after.
 ### Op vocabulary and the "ids + static flags" rule (gap 3)
 Each op moved off the CLI widens the privileged surface, so growth is governed by
 one explicit rule, enforced in `verify_request`'s schema check:
 > A broker op carries **only ids and enumerated static flags** — a bottle id, a
 > pool slot, a **content-addressed** image ref chosen from a fixed set, an op
 > name from a closed vocabulary. Never a free-form path, argv, command, or
 > caller-supplied filesystem location. If an operation cannot be expressed that
 > way, it does not become a broker op.
 Operations that fit and move off the CLI (all today in
 `backend/*/consolidated_launch.py`, driven by a short-lived CLI process):
 | Op | What it does | Fits the rule because |
 |---|---|---|
 | `launch` / `teardown` | existing | ids + slot + image ref |
 | `orchestrator.ensure_running` | start the infra container | no arguments |
 | `orchestrator.{start,restart,status}` | lifecycle (the #391 path) | no arguments |
 | `list_live` | enumerate running bottles for reconcile | no arguments; returns ids/IPs |
 | `allocate_ip` | `next_free_ip` over `_network_container_ips` | no arguments; returns an IP |
 | `provision_git_gate` | `cp`/`exec` a per-bottle deploy key into the gateway | bottle id + key handle, no path |
 | `reprovision` | `docker exec printenv <ENV_VAR_SECRET>` on a live agent | bottle id + secret *name* |
 Image **builds** stay with the orchestrator for v1 (PRD 0070 §Memory: builds run
 control-plane-side; a dedicated slim build unit is later, #468-adjacent), so no
 `build` broker op is added here.
 With `list_live` as an internal control-server call, `Orchestrator.reconcile`'s
 `live_source_ips` parameter goes away — the tell PRD 0070 called out that the
 orchestrator couldn't see the backend disappears with it.
 ### Secret provisioning (gap 2)
 The shared HS256 secret becomes a durable, out-of-band artifact via the
 **`TrustDomain`** seam (#476,
 [`trust_domain.py`](../../bot_bottle/trust_domain.py)):
 - The **launch-broker secret** is a `TrustDomain` whose key
  (`host_signing_key(<file>)`, minted 0600 on first use, durable under
  `bot_bottle_root()`) is provisioned to the orchestrator (signer) and the host
  controller (verifier). Durability across orchestrator restarts is what makes
  re-adoption work — a restart re-verifies against the same key.
 - The **host controller's own lifecycle endpoints** (the direct `cli -> host
  controller` path) get a **separate** `TrustDomain` key the orchestrator never
  holds — exactly the second domain #476's PRD reserves. The orchestrator must
  not be able to mint the credentials used to start and stop it.
 This reuses the seam #476 landed rather than re-deriving provisioning per
 backend (the PR #471 bug class).
 ### One daemon, structurally separate handlers (open decision 1)
 The audit writer and the broker live in **one daemon** for install simplicity,
 but with **no shared parsing** and **different credentials per handler**:
 - the **launch** handler requires the signed launch **JWT** (provenance +
  un-coercible schema);
 - the **audit-append** handler takes a plain **bearer token** and writes to the
  JSONL log.
 This does not defend against orchestrator compromise (it holds both creds) — it
 stops a bug in the boring audit path from reaching the privileged launch path.
 The launcher stays small enough to audit line-by-line, per PRD 0070.
 ### State ownership: split by owner and lifetime
 A single mounted DB is impossible — SQLite locking is not coherent across guest
 kernels over a share, which is why the macOS backend already uses a container-only
 volume (`INFRA_DB_VOLUME`). So state splits three ways (depends on #469, which
 gets `bot-bottle.db` off the data plane first):
 | Owner | State | Home | Shape |
 |---|---|---|---|
 | **Orchestrator** | `orchestrator_bottles` registry; `bottled_agent_secrets` (encrypted egress tokens); `supervise_proposals` / `supervise_responses` | volume nothing else mounts (generalizing the macOS design) | **SQLite** — mutable, transactional, queried |
 | **Host controller** | supervise audit entries; egress traffic log (today → container stderr); host-side config | host filesystem, survives orchestrator/volume destruction | **JSONL** — append-only |
 | **Gateway** | none | — | after #469 the data plane holds no DB state |
 The historical record is **JSONL, not SQLite**, because it is append-only, never
 updated, never transactionally queried: `O_APPEND` writes are atomic, there is no
 locking protocol to get wrong, hash-chaining for tamper-evidence is cheap, and it
 survives container-runtime volume pruning (the #450 lesson) and stays readable
 without the orchestrator running. Both halves of "the audit record" — supervise
 decisions and the egress traffic log — land in the one place.
 The orchestrator is **sole mounter and sole writer** of its SQLite volume; the
 host controller is **sole writer** of the JSONL log, over the authenticated
 audit-append channel.
 ## Implementation chunks
 Ordered, each independently mergeable:
 1. **`BrokerClient` + host launch server** over HTTP, reusing `verify_request`
   and the existing `DockerBroker` bodies. Wire `OrchestratorCore` to a
   `BrokerClient` behind a flag; keep `StubBroker` for the dev-harness. Closes
   gap 1.
 2. **Durable secret via `TrustDomain`** — provision the launch-broker key to
   signer + verifier; add the host controller's own lifecycle `TrustDomain`.
   Closes gap 2.
 3. **Grow the op vocabulary** one op at a time (`list_live` first — it also
   removes `reconcile`'s `live_source_ips`), each behind the ids + static-flags
   rule. Closes gap 3.
 4. **JSONL audit log** — the host-controller-owned, hash-chained historical
   record with the plain-bearer audit-append handler; redirect the egress traffic
   log into it.
 5. **Drop the Docker socket from the CLI** once every host-privileged op it used
   is a broker op — the payoff that unblocks the unprivileged Gitea runner user.
 ## Open questions
 1. **Schema-width rule enforcement.** The "ids + static flags" rule is stated;
   should `verify_request` reject unknown claim keys outright (strict schema) to
   keep the surface from drifting? Leaning yes.
 2. **Audit-append back-pressure.** What the audit handler does if the JSONL sink
   is unavailable (fail-closed vs. buffer) — resolve before shipping chunk 5.
 ## References
 - **PRD 0070** — the contract, the launch broker, and the state tiers this
  implements.
 - **#469** — get `bot-bottle.db` off the data plane (lands underneath this).
 - **#476** ([`prd-new-control-plane-auth-provisioning`](prd-new-control-plane-auth-provisioning.md))
  — the `TrustDomain` seam this plugs the host controller's key into.
 - **#391** — backend-agnostic orchestrator restart (the bootstrap path).
 - **#494** — enforce broker replay protection (`iat` window + `jti` cache); split
  out of this PRD as an independent in-process change.
 - **#386** — prebuilt images from the Gitea OCI registry (the fixed image set the
  broker validates against).
 - **#355** — generic `SecretProvider`.
 - **#478** — remote terminal design.