# PRD prd-new: Host control server

- **Status:** Draft
- **Author:** Claude
- **Created:** 2026-07-26
- **Issue:** #468

## Summary

Promote the in-process launch broker into a standalone **host control
server**: the single privileged component on the host. Both the CLI and the
orchestrator drive it over HTTP; it brokers agent launches, owns the
orchestrator's own lifecycle, and is the sole writer of host-durable state (the
tamper-evident audit record). This closes the four gaps between today's
well-formed broker *contract* ([`orchestrator/broker.py`](../../bot_bottle/orchestrator/broker.py))
and a real out-of-process service — transport, durable provisioned secret,
replay protection, and a disciplined op vocabulary — and splits host state by
owner and lifetime. The prize: **the CLI no longer needs the Docker socket**,
which is what finally lets a dedicated Gitea runner user drop the
root-equivalent `docker` group (PRD 0070, "Relationship to other work").

## Problem

Container launches run directly from a short-lived CLI process against the
Docker socket. That socket is root-equivalent, so every host that launches
bottles hands root to whoever invokes the CLI — including a CI runner user we
want to keep unprivileged. PRD 0070 already argues for replacing the fat socket
with a **thin, structured, auditable** launch broker, and the contract for that
broker exists and is tested in-process. But it is *only* in-process:
`LaunchBroker.submit(token)` is a method call from
`OrchestratorCore.launch_bottle` ([`service.py:116`](../../bot_bottle/orchestrator/service.py)),
and `DockerBroker` is on no production path — every backend starts the
orchestrator with `--broker stub` ([`__main__.py:54`](../../bot_bottle/orchestrator/__main__.py)).

Four gaps stand between that scaffold and a host service:

1. **No transport.** `submit` is an in-process call. A real service needs a
   `BrokerClient` that POSTs the signed token and a host-side HTTP server that
   verifies and acts.
2. **The signing secret is ephemeral and self-generated.**
   [`__main__.py:53`](../../bot_bottle/orchestrator/__main__.py) does
   `secrets.token_bytes(32)` and hands the *same value* to signer and verifier —
   viable only because they share a process. A separate daemon needs the secret
   provisioned out of band and durable across orchestrator restarts.
3. **Replay protection is claimed but not enforced.** `sign_request` emits `jti`
   and `iat`; `verify_request` reads neither — no expiry window, no `jti` cache
   ([`broker.py:88`](../../bot_bottle/orchestrator/broker.py)). Harmless
   in-process; over a wire a captured launch token replays indefinitely.
4. **The op vocabulary is `launch` / `teardown` only.** Everything else
   host-privileged still lives in the CLI, so the schema has to grow — carefully,
   since PRD 0070's security argument rests on "structured requests only, static
   flags + ids."

Separately, host state has no clear owner. `OrchestratorCore.reconcile` takes
`live_source_ips` as a parameter *only because the orchestrator cannot see the
backend* ([`service.py:137`](../../bot_bottle/orchestrator/service.py)); the
egress traffic log is written to the container's stderr; and there is no durable,
tamper-evident home for the audit record that survives orchestrator destruction.

## Goals / Success Criteria

- A standalone host control server that the CLI and orchestrator reach over
  **HTTP**, with three entry paths working end to end:
  - `web console -(iroh)-> orchestrator -(http)-> host controller -> launch`
  - `cli -(http)-> orchestrator -(http)-> host controller -> launch`
  - `cli -(http)-> host controller` — start / restart / status of the
    orchestrator **itself** (the bootstrap/recovery path #391 targets).
- The launch op is expressed as a **signed JWT of static flags + ids only** and
  verified with **replay protection enforced**: `iat` expiry window + a `jti`
  cache reject a captured token.
- The signing secret is **provisioned out of band and durable** across
  orchestrator restarts (a `TrustDomain` per #476, with a key the orchestrator
  never holds for the host controller's *own* endpoints).
- Host-privileged operations move off the CLI to the control server; **the CLI
  no longer opens the Docker socket** for bottle operations.
- `Orchestrator.reconcile` no longer takes `live_source_ips` — live-bottle
  enumeration becomes an internal control-server call.
- Host-durable state lands as an **append-only, hash-chained JSONL** audit log
  owned solely by the host controller; operational state stays SQLite owned
  solely by the orchestrator.

## Non-goals

- **Removing standing privilege.** This converts on-demand privilege (a CLI the
  user invokes) into standing privilege (a daemon under launchd/systemd). The
  win is that the privilege is *narrower* (structured requests vs. a raw socket),
  not that it disappears. "Always running" is an accepted new property.
- **Asymmetric signing.** We stay HS256 — see Design / "Signing stays
  symmetric."
- **Integrity against a live compromised orchestrator.** Host-location of the
  audit log does not buy this: the orchestrator makes the decisions being audited
  and can forge or omit entries wherever the file lives. An off-box copy is the
  answer, tracked separately.
- **A single unified DB for all state.** Impossible over a guest-kernel share
  (SQLite locking is not coherent); state is split by owner and lifetime instead.
- **The generic `SecretProvider` (#355)** and **remote terminal design (#478)** —
  both ride the same door but are their own work.

## Design

### Topology

The host controller is the sole privileged component. The orchestrator becomes a
client of it for launches, and the CLI becomes a client of it for *both* bottle
operations (indirectly, through the orchestrator) and orchestrator lifecycle
(directly, for bootstrap/recovery — startup can't route through the thing being
started).

```
web console ─(iroh)─▶ orchestrator ─┐
                                    ├─(http, signed JWT)─▶ host controller ─▶ launch
cli ────────(http)──▶ orchestrator ─┘
cli ────────(http, bearer)──────────────────────────────▶ host controller   (orchestrator lifecycle)
```

### Transport: `BrokerClient` + host server

`LaunchBroker.submit(token)` keeps its exact signature and semantics; only the
*wire* changes. A new `BrokerClient` implements the same submit contract by
POSTing the signed token to the host controller (stdlib `urllib`, like the
existing [`orchestrator/client.py`](../../bot_bottle/orchestrator/client.py)),
and the host controller's launch handler is the existing `verify_request` +
`_launch`/`_teardown` path, now reached over HTTP instead of a method call. The
in-process `StubBroker` stays for the dev-harness and tests; `DockerBroker`'s
`_launch`/`_teardown` bodies move behind the server unchanged. Because the client
satisfies the same interface `OrchestratorCore` already depends on, the core does
not change to gain a real backend.

### Signing stays symmetric (HS256)

PRD 0070 nominally specifies asymmetric; the code is HS256 and we keep it.
Asymmetric matters when the verifier is *less* privileged than the signer — here
it is the reverse: the host controller (verifier) is strictly more privileged
than the orchestrator (signer), and a controller that could forge orchestrator
requests gains nothing, since it is already the component that launches. Staying
symmetric also honors the no-runtime-deps policy (stdlib has no Ed25519). This
matches the reasoning already inlined in `broker.py`'s module docstring.

### Replay protection (gap 3)

`sign_request` already emits `jti` and `iat`; `verify_request` must start reading
them:

- **Expiry window.** Reject a token whose `iat` is outside a small window
  (proposed ±60s, one constant) — bounds how long a captured token is useful and
  how much clock skew is tolerated.
- **`jti` cache.** Keep a set of seen `jti`s for at least the expiry window;
  reject a repeat. The cache only needs to remember tokens that could still be
  within their window, so it self-trims by `iat` — no unbounded growth, no
  persistence needed (a token older than the window is already rejected by the
  expiry check, so a cache lost on restart cannot admit a replay).

Both are fail-closed additions to the existing `BrokerAuthError` path: a missing
or malformed `jti`/`iat` is a rejection, not a pass. In-process callers are
unaffected in practice (fresh token per submit), so this can land ahead of the
transport.

### Op vocabulary and the "ids + static flags" rule (gap 4)

Each op moved off the CLI widens the privileged surface, so growth is governed by
one explicit rule, enforced in `verify_request`'s schema check:

> A broker op carries **only ids and enumerated static flags** — a bottle id, a
> pool slot, a **content-addressed** image ref chosen from a fixed set, an op
> name from a closed vocabulary. Never a free-form path, argv, command, or
> caller-supplied filesystem location. If an operation cannot be expressed that
> way, it does not become a broker op.

Operations that fit and move off the CLI (all today in
`backend/*/consolidated_launch.py`, driven by a short-lived CLI process):

| Op | What it does | Fits the rule because |
|---|---|---|
| `launch` / `teardown` | existing | ids + slot + image ref |
| `orchestrator.ensure_running` | start the infra container | no arguments |
| `orchestrator.{start,restart,status}` | lifecycle (the #391 path) | no arguments |
| `list_live` | enumerate running bottles for reconcile | no arguments; returns ids/IPs |
| `allocate_ip` | `next_free_ip` over `_network_container_ips` | no arguments; returns an IP |
| `provision_git_gate` | `cp`/`exec` a per-bottle deploy key into the gateway | bottle id + key handle, no path |
| `reprovision` | `docker exec printenv <ENV_VAR_SECRET>` on a live agent | bottle id + secret *name* |

Image **builds** stay with the orchestrator for v1 (PRD 0070 §Memory: builds run
control-plane-side; a dedicated slim build unit is later, #468-adjacent), so no
`build` broker op is added here.

With `list_live` as an internal control-server call, `Orchestrator.reconcile`'s
`live_source_ips` parameter goes away — the tell PRD 0070 called out that the
orchestrator couldn't see the backend disappears with it.

### Secret provisioning (gap 2)

The shared HS256 secret becomes a durable, out-of-band artifact via the
**`TrustDomain`** seam (#476,
[`trust_domain.py`](../../bot_bottle/trust_domain.py)):

- The **launch-broker secret** is a `TrustDomain` whose key
  (`host_signing_key(<file>)`, minted 0600 on first use, durable under
  `bot_bottle_root()`) is provisioned to the orchestrator (signer) and the host
  controller (verifier). Durability across orchestrator restarts is what makes
  re-adoption work — a restart re-verifies against the same key.
- The **host controller's own lifecycle endpoints** (the direct `cli -> host
  controller` path) get a **separate** `TrustDomain` key the orchestrator never
  holds — exactly the second domain #476's PRD reserves. The orchestrator must
  not be able to mint the credentials used to start and stop it.

This reuses the seam #476 landed rather than re-deriving provisioning per
backend (the PR #471 bug class).

### One daemon, structurally separate handlers (open decision 1)

The audit writer and the broker live in **one daemon** for install simplicity,
but with **no shared parsing** and **different credentials per handler**:

- the **launch** handler requires the signed launch **JWT** (provenance +
  un-coercible schema);
- the **audit-append** handler takes a plain **bearer token** and writes to the
  JSONL log.

This does not defend against orchestrator compromise (it holds both creds) — it
stops a bug in the boring audit path from reaching the privileged launch path.
The launcher stays small enough to audit line-by-line, per PRD 0070.

### State ownership: split by owner and lifetime

A single mounted DB is impossible — SQLite locking is not coherent across guest
kernels over a share, which is why the macOS backend already uses a container-only
volume (`INFRA_DB_VOLUME`). So state splits three ways (depends on #469, which
gets `bot-bottle.db` off the data plane first):

| Owner | State | Home | Shape |
|---|---|---|---|
| **Orchestrator** | `orchestrator_bottles` registry; `bottled_agent_secrets` (encrypted egress tokens); `supervise_proposals` / `supervise_responses` | volume nothing else mounts (generalizing the macOS design) | **SQLite** — mutable, transactional, queried |
| **Host controller** | supervise audit entries; egress traffic log (today → container stderr); host-side config | host filesystem, survives orchestrator/volume destruction | **JSONL** — append-only |
| **Gateway** | none | — | after #469 the data plane holds no DB state |

The historical record is **JSONL, not SQLite**, because it is append-only, never
updated, never transactionally queried: `O_APPEND` writes are atomic, there is no
locking protocol to get wrong, hash-chaining for tamper-evidence is cheap, and it
survives container-runtime volume pruning (the #450 lesson) and stays readable
without the orchestrator running. Both halves of "the audit record" — supervise
decisions and the egress traffic log — land in the one place.

The orchestrator is **sole mounter and sole writer** of its SQLite volume; the
host controller is **sole writer** of the JSONL log, over the authenticated
audit-append channel.

## Implementation chunks

Ordered, each independently mergeable:

1. **Enforce replay protection** in `verify_request` (`iat` window + `jti`
   cache). Pure in-process change; no transport needed. Closes gap 3.
2. **`BrokerClient` + host launch server** over HTTP, reusing `verify_request`
   and the existing `DockerBroker` bodies. Wire `OrchestratorCore` to a
   `BrokerClient` behind a flag; keep `StubBroker` for the dev-harness. Closes
   gap 1.
3. **Durable secret via `TrustDomain`** — provision the launch-broker key to
   signer + verifier; add the host controller's own lifecycle `TrustDomain`.
   Closes gap 2.
4. **Grow the op vocabulary** one op at a time (`list_live` first — it also
   removes `reconcile`'s `live_source_ips`), each behind the ids + static-flags
   rule. Closes gap 4.
5. **JSONL audit log** — the host-controller-owned, hash-chained historical
   record with the plain-bearer audit-append handler; redirect the egress traffic
   log into it.
6. **Drop the Docker socket from the CLI** once every host-privileged op it used
   is a broker op — the payoff that unblocks the unprivileged Gitea runner user.

## Open questions

1. **Schema-width rule enforcement.** The "ids + static flags" rule is stated;
   should `verify_request` reject unknown claim keys outright (strict schema) to
   keep the surface from drifting? Leaning yes.
2. **`jti` cache scope under a plane split.** With one host controller per host
   the in-memory cache is sufficient; if lifecycle ever fans out, the cache and
   the expiry window need to be co-located with the single verifier.
3. **Audit-append back-pressure.** What the audit handler does if the JSONL sink
   is unavailable (fail-closed vs. buffer) — resolve before shipping chunk 5.

## References

- **PRD 0070** — the contract, the launch broker, and the state tiers this
  implements.
- **#469** — get `bot-bottle.db` off the data plane (lands underneath this).
- **#476** ([`prd-new-control-plane-auth-provisioning`](prd-new-control-plane-auth-provisioning.md))
  — the `TrustDomain` seam this plugs the host controller's key into.
- **#391** — backend-agnostic orchestrator restart (the bootstrap path).
- **#386** — prebuilt images from the Gitea OCI registry (the fixed image set the
  broker validates against).
- **#355** — generic `SecretProvider`.
- **#478** — remote terminal design.