bot-bottle/docs/prds/prd-new-host-control-server.md

PRD prd-new: Host control server

• Status: Draft
• Author: Claude
• Created: 2026-07-26
• Issue: #468

Summary

Promote the in-process launch broker into a standalone host control server: the single privileged component on the host. Both the CLI and the orchestrator drive it over HTTP; it brokers agent launches, owns the orchestrator's own lifecycle, and is the sole writer of host-durable state (the tamper-evident audit record). This closes the four gaps between today's well-formed broker contract (orchestrator/broker.py) and a real out-of-process service — transport, durable provisioned secret, replay protection, and a disciplined op vocabulary — and splits host state by owner and lifetime. The prize: the CLI no longer needs the Docker socket, which is what finally lets a dedicated Gitea runner user drop the root-equivalent docker group (PRD 0070, "Relationship to other work").

Problem

Container launches run directly from a short-lived CLI process against the Docker socket. That socket is root-equivalent, so every host that launches bottles hands root to whoever invokes the CLI — including a CI runner user we want to keep unprivileged. PRD 0070 already argues for replacing the fat socket with a thin, structured, auditable launch broker, and the contract for that broker exists and is tested in-process. But it is only in-process: LaunchBroker.submit(token) is a method call from OrchestratorCore.launch_bottle (service.py:116), and DockerBroker is on no production path — every backend starts the orchestrator with --broker stub (__main__.py:54).

Four gaps stand between that scaffold and a host service:

1. No transport. submit is an in-process call. A real service needs a BrokerClient that POSTs the signed token and a host-side HTTP server that verifies and acts.
2. The signing secret is ephemeral and self-generated. __main__.py:53 does secrets.token_bytes(32) and hands the same value to signer and verifier — viable only because they share a process. A separate daemon needs the secret provisioned out of band and durable across orchestrator restarts.
3. Replay protection is claimed but not enforced. sign_request emits jti and iat; verify_request reads neither — no expiry window, no jti cache (broker.py:88). Harmless in-process; over a wire a captured launch token replays indefinitely.
4. The op vocabulary is launch / teardown only. Everything else host-privileged still lives in the CLI, so the schema has to grow — carefully, since PRD 0070's security argument rests on "structured requests only, static flags + ids."

Separately, host state has no clear owner. OrchestratorCore.reconcile takes live_source_ips as a parameter only because the orchestrator cannot see the backend (service.py:137); the egress traffic log is written to the container's stderr; and there is no durable, tamper-evident home for the audit record that survives orchestrator destruction.

Goals / Success Criteria

• A standalone host control server that the CLI and orchestrator reach over HTTP, with three entry paths working end to end:
  • web console -(iroh)-> orchestrator -(http)-> host controller -> launch
  • cli -(http)-> orchestrator -(http)-> host controller -> launch
  • cli -(http)-> host controller — start / restart / status of the orchestrator itself (the bootstrap/recovery path #391 targets).
• The launch op is expressed as a signed JWT of static flags + ids only and verified with replay protection enforced: iat expiry window + a jti cache reject a captured token.
• The signing secret is provisioned out of band and durable across orchestrator restarts (a TrustDomain per #476, with a key the orchestrator never holds for the host controller's own endpoints).
• Host-privileged operations move off the CLI to the control server; the CLI no longer opens the Docker socket for bottle operations.
• Orchestrator.reconcile no longer takes live_source_ips — live-bottle enumeration becomes an internal control-server call.
• Host-durable state lands as an append-only, hash-chained JSONL audit log owned solely by the host controller; operational state stays SQLite owned solely by the orchestrator.

Non-goals

• Removing standing privilege. This converts on-demand privilege (a CLI the user invokes) into standing privilege (a daemon under launchd/systemd). The win is that the privilege is narrower (structured requests vs. a raw socket), not that it disappears. "Always running" is an accepted new property.
• Asymmetric signing. We stay HS256 — see Design / "Signing stays symmetric."
• Integrity against a live compromised orchestrator. Host-location of the audit log does not buy this: the orchestrator makes the decisions being audited and can forge or omit entries wherever the file lives. An off-box copy is the answer, tracked separately.
• A single unified DB for all state. Impossible over a guest-kernel share (SQLite locking is not coherent); state is split by owner and lifetime instead.
• The generic SecretProvider (#355) and remote terminal design (#478) — both ride the same door but are their own work.

Design

Topology

The host controller is the sole privileged component. The orchestrator becomes a client of it for launches, and the CLI becomes a client of it for both bottle operations (indirectly, through the orchestrator) and orchestrator lifecycle (directly, for bootstrap/recovery — startup can't route through the thing being started).

web console ─(iroh)─▶ orchestrator ─┐
                                    ├─(http, signed JWT)─▶ host controller ─▶ launch
cli ────────(http)──▶ orchestrator ─┘
cli ────────(http, bearer)──────────────────────────────▶ host controller   (orchestrator lifecycle)

Transport: BrokerClient + host server

LaunchBroker.submit(token) keeps its exact signature and semantics; only the wire changes. A new BrokerClient implements the same submit contract by POSTing the signed token to the host controller (stdlib urllib, like the existing orchestrator/client.py), and the host controller's launch handler is the existing verify_request + _launch/_teardown path, now reached over HTTP instead of a method call. The in-process StubBroker stays for the dev-harness and tests; DockerBroker's _launch/_teardown bodies move behind the server unchanged. Because the client satisfies the same interface OrchestratorCore already depends on, the core does not change to gain a real backend.

Signing stays symmetric (HS256)

PRD 0070 nominally specifies asymmetric; the code is HS256 and we keep it. Asymmetric matters when the verifier is less privileged than the signer — here it is the reverse: the host controller (verifier) is strictly more privileged than the orchestrator (signer), and a controller that could forge orchestrator requests gains nothing, since it is already the component that launches. Staying symmetric also honors the no-runtime-deps policy (stdlib has no Ed25519). This matches the reasoning already inlined in broker.py's module docstring.

Replay protection (gap 3)

sign_request already emits jti and iat; verify_request must start reading them:

• Expiry window. Reject a token whose iat is outside a small window (proposed ±60s, one constant) — bounds how long a captured token is useful and how much clock skew is tolerated.
• jti cache. Keep a set of seen jtis for at least the expiry window; reject a repeat. The cache only needs to remember tokens that could still be within their window, so it self-trims by iat — no unbounded growth, no persistence needed (a token older than the window is already rejected by the expiry check, so a cache lost on restart cannot admit a replay).

Both are fail-closed additions to the existing BrokerAuthError path: a missing or malformed jti/iat is a rejection, not a pass. In-process callers are unaffected in practice (fresh token per submit), so this can land ahead of the transport.

Op vocabulary and the "ids + static flags" rule (gap 4)

Each op moved off the CLI widens the privileged surface, so growth is governed by one explicit rule, enforced in verify_request's schema check:

A broker op carries only ids and enumerated static flags — a bottle id, a pool slot, a content-addressed image ref chosen from a fixed set, an op name from a closed vocabulary. Never a free-form path, argv, command, or caller-supplied filesystem location. If an operation cannot be expressed that way, it does not become a broker op.

Operations that fit and move off the CLI (all today in backend/*/consolidated_launch.py, driven by a short-lived CLI process):

Op	What it does	Fits the rule because
launch / teardown	existing	ids + slot + image ref
orchestrator.ensure_running	start the infra container	no arguments
orchestrator.{start,restart,status}	lifecycle (the #391 path)	no arguments
list_live	enumerate running bottles for reconcile	no arguments; returns ids/IPs
allocate_ip	next_free_ip over _network_container_ips	no arguments; returns an IP
provision_git_gate	cp/exec a per-bottle deploy key into the gateway	bottle id + key handle, no path
reprovision	docker exec printenv <ENV_VAR_SECRET> on a live agent	bottle id + secret name

Image builds stay with the orchestrator for v1 (PRD 0070 §Memory: builds run control-plane-side; a dedicated slim build unit is later, #468-adjacent), so no build broker op is added here.

With list_live as an internal control-server call, Orchestrator.reconcile's live_source_ips parameter goes away — the tell PRD 0070 called out that the orchestrator couldn't see the backend disappears with it.

Secret provisioning (gap 2)

The shared HS256 secret becomes a durable, out-of-band artifact via the TrustDomain seam (#476, trust_domain.py):

• The launch-broker secret is a TrustDomain whose key (host_signing_key(<file>), minted 0600 on first use, durable under bot_bottle_root()) is provisioned to the orchestrator (signer) and the host controller (verifier). Durability across orchestrator restarts is what makes re-adoption work — a restart re-verifies against the same key.
• The host controller's own lifecycle endpoints (the direct cli -> host controller path) get a separate TrustDomain key the orchestrator never holds — exactly the second domain #476's PRD reserves. The orchestrator must not be able to mint the credentials used to start and stop it.

This reuses the seam #476 landed rather than re-deriving provisioning per backend (the PR #471 bug class).

One daemon, structurally separate handlers (open decision 1)

The audit writer and the broker live in one daemon for install simplicity, but with no shared parsing and different credentials per handler:

• the launch handler requires the signed launch JWT (provenance + un-coercible schema);
• the audit-append handler takes a plain bearer token and writes to the JSONL log.

This does not defend against orchestrator compromise (it holds both creds) — it stops a bug in the boring audit path from reaching the privileged launch path. The launcher stays small enough to audit line-by-line, per PRD 0070.

State ownership: split by owner and lifetime

A single mounted DB is impossible — SQLite locking is not coherent across guest kernels over a share, which is why the macOS backend already uses a container-only volume (INFRA_DB_VOLUME). So state splits three ways (depends on #469, which gets bot-bottle.db off the data plane first):

Owner	State	Home	Shape
Orchestrator	orchestrator_bottles registry; bottled_agent_secrets (encrypted egress tokens); supervise_proposals / supervise_responses	volume nothing else mounts (generalizing the macOS design)	SQLite — mutable, transactional, queried
Host controller	supervise audit entries; egress traffic log (today → container stderr); host-side config	host filesystem, survives orchestrator/volume destruction	JSONL — append-only
Gateway	none	—	after #469 the data plane holds no DB state

The historical record is JSONL, not SQLite, because it is append-only, never updated, never transactionally queried: O_APPEND writes are atomic, there is no locking protocol to get wrong, hash-chaining for tamper-evidence is cheap, and it survives container-runtime volume pruning (the #450 lesson) and stays readable without the orchestrator running. Both halves of "the audit record" — supervise decisions and the egress traffic log — land in the one place.

The orchestrator is sole mounter and sole writer of its SQLite volume; the host controller is sole writer of the JSONL log, over the authenticated audit-append channel.

Implementation chunks

Ordered, each independently mergeable:

1. Enforce replay protection in verify_request (iat window + jti cache). Pure in-process change; no transport needed. Closes gap 3.
2. BrokerClient + host launch server over HTTP, reusing verify_request and the existing DockerBroker bodies. Wire OrchestratorCore to a BrokerClient behind a flag; keep StubBroker for the dev-harness. Closes gap 1.
3. Durable secret via TrustDomain — provision the launch-broker key to signer + verifier; add the host controller's own lifecycle TrustDomain. Closes gap 2.
4. Grow the op vocabulary one op at a time (list_live first — it also removes reconcile's live_source_ips), each behind the ids + static-flags rule. Closes gap 4.
5. JSONL audit log — the host-controller-owned, hash-chained historical record with the plain-bearer audit-append handler; redirect the egress traffic log into it.
6. Drop the Docker socket from the CLI once every host-privileged op it used is a broker op — the payoff that unblocks the unprivileged Gitea runner user.

Open questions

1. Schema-width rule enforcement. The "ids + static flags" rule is stated; should verify_request reject unknown claim keys outright (strict schema) to keep the surface from drifting? Leaning yes.
2. jti cache scope under a plane split. With one host controller per host the in-memory cache is sufficient; if lifecycle ever fans out, the cache and the expiry window need to be co-located with the single verifier.
3. Audit-append back-pressure. What the audit handler does if the JSONL sink is unavailable (fail-closed vs. buffer) — resolve before shipping chunk 5.

References

• PRD 0070 — the contract, the launch broker, and the state tiers this implements.
• #469 — get bot-bottle.db off the data plane (lands underneath this).
• #476 (prd-new-control-plane-auth-provisioning) — the TrustDomain seam this plugs the host controller's key into.
• #391 — backend-agnostic orchestrator restart (the bootstrap path).
• #386 — prebuilt images from the Gitea OCI registry (the fixed image set the broker validates against).
• #355 — generic SecretProvider.
• #478 — remote terminal design.