|
|
|
@@ -0,0 +1,273 @@
|
|
|
|
|
# PRD prd-new: Host control server
|
|
|
|
|
|
|
|
|
|
- **Status:** Draft
|
|
|
|
|
- **Author:** Claude
|
|
|
|
|
- **Created:** 2026-07-26
|
|
|
|
|
- **Issue:** #468
|
|
|
|
|
|
|
|
|
|
## Summary
|
|
|
|
|
|
|
|
|
|
Promote the in-process launch broker into a standalone **host control
|
|
|
|
|
server**: the single privileged component on the host. Both the CLI and the
|
|
|
|
|
orchestrator drive it over HTTP; it brokers agent launches, owns the
|
|
|
|
|
orchestrator's own lifecycle, and is the sole writer of host-durable state (the
|
|
|
|
|
tamper-evident audit record). This closes the three gaps between today's
|
|
|
|
|
well-formed broker *contract* ([`orchestrator/broker.py`](../../bot_bottle/orchestrator/broker.py))
|
|
|
|
|
and a real out-of-process service — transport, durable provisioned secret,
|
|
|
|
|
and a disciplined op vocabulary — and splits host state by
|
|
|
|
|
owner and lifetime. The prize: **the CLI no longer needs the Docker socket**,
|
|
|
|
|
which is what finally lets a dedicated Gitea runner user drop the
|
|
|
|
|
root-equivalent `docker` group (PRD 0070, "Relationship to other work").
|
|
|
|
|
|
|
|
|
|
## Problem
|
|
|
|
|
|
|
|
|
|
Container launches run directly from a short-lived CLI process against the
|
|
|
|
|
Docker socket. That socket is root-equivalent, so every host that launches
|
|
|
|
|
bottles hands root to whoever invokes the CLI — including a CI runner user we
|
|
|
|
|
want to keep unprivileged. PRD 0070 already argues for replacing the fat socket
|
|
|
|
|
with a **thin, structured, auditable** launch broker, and the contract for that
|
|
|
|
|
broker exists and is tested in-process. But it is *only* in-process:
|
|
|
|
|
`LaunchBroker.submit(token)` is a method call from
|
|
|
|
|
`OrchestratorCore.launch_bottle` ([`service.py:116`](../../bot_bottle/orchestrator/service.py)),
|
|
|
|
|
and `DockerBroker` is on no production path — every backend starts the
|
|
|
|
|
orchestrator with `--broker stub` ([`__main__.py:54`](../../bot_bottle/orchestrator/__main__.py)).
|
|
|
|
|
|
|
|
|
|
Three gaps stand between that scaffold and a host service:
|
|
|
|
|
|
|
|
|
|
1. **No transport.** `submit` is an in-process call. A real service needs a
|
|
|
|
|
`BrokerClient` that POSTs the signed token and a host-side HTTP server that
|
|
|
|
|
verifies and acts.
|
|
|
|
|
2. **The signing secret is ephemeral and self-generated.**
|
|
|
|
|
[`__main__.py:53`](../../bot_bottle/orchestrator/__main__.py) does
|
|
|
|
|
`secrets.token_bytes(32)` and hands the *same value* to signer and verifier —
|
|
|
|
|
viable only because they share a process. A separate daemon needs the secret
|
|
|
|
|
provisioned out of band and durable across orchestrator restarts.
|
|
|
|
|
3. **The op vocabulary is `launch` / `teardown` only.** Everything else
|
|
|
|
|
host-privileged still lives in the CLI, so the schema has to grow — carefully,
|
|
|
|
|
since PRD 0070's security argument rests on "structured requests only, static
|
|
|
|
|
flags + ids."
|
|
|
|
|
|
|
|
|
|
Separately, host state has no clear owner. `OrchestratorCore.reconcile` takes
|
|
|
|
|
`live_source_ips` as a parameter *only because the orchestrator cannot see the
|
|
|
|
|
backend* ([`service.py:137`](../../bot_bottle/orchestrator/service.py)); the
|
|
|
|
|
egress traffic log is written to the container's stderr; and there is no durable,
|
|
|
|
|
tamper-evident home for the audit record that survives orchestrator destruction.
|
|
|
|
|
|
|
|
|
|
## Goals / Success Criteria
|
|
|
|
|
|
|
|
|
|
- A standalone host control server that the CLI and orchestrator reach over
|
|
|
|
|
**HTTP**, with three entry paths working end to end:
|
|
|
|
|
- `web console -(iroh)-> orchestrator -(http)-> host controller -> launch`
|
|
|
|
|
- `cli -(http)-> orchestrator -(http)-> host controller -> launch`
|
|
|
|
|
- `cli -(http)-> host controller` — start / restart / status of the
|
|
|
|
|
orchestrator **itself** (the bootstrap/recovery path #391 targets).
|
|
|
|
|
- The launch op is expressed as a **signed JWT of static flags + ids only**,
|
|
|
|
|
verified against a closed schema.
|
|
|
|
|
- The signing secret is **provisioned out of band and durable** across
|
|
|
|
|
orchestrator restarts (a `TrustDomain` per #476, with a key the orchestrator
|
|
|
|
|
never holds for the host controller's *own* endpoints).
|
|
|
|
|
- Host-privileged operations move off the CLI to the control server; **the CLI
|
|
|
|
|
no longer opens the Docker socket** for bottle operations.
|
|
|
|
|
- `Orchestrator.reconcile` no longer takes `live_source_ips` — live-bottle
|
|
|
|
|
enumeration becomes an internal control-server call.
|
|
|
|
|
- Host-durable state lands as an **append-only, hash-chained JSONL** audit log
|
|
|
|
|
owned solely by the host controller; operational state stays SQLite owned
|
|
|
|
|
solely by the orchestrator.
|
|
|
|
|
|
|
|
|
|
## Non-goals
|
|
|
|
|
|
|
|
|
|
- **Removing standing privilege.** This converts on-demand privilege (a CLI the
|
|
|
|
|
user invokes) into standing privilege (a daemon under launchd/systemd). The
|
|
|
|
|
win is that the privilege is *narrower* (structured requests vs. a raw socket),
|
|
|
|
|
not that it disappears. "Always running" is an accepted new property.
|
|
|
|
|
- **Asymmetric signing.** We stay HS256 — see Design / "Signing stays
|
|
|
|
|
symmetric."
|
|
|
|
|
- **Integrity against a live compromised orchestrator.** Host-location of the
|
|
|
|
|
audit log does not buy this: the orchestrator makes the decisions being audited
|
|
|
|
|
and can forge or omit entries wherever the file lives. An off-box copy is the
|
|
|
|
|
answer, tracked separately.
|
|
|
|
|
- **A single unified DB for all state.** Impossible over a guest-kernel share
|
|
|
|
|
(SQLite locking is not coherent); state is split by owner and lifetime instead.
|
|
|
|
|
- **The generic `SecretProvider` (#355)** and **remote terminal design (#478)** —
|
|
|
|
|
both ride the same door but are their own work.
|
|
|
|
|
|
|
|
|
|
## Design
|
|
|
|
|
|
|
|
|
|
### Topology
|
|
|
|
|
|
|
|
|
|
The host controller is the sole privileged component. The orchestrator becomes a
|
|
|
|
|
client of it for launches, and the CLI becomes a client of it for *both* bottle
|
|
|
|
|
operations (indirectly, through the orchestrator) and orchestrator lifecycle
|
|
|
|
|
(directly, for bootstrap/recovery — startup can't route through the thing being
|
|
|
|
|
started).
|
|
|
|
|
|
|
|
|
|
```
|
|
|
|
|
web console ─(iroh)─▶ orchestrator ─┐
|
|
|
|
|
├─(http, signed JWT)─▶ host controller ─▶ launch
|
|
|
|
|
cli ────────(http)──▶ orchestrator ─┘
|
|
|
|
|
cli ────────(http, bearer)──────────────────────────────▶ host controller (orchestrator lifecycle)
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
### Transport: `BrokerClient` + host server
|
|
|
|
|
|
|
|
|
|
`LaunchBroker.submit(token)` keeps its exact signature and semantics; only the
|
|
|
|
|
*wire* changes. A new `BrokerClient` implements the same submit contract by
|
|
|
|
|
POSTing the signed token to the host controller (stdlib `urllib`, like the
|
|
|
|
|
existing [`orchestrator/client.py`](../../bot_bottle/orchestrator/client.py)),
|
|
|
|
|
and the host controller's launch handler is the existing `verify_request` +
|
|
|
|
|
`_launch`/`_teardown` path, now reached over HTTP instead of a method call. The
|
|
|
|
|
in-process `StubBroker` stays for the dev-harness and tests; `DockerBroker`'s
|
|
|
|
|
`_launch`/`_teardown` bodies move behind the server unchanged. Because the client
|
|
|
|
|
satisfies the same interface `OrchestratorCore` already depends on, the core does
|
|
|
|
|
not change to gain a real backend.
|
|
|
|
|
|
|
|
|
|
### Signing stays symmetric (HS256)
|
|
|
|
|
|
|
|
|
|
PRD 0070 nominally specifies asymmetric; the code is HS256 and we keep it.
|
|
|
|
|
Asymmetric matters when the verifier is *less* privileged than the signer — here
|
|
|
|
|
it is the reverse: the host controller (verifier) is strictly more privileged
|
|
|
|
|
than the orchestrator (signer), and a controller that could forge orchestrator
|
|
|
|
|
requests gains nothing, since it is already the component that launches. Staying
|
|
|
|
|
symmetric also honors the no-runtime-deps policy (stdlib has no Ed25519). This
|
|
|
|
|
matches the reasoning already inlined in `broker.py`'s module docstring.
|
|
|
|
|
|
|
|
|
|
### Replay protection is out of scope (tracked in #494)
|
|
|
|
|
|
|
|
|
|
Once the launch token travels over a wire, a captured token could be replayed —
|
|
|
|
|
`sign_request` already emits `jti`/`iat` but `verify_request` reads neither, so
|
|
|
|
|
there is no expiry window or `jti` cache today. Enforcing that (an `iat` window +
|
|
|
|
|
a self-trimming `jti` cache) is a pure in-process change that lands independently
|
|
|
|
|
of this work, and it is deferred to **#494** rather than gating the MVP of the
|
|
|
|
|
host control server. Nothing here depends on it; it can merge before or after.
|
|
|
|
|
|
|
|
|
|
### Op vocabulary and the "ids + static flags" rule (gap 3)
|
|
|
|
|
|
|
|
|
|
Each op moved off the CLI widens the privileged surface, so growth is governed by
|
|
|
|
|
one explicit rule, enforced in `verify_request`'s schema check:
|
|
|
|
|
|
|
|
|
|
> A broker op carries **only ids and enumerated static flags** — a bottle id, a
|
|
|
|
|
> pool slot, a **content-addressed** image ref chosen from a fixed set, an op
|
|
|
|
|
> name from a closed vocabulary. Never a free-form path, argv, command, or
|
|
|
|
|
> caller-supplied filesystem location. If an operation cannot be expressed that
|
|
|
|
|
> way, it does not become a broker op.
|
|
|
|
|
|
|
|
|
|
Operations that fit and move off the CLI (all today in
|
|
|
|
|
`backend/*/consolidated_launch.py`, driven by a short-lived CLI process):
|
|
|
|
|
|
|
|
|
|
| Op | What it does | Fits the rule because |
|
|
|
|
|
|---|---|---|
|
|
|
|
|
| `launch` / `teardown` | existing | ids + slot + image ref |
|
|
|
|
|
| `orchestrator.ensure_running` | start the infra container | no arguments |
|
|
|
|
|
| `orchestrator.{start,restart,status}` | lifecycle (the #391 path) | no arguments |
|
|
|
|
|
| `list_live` | enumerate running bottles for reconcile | no arguments; returns ids/IPs |
|
|
|
|
|
| `allocate_ip` | `next_free_ip` over `_network_container_ips` | no arguments; returns an IP |
|
|
|
|
|
| `provision_git_gate` | `cp`/`exec` a per-bottle deploy key into the gateway | bottle id + key handle, no path |
|
|
|
|
|
| `reprovision` | `docker exec printenv <ENV_VAR_SECRET>` on a live agent | bottle id + secret *name* |
|
|
|
|
|
|
|
|
|
|
Image **builds** stay with the orchestrator for v1 (PRD 0070 §Memory: builds run
|
|
|
|
|
control-plane-side; a dedicated slim build unit is later, #468-adjacent), so no
|
|
|
|
|
`build` broker op is added here.
|
|
|
|
|
|
|
|
|
|
With `list_live` as an internal control-server call, `Orchestrator.reconcile`'s
|
|
|
|
|
`live_source_ips` parameter goes away — the tell PRD 0070 called out that the
|
|
|
|
|
orchestrator couldn't see the backend disappears with it.
|
|
|
|
|
|
|
|
|
|
### Secret provisioning (gap 2)
|
|
|
|
|
|
|
|
|
|
The shared HS256 secret becomes a durable, out-of-band artifact via the
|
|
|
|
|
**`TrustDomain`** seam (#476,
|
|
|
|
|
[`trust_domain.py`](../../bot_bottle/trust_domain.py)):
|
|
|
|
|
|
|
|
|
|
- The **launch-broker secret** is a `TrustDomain` whose key
|
|
|
|
|
(`host_signing_key(<file>)`, minted 0600 on first use, durable under
|
|
|
|
|
`bot_bottle_root()`) is provisioned to the orchestrator (signer) and the host
|
|
|
|
|
controller (verifier). Durability across orchestrator restarts is what makes
|
|
|
|
|
re-adoption work — a restart re-verifies against the same key.
|
|
|
|
|
- The **host controller's own lifecycle endpoints** (the direct `cli -> host
|
|
|
|
|
controller` path) get a **separate** `TrustDomain` key the orchestrator never
|
|
|
|
|
holds — exactly the second domain #476's PRD reserves. The orchestrator must
|
|
|
|
|
not be able to mint the credentials used to start and stop it.
|
|
|
|
|
|
|
|
|
|
This reuses the seam #476 landed rather than re-deriving provisioning per
|
|
|
|
|
backend (the PR #471 bug class).
|
|
|
|
|
|
|
|
|
|
### One daemon, structurally separate handlers (open decision 1)
|
|
|
|
|
|
|
|
|
|
The audit writer and the broker live in **one daemon** for install simplicity,
|
|
|
|
|
but with **no shared parsing** and **different credentials per handler**:
|
|
|
|
|
|
|
|
|
|
- the **launch** handler requires the signed launch **JWT** (provenance +
|
|
|
|
|
un-coercible schema);
|
|
|
|
|
- the **audit-append** handler takes a plain **bearer token** and writes to the
|
|
|
|
|
JSONL log.
|
|
|
|
|
|
|
|
|
|
This does not defend against orchestrator compromise (it holds both creds) — it
|
|
|
|
|
stops a bug in the boring audit path from reaching the privileged launch path.
|
|
|
|
|
The launcher stays small enough to audit line-by-line, per PRD 0070.
|
|
|
|
|
|
|
|
|
|
### State ownership: split by owner and lifetime
|
|
|
|
|
|
|
|
|
|
A single mounted DB is impossible — SQLite locking is not coherent across guest
|
|
|
|
|
kernels over a share, which is why the macOS backend already uses a container-only
|
|
|
|
|
volume (`INFRA_DB_VOLUME`). So state splits three ways (depends on #469, which
|
|
|
|
|
gets `bot-bottle.db` off the data plane first):
|
|
|
|
|
|
|
|
|
|
| Owner | State | Home | Shape |
|
|
|
|
|
|---|---|---|---|
|
|
|
|
|
| **Orchestrator** | `orchestrator_bottles` registry; `bottled_agent_secrets` (encrypted egress tokens); `supervise_proposals` / `supervise_responses` | volume nothing else mounts (generalizing the macOS design) | **SQLite** — mutable, transactional, queried |
|
|
|
|
|
| **Host controller** | supervise audit entries; egress traffic log (today → container stderr); host-side config | host filesystem, survives orchestrator/volume destruction | **JSONL** — append-only |
|
|
|
|
|
| **Gateway** | none | — | after #469 the data plane holds no DB state |
|
|
|
|
|
|
|
|
|
|
The historical record is **JSONL, not SQLite**, because it is append-only, never
|
|
|
|
|
updated, never transactionally queried: `O_APPEND` writes are atomic, there is no
|
|
|
|
|
locking protocol to get wrong, hash-chaining for tamper-evidence is cheap, and it
|
|
|
|
|
survives container-runtime volume pruning (the #450 lesson) and stays readable
|
|
|
|
|
without the orchestrator running. Both halves of "the audit record" — supervise
|
|
|
|
|
decisions and the egress traffic log — land in the one place.
|
|
|
|
|
|
|
|
|
|
The orchestrator is **sole mounter and sole writer** of its SQLite volume; the
|
|
|
|
|
host controller is **sole writer** of the JSONL log, over the authenticated
|
|
|
|
|
audit-append channel.
|
|
|
|
|
|
|
|
|
|
## Implementation chunks
|
|
|
|
|
|
|
|
|
|
Ordered, each independently mergeable:
|
|
|
|
|
|
|
|
|
|
1. **`BrokerClient` + host launch server** over HTTP, reusing `verify_request`
|
|
|
|
|
and the existing `DockerBroker` bodies. Wire `OrchestratorCore` to a
|
|
|
|
|
`BrokerClient` behind a flag; keep `StubBroker` for the dev-harness. Closes
|
|
|
|
|
gap 1.
|
|
|
|
|
2. **Durable secret via `TrustDomain`** — provision the launch-broker key to
|
|
|
|
|
signer + verifier; add the host controller's own lifecycle `TrustDomain`.
|
|
|
|
|
Closes gap 2.
|
|
|
|
|
3. **Grow the op vocabulary** one op at a time (`list_live` first — it also
|
|
|
|
|
removes `reconcile`'s `live_source_ips`), each behind the ids + static-flags
|
|
|
|
|
rule. Closes gap 3.
|
|
|
|
|
4. **JSONL audit log** — the host-controller-owned, hash-chained historical
|
|
|
|
|
record with the plain-bearer audit-append handler; redirect the egress traffic
|
|
|
|
|
log into it.
|
|
|
|
|
5. **Drop the Docker socket from the CLI** once every host-privileged op it used
|
|
|
|
|
is a broker op — the payoff that unblocks the unprivileged Gitea runner user.
|
|
|
|
|
|
|
|
|
|
## Open questions
|
|
|
|
|
|
|
|
|
|
1. **Schema-width rule enforcement.** The "ids + static flags" rule is stated;
|
|
|
|
|
should `verify_request` reject unknown claim keys outright (strict schema) to
|
|
|
|
|
keep the surface from drifting? Leaning yes.
|
|
|
|
|
2. **Audit-append back-pressure.** What the audit handler does if the JSONL sink
|
|
|
|
|
is unavailable (fail-closed vs. buffer) — resolve before shipping chunk 5.
|
|
|
|
|
|
|
|
|
|
## References
|
|
|
|
|
|
|
|
|
|
- **PRD 0070** — the contract, the launch broker, and the state tiers this
|
|
|
|
|
implements.
|
|
|
|
|
- **#469** — get `bot-bottle.db` off the data plane (lands underneath this).
|
|
|
|
|
- **#476** ([`prd-new-control-plane-auth-provisioning`](prd-new-control-plane-auth-provisioning.md))
|
|
|
|
|
— the `TrustDomain` seam this plugs the host controller's key into.
|
|
|
|
|
- **#391** — backend-agnostic orchestrator restart (the bootstrap path).
|
|
|
|
|
- **#494** — enforce broker replay protection (`iat` window + `jti` cache); split
|
|
|
|
|
out of this PRD as an independent in-process change.
|
|
|
|
|
- **#386** — prebuilt images from the Gitea OCI registry (the fixed image set the
|
|
|
|
|
broker validates against).
|
|
|
|
|
- **#355** — generic `SecretProvider`.
|
|
|
|
|
- **#478** — remote terminal design.
|
didericis-claude
Quality Badge Bot