docs(orchestrator): tighten auth-provisioning wording to concrete services

Address review on #482: drop the generic "credential boundary" framing in the
PRD and docstrings and talk about the specific services — the orchestrator holds
the control-plane key; the host controller (#468) gets a separate key the
orchestrator never holds, so the orchestrator can't mint the credentials it uses
to talk to the host controller that owns its lifecycle. Lead with that concrete
win. No code behaviour change.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

bot_bottle/orchestrator_auth.py

@@ -62,15 +62,13 @@ _HEADER_SEGMENT = _b64url_encode(
 def mint(role: str, secret: str, *, roles: frozenset[str] = ROLES) -> str:
     """A compact HS256 token asserting `role`, signed with `secret`.
 
-    `roles` is the role set the caller's *trust domain* recognises (default: the
+    `roles` is the set the signing key is allowed to sign (default: the
-    orchestrator control plane's `{gateway, cli}`). A domain names its own set so
+    orchestrator's `{gateway, cli}`). A separate service (e.g. the host
-    each credential boundary mints only its own roles — a separate boundary
+    controller) passes its own key + role set so its tokens can't be forged with
-    (e.g. a host controller) instantiates a distinct domain with a distinct key
+    the orchestrator's key — see `trust_domain.py`, issues #476/#468.
-    and role set rather than adding a role here, so its key cannot forge the
-    other domain's tokens (see `trust_domain.py`, issues #476/#468).
 
-    Raises ValueError for a role outside `roles` (mint only what that domain will
+    Raises ValueError for a role outside `roles`, or an empty signing key (an
-    accept) or an empty signing key (an unsigned credential is never valid)."""
+    unsigned credential is never valid)."""
     if role not in roles:
         raise ValueError(f"unknown control-plane role {role!r}")
     if not secret:

bot_bottle/paths.py

@@ -101,14 +101,12 @@ def host_signing_key(filename: str) -> str:
     """A per-host signing key at `<root>/<filename>`, minted (256-bit, url-safe)
     and persisted 0600 on first use, then reused.
 
-    The generic form of `host_orchestrator_token()`: a *trust domain*
+    The generic form of `host_orchestrator_token()`: each service names its own
-    (`trust_domain.py`) names its own key file so each credential boundary gets a
+    key file (`trust_domain.py`), so the orchestrator and a separate service like
-    distinct host-canonical key — the orchestrator control plane names one file,
+    the host controller (#468) get distinct keys neither can read. It is a *host*
-    a separate boundary (e.g. a host controller) names another, and neither can
+    artifact — the file lives under the root the agent never mounts, and its value
-    read the other's key (issues #476/#468). It is a *host* artifact: the file
+    is injected only into the trusted control-plane process — so reading it here
-    lives under the root the agent never mounts, and its value is injected only
+    is safe on the launch path but the value never reaches a bottle."""
-    into the trusted control-plane process, so reading it here is safe on the
-    host launch path but the value never reaches a bottle."""
     path = bot_bottle_root() / filename
     try:
         existing = path.read_text().strip()

bot_bottle/trust_domain.py

@@ -1,29 +1,25 @@
-"""Trust domains: the unit of control-plane auth provisioning (issue #476).
+"""Per-service control-plane signing keys (issue #476).
 
-A *trust domain* is one **credential boundary** — a single host-canonical
+A `TrustDomain` is one service's signing material: its host-canonical key file,
-signing key plus the role set that key is allowed to sign, plus the env vars the
+the roles that key may sign, and the env vars its key and a pre-minted token ride
-key and a pre-minted token are carried in. Provisioning is parameterized *per
+in. Scoping `mint`/`verify` to a domain's roles keeps one service's key from
-domain*, not per key: the orchestrator control plane is one domain
+signing (or accepting) another service's tokens.
-(`CONTROL_PLANE` — key `orchestrator-token`, roles `{gateway, cli}`); a future
-boundary (e.g. the host controller of #468) instantiates its **own** domain with
-its **own** key, verifier, and role set rather than adding a role to this one.
 
-That distinction is the security invariant behind the split: adding a `host`
+Today there is one domain, `CONTROL_PLANE` — the orchestrator's key (roles
-role to the control plane's `ROLES` frozenset would let anything holding the
+`{gateway, cli}`): the orchestrator holds it and mints the gateway's and CLI's
-control-plane signing key (the orchestrator itself) mint host-controller tokens,
+tokens. The host controller (#468) will add a **second** domain with its own key
-collapsing the boundary #468 needs — the host controller owns the orchestrator's
+the orchestrator never holds. That is the point: the host controller starts and
-lifecycle, so it must not be forgeable *by* the orchestrator. Two keys, two
+stops the orchestrator, so the orchestrator must not be able to mint the
-verifiers, two role sets.
+credentials it uses to talk to it. Adding a `host` role to `CONTROL_PLANE`
+instead would defeat that — the orchestrator holds that key, so it could forge
+`host` tokens.
 
-`ControlPlaneProvisioning` is the single shared contract every backend launcher
+`ControlPlaneProvisioning` is the one seam every backend launcher uses to get the
-satisfies instead of re-deriving, by hand, how to generate the signing key,
+orchestrator its key and the gateway its token, instead of re-deriving that
-scope it to the orchestrator process, mint the gateway JWT, and keep the host
+wiring per backend (the bug class behind PR #471 — see
-key canonical (the bug class that took PR #471 three review rounds — see
 `docs/prds/prd-new-control-plane-auth-provisioning.md`).
 
-Stdlib-only; the crypto lives in `orchestrator_auth` (untouched HMAC), the key
+Stdlib-only: the HMAC lives in `orchestrator_auth`, the key file in `paths`.
-file lives in `paths` (no bot-bottle imports, safe to copy flat), and this module
-composes the two into the provisioning seam.
 """
 
 from __future__ import annotations
@@ -49,15 +45,14 @@ class ProvisioningError(RuntimeError):
 
 @dataclass(frozen=True)
 class TrustDomain:
-    """One credential boundary: a host-canonical signing key + the role set it
+    """One service's signing material: a host-canonical key file, the roles that
-    signs + the env vars its key and a pre-minted token ride in.
+    key may sign, and the env vars its key and a minted token ride in.
 
-    `signing_key()` reads-or-mints the host-canonical key (never a guest's); the
+    The service that *owns* the domain (e.g. the orchestrator) receives the raw
-    orchestrator process that *owns* the domain receives that raw key (via
+    key via `key_env`; a delegate (e.g. the gateway) receives only a pre-minted,
-    `key_env`), while a delegate (the data plane) receives only a pre-minted,
+    role-scoped token via `token_env` it cannot rewrite. `mint`/`verify` are
-    role-scoped token (via `token_env`) it cannot rewrite. `mint`/`verify` are
+    scoped to `roles`, so this service's key can neither sign nor accept another
-    scoped to this domain's `roles`, so a token minted here neither carries nor
+    service's role."""
-    verifies a role from another domain."""
 
     name: str
     key_filename: str
@@ -66,37 +61,35 @@ class TrustDomain:
     token_env: str
 
     def signing_key(self) -> str:
-        """The host-canonical signing key for this domain (minted 0600 on first
+        """This service's host-canonical signing key (minted 0600 on first use).
-        use). Host-side only — the value is injected into the owning process, not
+        Host-side only — the value is injected into the owning process."""
-        read there."""
         return host_signing_key(self.key_filename)
 
     def key_from_env(self, environ: Mapping[str, str] | None = None) -> str:
-        """The signing key as seen by the *owning process* — read from `key_env`
+        """The signing key as the owning process sees it — read from `key_env`
-        in the environment (default `os.environ`). "" when unset: the caller
+        (default `os.environ`). "" when unset; the caller decides whether that is
-        decides whether that is fatal (see `OrchestratorServer`'s open-mode
+        fatal (`ControlPlaneProvisioning`) or the open-mode fallback
-        fallback) or fail-closed (see `ControlPlaneProvisioning`)."""
+        (`OrchestratorServer`)."""
         env = os.environ if environ is None else environ
         return env.get(self.key_env, "").strip()
 
     def mint(self, role: str) -> str:
-        """A role-scoped token for a delegate, signed with this domain's key.
+        """A role-scoped token for a delegate, signed with this service's key.
-        Raises ValueError for a role outside this domain (mint only what this
+        Raises ValueError for a role this service doesn't sign."""
-        boundary accepts)."""
         if role not in self.roles:
             raise ValueError(f"role {role!r} is not in trust domain {self.name!r}")
         return orchestrator_auth.mint(role, self.signing_key(), roles=self.roles)
 
     def verify(self, token: str, key: str) -> str | None:
-        """The role `token` carries under `key`, or None. `key` is passed
+        """The role `token` carries under `key`, or None. `key` is passed in
-        explicitly (not read from the host file) because the verifier — the
+        (not read from disk) because the verifier — the control-plane process —
-        control-plane process — holds it in `key_env`, not on disk in its guest."""
+        holds it in `key_env`, not on disk in its guest."""
         return orchestrator_auth.verify(token, key, roles=self.roles)
 
 
-# The orchestrator control-plane domain: the signing key held by the
+# The orchestrator's domain: the key the orchestrator (and host CLI) holds, the
-# orchestrator + host CLI, the `gateway` token handed to the data plane, and the
+# `gateway` token it mints for the data plane, and the `cli` token the CLI mints
-# `cli` token the CLI mints for itself.
+# for itself. #468's host controller will add a second, separate domain.
 CONTROL_PLANE = TrustDomain(
     name="control-plane",
     key_filename=ORCHESTRATOR_TOKEN_FILENAME,
@@ -108,21 +101,19 @@ CONTROL_PLANE = TrustDomain(
 
 @dataclass(frozen=True)
 class Topology:
-    """What a backend *is*, for control-plane auth provisioning (#476) — the
+    """Where a backend runs the two planes, so the provisioning seam can decide
-    backend declares this instead of encoding the provisioning decision by hand
+    whether an open control plane is dangerous — the backend declares this
-    in its launcher.
+    instead of hardcoding the decision in its launcher.
 
-    `data_plane_shares_control_host` — the data plane runs on the same host/VM as
+    `data_plane_shares_control_host` — the gateway runs on the same host/VM as
-    the control plane, so a reachable-but-OPEN control plane would hand that
+    the orchestrator, so an open orchestrator would hand the co-located gateway
-    co-located data plane full `cli`. This is the default and the case that makes
+    full `cli`. The default, and what makes the signing key mandatory. Every
-    the signing key **mandatory** (open mode unreachable). Every current backend
+    current backend is co-located (docker/macOS: two containers on one host;
-    is co-located (docker/macOS: two containers on one host; firecracker: two VMs
+    firecracker: two VMs on one host, agents L3-isolated); an isolated control
-    on one host, agents L3-isolated). A backend with a genuinely isolated control
     plane on a separate trusted host may declare False.
 
-    `combined_guest` — control plane and data plane share a single guest (the
+    `combined_guest` — both planes in one guest (the retired combined infra VM).
-    retired combined infra VM). Informational today; kept so a future combined
+    Informational; kept so a future combined backend declares it."""
-    backend *declares* it rather than rediscovering the provisioning."""
 
     data_plane_shares_control_host: bool = True
     combined_guest: bool = False
@@ -135,47 +126,34 @@ COLOCATED = Topology(data_plane_shares_control_host=True)
 
 @dataclass(frozen=True)
 class ControlPlaneProvisioning:
-    """The single shared control-plane auth provisioning contract (#476).
+    """The one seam every backend launcher uses to provision control-plane auth,
-
+    instead of re-deriving the four invariants that each cost a PR #471 review
-    A backend launcher satisfies THIS instead of re-deriving the four invariants
+    round: the orchestrator gets the raw key (`orchestrator_key`), the gateway
-    that each took a PR #471 review round to get right:
+    gets a minted `gateway` token (`gateway_token`), the host CLI mints its own
-
+    `cli` token from the same host-canonical key, and the orchestrator never
-      1. **Host-canonical key.** The signing key is `domain.signing_key()` — a
+    starts open where its gateway is co-located."""
-         guest is *handed* it, never generates or overwrites it (round 3's bug:
-         the Firecracker guest clobbered the host key).
-      2. **Split credential.** Only the orchestrator process gets the raw key
-         (`orchestrator_key`); the data plane gets a pre-minted, role-scoped
-         token (`gateway_token`) it cannot rewrite into `cli` (round 1's bug).
-      3. **CLI validity across backends.** The host CLI mints its `cli` token
-         from this same canonical key, so it stays valid no matter which backend
-         (or how many) are co-running.
-      4. **No open mode.** `orchestrator_key` fail-closes for any topology whose
-         data plane shares the control plane's host/VM, so a launcher cannot
-         start the control plane OPEN (round 2's bug)."""
 
     domain: TrustDomain = CONTROL_PLANE
     topology: Topology = field(default=COLOCATED)
 
     def orchestrator_key(self) -> str:
-        """The raw signing key the control-plane *process* must receive (carry it
+        """The raw signing key the orchestrator process must receive (carry it in
-        in `domain.key_env`). Fail-closed: raises `ProvisioningError` rather than
+        `domain.key_env`). Fail-closed: raises rather than return "" for a
-        returning "" for a co-located topology, because an empty key makes the
+        co-located topology, since an empty key runs the server open and hands
-        server run OPEN and hand the co-located data plane full `cli` (#476
+        the co-located gateway full `cli`."""
-        invariant 4)."""
         key = self.domain.signing_key()
         if not key and self.topology.data_plane_shares_control_host:
             raise ProvisioningError(
-                f"refusing to provision the {self.domain.name} control plane "
+                f"refusing to start the {self.domain.name} orchestrator without "
-                "without a signing key: its data plane shares this host/VM, so "
+                "a signing key: its gateway shares this host/VM, so an open "
-                "an OPEN control plane would grant that data plane full `cli` "
+                "orchestrator would grant that gateway full `cli` (#476)"
-                "(#476)"
             )
         return key
 
     def gateway_token(self) -> str:
-        """The pre-minted `gateway`-role token the data plane receives (carry it
+        """The `gateway`-role token the gateway receives (carry it in
-        in `domain.token_env`) — minted from the canonical key, never the key
+        `domain.token_env`) — minted from the key, never the key itself, so a
-        itself, so a compromised data plane cannot forge a `cli` token."""
+        compromised gateway cannot forge a `cli` token."""
         return self.domain.mint(ROLE_GATEWAY)
 
 

docs/prds/prd-new-control-plane-auth-provisioning.md

@@ -1,4 +1,4 @@
-# PRD prd-new: Uniform control-plane auth provisioning
+# PRD prd-new: Per-service signing keys for control-plane auth
 
 - **Status:** Draft
 - **Author:** claude
@@ -7,135 +7,82 @@
 
 ## Summary
 
-Hoist control-plane auth provisioning out of the per-backend launchers into a
+Provision control-plane signing keys **per service**, through one shared seam, so
-single shared contract, parameterized per **trust domain**. A backend obtains
+no service can mint another's credentials. Concretely: the orchestrator holds the
-its signing key and the data plane's token through one seam
+control-plane key and mints the gateway's and CLI's tokens; the host controller
-(`trust_domain.ControlPlaneProvisioning`) instead of re-deriving, by hand, how
+(#468, next) gets a **separate** key the orchestrator never holds — so the
-to generate the signing key, scope it to the orchestrator, mint the `gateway`
+orchestrator cannot forge the credentials it uses to talk to the host controller
-JWT, and keep the host key canonical. This removes the integration-bug class
+that starts and stops it. Landing this seam also retires the per-backend
-that took PR #471 three review rounds to land, and gives issue #468's host
+provisioning duplication that made PR #471 take three review rounds.
-controller a clean seam to instantiate its **own** domain (own key, own roles)
-without weakening the control plane's.
 
 ## Problem
 
-Each launcher (`docker` gateway, `docker` infra, `macos` infra, `firecracker`
+**1. The orchestrator could forge host-controller credentials.** The
-infra) implemented the control-plane auth invariants independently. Every
+orchestrator's key signs roles `{gateway, cli}`. The tempting way to add the host
-blocking finding in PR #471 was the same class of *integration* bug — not a flaw
+controller (#468) is a third role, `host`, on that same key. But then the
-in the auth primitive (`orchestrator_auth.mint`/`verify`), but in how each
+orchestrator — which holds the key — can mint `host` tokens, and the host
-backend wired it:
+controller, which owns the orchestrator's lifecycle, must not trust anything the
+orchestrator can mint. The two services need separate keys.
 
-- **Round 1 (High):** the data plane got the full-power control-plane token, so
+**2. Every backend provisioned auth by hand.** Each launcher (docker
-  a compromised egress/git-gate could approve its own supervise proposals. Fixed
+gateway/infra, macOS infra, firecracker infra) re-derived how to generate the
-  with role-scoped JWTs (`gateway` vs `cli`).
+signing key, scope it to the orchestrator, mint the gateway JWT, and keep the
-- **Round 2 (High):** the Firecracker infra VM never provisioned the signing key
+host key file canonical. All three PR #471 High-severity findings were this one
-  or a `gateway` JWT, so the control plane fell into **open mode** and handed
+integration bug in different launchers: the data plane got the full `cli` token;
-  every unauthenticated caller the `cli` role.
+the firecracker control plane ran open; the firecracker guest clobbered the host
-- **Round 3 (High):** the Firecracker fix then clobbered the host-canonical
+key.
-  `orchestrator-token` with its guest key, 401'ing every already-running
-  Docker/macOS orchestrator.
-
-Miss one step per bespoke launcher and it's either a security hole or a
-cross-backend coexistence regression — and the tests didn't catch it because each
-backend's provisioning was hand-rolled.
 
 ## Goals / Success Criteria
 
-- One shared seam every backend satisfies for control-plane auth provisioning;
+- The orchestrator and the host controller sign with **different** keys; neither
-  no launcher re-derives the four invariants.
+  can mint the other's tokens. (This PR provisions the orchestrator's key and
-- The signing key is **host-canonical** — a guest is handed it, never generates
+  leaves a drop-in seam for the host controller's.)
-  or overwrites it.
+- One shared provisioning seam every backend uses — a new backend or daemon
-- Only the orchestrator process receives the raw key; the data plane receives a
+  implements it instead of rediscovering these four invariants:
-  pre-minted, role-scoped `gateway` token it cannot rewrite into `cli`.
+  1. the signing key is host-canonical: a guest is handed it, never generates or
-- The host CLI's `cli` token is minted from the same canonical key, so it stays
+     overwrites it;
-  valid across simultaneously-running backends.
+  2. only the orchestrator process gets the raw key; the gateway gets a
-- Open mode is unreachable for any backend whose data plane shares a host/VM
+     pre-minted `gateway` token it can't rewrite into `cli`;
-  with the control plane (fail-closed by default).
+  3. the host CLI's `cli` token is minted from the same key, so it stays valid
-- Provisioning is parameterized **per trust domain**, so #468 adds a separate
+     across co-running backends;
-  host-controller domain (own key, own verifier, own role set) rather than a
+  4. the control plane never runs open where its data plane shares the host/VM.
-  `host` role on the control plane's frozenset.
-- Adding a backend or a data-plane daemon means *implementing the contract*, not
-  rediscovering the invariants.
 
 ## Non-goals
 
-- Not a rewrite of the auth primitive. `orchestrator_auth`'s HMAC mint/verify is
+- The host controller itself (#468) — this only provisions the orchestrator's
-  unchanged except for an optional `roles=` argument (default preserved) so a
+  key and the seam #468 plugs into.
-  domain can scope its own role set.
+- Rewriting the HMAC primitive: `orchestrator_auth.mint/verify` gain an optional
-- Not the #468 host-controller domain itself — this only provides the seam it
+  `roles=` arg (default unchanged) so a key can carry a different role set;
-  will instantiate.
+  nothing else changes.
-- Not a change to network topology, the plane split (#469), or the server's
+- Network topology, the plane split (#469), or the server's open-mode fallback
-  documented open-mode fallback for tests/isolated control planes.
+  for tests.
-- The complementary #469 hardening (distinct non-root UIDs for co-located
-  daemons) stays separate.
 
 ## Design
 
-### Trust domain — the unit of provisioning
+A **`TrustDomain`** is one service's signing material: its host-canonical key
+file, the roles that key may sign, and the env vars its key and a pre-minted
+token ride in. `mint`/`verify` are scoped to that domain's roles, so a token
+signed by one service's key neither carries nor verifies another service's role.
 
-A **trust domain** (`trust_domain.TrustDomain`) is one credential boundary: a
+- `CONTROL_PLANE` — the orchestrator's domain: key `orchestrator-token`, roles
-host-canonical signing key file, the role set that key may sign, and the env
+  `{gateway, cli}`. The orchestrator process holds the key; the gateway holds
-vars the raw key and a pre-minted token ride in. `mint`/`verify` are scoped to
+  only a minted `gateway` token; the host CLI mints its own `cli` token.
-the domain's roles, so a token minted in one domain neither carries nor verifies
+- The host controller (#468) will add a second `TrustDomain` — its own key file
-a role from another.
+  and role(s) — that the orchestrator never holds.
 
-The orchestrator control plane is one domain, `CONTROL_PLANE` (key
+**`ControlPlaneProvisioning`** is the seam the backends call.
-`orchestrator-token`, roles `{gateway, cli}`). The security reason provisioning
+`orchestrator_key()` returns the raw key for the control-plane process
-is per-domain and not per-key: adding a `host` role to `CONTROL_PLANE.roles`
+(fail-closed: it raises rather than hand back an empty key that would run the
-would let anything holding the control-plane key (the orchestrator itself) mint
+server open where the data plane is co-located). `gateway_token()` mints the
-host-controller tokens, collapsing the boundary #468 needs — the host controller
+gateway's token. Each backend applies these through its own transport —
-owns the orchestrator's lifecycle, so it must not be forgeable *by* the
+docker/macOS inject env vars, firecracker pushes over SSH — but none re-derives
-orchestrator. Two keys, two verifiers, two role sets.
+*which* key or role.
 
-`paths.host_signing_key(filename)` generalizes the old
+`paths.host_signing_key(filename)` generalizes `host_orchestrator_token()` so each
-`host_orchestrator_token()` (now a thin specialization) so each domain names its
+domain names its own key file.
-own host-canonical key file.
-
-### The provisioning contract
-
-`ControlPlaneProvisioning` composes a domain with a declared `Topology` and
-answers the four invariants once:
-
-- `orchestrator_key()` → the raw key the control-plane **process** receives.
-  Fail-closed: raises `ProvisioningError` for a co-located topology when the key
-  is empty (which would run the server OPEN).
-- `gateway_token()` → the pre-minted `gateway` token the data plane receives,
-  minted from the canonical key, never the key itself.
-
-The `Orchestrator` ABC (`orchestrator/lifecycle.py`) holds one
-`ControlPlaneProvisioning` and exposes `control_plane_key()` and
-`mint_gateway_token()` over it. Each backend's orchestrator obtains its key
-through `control_plane_key()` and applies it via its own transport (docker/macOS:
-env var `key_env`; firecracker: SSH push to the guest) — the transport differs,
-the derivation no longer does.
-
-### Topology — the backend declares what it is
-
-`Topology` captures the provisioning-relevant dimensions the issue names
-(combined-guest vs standalone, data plane co-located vs isolated). The default,
-`COLOCATED`, makes the signing key mandatory (fail-closed). Every current backend
-is co-located (docker/macOS: two containers on one host; firecracker: two VMs on
-one host, agents L3-isolated), so none needs to redeclare it — the safe posture
-is the default, and a genuinely isolated control plane opts out explicitly.
-
-### Data flow
-
-```
-host key file (per-domain, 0600, host-canonical)
-        │  paths.host_signing_key(domain.key_filename)
-        ▼
-TrustDomain ── mint(role) ─────────────► gateway token ─► data-plane process (token_env / SSH)
-        │  signing_key()                 (gateway role, unrewritable)
-        ▼
-ControlPlaneProvisioning.orchestrator_key()  ─► control-plane process (key_env / SSH)
-        │  (fail-closed for co-located topology)
-        ▼
-OrchestratorClient ── CONTROL_PLANE.mint(cli) ─► host CLI's own operator token
-```
 
 ## Open questions
 
-None blocking. #468 will add its host-controller domain as a second
+None blocking. #468 adds its `TrustDomain` and a second
-`TrustDomain` + `ControlPlaneProvisioning`-shaped consumer; whether the
+`ControlPlaneProvisioning`-shaped consumer; renaming that class to something
-provisioning class is renamed to a domain-neutral `DomainProvisioning` at that
+service-neutral is a cosmetic call to make then.
-point is a cosmetic call to make when #468 lands.