diff --git a/bot_bottle/orchestrator_auth.py b/bot_bottle/orchestrator_auth.py index 5b07e40e..687dabff 100644 --- a/bot_bottle/orchestrator_auth.py +++ b/bot_bottle/orchestrator_auth.py @@ -62,15 +62,13 @@ _HEADER_SEGMENT = _b64url_encode( def mint(role: str, secret: str, *, roles: frozenset[str] = ROLES) -> str: """A compact HS256 token asserting `role`, signed with `secret`. - `roles` is the role set the caller's *trust domain* recognises (default: the - orchestrator control plane's `{gateway, cli}`). A domain names its own set so - each credential boundary mints only its own roles — a separate boundary - (e.g. a host controller) instantiates a distinct domain with a distinct key - and role set rather than adding a role here, so its key cannot forge the - other domain's tokens (see `trust_domain.py`, issues #476/#468). + `roles` is the set the signing key is allowed to sign (default: the + orchestrator's `{gateway, cli}`). A separate service (e.g. the host + controller) passes its own key + role set so its tokens can't be forged with + the orchestrator's key — see `trust_domain.py`, issues #476/#468. - Raises ValueError for a role outside `roles` (mint only what that domain will - accept) or an empty signing key (an unsigned credential is never valid).""" + Raises ValueError for a role outside `roles`, or an empty signing key (an + unsigned credential is never valid).""" if role not in roles: raise ValueError(f"unknown control-plane role {role!r}") if not secret: diff --git a/bot_bottle/paths.py b/bot_bottle/paths.py index 7f7841e4..8cd69e04 100644 --- a/bot_bottle/paths.py +++ b/bot_bottle/paths.py @@ -101,14 +101,12 @@ def host_signing_key(filename: str) -> str: """A per-host signing key at `/`, minted (256-bit, url-safe) and persisted 0600 on first use, then reused. - The generic form of `host_orchestrator_token()`: a *trust domain* - (`trust_domain.py`) names its own key file so each credential boundary gets a - distinct host-canonical key — the orchestrator control plane names one file, - a separate boundary (e.g. a host controller) names another, and neither can - read the other's key (issues #476/#468). It is a *host* artifact: the file - lives under the root the agent never mounts, and its value is injected only - into the trusted control-plane process, so reading it here is safe on the - host launch path but the value never reaches a bottle.""" + The generic form of `host_orchestrator_token()`: each service names its own + key file (`trust_domain.py`), so the orchestrator and a separate service like + the host controller (#468) get distinct keys neither can read. It is a *host* + artifact — the file lives under the root the agent never mounts, and its value + is injected only into the trusted control-plane process — so reading it here + is safe on the launch path but the value never reaches a bottle.""" path = bot_bottle_root() / filename try: existing = path.read_text().strip() diff --git a/bot_bottle/trust_domain.py b/bot_bottle/trust_domain.py index 66d64c2e..428df119 100644 --- a/bot_bottle/trust_domain.py +++ b/bot_bottle/trust_domain.py @@ -1,29 +1,25 @@ -"""Trust domains: the unit of control-plane auth provisioning (issue #476). +"""Per-service control-plane signing keys (issue #476). -A *trust domain* is one **credential boundary** — a single host-canonical -signing key plus the role set that key is allowed to sign, plus the env vars the -key and a pre-minted token are carried in. Provisioning is parameterized *per -domain*, not per key: the orchestrator control plane is one domain -(`CONTROL_PLANE` — key `orchestrator-token`, roles `{gateway, cli}`); a future -boundary (e.g. the host controller of #468) instantiates its **own** domain with -its **own** key, verifier, and role set rather than adding a role to this one. +A `TrustDomain` is one service's signing material: its host-canonical key file, +the roles that key may sign, and the env vars its key and a pre-minted token ride +in. Scoping `mint`/`verify` to a domain's roles keeps one service's key from +signing (or accepting) another service's tokens. -That distinction is the security invariant behind the split: adding a `host` -role to the control plane's `ROLES` frozenset would let anything holding the -control-plane signing key (the orchestrator itself) mint host-controller tokens, -collapsing the boundary #468 needs — the host controller owns the orchestrator's -lifecycle, so it must not be forgeable *by* the orchestrator. Two keys, two -verifiers, two role sets. +Today there is one domain, `CONTROL_PLANE` — the orchestrator's key (roles +`{gateway, cli}`): the orchestrator holds it and mints the gateway's and CLI's +tokens. The host controller (#468) will add a **second** domain with its own key +the orchestrator never holds. That is the point: the host controller starts and +stops the orchestrator, so the orchestrator must not be able to mint the +credentials it uses to talk to it. Adding a `host` role to `CONTROL_PLANE` +instead would defeat that — the orchestrator holds that key, so it could forge +`host` tokens. -`ControlPlaneProvisioning` is the single shared contract every backend launcher -satisfies instead of re-deriving, by hand, how to generate the signing key, -scope it to the orchestrator process, mint the gateway JWT, and keep the host -key canonical (the bug class that took PR #471 three review rounds — see +`ControlPlaneProvisioning` is the one seam every backend launcher uses to get the +orchestrator its key and the gateway its token, instead of re-deriving that +wiring per backend (the bug class behind PR #471 — see `docs/prds/prd-new-control-plane-auth-provisioning.md`). -Stdlib-only; the crypto lives in `orchestrator_auth` (untouched HMAC), the key -file lives in `paths` (no bot-bottle imports, safe to copy flat), and this module -composes the two into the provisioning seam. +Stdlib-only: the HMAC lives in `orchestrator_auth`, the key file in `paths`. """ from __future__ import annotations @@ -49,15 +45,14 @@ class ProvisioningError(RuntimeError): @dataclass(frozen=True) class TrustDomain: - """One credential boundary: a host-canonical signing key + the role set it - signs + the env vars its key and a pre-minted token ride in. + """One service's signing material: a host-canonical key file, the roles that + key may sign, and the env vars its key and a minted token ride in. - `signing_key()` reads-or-mints the host-canonical key (never a guest's); the - orchestrator process that *owns* the domain receives that raw key (via - `key_env`), while a delegate (the data plane) receives only a pre-minted, - role-scoped token (via `token_env`) it cannot rewrite. `mint`/`verify` are - scoped to this domain's `roles`, so a token minted here neither carries nor - verifies a role from another domain.""" + The service that *owns* the domain (e.g. the orchestrator) receives the raw + key via `key_env`; a delegate (e.g. the gateway) receives only a pre-minted, + role-scoped token via `token_env` it cannot rewrite. `mint`/`verify` are + scoped to `roles`, so this service's key can neither sign nor accept another + service's role.""" name: str key_filename: str @@ -66,37 +61,35 @@ class TrustDomain: token_env: str def signing_key(self) -> str: - """The host-canonical signing key for this domain (minted 0600 on first - use). Host-side only — the value is injected into the owning process, not - read there.""" + """This service's host-canonical signing key (minted 0600 on first use). + Host-side only — the value is injected into the owning process.""" return host_signing_key(self.key_filename) def key_from_env(self, environ: Mapping[str, str] | None = None) -> str: - """The signing key as seen by the *owning process* — read from `key_env` - in the environment (default `os.environ`). "" when unset: the caller - decides whether that is fatal (see `OrchestratorServer`'s open-mode - fallback) or fail-closed (see `ControlPlaneProvisioning`).""" + """The signing key as the owning process sees it — read from `key_env` + (default `os.environ`). "" when unset; the caller decides whether that is + fatal (`ControlPlaneProvisioning`) or the open-mode fallback + (`OrchestratorServer`).""" env = os.environ if environ is None else environ return env.get(self.key_env, "").strip() def mint(self, role: str) -> str: - """A role-scoped token for a delegate, signed with this domain's key. - Raises ValueError for a role outside this domain (mint only what this - boundary accepts).""" + """A role-scoped token for a delegate, signed with this service's key. + Raises ValueError for a role this service doesn't sign.""" if role not in self.roles: raise ValueError(f"role {role!r} is not in trust domain {self.name!r}") return orchestrator_auth.mint(role, self.signing_key(), roles=self.roles) def verify(self, token: str, key: str) -> str | None: - """The role `token` carries under `key`, or None. `key` is passed - explicitly (not read from the host file) because the verifier — the - control-plane process — holds it in `key_env`, not on disk in its guest.""" + """The role `token` carries under `key`, or None. `key` is passed in + (not read from disk) because the verifier — the control-plane process — + holds it in `key_env`, not on disk in its guest.""" return orchestrator_auth.verify(token, key, roles=self.roles) -# The orchestrator control-plane domain: the signing key held by the -# orchestrator + host CLI, the `gateway` token handed to the data plane, and the -# `cli` token the CLI mints for itself. +# The orchestrator's domain: the key the orchestrator (and host CLI) holds, the +# `gateway` token it mints for the data plane, and the `cli` token the CLI mints +# for itself. #468's host controller will add a second, separate domain. CONTROL_PLANE = TrustDomain( name="control-plane", key_filename=ORCHESTRATOR_TOKEN_FILENAME, @@ -108,21 +101,19 @@ CONTROL_PLANE = TrustDomain( @dataclass(frozen=True) class Topology: - """What a backend *is*, for control-plane auth provisioning (#476) — the - backend declares this instead of encoding the provisioning decision by hand - in its launcher. + """Where a backend runs the two planes, so the provisioning seam can decide + whether an open control plane is dangerous — the backend declares this + instead of hardcoding the decision in its launcher. - `data_plane_shares_control_host` — the data plane runs on the same host/VM as - the control plane, so a reachable-but-OPEN control plane would hand that - co-located data plane full `cli`. This is the default and the case that makes - the signing key **mandatory** (open mode unreachable). Every current backend - is co-located (docker/macOS: two containers on one host; firecracker: two VMs - on one host, agents L3-isolated). A backend with a genuinely isolated control + `data_plane_shares_control_host` — the gateway runs on the same host/VM as + the orchestrator, so an open orchestrator would hand the co-located gateway + full `cli`. The default, and what makes the signing key mandatory. Every + current backend is co-located (docker/macOS: two containers on one host; + firecracker: two VMs on one host, agents L3-isolated); an isolated control plane on a separate trusted host may declare False. - `combined_guest` — control plane and data plane share a single guest (the - retired combined infra VM). Informational today; kept so a future combined - backend *declares* it rather than rediscovering the provisioning.""" + `combined_guest` — both planes in one guest (the retired combined infra VM). + Informational; kept so a future combined backend declares it.""" data_plane_shares_control_host: bool = True combined_guest: bool = False @@ -135,47 +126,34 @@ COLOCATED = Topology(data_plane_shares_control_host=True) @dataclass(frozen=True) class ControlPlaneProvisioning: - """The single shared control-plane auth provisioning contract (#476). - - A backend launcher satisfies THIS instead of re-deriving the four invariants - that each took a PR #471 review round to get right: - - 1. **Host-canonical key.** The signing key is `domain.signing_key()` — a - guest is *handed* it, never generates or overwrites it (round 3's bug: - the Firecracker guest clobbered the host key). - 2. **Split credential.** Only the orchestrator process gets the raw key - (`orchestrator_key`); the data plane gets a pre-minted, role-scoped - token (`gateway_token`) it cannot rewrite into `cli` (round 1's bug). - 3. **CLI validity across backends.** The host CLI mints its `cli` token - from this same canonical key, so it stays valid no matter which backend - (or how many) are co-running. - 4. **No open mode.** `orchestrator_key` fail-closes for any topology whose - data plane shares the control plane's host/VM, so a launcher cannot - start the control plane OPEN (round 2's bug).""" + """The one seam every backend launcher uses to provision control-plane auth, + instead of re-deriving the four invariants that each cost a PR #471 review + round: the orchestrator gets the raw key (`orchestrator_key`), the gateway + gets a minted `gateway` token (`gateway_token`), the host CLI mints its own + `cli` token from the same host-canonical key, and the orchestrator never + starts open where its gateway is co-located.""" domain: TrustDomain = CONTROL_PLANE topology: Topology = field(default=COLOCATED) def orchestrator_key(self) -> str: - """The raw signing key the control-plane *process* must receive (carry it - in `domain.key_env`). Fail-closed: raises `ProvisioningError` rather than - returning "" for a co-located topology, because an empty key makes the - server run OPEN and hand the co-located data plane full `cli` (#476 - invariant 4).""" + """The raw signing key the orchestrator process must receive (carry it in + `domain.key_env`). Fail-closed: raises rather than return "" for a + co-located topology, since an empty key runs the server open and hands + the co-located gateway full `cli`.""" key = self.domain.signing_key() if not key and self.topology.data_plane_shares_control_host: raise ProvisioningError( - f"refusing to provision the {self.domain.name} control plane " - "without a signing key: its data plane shares this host/VM, so " - "an OPEN control plane would grant that data plane full `cli` " - "(#476)" + f"refusing to start the {self.domain.name} orchestrator without " + "a signing key: its gateway shares this host/VM, so an open " + "orchestrator would grant that gateway full `cli` (#476)" ) return key def gateway_token(self) -> str: - """The pre-minted `gateway`-role token the data plane receives (carry it - in `domain.token_env`) — minted from the canonical key, never the key - itself, so a compromised data plane cannot forge a `cli` token.""" + """The `gateway`-role token the gateway receives (carry it in + `domain.token_env`) — minted from the key, never the key itself, so a + compromised gateway cannot forge a `cli` token.""" return self.domain.mint(ROLE_GATEWAY) diff --git a/docs/prds/prd-new-control-plane-auth-provisioning.md b/docs/prds/prd-new-control-plane-auth-provisioning.md index da80ef9c..865cd396 100644 --- a/docs/prds/prd-new-control-plane-auth-provisioning.md +++ b/docs/prds/prd-new-control-plane-auth-provisioning.md @@ -1,4 +1,4 @@ -# PRD prd-new: Uniform control-plane auth provisioning +# PRD prd-new: Per-service signing keys for control-plane auth - **Status:** Draft - **Author:** claude @@ -7,135 +7,82 @@ ## Summary -Hoist control-plane auth provisioning out of the per-backend launchers into a -single shared contract, parameterized per **trust domain**. A backend obtains -its signing key and the data plane's token through one seam -(`trust_domain.ControlPlaneProvisioning`) instead of re-deriving, by hand, how -to generate the signing key, scope it to the orchestrator, mint the `gateway` -JWT, and keep the host key canonical. This removes the integration-bug class -that took PR #471 three review rounds to land, and gives issue #468's host -controller a clean seam to instantiate its **own** domain (own key, own roles) -without weakening the control plane's. +Provision control-plane signing keys **per service**, through one shared seam, so +no service can mint another's credentials. Concretely: the orchestrator holds the +control-plane key and mints the gateway's and CLI's tokens; the host controller +(#468, next) gets a **separate** key the orchestrator never holds — so the +orchestrator cannot forge the credentials it uses to talk to the host controller +that starts and stops it. Landing this seam also retires the per-backend +provisioning duplication that made PR #471 take three review rounds. ## Problem -Each launcher (`docker` gateway, `docker` infra, `macos` infra, `firecracker` -infra) implemented the control-plane auth invariants independently. Every -blocking finding in PR #471 was the same class of *integration* bug — not a flaw -in the auth primitive (`orchestrator_auth.mint`/`verify`), but in how each -backend wired it: +**1. The orchestrator could forge host-controller credentials.** The +orchestrator's key signs roles `{gateway, cli}`. The tempting way to add the host +controller (#468) is a third role, `host`, on that same key. But then the +orchestrator — which holds the key — can mint `host` tokens, and the host +controller, which owns the orchestrator's lifecycle, must not trust anything the +orchestrator can mint. The two services need separate keys. -- **Round 1 (High):** the data plane got the full-power control-plane token, so - a compromised egress/git-gate could approve its own supervise proposals. Fixed - with role-scoped JWTs (`gateway` vs `cli`). -- **Round 2 (High):** the Firecracker infra VM never provisioned the signing key - or a `gateway` JWT, so the control plane fell into **open mode** and handed - every unauthenticated caller the `cli` role. -- **Round 3 (High):** the Firecracker fix then clobbered the host-canonical - `orchestrator-token` with its guest key, 401'ing every already-running - Docker/macOS orchestrator. - -Miss one step per bespoke launcher and it's either a security hole or a -cross-backend coexistence regression — and the tests didn't catch it because each -backend's provisioning was hand-rolled. +**2. Every backend provisioned auth by hand.** Each launcher (docker +gateway/infra, macOS infra, firecracker infra) re-derived how to generate the +signing key, scope it to the orchestrator, mint the gateway JWT, and keep the +host key file canonical. All three PR #471 High-severity findings were this one +integration bug in different launchers: the data plane got the full `cli` token; +the firecracker control plane ran open; the firecracker guest clobbered the host +key. ## Goals / Success Criteria -- One shared seam every backend satisfies for control-plane auth provisioning; - no launcher re-derives the four invariants. -- The signing key is **host-canonical** — a guest is handed it, never generates - or overwrites it. -- Only the orchestrator process receives the raw key; the data plane receives a - pre-minted, role-scoped `gateway` token it cannot rewrite into `cli`. -- The host CLI's `cli` token is minted from the same canonical key, so it stays - valid across simultaneously-running backends. -- Open mode is unreachable for any backend whose data plane shares a host/VM - with the control plane (fail-closed by default). -- Provisioning is parameterized **per trust domain**, so #468 adds a separate - host-controller domain (own key, own verifier, own role set) rather than a - `host` role on the control plane's frozenset. -- Adding a backend or a data-plane daemon means *implementing the contract*, not - rediscovering the invariants. +- The orchestrator and the host controller sign with **different** keys; neither + can mint the other's tokens. (This PR provisions the orchestrator's key and + leaves a drop-in seam for the host controller's.) +- One shared provisioning seam every backend uses — a new backend or daemon + implements it instead of rediscovering these four invariants: + 1. the signing key is host-canonical: a guest is handed it, never generates or + overwrites it; + 2. only the orchestrator process gets the raw key; the gateway gets a + pre-minted `gateway` token it can't rewrite into `cli`; + 3. the host CLI's `cli` token is minted from the same key, so it stays valid + across co-running backends; + 4. the control plane never runs open where its data plane shares the host/VM. ## Non-goals -- Not a rewrite of the auth primitive. `orchestrator_auth`'s HMAC mint/verify is - unchanged except for an optional `roles=` argument (default preserved) so a - domain can scope its own role set. -- Not the #468 host-controller domain itself — this only provides the seam it - will instantiate. -- Not a change to network topology, the plane split (#469), or the server's - documented open-mode fallback for tests/isolated control planes. -- The complementary #469 hardening (distinct non-root UIDs for co-located - daemons) stays separate. +- The host controller itself (#468) — this only provisions the orchestrator's + key and the seam #468 plugs into. +- Rewriting the HMAC primitive: `orchestrator_auth.mint/verify` gain an optional + `roles=` arg (default unchanged) so a key can carry a different role set; + nothing else changes. +- Network topology, the plane split (#469), or the server's open-mode fallback + for tests. ## Design -### Trust domain — the unit of provisioning +A **`TrustDomain`** is one service's signing material: its host-canonical key +file, the roles that key may sign, and the env vars its key and a pre-minted +token ride in. `mint`/`verify` are scoped to that domain's roles, so a token +signed by one service's key neither carries nor verifies another service's role. -A **trust domain** (`trust_domain.TrustDomain`) is one credential boundary: a -host-canonical signing key file, the role set that key may sign, and the env -vars the raw key and a pre-minted token ride in. `mint`/`verify` are scoped to -the domain's roles, so a token minted in one domain neither carries nor verifies -a role from another. +- `CONTROL_PLANE` — the orchestrator's domain: key `orchestrator-token`, roles + `{gateway, cli}`. The orchestrator process holds the key; the gateway holds + only a minted `gateway` token; the host CLI mints its own `cli` token. +- The host controller (#468) will add a second `TrustDomain` — its own key file + and role(s) — that the orchestrator never holds. -The orchestrator control plane is one domain, `CONTROL_PLANE` (key -`orchestrator-token`, roles `{gateway, cli}`). The security reason provisioning -is per-domain and not per-key: adding a `host` role to `CONTROL_PLANE.roles` -would let anything holding the control-plane key (the orchestrator itself) mint -host-controller tokens, collapsing the boundary #468 needs — the host controller -owns the orchestrator's lifecycle, so it must not be forgeable *by* the -orchestrator. Two keys, two verifiers, two role sets. +**`ControlPlaneProvisioning`** is the seam the backends call. +`orchestrator_key()` returns the raw key for the control-plane process +(fail-closed: it raises rather than hand back an empty key that would run the +server open where the data plane is co-located). `gateway_token()` mints the +gateway's token. Each backend applies these through its own transport — +docker/macOS inject env vars, firecracker pushes over SSH — but none re-derives +*which* key or role. -`paths.host_signing_key(filename)` generalizes the old -`host_orchestrator_token()` (now a thin specialization) so each domain names its -own host-canonical key file. - -### The provisioning contract - -`ControlPlaneProvisioning` composes a domain with a declared `Topology` and -answers the four invariants once: - -- `orchestrator_key()` → the raw key the control-plane **process** receives. - Fail-closed: raises `ProvisioningError` for a co-located topology when the key - is empty (which would run the server OPEN). -- `gateway_token()` → the pre-minted `gateway` token the data plane receives, - minted from the canonical key, never the key itself. - -The `Orchestrator` ABC (`orchestrator/lifecycle.py`) holds one -`ControlPlaneProvisioning` and exposes `control_plane_key()` and -`mint_gateway_token()` over it. Each backend's orchestrator obtains its key -through `control_plane_key()` and applies it via its own transport (docker/macOS: -env var `key_env`; firecracker: SSH push to the guest) — the transport differs, -the derivation no longer does. - -### Topology — the backend declares what it is - -`Topology` captures the provisioning-relevant dimensions the issue names -(combined-guest vs standalone, data plane co-located vs isolated). The default, -`COLOCATED`, makes the signing key mandatory (fail-closed). Every current backend -is co-located (docker/macOS: two containers on one host; firecracker: two VMs on -one host, agents L3-isolated), so none needs to redeclare it — the safe posture -is the default, and a genuinely isolated control plane opts out explicitly. - -### Data flow - -``` -host key file (per-domain, 0600, host-canonical) - │ paths.host_signing_key(domain.key_filename) - ▼ -TrustDomain ── mint(role) ─────────────► gateway token ─► data-plane process (token_env / SSH) - │ signing_key() (gateway role, unrewritable) - ▼ -ControlPlaneProvisioning.orchestrator_key() ─► control-plane process (key_env / SSH) - │ (fail-closed for co-located topology) - ▼ -OrchestratorClient ── CONTROL_PLANE.mint(cli) ─► host CLI's own operator token -``` +`paths.host_signing_key(filename)` generalizes `host_orchestrator_token()` so each +domain names its own key file. ## Open questions -None blocking. #468 will add its host-controller domain as a second -`TrustDomain` + `ControlPlaneProvisioning`-shaped consumer; whether the -provisioning class is renamed to a domain-neutral `DomainProvisioning` at that -point is a cosmetic call to make when #468 lands. +None blocking. #468 adds its `TrustDomain` and a second +`ControlPlaneProvisioning`-shaped consumer; renaming that class to something +service-neutral is a cosmetic call to make then.