From 7c57785f4812b53de5418ad7b0edc809a20bd3fc Mon Sep 17 00:00:00 2001 From: didericis Date: Mon, 13 Jul 2026 12:39:06 -0400 Subject: [PATCH] docs(prd): 0070 fold in review decisions (#352) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Resolve open questions from review: consolidate egress (hardened minimal surface + identity token + vault mitigations); HTTP control-plane transport; broker schema = signed-JWT JSON of static flags+ids (provenance + un-coercible); state re-adoption procedure (singleton orchestrator → wait-healthy → adopt via SQLite + process inspection before serving, with write-ahead intent to close the in-flight-launch race). Add a per-bottle identity-token defense-in-depth layer on the attribution invariant. Remaining open: VM-to-VM routing (per-backend wire(), pending spike link), live-reload protocol, identity-token delivery. Co-Authored-By: Claude Opus 4.8 Claude-Session: https://claude.ai/code/session_01WBMWTEtQdJ4W5UrWuLHCck --- docs/prds/0070-per-host-orchestrator.md | 80 +++++++++++++++++++++---- 1 file changed, 67 insertions(+), 13 deletions(-) diff --git a/docs/prds/0070-per-host-orchestrator.md b/docs/prds/0070-per-host-orchestrator.md index 701a33a..77e2af6 100644 --- a/docs/prds/0070-per-host-orchestrator.md +++ b/docs/prds/0070-per-host-orchestrator.md @@ -117,6 +117,27 @@ If a backend can't honor the invariant, source-IP consolidation is not safe there and that backend keeps per-bottle sidecars. The invariant is a hard precondition, not an aspiration. +**Defense-in-depth — a per-bottle identity token.** On top of the +network-layer invariant, inject a per-bottle secret token into every +request the agent makes to the orchestrator (the agent already egresses +through the sidecar proxy, so this is cheap to add). It gives an +**application-layer** proof of identity independent of the network layer: + +- On **Firecracker** the `/31` + nft already make source IP unspoofable + *by construction*, so the token is belt-and-suspenders there — but cheap + insurance against a misconfigured invariant. +- On **weaker backends** (Docker) it is load-bearing, providing attribution + that doesn't lean on network anti-spoof. + +Requirements: the token is **per-bottle, unguessable, and +non-cross-leakable** — a bottle can only ever prove it is *itself* (it +can't learn another bottle's token, given bottle isolation), so a hostile +agent gains nothing by presenting it. The orchestrator provisions the +token at launch and the sidecar requires it to attribute + authorize. +Note this hardens *attribution*, not *secret exposure*: it's app-layer, so +a compromised orchestrator still sees every token (that's the +concentration problem, addressed separately under Secret handling). + ### Secret handling — a FUTURE pattern (not v1) > **Status: future / not required for the initial orchestrator.** The @@ -170,7 +191,9 @@ Three surfaces; only one is per-backend. `deregister_bottle`, `supervise_queue`. Fully backend-agnostic. Both the local `cli.py` and the remote console funnel through it, so policy is uniform and `cli.py` becomes a thin client rather than a parallel - launcher. + launcher. **Transport: HTTP** — the most universal/reliable choice on + every host (no vsock/unix-socket portability caveats); a local unix + socket is a fine optimization, but HTTP is the wire contract. 2. **Data plane (agent → orchestrator)** — the egress / git / supervise endpoints. Already agnostic today (agents dial `http://sidecar:9099`); only the *address* and *how packets get there* are per-backend. @@ -218,6 +241,17 @@ exposes a broker the orchestrator calls to launch an agent: Stage 3 removes); a narrower broker later. - **Apple** — the `container` CLI/daemon. +**Broker request schema:** human-readable JSON of **static flags + ids +only** (never free-form paths/argv — that's what keeps it un-coercible +into arbitrary launches), wrapped as a **signed JWT** so the broker +verifies *provenance*: the request came from the real orchestrator, not a +forged one from a compromised co-located component. The orchestrator signs; +the broker verifies with the orchestrator's public key (provisioned at +broker install). This is the concrete form of security #3's "structured +requests only." Same JSON-with-ids + JWT shape for all +orchestrator↔broker/sidecar communication; cases that don't fit get +handled as they arise, with this as the default. + If `BottleBackend` bloats, the pressure valve is composition one level down: vend a `backend.network()` / `Wiring` collaborator rather than piling methods on — the same discipline, recursed. @@ -297,17 +331,37 @@ Keep the sidecar **service one shared thing** throughout. - **PR #350 (netpool single-source):** the same "one source per fact, composition over parallel hierarchies" discipline the contract follows. +## Decisions (review 2026-07-13) + +- **Egress sharing:** **consolidate egress** (worth it). Treat it as a + minimal, hardened attack surface for a malicious agent rather than + keeping it per-bottle; pair it with the identity token above and the + short-lived-vault-token mitigations (Secret handling / #355). +- **Control-plane transport:** **HTTP** — most universal/reliable on every + host; unix socket is an optional local optimization (see the contract). +- **Broker request schema:** **signed-JWT JSON, static flags + ids only** + (see the launch broker) — provenance + un-coercible by construction. +- **State re-adoption:** the restart procedure is: + 1. **Singleton:** a new orchestrator launch requires no pre-existing + orchestrator; any found (healthy or not) is fully shut down first. + 2. Re-adoption **waits for the new orchestrator to be healthy**. + 3. Once healthy, it discovers all agents needing adoption via **both the + SQLite registry and live process/VM inspection** *before serving any + other request*. The two-source sweep is what closes the *in-flight + launch* race — a launch that started (VM booting / slot claimed) but + hadn't committed to SQLite when the old orchestrator died would be + invisible to a SQLite-only sweep; process inspection catches it. + Launches should also write an **intent record ahead of committing + resources** so the sweep can reconcile intent vs. actual. + ## Open questions -- **Egress sharing tradeoff:** is the secret-concentration blast radius of - one shared mitmproxy worth the resource win, or share only supervise - (near-zero secrets) and keep egress + git-gate per-bottle initially? -- **Control-plane shape:** RPC transport (unix socket / vsock / HTTP over - the TAP) and the live-reload protocol for per-bottle policy. -- **State re-adoption:** exact scheme for an orchestrator restart to - re-adopt running agent VMs from the SQLite registry without racing - in-flight launches. -- **VM-to-VM routing:** the nft forward rules + addressing for a - per-host orchestrator VM on its own TAP. -- **Broker request schema:** the exact structured contract that stays - auditable and can't be coerced into launching arbitrary payloads. +- **VM-to-VM routing:** per-backend, and in the design it *is* + `BottleBackend.wire()` (DNAT+forward for fc, shared-net for + docker/apple), not orchestrator logic. Likely already prototyped in the + earlier "sidecars in VMs" spike — **link the spike** and this drops from + open question to "implement per the spike." +- **Live-reload protocol** for per-bottle policy over the HTTP control + plane (add/remove routes/keys/proposals without a restart). +- **Identity-token delivery:** exactly how the per-bottle token is placed + where the agent can present it but not swap in another bottle's.