docs(prd): 0070 fold in review decisions (#352)
Resolve open questions from review: consolidate egress (hardened minimal surface + identity token + vault mitigations); HTTP control-plane transport; broker schema = signed-JWT JSON of static flags+ids (provenance + un-coercible); state re-adoption procedure (singleton orchestrator → wait-healthy → adopt via SQLite + process inspection before serving, with write-ahead intent to close the in-flight-launch race). Add a per-bottle identity-token defense-in-depth layer on the attribution invariant. Remaining open: VM-to-VM routing (per-backend wire(), pending spike link), live-reload protocol, identity-token delivery. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WBMWTEtQdJ4W5UrWuLHCck
This commit is contained in:
@@ -117,6 +117,27 @@ If a backend can't honor the invariant, source-IP consolidation is not
|
||||
safe there and that backend keeps per-bottle sidecars. The invariant is a
|
||||
hard precondition, not an aspiration.
|
||||
|
||||
**Defense-in-depth — a per-bottle identity token.** On top of the
|
||||
network-layer invariant, inject a per-bottle secret token into every
|
||||
request the agent makes to the orchestrator (the agent already egresses
|
||||
through the sidecar proxy, so this is cheap to add). It gives an
|
||||
**application-layer** proof of identity independent of the network layer:
|
||||
|
||||
- On **Firecracker** the `/31` + nft already make source IP unspoofable
|
||||
*by construction*, so the token is belt-and-suspenders there — but cheap
|
||||
insurance against a misconfigured invariant.
|
||||
- On **weaker backends** (Docker) it is load-bearing, providing attribution
|
||||
that doesn't lean on network anti-spoof.
|
||||
|
||||
Requirements: the token is **per-bottle, unguessable, and
|
||||
non-cross-leakable** — a bottle can only ever prove it is *itself* (it
|
||||
can't learn another bottle's token, given bottle isolation), so a hostile
|
||||
agent gains nothing by presenting it. The orchestrator provisions the
|
||||
token at launch and the sidecar requires it to attribute + authorize.
|
||||
Note this hardens *attribution*, not *secret exposure*: it's app-layer, so
|
||||
a compromised orchestrator still sees every token (that's the
|
||||
concentration problem, addressed separately under Secret handling).
|
||||
|
||||
### Secret handling — a FUTURE pattern (not v1)
|
||||
|
||||
> **Status: future / not required for the initial orchestrator.** The
|
||||
@@ -170,7 +191,9 @@ Three surfaces; only one is per-backend.
|
||||
`deregister_bottle`, `supervise_queue`. Fully backend-agnostic. Both the
|
||||
local `cli.py` and the remote console funnel through it, so policy is
|
||||
uniform and `cli.py` becomes a thin client rather than a parallel
|
||||
launcher.
|
||||
launcher. **Transport: HTTP** — the most universal/reliable choice on
|
||||
every host (no vsock/unix-socket portability caveats); a local unix
|
||||
socket is a fine optimization, but HTTP is the wire contract.
|
||||
2. **Data plane (agent → orchestrator)** — the egress / git / supervise
|
||||
endpoints. Already agnostic today (agents dial `http://sidecar:9099`);
|
||||
only the *address* and *how packets get there* are per-backend.
|
||||
@@ -218,6 +241,17 @@ exposes a broker the orchestrator calls to launch an agent:
|
||||
Stage 3 removes); a narrower broker later.
|
||||
- **Apple** — the `container` CLI/daemon.
|
||||
|
||||
**Broker request schema:** human-readable JSON of **static flags + ids
|
||||
only** (never free-form paths/argv — that's what keeps it un-coercible
|
||||
into arbitrary launches), wrapped as a **signed JWT** so the broker
|
||||
verifies *provenance*: the request came from the real orchestrator, not a
|
||||
forged one from a compromised co-located component. The orchestrator signs;
|
||||
the broker verifies with the orchestrator's public key (provisioned at
|
||||
broker install). This is the concrete form of security #3's "structured
|
||||
requests only." Same JSON-with-ids + JWT shape for all
|
||||
orchestrator↔broker/sidecar communication; cases that don't fit get
|
||||
handled as they arise, with this as the default.
|
||||
|
||||
If `BottleBackend` bloats, the pressure valve is composition one level
|
||||
down: vend a `backend.network()` / `Wiring` collaborator rather than
|
||||
piling methods on — the same discipline, recursed.
|
||||
@@ -297,17 +331,37 @@ Keep the sidecar **service one shared thing** throughout.
|
||||
- **PR #350 (netpool single-source):** the same "one source per fact,
|
||||
composition over parallel hierarchies" discipline the contract follows.
|
||||
|
||||
## Decisions (review 2026-07-13)
|
||||
|
||||
- **Egress sharing:** **consolidate egress** (worth it). Treat it as a
|
||||
minimal, hardened attack surface for a malicious agent rather than
|
||||
keeping it per-bottle; pair it with the identity token above and the
|
||||
short-lived-vault-token mitigations (Secret handling / #355).
|
||||
- **Control-plane transport:** **HTTP** — most universal/reliable on every
|
||||
host; unix socket is an optional local optimization (see the contract).
|
||||
- **Broker request schema:** **signed-JWT JSON, static flags + ids only**
|
||||
(see the launch broker) — provenance + un-coercible by construction.
|
||||
- **State re-adoption:** the restart procedure is:
|
||||
1. **Singleton:** a new orchestrator launch requires no pre-existing
|
||||
orchestrator; any found (healthy or not) is fully shut down first.
|
||||
2. Re-adoption **waits for the new orchestrator to be healthy**.
|
||||
3. Once healthy, it discovers all agents needing adoption via **both the
|
||||
SQLite registry and live process/VM inspection** *before serving any
|
||||
other request*. The two-source sweep is what closes the *in-flight
|
||||
launch* race — a launch that started (VM booting / slot claimed) but
|
||||
hadn't committed to SQLite when the old orchestrator died would be
|
||||
invisible to a SQLite-only sweep; process inspection catches it.
|
||||
Launches should also write an **intent record ahead of committing
|
||||
resources** so the sweep can reconcile intent vs. actual.
|
||||
|
||||
## Open questions
|
||||
|
||||
- **Egress sharing tradeoff:** is the secret-concentration blast radius of
|
||||
one shared mitmproxy worth the resource win, or share only supervise
|
||||
(near-zero secrets) and keep egress + git-gate per-bottle initially?
|
||||
- **Control-plane shape:** RPC transport (unix socket / vsock / HTTP over
|
||||
the TAP) and the live-reload protocol for per-bottle policy.
|
||||
- **State re-adoption:** exact scheme for an orchestrator restart to
|
||||
re-adopt running agent VMs from the SQLite registry without racing
|
||||
in-flight launches.
|
||||
- **VM-to-VM routing:** the nft forward rules + addressing for a
|
||||
per-host orchestrator VM on its own TAP.
|
||||
- **Broker request schema:** the exact structured contract that stays
|
||||
auditable and can't be coerced into launching arbitrary payloads.
|
||||
- **VM-to-VM routing:** per-backend, and in the design it *is*
|
||||
`BottleBackend.wire()` (DNAT+forward for fc, shared-net for
|
||||
docker/apple), not orchestrator logic. Likely already prototyped in the
|
||||
earlier "sidecars in VMs" spike — **link the spike** and this drops from
|
||||
open question to "implement per the spike."
|
||||
- **Live-reload protocol** for per-bottle policy over the HTTP control
|
||||
plane (add/remove routes/keys/proposals without a restart).
|
||||
- **Identity-token delivery:** exactly how the per-bottle token is placed
|
||||
where the agent can present it but not swap in another bottle's.
|
||||
|
||||
Reference in New Issue
Block a user