diff --git a/docs/prds/prd-new-audit-event-schema.md b/docs/prds/prd-new-audit-event-schema.md index d458ac88..920d5652 100644 --- a/docs/prds/prd-new-audit-event-schema.md +++ b/docs/prds/prd-new-audit-event-schema.md @@ -66,10 +66,11 @@ shared schema. Concretely: ## Goals / Success Criteria - A single `AuditEvent` envelope type, versioned, that every producer - emits. Fields are split into a **`trusted`** block (host-supplied: - bottled-agent slug from source-IP attribution, host wall-clock, producer - identity) and an **`untrusted`** block (anything the agent or a remote - claimed), and the split is structural, not a convention. + emits. The trust boundary is **one `untrusted` region**: everything + outside it is host-established and trusted (source-IP → bottled-agent + attribution, host wall-clock, producer identity, chain metadata); + `untrusted` is the sole place anything an agent or a remote claimed may + go. The boundary is structural, not a convention. - **Canonical serialization** (`sort_keys`, `(",", ":")` separators, UTF-8, `ensure_ascii=False`) is defined once and reused, so the same logical event always hashes identically across producers and hosts. @@ -86,6 +87,11 @@ shared schema. Concretely: - A **redaction rule** is enforced at the envelope boundary: known credential-shaped fields are rejected/redacted before a record is written; the writer refuses raw secrets rather than storing them. +- The envelope **projects onto the OpenTelemetry Logs data model and a + CloudEvents JSON envelope** by field re-mapping alone (no reformat), + preserving the trust boundary and carrying the integrity fields — per + #487's export/interop requirement. (The export adapters are follow-up; + the *schema* must make them a re-map.) - The **host controller (#468)** emits `lifecycle.*` events through this contract as the first consumer; existing supervise/egress producers are migrated behind the same envelope without changing operator-facing @@ -120,26 +126,21 @@ stable top level: ``` { + // --- everything at the top level is host-established (trusted) --- "v": 1, // schema version — bumped only on a breaking change "id": "", // globally unique event id "type": "egress.decision", // dotted event type from the registry - - // --- chain / ordering (structural, host-owned) --- "epoch": 7, // writer-boot counter, bumped once per host-controller (writer) start "seq": 1287, // monotonic sequence within this epoch (gap-detectable) "prev": "", // hash of the previous record in the chain ("" for genesis) "hash": "", // sha256(prev + canonical(this event with hash="")) + "producer": "host-controller", // which host component wrote this + "host": "mac-studio-1", + "bottled_agent": "amber-fox-12", // slug from source-IP attribution (null for host-level events) + "ts_wall": "2026-07-26T18:22:04.113Z", // host wall-clock, RFC3339 UTC + "ts_mono": 90142.55, // host monotonic secs since this epoch's boot (intra-epoch ordering only) - // --- trusted: everything the *host* established; authoritative --- - "trusted": { - "producer": "host-controller", // which host component wrote this - "host": "mac-studio-1", - "bottled_agent": "amber-fox-12", // slug from source-IP attribution (null for host-level events) - "ts_wall": "2026-07-26T18:22:04.113Z", // host wall-clock, RFC3339 UTC - "ts_mono": 90142.55 // host monotonic secs since this epoch's boot (intra-epoch ordering only) - }, - - // --- untrusted: anything the agent or a remote claimed; never authoritative --- + // --- the ONLY untrusted region: anything the agent or a remote claimed --- "untrusted": { "reason": "npm install needs registry.npmjs.org", "target": "registry.npmjs.org:443" @@ -147,22 +148,22 @@ stable top level: } ``` -The **`trusted` / `untrusted` split is the core invariant.** A producer may -only place a field in `trusted` if the *host* established it: the -source-IP → `bottled_agent` slug attribution, the host's own clock -(`ts_wall`/`ts_mono`), and the producer's own identity. **`producer` and -`ts_*` live inside `trusted` on purpose** — they are host-supplied, so -grouping them there (rather than as loose top-level fields) keeps the -"authoritative ⇔ inside `trusted`" rule structural, with nothing -host-established leaking outside it. Everything an agent or a remote said -goes in `untrusted`. A reader (or a future policy engine) can therefore -trust `trusted.bottled_agent` for attribution and treat `untrusted.*` as -adversarial claims — the distinction the current stores lack. +The **trust boundary is a single region, not a split.** Everything outside +`untrusted` is trusted by construction — the host established it: the +schema/chain metadata (`v`, `id`, `type`, `epoch`, `seq`, `prev`, `hash`), +the producer and host identity, the source-IP → `bottled_agent` slug +attribution, and the host clock (`ts_wall`/`ts_mono`). `untrusted` is the +**one** place anything an agent or a remote claimed may go. A reader (or a +future policy engine) trusts every top-level field for attribution and +treats `untrusted.*` — and only `untrusted.*` — as adversarial claims. -Only the small structural set — `v`, `id`, `type`, `epoch`, `seq`, `prev`, -`hash` — sits at the top level; it is host-owned too, but it is chain -metadata rather than event data, so it stays out of the `trusted` body to -keep that body purely about *what happened*. +Framing it as "one untrusted region, everything else trusted" (rather than +two parallel `trusted`/`untrusted` blocks) removes the mistake where a +producer forgets to nest a host field under `trusted`: a field is trusted +unless it is deliberately placed inside `untrusted`. The construction API +enforces this — producers pass trusted fields positionally and hand all +agent/remote-claimed data as the single `untrusted` mapping, so there is no +way to emit a top-level field that *looks* authoritative but isn't. ### Canonical serialization + hash chain @@ -235,7 +236,7 @@ required `untrusted` keys so producers and the verifier agree on shape: - **lifecycle.*** — `lifecycle.bottled_agent_start`, `lifecycle.bottled_agent_stop`, `lifecycle.bottled_agent_crash` (producer: host-controller, #468). Leaf names use `bottled_agent` to match - the `trusted.bottled_agent` field — one term for the subject everywhere. + the top-level `bottled_agent` field — one term for the subject everywhere. - **decision.*** — `decision.proposed`, `decision.resolved` (producer: supervise; carries operator action + justification, replacing PRD 0013's row shape). @@ -287,6 +288,56 @@ and keep the event) rather than drop, so a producer bug can never make an audit event vanish; the key deny-list stays a hard refusal because a credential in a named field is always a producer bug worth surfacing. +### Export / interoperability (CloudEvents, OpenTelemetry Logs) + +#487 requires the envelope to map onto the **OpenTelemetry Logs data +model** and/or a **CloudEvents JSON** envelope *without losing integrity or +attribution semantics*. The flattened shape (one `untrusted` region, +everything else trusted at top level) does **not** conflict with either — it +maps *more* cleanly than a nested `trusted`/`untrusted` pair would, because +both target models expect a flat set of top-level fields plus one payload +subtree. + +**CloudEvents.** Context attributes MUST be scalar simple types — a map +cannot be a context attribute — so a nested `trusted` block would have had +to be flattened for CloudEvents anyway. Our flat top level maps directly: +`id`→`id`, `type`→`type`, `producer`+`host`→`source`, +`bottled_agent`→`subject`, `ts_wall`→`time`; the integrity/chain fields +(`epoch`, `seq`, `prev`, `hash`, `v`) ride as **extension attributes** +(scalars — legal). The `untrusted` map goes in `data`. Only mechanical +transform needed: extension attribute names must be lowercase-alphanumeric, +so `bottled_agent`/`ts_mono`/etc. are renamed at export (e.g. a +`botbottle`-prefixed form) — a naming rule, not a schema conflict. + +**OpenTelemetry Logs.** `ts_wall`→`Timestamp`; `type`→the `event.name` +attribute; the flat trusted fields → `Attributes` under a `botbottle.*` +namespace (`botbottle.bottled_agent`, `botbottle.producer`, +`botbottle.chain.hash`, …); `untrusted.*` → `Attributes` under +`botbottle.untrusted.*` (or `Body`). OTel attributes are a dotted map that +happily carries the nested subtree. + +**Attribution is preserved** precisely because the boundary is now +structural: on export, top-level fields become trusted context/attributes +and the `untrusted` subtree stays a single, clearly-named region — so a +downstream consumer still sees exactly which fields an agent claimed. +Nothing agent-claimed is promoted to a trusted-looking position. + +**Integrity has one deliberate caveat.** CloudEvents/OTel are +representation envelopes with their own (or no) canonicalization; `hash` +and `prev` are computed over **our** canonical JSON, not over the exported +form. So the chain fields travel *as data* for reference, but +tamper-evidence is always verified against the **native journal** (the +source of truth) — never re-derived from an exported CloudEvents/OTel +record, whose key ordering / number formatting the exporter may change. +Export is thus a lossless-for-attribution **projection** that carries the +integrity fields along; verification stays on the canonical journal. This +satisfies "without losing integrity or attribution semantics": both are +carried, neither is *relied upon* in the foreign format. + +The export adapters themselves (and #324's webhook delivery / causal +ordering) are follow-up implementation — this PRD fixes the *schema* so +that projection is a field re-map, never a reformat. + ## Implementation chunks 1. **(this PR — PRD only.)** The contract above. No code; scheduled to land @@ -308,7 +359,10 @@ credential in a named field is always a producer bug worth surfacing. 5. **Migrate existing producers.** Re-emit supervise `decision.*` (retiring the standalone `supervise_audit_entries` shape behind the index), egress `egress.*`, git-gate `forge.*`, control-plane `auth.*`. -6. **(follow-up.)** Cross-host merge transport (#324); per-writer signing + +6. **CloudEvents / OTel export adapters.** A projection layer emitting each + event as a CloudEvents JSON envelope and/or an OTel LogRecord (field + re-map per *Export / interoperability*); feeds #324's webhook delivery. +7. **(follow-up.)** Cross-host merge transport (#324); per-writer signing + external anchoring on the chain head; retention/rotation policy. ## Resolved in review (#495) @@ -320,10 +374,12 @@ credential in a named field is always a producer bug worth surfacing. carrying the last chain head as the next `prev` gives a total order of `(epoch, seq)` that survives restarts; `ts_mono` orders only within an epoch. (Design → *ordering across restarts*.) -- **`ts_*` and `producer` belong in `trusted`.** They are host-supplied, so - they now sit inside the `trusted` block; only chain metadata stays at the - top level. (Design → *The envelope*.) -- **Subject term is `bottled_agent` everywhere** — the `trusted` field and +- **Flatten to one `untrusted` region — decided.** Everything outside + `untrusted` (chain metadata, `producer`/`host`, `bottled_agent`, `ts_*`) + is trusted by construction, so the separate `trusted` sub-block is + removed; a field is trusted unless deliberately placed under `untrusted`. + (Design → *The envelope*.) +- **Subject term is `bottled_agent` everywhere** — the top-level field and the `lifecycle.bottled_agent_*` leaf names. (Design → *The envelope* / *Event registry*.) - **Retention head-carry — yes.** When a journal segment is rotated out,