diff --git a/docs/prds/prd-new-audit-event-schema.md b/docs/prds/prd-new-audit-event-schema.md index 9b781ea2..073d2ac7 100644 --- a/docs/prds/prd-new-audit-event-schema.md +++ b/docs/prds/prd-new-audit-event-schema.md @@ -37,7 +37,8 @@ emits into: and signed-commit (#480), auth/authz, and audit self-events — each with its trusted-vs-claimed fields and redaction rules. 5. A **stable export projection** (CloudEvents / OpenTelemetry Logs) and - the **#324 delivery contract** (payload, `(epoch, seq)` cursor, dedup, + the **#324 delivery contract** (payload, per-chain + `(host, epoch, seq)` cursor, dedup, backpressure, retention ordering). It is explicitly scheduled to land **immediately after the host @@ -142,7 +143,7 @@ stable top level: "epoch": 7, // writer-boot counter, bumped once per host-controller (writer) start "seq": 1287, // monotonic sequence within this epoch (gap-detectable) "segment": "20260726T000000Z", // journal segment id (rotation boundary); chain continues across segments - "prev": "", // hash of the previous record in the chain ("" for a segment genesis) + "prev": "", // prior record hash ("" only for the first record of a new chain) "hash": "", // sha256(prev + canonical(this event with hash="")) // ---- timestamps (host-owned) ---- @@ -270,7 +271,9 @@ The `hash` field is computed over the canonical form of the event **with digest = sha256_hex(prev_hash + canonical({**event, "hash": ""})) ``` -`prev` is the prior record's `hash`; a segment genesis uses `prev = ""`. +`prev` is the prior record's `hash`; only the first record of a brand-new +chain uses `prev = ""`. The first record of a rotated segment carries the +preceding segment's head, as specified under *Rotation* below. Two exact rules pin the bytes so the chain is reproducible anywhere: 1. **Serialize the record with its own `hash` field set to `""`** (present, @@ -296,7 +299,7 @@ serialization and linkage. An implementation is conformant iff it reproduces these bytes and hashes. ``` -# Record 0 — segment genesis (prev = "") +# Record 0 — brand-new chain genesis (prev = "") canonical(record0, hash=""): {"hash":"","id":"11111111-1111-4111-8111-111111111111","prev":"","seq":0,"type":"audit.segment_open"} hash0 = sha256("" + canonical) = @@ -319,17 +322,28 @@ bump that changes the bytes fails a golden test loudly. ### Ordering, idempotency, and duplicate handling -- **Ordering.** `(epoch, seq)` is a strict total order per host and, because +- **Ordering.** `(epoch, seq)` is a strict total order within one host's + native chain and, because the writer is single, a strict order per bottle/activation within that host — satisfying "at least strict causal order per activation/bottle". `causation_id` records the explicit cause edges (a DAG) on top of the total order, so a consumer can reconstruct request → decision → cutoff - even if unrelated events interleave between them. + even if unrelated events interleave between them. There is deliberately + no invented total order across imported host chains; a cross-host key is + `(host, epoch, seq)`, and consumers use correlation/causation edges where + causal ordering across hosts is known. - **Idempotency.** `id` is the idempotency key. A producer that retries an emit (e.g. after a writer restart mid-handoff) **reuses the same `id`**; - the writer drops a second append bearing an `id` already present in the - current segment's in-memory set, and the index `UPSERT`s by `id`, so a - duplicate never double-counts or forks the chain. + before appending, the writer checks a durable, host-wide id ledger that + spans every segment and survives restart/retention, and drops an id + already present. The ledger is operational metadata, not an audit source + of truth: after a crash it is reconciled from the journal before appends + resume, and retention preserves id tombstones after journal segments are + pruned. An in-memory set may cache the ledger but is never the authority. + The index additionally has a unique key on `id`; its `UPSERT` is + defensive and does not substitute for the pre-append check. Thus a + duplicate never enters the source-of-truth journal, double-counts, or + forks the chain. - **Deduplication downstream.** Because `id` is stable across export and replay, #324's cursor replay and any cross-host merge dedup on `id` — no consumer needs to invent a second identity. @@ -338,18 +352,22 @@ bump that changes the bytes fails a golden test loudly. - **Rotation.** At a segment boundary the writer opens a new segment file, sets its `segment` id, and carries the rotated-out segment's head as the - new genesis `prev` — so the chain is continuous *across* segments while + new segment's first `prev` — so the chain is continuous *across* segments + (`prev = ""` is reserved for the first record of a brand-new chain) while each file stays independently openable. `verify` walks segments in order - and checks the head-to-genesis link at each seam. + and checks the preceding-head-to-first-record link at each seam. - **Restart.** Covered above: read last line → adopt its `hash` as `prev`, bump `epoch`, reset `seq`. The chain never restarts even though the counters do. - **Import.** `audit import ` appends an externally supplied segment (e.g. recovered from another host or a backup). Import verifies - the incoming chain in isolation first, then links it only if its genesis - `prev` matches a known head or is explicitly grafted; imported records - keep their original `id` (dedup) and are marked with their origin host so - attribution is not laundered. + the incoming chain in isolation first. A continuation whose first `prev` + matches a known head extends that chain. A foreign chain is registered as + a separate immutable chain namespace rather than rewriting or grafting + its records (which would invalidate their hashes). Imported records keep + their original `host`, `id`, `epoch`, and `seq`; the index keys their + native order by `(host, epoch, seq)` so attribution is not laundered and + tuples from different hosts cannot collide. - **Truncation.** A crash can leave a partial final line; `verify` reports it as `truncated-tail` (recoverable — replay resumes from the last intact record). A chain that ends before a persisted head, or a missing interior @@ -376,11 +394,13 @@ start the writer: 2. bumps `epoch` (persisted alongside the chain head) and resets `seq` to 0 for the new boot. -Total order is therefore `(epoch, seq)` — monotonic across restarts by -construction — with `ts_event`/`ts_recorded` for human reading and `ts_mono` for -sub-second ordering inside an epoch. A crash mid-append truncates at most -the last (partial) line; the verifier flags it and replay resumes from the -last intact record. +Total order within this host's native chain is therefore `(epoch, seq)` — +monotonic across restarts by construction — with +`ts_event`/`ts_recorded` for human reading and `ts_mono` for sub-second +ordering inside an epoch. Imported chains retain their own +`(host, epoch, seq)` order and do not acquire a fictional order relative to +the local chain. A crash mid-append truncates at most the last (partial) +line; the verifier flags it and replay resumes from the last intact record. ### Journal (source of truth) + SQLite index (rebuildable) @@ -401,8 +421,11 @@ last intact record. `outcome`, `sensitivity`, `correlation_id`, `causation_id`, plus two event-specific projections promoted out of `payload` for query — `repository` and `commit_sha` (populated for `forge.*`/`commit.*`, null -otherwise). The full canonical record is stored verbatim in a `raw` column -so the index never loses fidelity to the journal. +otherwise). The unique event id and native-order index are respectively +`id` and `(host, epoch, seq)`; a local-only `ingest_seq` provides stable +display order when a query intentionally mixes chains without pretending +that it is causal order. The full canonical record is stored verbatim in a +`raw` column so the index never loses fidelity to the journal. **Local query surface** — `audit query`, no egress, no paid platform: @@ -411,7 +434,7 @@ audit query \ [--since T] [--until T] [--type egress.*] [--host H] [--bottle B] \ [--activation A] [--agent SLUG] [--actor ID] [--outcome blocked] \ [--repository R] [--correlation-id C] [--commit SHA] \ - [--follow BOTTLE] # a bottled agent's events in (epoch, seq) order + [--follow BOTTLE] # one host chain's events in (epoch, seq) order [--json | --table] audit verify [--segment S] # offline chain check; exit non-zero on any break audit rebuild # drop + replay journal → index @@ -578,9 +601,12 @@ PRD fixes the contract it depends on: exact bytes the hash covers), optionally wrapped in the CloudEvents projection whose `data` *is* that record. Either way the integrity fields travel intact and the receiver can verify against the same bytes. -- **Cursor.** The export cursor is `(epoch, seq)` (equivalently the last - exported `hash`). It advances **only on acknowledgement**, so delivery is - at-least-once and gap-free; a crash re-sends from the last acked cursor. +- **Cursor.** Export maintains one cursor per native host chain: + `(host, epoch, seq)` (equivalently that chain's last exported `hash`). + It advances **only on acknowledgement**, so delivery is at-least-once and + gap-free; a crash re-sends from the last acked cursor. Imported foreign + chains use independent cursors and never share a bare `(epoch, seq)` + namespace with the local chain. - **Idempotency / replay.** Dedup is on `id` (stable across replay), so at-least-once delivery is safe — the receiver collapses re-sends. - **Backpressure.** The outbox is the journal itself plus a cursor; when @@ -683,8 +709,8 @@ Nothing in the contract is left undefined — only its code is deferred. the `lifecycle.bottled_agent_*` leaf names. (Design → *The envelope* / *Event registry*.) - **Retention head-carry — yes.** When a journal segment is rotated out, - the new segment's genesis `prev` is the rotated-out head, so the verifier - still trusts the current head across a rotation. (Folds into the + the new segment's first record carries the rotated-out head as `prev`, so + the verifier still trusts the current head across a rotation. (Folds into the retention follow-up.) - **Redaction reuses the egress detectors — yes, scoped.** Reuse the deterministic `dlp_detectors` (`scan_token_patterns` / `redact_tokens` /