docs(prd): tighten audit ordering and chain guarantees
This commit is contained in:
@@ -37,7 +37,8 @@ emits into:
|
|||||||
and signed-commit (#480), auth/authz, and audit self-events — each with
|
and signed-commit (#480), auth/authz, and audit self-events — each with
|
||||||
its trusted-vs-claimed fields and redaction rules.
|
its trusted-vs-claimed fields and redaction rules.
|
||||||
5. A **stable export projection** (CloudEvents / OpenTelemetry Logs) and
|
5. A **stable export projection** (CloudEvents / OpenTelemetry Logs) and
|
||||||
the **#324 delivery contract** (payload, `(epoch, seq)` cursor, dedup,
|
the **#324 delivery contract** (payload, per-chain
|
||||||
|
`(host, epoch, seq)` cursor, dedup,
|
||||||
backpressure, retention ordering).
|
backpressure, retention ordering).
|
||||||
|
|
||||||
It is explicitly scheduled to land **immediately after the host
|
It is explicitly scheduled to land **immediately after the host
|
||||||
@@ -142,7 +143,7 @@ stable top level:
|
|||||||
"epoch": 7, // writer-boot counter, bumped once per host-controller (writer) start
|
"epoch": 7, // writer-boot counter, bumped once per host-controller (writer) start
|
||||||
"seq": 1287, // monotonic sequence within this epoch (gap-detectable)
|
"seq": 1287, // monotonic sequence within this epoch (gap-detectable)
|
||||||
"segment": "20260726T000000Z", // journal segment id (rotation boundary); chain continues across segments
|
"segment": "20260726T000000Z", // journal segment id (rotation boundary); chain continues across segments
|
||||||
"prev": "<hex>", // hash of the previous record in the chain ("" for a segment genesis)
|
"prev": "<hex>", // prior record hash ("" only for the first record of a new chain)
|
||||||
"hash": "<hex>", // sha256(prev + canonical(this event with hash=""))
|
"hash": "<hex>", // sha256(prev + canonical(this event with hash=""))
|
||||||
|
|
||||||
// ---- timestamps (host-owned) ----
|
// ---- timestamps (host-owned) ----
|
||||||
@@ -270,7 +271,9 @@ The `hash` field is computed over the canonical form of the event **with
|
|||||||
digest = sha256_hex(prev_hash + canonical({**event, "hash": ""}))
|
digest = sha256_hex(prev_hash + canonical({**event, "hash": ""}))
|
||||||
```
|
```
|
||||||
|
|
||||||
`prev` is the prior record's `hash`; a segment genesis uses `prev = ""`.
|
`prev` is the prior record's `hash`; only the first record of a brand-new
|
||||||
|
chain uses `prev = ""`. The first record of a rotated segment carries the
|
||||||
|
preceding segment's head, as specified under *Rotation* below.
|
||||||
Two exact rules pin the bytes so the chain is reproducible anywhere:
|
Two exact rules pin the bytes so the chain is reproducible anywhere:
|
||||||
|
|
||||||
1. **Serialize the record with its own `hash` field set to `""`** (present,
|
1. **Serialize the record with its own `hash` field set to `""`** (present,
|
||||||
@@ -296,7 +299,7 @@ serialization and linkage. An implementation is conformant iff it
|
|||||||
reproduces these bytes and hashes.
|
reproduces these bytes and hashes.
|
||||||
|
|
||||||
```
|
```
|
||||||
# Record 0 — segment genesis (prev = "")
|
# Record 0 — brand-new chain genesis (prev = "")
|
||||||
canonical(record0, hash=""):
|
canonical(record0, hash=""):
|
||||||
{"hash":"","id":"11111111-1111-4111-8111-111111111111","prev":"","seq":0,"type":"audit.segment_open"}
|
{"hash":"","id":"11111111-1111-4111-8111-111111111111","prev":"","seq":0,"type":"audit.segment_open"}
|
||||||
hash0 = sha256("" + canonical) =
|
hash0 = sha256("" + canonical) =
|
||||||
@@ -319,17 +322,28 @@ bump that changes the bytes fails a golden test loudly.
|
|||||||
|
|
||||||
### Ordering, idempotency, and duplicate handling
|
### Ordering, idempotency, and duplicate handling
|
||||||
|
|
||||||
- **Ordering.** `(epoch, seq)` is a strict total order per host and, because
|
- **Ordering.** `(epoch, seq)` is a strict total order within one host's
|
||||||
|
native chain and, because
|
||||||
the writer is single, a strict order per bottle/activation within that
|
the writer is single, a strict order per bottle/activation within that
|
||||||
host — satisfying "at least strict causal order per activation/bottle".
|
host — satisfying "at least strict causal order per activation/bottle".
|
||||||
`causation_id` records the explicit cause edges (a DAG) on top of the
|
`causation_id` records the explicit cause edges (a DAG) on top of the
|
||||||
total order, so a consumer can reconstruct request → decision → cutoff
|
total order, so a consumer can reconstruct request → decision → cutoff
|
||||||
even if unrelated events interleave between them.
|
even if unrelated events interleave between them. There is deliberately
|
||||||
|
no invented total order across imported host chains; a cross-host key is
|
||||||
|
`(host, epoch, seq)`, and consumers use correlation/causation edges where
|
||||||
|
causal ordering across hosts is known.
|
||||||
- **Idempotency.** `id` is the idempotency key. A producer that retries an
|
- **Idempotency.** `id` is the idempotency key. A producer that retries an
|
||||||
emit (e.g. after a writer restart mid-handoff) **reuses the same `id`**;
|
emit (e.g. after a writer restart mid-handoff) **reuses the same `id`**;
|
||||||
the writer drops a second append bearing an `id` already present in the
|
before appending, the writer checks a durable, host-wide id ledger that
|
||||||
current segment's in-memory set, and the index `UPSERT`s by `id`, so a
|
spans every segment and survives restart/retention, and drops an id
|
||||||
duplicate never double-counts or forks the chain.
|
already present. The ledger is operational metadata, not an audit source
|
||||||
|
of truth: after a crash it is reconciled from the journal before appends
|
||||||
|
resume, and retention preserves id tombstones after journal segments are
|
||||||
|
pruned. An in-memory set may cache the ledger but is never the authority.
|
||||||
|
The index additionally has a unique key on `id`; its `UPSERT` is
|
||||||
|
defensive and does not substitute for the pre-append check. Thus a
|
||||||
|
duplicate never enters the source-of-truth journal, double-counts, or
|
||||||
|
forks the chain.
|
||||||
- **Deduplication downstream.** Because `id` is stable across export and
|
- **Deduplication downstream.** Because `id` is stable across export and
|
||||||
replay, #324's cursor replay and any cross-host merge dedup on `id` — no
|
replay, #324's cursor replay and any cross-host merge dedup on `id` — no
|
||||||
consumer needs to invent a second identity.
|
consumer needs to invent a second identity.
|
||||||
@@ -338,18 +352,22 @@ bump that changes the bytes fails a golden test loudly.
|
|||||||
|
|
||||||
- **Rotation.** At a segment boundary the writer opens a new segment file,
|
- **Rotation.** At a segment boundary the writer opens a new segment file,
|
||||||
sets its `segment` id, and carries the rotated-out segment's head as the
|
sets its `segment` id, and carries the rotated-out segment's head as the
|
||||||
new genesis `prev` — so the chain is continuous *across* segments while
|
new segment's first `prev` — so the chain is continuous *across* segments
|
||||||
|
(`prev = ""` is reserved for the first record of a brand-new chain) while
|
||||||
each file stays independently openable. `verify` walks segments in order
|
each file stays independently openable. `verify` walks segments in order
|
||||||
and checks the head-to-genesis link at each seam.
|
and checks the preceding-head-to-first-record link at each seam.
|
||||||
- **Restart.** Covered above: read last line → adopt its `hash` as `prev`,
|
- **Restart.** Covered above: read last line → adopt its `hash` as `prev`,
|
||||||
bump `epoch`, reset `seq`. The chain never restarts even though the
|
bump `epoch`, reset `seq`. The chain never restarts even though the
|
||||||
counters do.
|
counters do.
|
||||||
- **Import.** `audit import <segment>` appends an externally supplied
|
- **Import.** `audit import <segment>` appends an externally supplied
|
||||||
segment (e.g. recovered from another host or a backup). Import verifies
|
segment (e.g. recovered from another host or a backup). Import verifies
|
||||||
the incoming chain in isolation first, then links it only if its genesis
|
the incoming chain in isolation first. A continuation whose first `prev`
|
||||||
`prev` matches a known head or is explicitly grafted; imported records
|
matches a known head extends that chain. A foreign chain is registered as
|
||||||
keep their original `id` (dedup) and are marked with their origin host so
|
a separate immutable chain namespace rather than rewriting or grafting
|
||||||
attribution is not laundered.
|
its records (which would invalidate their hashes). Imported records keep
|
||||||
|
their original `host`, `id`, `epoch`, and `seq`; the index keys their
|
||||||
|
native order by `(host, epoch, seq)` so attribution is not laundered and
|
||||||
|
tuples from different hosts cannot collide.
|
||||||
- **Truncation.** A crash can leave a partial final line; `verify` reports
|
- **Truncation.** A crash can leave a partial final line; `verify` reports
|
||||||
it as `truncated-tail` (recoverable — replay resumes from the last intact
|
it as `truncated-tail` (recoverable — replay resumes from the last intact
|
||||||
record). A chain that ends before a persisted head, or a missing interior
|
record). A chain that ends before a persisted head, or a missing interior
|
||||||
@@ -376,11 +394,13 @@ start the writer:
|
|||||||
2. bumps `epoch` (persisted alongside the chain head) and resets `seq` to
|
2. bumps `epoch` (persisted alongside the chain head) and resets `seq` to
|
||||||
0 for the new boot.
|
0 for the new boot.
|
||||||
|
|
||||||
Total order is therefore `(epoch, seq)` — monotonic across restarts by
|
Total order within this host's native chain is therefore `(epoch, seq)` —
|
||||||
construction — with `ts_event`/`ts_recorded` for human reading and `ts_mono` for
|
monotonic across restarts by construction — with
|
||||||
sub-second ordering inside an epoch. A crash mid-append truncates at most
|
`ts_event`/`ts_recorded` for human reading and `ts_mono` for sub-second
|
||||||
the last (partial) line; the verifier flags it and replay resumes from the
|
ordering inside an epoch. Imported chains retain their own
|
||||||
last intact record.
|
`(host, epoch, seq)` order and do not acquire a fictional order relative to
|
||||||
|
the local chain. A crash mid-append truncates at most the last (partial)
|
||||||
|
line; the verifier flags it and replay resumes from the last intact record.
|
||||||
|
|
||||||
### Journal (source of truth) + SQLite index (rebuildable)
|
### Journal (source of truth) + SQLite index (rebuildable)
|
||||||
|
|
||||||
@@ -401,8 +421,11 @@ last intact record.
|
|||||||
`outcome`, `sensitivity`, `correlation_id`, `causation_id`, plus two
|
`outcome`, `sensitivity`, `correlation_id`, `causation_id`, plus two
|
||||||
event-specific projections promoted out of `payload` for query —
|
event-specific projections promoted out of `payload` for query —
|
||||||
`repository` and `commit_sha` (populated for `forge.*`/`commit.*`, null
|
`repository` and `commit_sha` (populated for `forge.*`/`commit.*`, null
|
||||||
otherwise). The full canonical record is stored verbatim in a `raw` column
|
otherwise). The unique event id and native-order index are respectively
|
||||||
so the index never loses fidelity to the journal.
|
`id` and `(host, epoch, seq)`; a local-only `ingest_seq` provides stable
|
||||||
|
display order when a query intentionally mixes chains without pretending
|
||||||
|
that it is causal order. The full canonical record is stored verbatim in a
|
||||||
|
`raw` column so the index never loses fidelity to the journal.
|
||||||
|
|
||||||
**Local query surface** — `audit query`, no egress, no paid platform:
|
**Local query surface** — `audit query`, no egress, no paid platform:
|
||||||
|
|
||||||
@@ -411,7 +434,7 @@ audit query \
|
|||||||
[--since T] [--until T] [--type egress.*] [--host H] [--bottle B] \
|
[--since T] [--until T] [--type egress.*] [--host H] [--bottle B] \
|
||||||
[--activation A] [--agent SLUG] [--actor ID] [--outcome blocked] \
|
[--activation A] [--agent SLUG] [--actor ID] [--outcome blocked] \
|
||||||
[--repository R] [--correlation-id C] [--commit SHA] \
|
[--repository R] [--correlation-id C] [--commit SHA] \
|
||||||
[--follow BOTTLE] # a bottled agent's events in (epoch, seq) order
|
[--follow BOTTLE] # one host chain's events in (epoch, seq) order
|
||||||
[--json | --table]
|
[--json | --table]
|
||||||
audit verify [--segment S] # offline chain check; exit non-zero on any break
|
audit verify [--segment S] # offline chain check; exit non-zero on any break
|
||||||
audit rebuild # drop + replay journal → index
|
audit rebuild # drop + replay journal → index
|
||||||
@@ -578,9 +601,12 @@ PRD fixes the contract it depends on:
|
|||||||
exact bytes the hash covers), optionally wrapped in the CloudEvents
|
exact bytes the hash covers), optionally wrapped in the CloudEvents
|
||||||
projection whose `data` *is* that record. Either way the integrity fields
|
projection whose `data` *is* that record. Either way the integrity fields
|
||||||
travel intact and the receiver can verify against the same bytes.
|
travel intact and the receiver can verify against the same bytes.
|
||||||
- **Cursor.** The export cursor is `(epoch, seq)` (equivalently the last
|
- **Cursor.** Export maintains one cursor per native host chain:
|
||||||
exported `hash`). It advances **only on acknowledgement**, so delivery is
|
`(host, epoch, seq)` (equivalently that chain's last exported `hash`).
|
||||||
at-least-once and gap-free; a crash re-sends from the last acked cursor.
|
It advances **only on acknowledgement**, so delivery is at-least-once and
|
||||||
|
gap-free; a crash re-sends from the last acked cursor. Imported foreign
|
||||||
|
chains use independent cursors and never share a bare `(epoch, seq)`
|
||||||
|
namespace with the local chain.
|
||||||
- **Idempotency / replay.** Dedup is on `id` (stable across replay), so
|
- **Idempotency / replay.** Dedup is on `id` (stable across replay), so
|
||||||
at-least-once delivery is safe — the receiver collapses re-sends.
|
at-least-once delivery is safe — the receiver collapses re-sends.
|
||||||
- **Backpressure.** The outbox is the journal itself plus a cursor; when
|
- **Backpressure.** The outbox is the journal itself plus a cursor; when
|
||||||
@@ -683,8 +709,8 @@ Nothing in the contract is left undefined — only its code is deferred.
|
|||||||
the `lifecycle.bottled_agent_*` leaf names. (Design → *The envelope* /
|
the `lifecycle.bottled_agent_*` leaf names. (Design → *The envelope* /
|
||||||
*Event registry*.)
|
*Event registry*.)
|
||||||
- **Retention head-carry — yes.** When a journal segment is rotated out,
|
- **Retention head-carry — yes.** When a journal segment is rotated out,
|
||||||
the new segment's genesis `prev` is the rotated-out head, so the verifier
|
the new segment's first record carries the rotated-out head as `prev`, so
|
||||||
still trusts the current head across a rotation. (Folds into the
|
the verifier still trusts the current head across a rotation. (Folds into the
|
||||||
retention follow-up.)
|
retention follow-up.)
|
||||||
- **Redaction reuses the egress detectors — yes, scoped.** Reuse the
|
- **Redaction reuses the egress detectors — yes, scoped.** Reuse the
|
||||||
deterministic `dlp_detectors` (`scan_token_patterns` / `redact_tokens` /
|
deterministic `dlp_detectors` (`scan_token_patterns` / `redact_tokens` /
|
||||||
|
|||||||
Reference in New Issue
Block a user