PRD: canonical tamper-evident audit-event schema (#487) #495

Closed
didericis wants to merge 5 commits from didericis/prd-audit-event-schema AGit into main
Showing only changes of commit 88b82a169e - Show all commits
+365 -82
View File
@@ -19,17 +19,25 @@ field names, timestamps, or how a bottled agent is identified.
This PRD defines **one canonical audit-event contract** every producer
emits into:
1. A **versioned envelope** — schema version, event id, event type,
monotonic + wall-clock timestamps, and an explicit **trust boundary**
between host-supplied and agent-claimed fields.
2. **Canonical JSON serialization + a per-writer hash chain**, so any
deletion or edit of a past record breaks the chain and is detectable
offline.
1. A **versioned envelope** — schema version, event id, event/observed
timestamps, host-attributed identity (`bottle`/`bottled_agent`/
`activation`), provenance (`manifest_digest`, `policy_version`),
host-observed `actor`/`action`/`resource`/`outcome`, correlation/
causation ids, a sensitivity class, a typed payload, and an explicit
**trust boundary** between host-supplied and agent-claimed fields.
2. **Canonical JSON serialization + a per-writer hash chain** (with
normative test vectors), so any deletion, edit, or reorder of a past
record breaks the chain and is detectable offline.
3. An **append-only JSONL journal as the source of truth**, with a
**rebuildable SQLite index** for local search — no paid platform, no
network dependency.
4. An **initial event registry** covering lifecycle, decision, egress,
auth, and forge events.
**rebuildable SQLite index** and a local `audit query`/`verify` surface
— no paid platform, no network dependency.
4. An **initial event registry** covering lifecycle, host-controller,
supervise decision, egress (request/decision/cutoff/anomaly), git-gate
and signed-commit (#480), auth/authz, and audit self-events — each with
its trusted-vs-claimed fields and redaction rules.
5. A **stable export projection** (CloudEvents / OpenTelemetry Logs) and
the **#324 delivery contract** (payload, `(epoch, seq)` cursor, dedup,
backpressure, retention ordering).
It is explicitly scheduled to land **immediately after the host
controller (#468)** so the host controller's lifecycle transitions are the
8
@@ -126,44 +134,108 @@ stable top level:
```
{
// --- everything at the top level is host-established (trusted) ---
// ---- schema + integrity (host-owned) ----
"v": 1, // schema version — bumped only on a breaking change
"id": "<uuid4>", // globally unique event id
"id": "<uuid4>", // globally unique event id; stable across export/replay (dedup key)
"type": "egress.decision", // dotted event type from the registry
"epoch": 7, // writer-boot counter, bumped once per host-controller (writer) start
"seq": 1287, // monotonic sequence within this epoch (gap-detectable)
"prev": "<hex>", // hash of the previous record in the chain ("" for genesis)
"segment": "20260726T000000Z", // journal segment id (rotation boundary); chain continues across segments
"prev": "<hex>", // hash of the previous record in the chain ("" for a segment genesis)
"hash": "<hex>", // sha256(prev + canonical(this event with hash=""))
"producer": "host-controller", // which host component wrote this
"host": "mac-studio-1",
"bottled_agent": "amber-fox-12", // slug from source-IP attribution (null for host-level events)
"ts_wall": "2026-07-26T18:22:04.113Z", // host wall-clock, RFC3339 UTC
"ts_mono": 90142.55, // host monotonic secs since this epoch's boot (intra-epoch ordering only)
// --- the ONLY untrusted region: anything the agent or a remote claimed ---
// ---- timestamps (host-owned) ----
"ts_event": "2026-07-26T18:22:04.061Z", // when the underlying event occurred at the boundary
"ts_recorded": "2026-07-26T18:22:04.113Z", // when the single writer appended it (authoritative)
"ts_mono": 90142.55, // monotonic secs since this epoch's boot (intra-epoch ordering only)
// ---- attribution + provenance (host-established) ----
"producer": "egress", // host component that emitted the event
"host": "mac-studio-1",
"bottle": "amber-fox", // bottle (container/VM) identity
"bottled_agent": "amber-fox-12", // bottled-agent slug from source-IP attribution (null for host-level events)
"activation": "01J8Z...", // activation id: one run/session of the bottled agent (null if n/a)
"manifest_digest": "sha256:9f2…", // digest of the manifest the bottled agent is running (null if n/a)
"policy_version": "egress-2026-07-20", // policy/config version in force at decision time (null if n/a)
// ---- semantics: host-observed facts of what happened ----
"actor": "bottled-agent:amber-fox-12", // who acted, as a host-attributed identity
"action": "egress.connect", // what was attempted / done
"resource": "registry.npmjs.org:443", // what it acted on, as observed at the boundary
"outcome": "blocked", // host-decided result: allowed|blocked|deferred|success|failure
"sensitivity": "security", // classification: normal|security|restricted (drives redaction + export)
// ---- correlation (host-assigned) ----
"correlation_id": "flow-9c2a…", // groups a related flow (request → decision → cutoff)
"causation_id": "<event id>", // the event that directly caused this one ("" if root)
// ---- typed, trusted, event-specific payload (shape fixed per type in the registry) ----
"payload": {
"route_id": 4,
"detector": "token_patterns"
},
// ---- the ONLY untrusted region: agent- or remote-claimed data ----
"untrusted": {
"reason": "npm install needs registry.npmjs.org",
"target": "registry.npmjs.org:443"
"reason": "npm install needs registry.npmjs.org" // the agent's stated justification
}
}
```
The **trust boundary is a single region, not a split.** Everything outside
`untrusted` is trusted by construction — the host established it: the
schema/chain metadata (`v`, `id`, `type`, `epoch`, `seq`, `prev`, `hash`),
the producer and host identity, the source-IP → `bottled_agent` slug
attribution, and the host clock (`ts_wall`/`ts_mono`). `untrusted` is the
**one** place anything an agent or a remote claimed may go. A reader (or a
future policy engine) trusts every top-level field for attribution and
treats `untrusted.*` — and only `untrusted.*` — as adversarial claims.
`untrusted` is trusted by construction — the host established it: schema and
chain metadata, both timestamps, the attribution/provenance fields
(source-IP → `bottled_agent`/`bottle`/`activation`, `manifest_digest`,
`policy_version`), the host-observed semantics
(`actor`/`action`/`resource`/`outcome`), the host-assigned correlation ids,
and the typed `payload`. `untrusted` is the **one** place anything an agent
or a remote claimed may go (e.g. the agent's free-text `reason`). A reader
(or a future policy engine) trusts every field outside `untrusted` for
attribution and treats `untrusted.*` — and only `untrusted.*` — as
adversarial claims.
Framing it as "one untrusted region, everything else trusted" (rather than
two parallel `trusted`/`untrusted` blocks) removes the mistake where a
producer forgets to nest a host field under `trusted`: a field is trusted
unless it is deliberately placed inside `untrusted`. The construction API
enforces this — producers pass trusted fields positionally and hand all
agent/remote-claimed data as the single `untrusted` mapping, so there is no
way to emit a top-level field that *looks* authoritative but isn't.
Framing it as "one untrusted region, everything else trusted" removes the
mistake where a producer forgets to mark a claimed field: a field is
trusted unless it is deliberately placed inside `untrusted`. The
construction API enforces this — producers pass trusted fields explicitly
Review

annoying, but should be consistent: bottled_agent_start, bottled_agent_stop, bottled_agent_crash

annoying, but should be consistent: bottled_agent_start, bottled_agent_stop, bottled_agent_crash
Review

Done: lifecycle.bottled_agent_start / _stop / _crash, matching the bottled_agent field name.

Done: `lifecycle.bottled_agent_start` / `_stop` / `_crash`, matching the `bottled_agent` field name.
and hand all agent/remote-claimed data as the single `untrusted` mapping,
so there is no way to emit a top-level field that *looks* authoritative but
isn't.
**Two timestamps** because they answer different questions and can diverge
under backpressure: `ts_event` is when the thing happened at the boundary
(the proxy saw the connect, the gate saw the push); `ts_recorded` is when
the single writer durably appended it. Ordering and the chain use
`(epoch, seq)`, never either wall clock. Both are host-set — a bottled
agent never supplies a timestamp.
**Optionality.** `bottle`/`bottled_agent`/`activation`, `manifest_digest`,
`policy_version`, and `payload`/`untrusted` are `null`/absent for events
that have no such subject (a host-level `hostctl.*` or `audit.*` event has
no bottled agent). Absent ≠ empty: a reader distinguishes "no subject" from
"unknown". `id`, `type`, the chain fields, both timestamps, `producer`,
`host`, `actor`, `action`, `outcome`, and `sensitivity` are always present.
didericis marked this conversation as resolved
Review

Would be nice if we use the same credential shape detectors we use on egress, but not sure whether it's worth it or if that would be brittle/concerns would diverge. Evaluate how reasonable that is.

Would be nice if we use the same credential shape detectors we use on egress, but not sure whether it's worth it or if that would be brittle/concerns would diverge. Evaluate how reasonable that is.
Review

Evaluated: reuse the deterministic detectors in gateway/egress/dlp_detectors.py (scan_token_patterns / redact_tokens, plus scan_known_secrets) — pure-Python and already the project's source of truth for credential shapes, so wire-egress and journal-redaction can't drift. Excluded scan_entropy: it's tuned for large streamed bodies and would false-positive on the short high-entropy values audit records legitimately carry (hashes, uuids, base64 ids), redacting the very fingerprints we want to keep. So: share the deterministic detectors, keep entropy egress-only.

Evaluated: reuse the deterministic detectors in `gateway/egress/dlp_detectors.py` (`scan_token_patterns` / `redact_tokens`, plus `scan_known_secrets`) — pure-Python and already the project's source of truth for credential shapes, so wire-egress and journal-redaction can't drift. **Excluded `scan_entropy`**: it's tuned for large streamed bodies and would false-positive on the short high-entropy values audit records legitimately carry (hashes, uuids, base64 ids), redacting the very fingerprints we want to keep. So: share the deterministic detectors, keep entropy egress-only.
#### Trust provenance of every common field
| Field | Trust | Set by |
|---|---|---|
| `v` `id` `type` `epoch` `seq` `segment` `prev` `hash` | trusted | the single writer |
| `ts_event` | trusted | emitting host component (boundary) |
| `ts_recorded` `ts_mono` | trusted | the single writer |
| `producer` `host` | trusted | the single writer |
| `bottle` `bottled_agent` `activation` | trusted | gateway source-IP → slug attribution |
| `manifest_digest` `policy_version` | trusted | control plane (what was in force) |
| `actor` `action` `resource` `outcome` | trusted | host component that observed/decided it |
| `sensitivity` | trusted | registry default for `type`, overridable up (never down) by the producer |
| `correlation_id` `causation_id` | trusted | the single writer (assigned as it threads the flow) |
| `payload.*` | trusted | emitting host component (shape fixed per `type`) |
| `untrusted.*` | **claimed** | copied verbatim from a bottle / gateway / forge / remote |
Every registry entry (below) restates, per event type, which `payload`
keys are required and names any `untrusted` keys it carries — so "trusted
vs claimed" is explicit for every event-specific attribute, not just the
common ones.
### Canonical serialization + hash chain
2
@@ -183,11 +255,92 @@ The `hash` field is computed over the canonical form of the event **with
digest = sha256_hex(prev_hash + canonical({**event, "hash": ""}))
```
`prev` is the prior record's `hash`; genesis uses `prev = ""`. This makes
the journal an append-only Merkle-style chain: editing or deleting record
*n* changes its hash, so record *n+1*'s `prev` no longer matches — the
break is local and points at the tampered record. Verification needs only
the journal itself (no keys), so it runs offline and in CI.
`prev` is the prior record's `hash`; a segment genesis uses `prev = ""`.
Two exact rules pin the bytes so the chain is reproducible anywhere:
1. **Serialize the record with its own `hash` field set to `""`** (present,
empty), never omitted — the key set is identical before and after
Review

Yes

Yes
Review

Adopted — the rotated-out head becomes the new segment's genesis prev, so the verifier still trusts the current head across a rotation. Folded into the retention follow-up.

Adopted — the rotated-out head becomes the new segment's genesis `prev`, so the verifier still trusts the current head across a rotation. Folded into the retention follow-up.
hashing.
2. **Digest = `sha256_hex(prev + canonical(record_with_empty_hash))`**,
where `prev` is the previous record's `hash` string (`""` at genesis),
`+` is string concatenation, and `canonical` is the function above.
`ts_mono`, being a float, is serialized by Python's shortest-round-trip
`repr` via `json.dumps`; producers therefore emit it as a JSON number
they do not post-process. (All other fields are strings/ints/objects,
which serialize unambiguously.)
Editing or deleting record *n* changes its hash, so record *n+1*'s `prev`
no longer matches — the break is local and names the tampered record.
Verification needs only the journal itself (no keys), so it runs offline
and in CI.
#### Test vectors (normative)
Two records, reduced to the chain-relevant fields, demonstrate the exact
serialization and linkage. An implementation is conformant iff it
reproduces these bytes and hashes.
```
# Record 0 — segment genesis (prev = "")
canonical(record0, hash=""):
{"hash":"","id":"11111111-1111-4111-8111-111111111111","prev":"","seq":0,"type":"audit.segment_open"}
hash0 = sha256("" + canonical) =
942ea5729bcac6efdbdea942396bfa574ab0d6ebf5615402595359422f2aeb83
# Record 1 — chains onto record 0 (prev = hash0)
canonical(record1, hash=""):
{"hash":"","id":"22222222-2222-4222-8222-222222222222","prev":"942ea5729bcac6efdbdea942396bfa574ab0d6ebf5615402595359422f2aeb83","seq":1,"type":"lifecycle.bottled_agent_start"}
hash1 = sha256(hash0 + canonical) =
bc082347680405fee50b60a9c304611aa026950b15d869b7e3ae56e1c451b856
# Tamper check: flip record0.type → recompute →
# 4553eda647f33f0c608cfea44be28efbbaca45ed30b873fcbd4405fa5ce737ed
# which no longer equals record1.prev (942ea5…) — the break is detected at record1.
```
The implementation PR ships these plus full-envelope vectors (every field
populated, and a redaction case) as committed fixtures, so a schema-version
bump that changes the bytes fails a golden test loudly.
### Ordering, idempotency, and duplicate handling
- **Ordering.** `(epoch, seq)` is a strict total order per host and, because
the writer is single, a strict order per bottle/activation within that
host — satisfying "at least strict causal order per activation/bottle".
`causation_id` records the explicit cause edges (a DAG) on top of the
total order, so a consumer can reconstruct request → decision → cutoff
even if unrelated events interleave between them.
- **Idempotency.** `id` is the idempotency key. A producer that retries an
emit (e.g. after a writer restart mid-handoff) **reuses the same `id`**;
the writer drops a second append bearing an `id` already present in the
current segment's in-memory set, and the index `UPSERT`s by `id`, so a
duplicate never double-counts or forks the chain.
- **Deduplication downstream.** Because `id` is stable across export and
replay, #324's cursor replay and any cross-host merge dedup on `id` — no
consumer needs to invent a second identity.
### Behavior across rotation, restart, import, truncation
- **Rotation.** At a segment boundary the writer opens a new segment file,
sets its `segment` id, and carries the rotated-out segment's head as the
new genesis `prev` — so the chain is continuous *across* segments while
each file stays independently openable. `verify` walks segments in order
and checks the head-to-genesis link at each seam.
- **Restart.** Covered above: read last line → adopt its `hash` as `prev`,
bump `epoch`, reset `seq`. The chain never restarts even though the
counters do.
- **Import.** `audit import <segment>` appends an externally supplied
segment (e.g. recovered from another host or a backup). Import verifies
the incoming chain in isolation first, then links it only if its genesis
`prev` matches a known head or is explicitly grafted; imported records
keep their original `id` (dedup) and are marked with their origin host so
attribution is not laundered.
- **Truncation.** A crash can leave a partial final line; `verify` reports
it as `truncated-tail` (recoverable — replay resumes from the last intact
record). A chain that ends before a persisted head, or a missing interior
`seq`, is reported as `gap`/`missing-suffix` (evidence of deletion, not a
clean crash). The two are distinguished so an operator can tell "power
loss" from "someone trimmed the log".
### Single writer; ordering across restarts
@@ -209,7 +362,7 @@ start the writer:
0 for the new boot.
Total order is therefore `(epoch, seq)` — monotonic across restarts by
construction — with `ts_wall` for human reading and `ts_mono` for
construction — with `ts_event`/`ts_recorded` for human reading and `ts_mono` for
sub-second ordering inside an epoch. A crash mid-append truncates at most
the last (partial) line; the verifier flags it and replay resumes from the
last intact record.
@@ -220,35 +373,91 @@ last intact record.
alongside `host_db_path()`), one canonical event per line, opened
`O_APPEND`. This is authoritative.
- **Index:** a new `audit_events` table via the existing `DbStore` /
`TableMigrations` machinery, holding the envelope columns plus JSON
blobs, indexed on `(bottled_agent, type, ts_wall)`. It is a **derived
cache**: `audit rebuild` truncates and replays the journal, re-verifying
the chain as it goes. If the DB is deleted or drifts, it is regenerated
from the journal with no data loss. (This supersedes the free-standing
`supervise_audit_entries` table, which becomes a view/producer onto the
new index.)
`TableMigrations` machinery. It is a **derived cache, not a second source
of truth**: `audit rebuild` truncates and replays the journal,
re-verifying the chain as it goes, so a deleted or drifted DB is
regenerated from the journal with no data loss. On a `verify` failure
during rebuild it stops and reports rather than indexing past a break.
(This supersedes the free-standing `supervise_audit_entries` table, which
becomes a producer onto the new index.)
**Indexable fields** (columns + indices): `ts_event`, `ts_recorded`,
`type`, `host`, `bottle`, `bottled_agent`, `activation`, `actor`,
`outcome`, `sensitivity`, `correlation_id`, `causation_id`, plus two
event-specific projections promoted out of `payload` for query —
`repository` and `commit_sha` (populated for `forge.*`/`commit.*`, null
otherwise). The full canonical record is stored verbatim in a `raw` column
so the index never loses fidelity to the journal.
**Local query surface** — `audit query`, no egress, no paid platform:
```
audit query \
[--since T] [--until T] [--type egress.*] [--host H] [--bottle B] \
[--activation A] [--agent SLUG] [--actor ID] [--outcome blocked] \
[--repository R] [--correlation-id C] [--commit SHA] \
[--follow BOTTLE] # a bottled agent's events in (epoch, seq) order
[--json | --table]
audit verify [--segment S] # offline chain check; exit non-zero on any break
audit rebuild # drop + replay journal → index
audit import <segment> # graft an external segment (see above)
```
Type filters accept a `group.*` glob. A read-only local HTTP endpoint
mirrors the same filters for the future review console; both are pure reads
over the index and can never mutate the journal.
### Event registry (initial)
Dotted `type` names, grouped; the registry is a table mapping type
required `untrusted` keys so producers and the verifier agree on shape:
Dotted `type` names, grouped. The registry is a table mapping each type to
its required `payload` keys, its `untrusted` keys (if any), a default
`sensitivity`, and its correlation behavior — so producers and the verifier
agree on shape and "trusted vs claimed" is pinned per type. Initial
coverage (the issue's mandated set):
- **lifecycle.*** — `lifecycle.bottled_agent_start`,
`lifecycle.bottled_agent_stop`, `lifecycle.bottled_agent_crash`
(producer: host-controller, #468). Leaf names use `bottled_agent` to match
the top-level `bottled_agent` field — one term for the subject everywhere.
- **decision.*** — `decision.proposed`, `decision.resolved`
(producer: supervise; carries operator action + justification, replacing
PRD 0013's row shape).
- **egress.*** — `egress.decision` (allow/block at the proxy),
`egress.route_added`.
- **auth.*** — `auth.token_minted`, `auth.token_rejected` (control-plane;
**never** the token itself — see redaction).
- **forge.*** — `forge.push_accepted`, `forge.push_rejected` (git-gate),
`forge.pr_opened`.
| Group / type | Producer | Required `payload` (trusted) | `untrusted` | Default sensitivity |
|---|---|---|---|---|
| **lifecycle.*** — `bottled_agent_start` / `_stop` / `_crash` | host-controller (#468) | `manifest_digest`, `exit` (for stop/crash) | — | normal |
| **hostctl.*** — `broker_launch`, `broker_teardown`, `broker_reject` | host-controller (#468) | `op`, `request_digest` | — | security |
| **decision.*** — `proposed`, `resolved` | supervise | `tool`, `operator_action`, `justification`, `diff_digest` | `agent_rationale` | security |
| **egress.*** — `request`, `decision`, `cutoff`, `anomaly` | egress proxy | `route_id`, `detector` (on match), `bytes` (cutoff) | `reason`, `target_claimed` | security |
| **forge.*** — `push_accepted`, `push_rejected`, `pr_opened` | git-gate | `repository`, `ref`, `gitleaks_result` | `title`, `description` | security |
| **commit.signed** (#480) | git-gate | `repository`, `commit_sha`, `activation_key_id`, `signature_ref` | `commit_message` | security |
| **auth.*** — `token_minted`, `token_rejected`, `authz_denied` | control plane | `role`, `token_id`, `reason_code` | — | security |
| **audit.*** — `segment_open`, `verify_failed`, `truncation_detected`, `export_failed` | audit writer/verifier | `segment`, `detail` | — | security |
New types are additive; adding one does not bump `v`. Removing or
re-typing a field bumps `v`.
Notes:
- **`egress.request` vs `egress.decision`** share a `correlation_id`; the
`decision`'s `causation_id` points at the `request`, and a later `cutoff`
chains onto the `decision` — so a flow is reconstructable.
- **`audit.*` self-events** make the audit subsystem audit itself: a failed
verification, a detected truncation, or a dropped export is itself a
chained, tamper-evident record — you cannot silence the alarm without
breaking the chain that carries it.
- **Free-text and remote-echoed fields are always `untrusted`** (`reason`,
`agent_rationale`, PR `title`/`description`, `target_claimed`), because
they originate in the bottle or a remote response; the host-observed
counterpart (`resource`, `outcome`, `gitleaks_result`) is the trusted
fact.
**Sensitivity + redaction per type.** Every type's default `sensitivity`
is listed above; a producer may raise it (never lower it). `restricted`
events keep their `payload` in the journal but the export projection ships
only the envelope + a payload digest unless the consumer is authorized —
so a `security`/`restricted` record is still counted and correlated
downstream without leaking its body. The credential-shape redaction rules
(next) apply to **every** type regardless of sensitivity.
**Schema evolution & backward-compatible readers.** The registry is
append-only: **adding** a type, an optional `payload` key, or an
`untrusted` key does **not** bump `v`; readers ignore unknown fields
(forward-compatible) and treat absent optional fields as `null`.
**Removing** or **re-typing** a field, or making an optional field
required, bumps `v`. A reader declares the max `v` it understands and
refuses to *interpret* a higher-`v` record, but the **verifier is
version-agnostic** — the hash covers whatever fields exist, so chain
integrity is checkable across versions without understanding semantics.
Every `v` bump ships a migration note and updated golden vectors.
### Redaction rule
@@ -288,6 +497,14 @@ and keep the event) rather than drop, so a producer bug can never make an
audit event vanish; the key deny-list stays a hard refusal because a
credential in a named field is always a producer bug worth surfacing.
**No raw-payload capture by default.** The envelope carries *decisions and
metadata*, not traffic. Prompts, model responses, request/response bodies,
and file contents are **not** recorded unless a producer opts a specific,
reviewed field in — and such a field is `untrusted` and subject to both
redaction layers. This keeps the audit log from becoming a covert copy of
the very data the sandbox exists to contain (the issue's "unsafe payload
capture" non-goal).
### Export / interoperability (CloudEvents, OpenTelemetry Logs)
#487 requires the envelope to map onto the **OpenTelemetry Logs data
@@ -302,14 +519,14 @@ subtree.
cannot be a context attribute — so a nested `trusted` block would have had
to be flattened for CloudEvents anyway. Our flat top level maps directly:
`id`→`id`, `type`→`type`, `producer`+`host`→`source`,
`bottled_agent`→`subject`, `ts_wall`→`time`; the integrity/chain fields
`bottled_agent`→`subject`, `ts_event`→`time` (`ts_recorded` as an extension); the integrity/chain fields
(`epoch`, `seq`, `prev`, `hash`, `v`) ride as **extension attributes**
(scalars — legal). The `untrusted` map goes in `data`. Only mechanical
transform needed: extension attribute names must be lowercase-alphanumeric,
so `bottled_agent`/`ts_mono`/etc. are renamed at export (e.g. a
`botbottle`-prefixed form) — a naming rule, not a schema conflict.
**OpenTelemetry Logs.** `ts_wall`→`Timestamp`; `type`→the `event.name`
**OpenTelemetry Logs.** `ts_event`→`Timestamp`, `ts_recorded`→`ObservedTimestamp`; `type`→the `event.name`
attribute; the flat trusted fields → `Attributes` under a `botbottle.*`
namespace (`botbottle.bottled_agent`, `botbottle.producer`,
`botbottle.chain.hash`, …); `untrusted.*` → `Attributes` under
@@ -334,9 +551,47 @@ integrity fields along; verification stays on the canonical journal. This
satisfies "without losing integrity or attribution semantics": both are
carried, neither is *relied upon* in the foreign format.
The export adapters themselves (and #324's webhook delivery / causal
ordering) are follow-up implementation — this PRD fixes the *schema* so
that projection is a field re-map, never a reformat.
The export adapters themselves are follow-up implementation — this PRD
fixes the *schema* so that projection is a field re-map, never a reformat.
#### The #324 delivery contract (payload, cursor, backpressure)
#324 transports events off-box; it must not invent a second envelope. This
PRD fixes the contract it depends on:
- **Payload.** #324 ships the **native canonical record verbatim** (the
exact bytes the hash covers), optionally wrapped in the CloudEvents
projection whose `data` *is* that record. Either way the integrity fields
travel intact and the receiver can verify against the same bytes.
- **Cursor.** The export cursor is `(epoch, seq)` (equivalently the last
exported `hash`). It advances **only on acknowledgement**, so delivery is
at-least-once and gap-free; a crash re-sends from the last acked cursor.
- **Idempotency / replay.** Dedup is on `id` (stable across replay), so
at-least-once delivery is safe — the receiver collapses re-sends.
- **Backpressure.** The outbox is the journal itself plus a cursor; when
the endpoint is slow the cursor simply lags — the writer never blocks on
export, and audit never applies backpressure to the data plane it
records.
- **Retention interaction.** Retention/rotation **must not** prune a
segment whose records are still behind the export cursor; the reaper
honors `min(cursor)` across all configured consumers. (The schedule
itself stays the retention follow-up; this is the *ordering* constraint
that follow-up must respect.)
#### #480 signed-commit attribution maps in without weakening it
#480 binds a commit's bytes to a per-activation signing key. It maps to the
`commit.signed` event: `payload` carries `repository`, `commit_sha`,
`activation_key_id`, and a `signature_ref` (the detached-signature
location or its digest) — **not** the private key and not a re-derived
signature. The audit event therefore *references and timestamps* #480's
existing byte-to-activation-key proof inside the tamper-evident chain; it
does not re-implement or replace it, so #480's guarantee is unweakened —
the signature still verifies against the commit bytes independently, and
the audit record adds only "this binding was observed at this point in the
chain". The trusted `actor`/`activation` fields and the `commit_sha`
payload are host-observed at the gate, so attribution cannot be forged by
the committing agent.
## Implementation chunks
@@ -349,21 +604,49 @@ that projection is a field re-map, never a reformat.
`redact_tokens`); unit tests for determinism, chain-break detection,
`epoch`/`seq` continuity across a simulated restart, and redaction of
both a deny-listed key and a token-shaped value.
3. **SQLite index + `audit rebuild` / `audit verify` CLI.** New
`audit_events` migration; replay-from-journal; offline chain verifier;
local query commands (by bottled-agent / type / time / producer).
3. **SQLite index + `audit` CLI.** New `audit_events` migration (indexable
fields above); replay-from-journal; offline chain verifier
(`truncated-tail` vs `gap`); `query` / `verify` / `rebuild` / `import`;
idempotent `UPSERT` by `id`.
4. **Host controller as first producer (#468).** Wire
`lifecycle.bottled_agent_*` emission into the host controller's
start/stop/crash paths; establish the `epoch` bump + chain-head carry on
writer restart here (the host controller owns the single writer).
`lifecycle.bottled_agent_*` and `hostctl.*` emission into the host
controller; establish the `epoch` bump + chain-head carry + segment
rotation on writer restart here (it owns the single writer).
5. **Migrate existing producers.** Re-emit supervise `decision.*` (retiring
the standalone `supervise_audit_entries` shape behind the index), egress
`egress.*`, git-gate `forge.*`, control-plane `auth.*`.
6. **CloudEvents / OTel export adapters.** A projection layer emitting each
event as a CloudEvents JSON envelope and/or an OTel LogRecord (field
re-map per *Export / interoperability*); feeds #324's webhook delivery.
7. **(follow-up.)** Cross-host merge transport (#324); per-writer signing +
external anchoring on the chain head; retention/rotation policy.
`egress.*`, git-gate `forge.*` + `commit.signed` (#480), control-plane
`auth.*`; add the `audit.*` self-events (verify/truncation/export
failure).
6. **CloudEvents / OTel export adapters + #324 delivery.** Projection layer
(field re-map per *Export / interoperability*) plus the outbox cursor,
ack-driven advance, and retention-ordering guard the #324 contract
specifies.
7. **(follow-up.)** Cross-host merge transport; per-writer signing +
external anchoring on the chain head; retention/rotation *schedule*.
## Acceptance-criteria coverage (#487)
The issue defines the contract; implementation is explicitly split into
follow-up PRs. This PRD is the durable decision record; each acceptance box
maps to a section:
| #487 acceptance criterion | Where |
|---|---|
| Durable PRD defines versioned envelope + initial registry | *The envelope*, *Event registry* |
| Canonical JSON + hash-chain rules, unambiguous, with test vectors | *Canonical serialization + hash chain* → *Test vectors* |
| Trust provenance explicit for every common + event-specific field | *Trust provenance of every common field*; per-type `untrusted` in *Event registry* |
| Redaction prohibits credentials / raw secrets / unsafe capture by default | *Redaction rule*; `untrusted`-only claims; sensitivity classes |
| JSONL journal canonical; SQLite index fully rebuildable | *Journal + SQLite index* (`audit rebuild`) |
| Minimum local search/query contract | *Journal + SQLite index* → *Local query surface* |
| #324 can transport/replay without a second envelope | *The #324 delivery contract* |
| #480 maps in without weakening its byte-to-activation-key guarantee | *#480 signed-commit attribution maps in…* |
| Schema evolution + backward-compatible readers | *Schema evolution & backward-compatible readers* |
| Integrity detects modification / deletion / reorder / bad continuation | *Test vectors* (tamper), *Behavior across rotation…truncation*, `audit verify` |
Two acceptance items are **specified here, implemented later** by design
(the issue permits this): the concrete test-vector *fixtures* and the
`audit` CLI land in impl chunks 23; the #324 outbox lands in chunk 6.
Nothing in the contract is left undefined — only its code is deferred.
## Resolved in review (#495)