bot-bottle/docs/prds/prd-new-audit-event-schema.md

# PRD prd-new: Canonical tamper-evident audit-event schema and local query contract

- **Status:** Draft
- **Author:** didericis-claude
- **Created:** 2026-07-26
- **Issue:** #487

## Summary

bot-bottle already emits security- and provenance-relevant events from
several producers — supervise operator decisions (PRD 0013's
`AuditStore`), egress allow/block enforcement, git-gate push decisions,
control-plane token minting, and (next) host-controller lifecycle
transitions — but each writes its own shape to its own sink. There is no
shared envelope, no tamper-evidence, and no single place to search. Local
incident reconstruction means grepping several stores that don't agree on
field names, timestamps, or how a bottled agent is identified.

This PRD defines **one canonical audit-event contract** every producer
emits into:

1. A **versioned envelope** — schema version, event id, event type,
   monotonic + wall-clock timestamps, and an explicit **trust boundary**
   between host-supplied and agent-claimed fields.
2. **Canonical JSON serialization + a per-writer hash chain**, so any
   deletion or edit of a past record breaks the chain and is detectable
   offline.
3. An **append-only JSONL journal as the source of truth**, with a
   **rebuildable SQLite index** for local search — no paid platform, no
   network dependency.
4. An **initial event registry** covering lifecycle, decision, egress,
   auth, and forge events.

It is explicitly scheduled to land **immediately after the host
controller (#468)** so the host controller's lifecycle transitions are the
first producer wired onto the new contract (per the directive on #487).

## Problem

Audit infrastructure is fragmented across #468, #324, and #480 with no
shared schema. Concretely:

- **No shared envelope.** `supervise_audit_entries` (PRD 0013) has
  `timestamp, bottle_slug, component, operator_action, ...`. The egress
  proxy and git-gate log their own ad-hoc lines. There is no common
  `event_id`, `event_type`, or version, so cross-producer correlation
  ("what did bottled agent X do between its start and this rejected push?") is
  manual and lossy.
- **No tamper-evidence.** The audit store is a plain SQLite table. Anyone
  who can write the DB can delete or rewrite a row and leave no trace.
  Audit that an attacker (or a buggy agent) can silently rewrite is not
  audit.
- **Trusted and untrusted data are mixed.** A bottled agent is attributed by
  **source IP → slug** at the gateway (host-supplied, trustworthy). An
  agent can also *claim* things about itself in a tool call
  (agent-claimed, adversarial). Today nothing in the record marks which is
  which, so a reader can be misled by an agent-supplied field that looks
  authoritative.
- **No local search.** Reconstructing an incident means reading multiple
  sinks with different schemas. There is no query contract and no promise
  that the index can be rebuilt from the journal if it drifts or is lost.
- **No redaction rule.** Nothing prohibits a producer from writing a raw
  token or secret into an audit record, which would turn the audit log
  itself into a credential store.

## Goals / Success Criteria

- A single `AuditEvent` envelope type, versioned, that every producer
  emits. The trust boundary is **one `untrusted` region**: everything
  outside it is host-established and trusted (source-IP → bottled-agent
  attribution, host wall-clock, producer identity, chain metadata);
  `untrusted` is the sole place anything an agent or a remote claimed may
  go. The boundary is structural, not a convention.
- **Canonical serialization** (`sort_keys`, `(",", ":")` separators,
  UTF-8, `ensure_ascii=False`) is defined once and reused, so the same
  logical event always hashes identically across producers and hosts.
- Each writer maintains a **hash chain**: `hash = sha256(prev_hash ||
  canonical(event))`. Deleting or editing any past record breaks every
  subsequent link; a standalone verifier detects the break offline with no
  secret material.
- The **JSONL journal is the source of truth**; the **SQLite index is
  fully rebuildable** from it (`audit rebuild` reconstructs the DB and
  re-verifies the chain).
- **Local query** works with no paid platform and no egress: filter by
  bottled agent, event type, time range, and producer, and follow a bottled
  agent's events in order.
- A **redaction rule** is enforced at the envelope boundary: known
  credential-shaped fields are rejected/redacted before a record is
  written; the writer refuses raw secrets rather than storing them.
- The envelope **projects onto the OpenTelemetry Logs data model and a
  CloudEvents JSON envelope** by field re-mapping alone (no reformat),
  preserving the trust boundary and carrying the integrity fields — per
  #487's export/interop requirement. (The export adapters are follow-up;
  the *schema* must make them a re-map.)
- The **host controller (#468)** emits `lifecycle.*` events through this
  contract as the first consumer; existing supervise/egress producers are
  migrated behind the same envelope without changing operator-facing
  behavior.

## Non-goals

- **Cross-host aggregation / shipping.** This PRD makes each host's journal
  canonical and correlatable *by construction* (stable ids, hash chain),
  but the transport that merges multiple hosts into one timeline is a
  follow-up (#324). The schema is designed so that merge is a later append,
  not a reformat.
- **Cryptographic signing / external anchoring.** Hash-chaining gives
  tamper-**evidence** (you can detect edits), not tamper-**resistance**
  against an attacker who can rewrite the whole chain. Per-writer signing
  keys and periodic external anchoring are a follow-up; the chain-head hash
  is the seam they attach to.
- **Real-time alerting / SIEM rules.** Query is local and pull-based here.
- **Retention / rotation policy.** Journal rotation and TTL are operator
  policy, tracked separately; the format must survive rotation (chain head
  carried across segments) but this PRD does not set the schedule.
- **Replacing PRD 0013's operator queue.** The supervise proposal/response
  queue is unchanged; only its terminal *audit* record is re-emitted onto
  the new envelope.

## Design

### The envelope

One dataclass, `AuditEvent`, serialized to a JSON object with a small,
stable top level:

```
{
  // --- everything at the top level is host-established (trusted) ---
  "v": 1,                          // schema version — bumped only on a breaking change
  "id": "<uuid4>",                 // globally unique event id
  "type": "egress.decision",       // dotted event type from the registry
  "epoch": 7,                      // writer-boot counter, bumped once per host-controller (writer) start
  "seq": 1287,                     // monotonic sequence within this epoch (gap-detectable)
  "prev": "<hex>",                 // hash of the previous record in the chain ("" for genesis)
  "hash": "<hex>",                 // sha256(prev + canonical(this event with hash=""))
  "producer": "host-controller",   // which host component wrote this
  "host": "mac-studio-1",
  "bottled_agent": "amber-fox-12", // slug from source-IP attribution (null for host-level events)
  "ts_wall": "2026-07-26T18:22:04.113Z", // host wall-clock, RFC3339 UTC
  "ts_mono": 90142.55,             // host monotonic secs since this epoch's boot (intra-epoch ordering only)

  // --- the ONLY untrusted region: anything the agent or a remote claimed ---
  "untrusted": {
    "reason": "npm install needs registry.npmjs.org",
    "target": "registry.npmjs.org:443"
  }
}
```

The **trust boundary is a single region, not a split.** Everything outside
`untrusted` is trusted by construction — the host established it: the
schema/chain metadata (`v`, `id`, `type`, `epoch`, `seq`, `prev`, `hash`),
the producer and host identity, the source-IP → `bottled_agent` slug
attribution, and the host clock (`ts_wall`/`ts_mono`). `untrusted` is the
**one** place anything an agent or a remote claimed may go. A reader (or a
future policy engine) trusts every top-level field for attribution and
treats `untrusted.*` — and only `untrusted.*` — as adversarial claims.

Framing it as "one untrusted region, everything else trusted" (rather than
two parallel `trusted`/`untrusted` blocks) removes the mistake where a
producer forgets to nest a host field under `trusted`: a field is trusted
unless it is deliberately placed inside `untrusted`. The construction API
enforces this — producers pass trusted fields positionally and hand all
agent/remote-claimed data as the single `untrusted` mapping, so there is no
way to emit a top-level field that *looks* authoritative but isn't.

### Canonical serialization + hash chain

Serialization is defined once (extends the existing `sha256_hex` /
`util.py` helpers):

```
def canonical(event: dict) -> str:
    return json.dumps(event, sort_keys=True, separators=(",", ":"),
                      ensure_ascii=False)
```

The `hash` field is computed over the canonical form of the event **with
`hash` set to `""`**, prefixed by the previous record's hash:

```
digest = sha256_hex(prev_hash + canonical({**event, "hash": ""}))
```

`prev` is the prior record's `hash`; genesis uses `prev = ""`. This makes
the journal an append-only Merkle-style chain: editing or deleting record
*n* changes its hash, so record *n+1*'s `prev` no longer matches — the
break is local and points at the tampered record. Verification needs only
the journal itself (no keys), so it runs offline and in CI.

### Single writer; ordering across restarts

**Decided: one writer per host** (reviewed — the host controller owns it).
Producers hand events to the host controller, which is the sole appender,
so the chain has one well-defined total order and one `seq`/`epoch`
counter. This ties audit availability to the host controller being up,
which is acceptable because the host controller already gates every
lifecycle transition; per-producer chains are noted only as a future
scaling path, not built now.

**Restarts** are handled by the chain, not the clock. `ts_mono` resets to
~0 on every writer start, so it orders events only *within* one boot. On
start the writer:

1. reads the last line of the journal, adopts its `hash` as the next
   record's `prev` (the chain is continuous across the restart), and
2. bumps `epoch` (persisted alongside the chain head) and resets `seq` to
   0 for the new boot.

Total order is therefore `(epoch, seq)` — monotonic across restarts by
construction — with `ts_wall` for human reading and `ts_mono` for
sub-second ordering inside an epoch. A crash mid-append truncates at most
the last (partial) line; the verifier flags it and replay resumes from the
last intact record.

### Journal (source of truth) + SQLite index (rebuildable)

- **Journal:** one append-only JSONL file per host (path from `paths.py`,
  alongside `host_db_path()`), one canonical event per line, opened
  `O_APPEND`. This is authoritative.
- **Index:** a new `audit_events` table via the existing `DbStore` /
  `TableMigrations` machinery, holding the envelope columns plus JSON
  blobs, indexed on `(bottled_agent, type, ts_wall)`. It is a **derived
  cache**: `audit rebuild` truncates and replays the journal, re-verifying
  the chain as it goes. If the DB is deleted or drifts, it is regenerated
  from the journal with no data loss. (This supersedes the free-standing
  `supervise_audit_entries` table, which becomes a view/producer onto the
  new index.)

### Event registry (initial)

Dotted `type` names, grouped; the registry is a table mapping type →
required `untrusted` keys so producers and the verifier agree on shape:

- **lifecycle.*** — `lifecycle.bottled_agent_start`,
  `lifecycle.bottled_agent_stop`, `lifecycle.bottled_agent_crash`
  (producer: host-controller, #468). Leaf names use `bottled_agent` to match
  the top-level `bottled_agent` field — one term for the subject everywhere.
- **decision.*** — `decision.proposed`, `decision.resolved`
  (producer: supervise; carries operator action + justification, replacing
  PRD 0013's row shape).
- **egress.*** — `egress.decision` (allow/block at the proxy),
  `egress.route_added`.
- **auth.*** — `auth.token_minted`, `auth.token_rejected` (control-plane;
  **never** the token itself — see redaction).
- **forge.*** — `forge.push_accepted`, `forge.push_rejected` (git-gate),
  `forge.pr_opened`.

New types are additive; adding one does not bump `v`. Removing or
re-typing a field bumps `v`.

### Redaction rule

Redaction runs at the envelope boundary, before a record is written, in two
layers:

1. **Key deny-list (structural).** A field whose *key* matches a known
   credential shape (`token`, `secret`, `password`, `authorization`,
   `*_key`) is refused — the producer must pass a reference (a token *id*
   or `sha256` fingerprint), never the raw value. `auth.token_minted`
   therefore records the token id and role, not the JWT. This is the
   primary guard: it is cheap, deterministic, and catches the intended
   mistake (a producer stuffing a credential into a named field).

2. **Value scan — reuse the egress DLP detectors.** Per review, the value
   layer reuses the *same* deterministic credential-shape detectors the
   egress proxy already ships:
   `bot_bottle/gateway/egress/dlp_detectors.py` —
   `scan_token_patterns` / `redact_tokens` (and `scan_known_secrets` for
   host-known secret material). They are pure-Python, mitmproxy-free, and
   already the project's source of truth for "what a leaked credential
   looks like," so a single detector set governs both what may leave over
   the wire and what may land in the journal — they can't drift apart.

   **Scoped deliberately:** only the pattern/known-secret detectors are
   reused, **not** `scan_entropy`. Entropy scoring is tuned for large
   streamed request bodies; on the short, high-entropy structured values an
   audit event legitimately carries (hashes, uuids, base64 ids) it would
   false-positive and start redacting the very fingerprints the log needs.
   So the shared layer is the deterministic detectors; entropy stays an
   egress-only concern. (This is the "evaluate how reasonable that is" from
   review: reuse the deterministic detectors — yes; share the entropy
   heuristic — no.)

On a value-layer match the default is **redact** (scrub to a placeholder
and keep the event) rather than drop, so a producer bug can never make an
audit event vanish; the key deny-list stays a hard refusal because a
credential in a named field is always a producer bug worth surfacing.

### Export / interoperability (CloudEvents, OpenTelemetry Logs)

#487 requires the envelope to map onto the **OpenTelemetry Logs data
model** and/or a **CloudEvents JSON** envelope *without losing integrity or
attribution semantics*. The flattened shape (one `untrusted` region,
everything else trusted at top level) does **not** conflict with either — it
maps *more* cleanly than a nested `trusted`/`untrusted` pair would, because
both target models expect a flat set of top-level fields plus one payload
subtree.

**CloudEvents.** Context attributes MUST be scalar simple types — a map
cannot be a context attribute — so a nested `trusted` block would have had
to be flattened for CloudEvents anyway. Our flat top level maps directly:
`id`→`id`, `type`→`type`, `producer`+`host`→`source`,
`bottled_agent`→`subject`, `ts_wall`→`time`; the integrity/chain fields
(`epoch`, `seq`, `prev`, `hash`, `v`) ride as **extension attributes**
(scalars — legal). The `untrusted` map goes in `data`. Only mechanical
transform needed: extension attribute names must be lowercase-alphanumeric,
so `bottled_agent`/`ts_mono`/etc. are renamed at export (e.g. a
`botbottle`-prefixed form) — a naming rule, not a schema conflict.

**OpenTelemetry Logs.** `ts_wall`→`Timestamp`; `type`→the `event.name`
attribute; the flat trusted fields → `Attributes` under a `botbottle.*`
namespace (`botbottle.bottled_agent`, `botbottle.producer`,
`botbottle.chain.hash`, …); `untrusted.*` → `Attributes` under
`botbottle.untrusted.*` (or `Body`). OTel attributes are a dotted map that
happily carries the nested subtree.

**Attribution is preserved** precisely because the boundary is now
structural: on export, top-level fields become trusted context/attributes
and the `untrusted` subtree stays a single, clearly-named region — so a
downstream consumer still sees exactly which fields an agent claimed.
Nothing agent-claimed is promoted to a trusted-looking position.

**Integrity has one deliberate caveat.** CloudEvents/OTel are
representation envelopes with their own (or no) canonicalization; `hash`
and `prev` are computed over **our** canonical JSON, not over the exported
form. So the chain fields travel *as data* for reference, but
tamper-evidence is always verified against the **native journal** (the
source of truth) — never re-derived from an exported CloudEvents/OTel
record, whose key ordering / number formatting the exporter may change.
Export is thus a lossless-for-attribution **projection** that carries the
integrity fields along; verification stays on the canonical journal. This
satisfies "without losing integrity or attribution semantics": both are
carried, neither is *relied upon* in the foreign format.

The export adapters themselves (and #324's webhook delivery / causal
ordering) are follow-up implementation — this PRD fixes the *schema* so
that projection is a field re-map, never a reformat.

## Implementation chunks

1. **(this PR — PRD only.)** The contract above. No code; scheduled to land
   right after #468.
2. **Envelope + canonical + chain core.** `AuditEvent` dataclass,
   `canonical()`, chain hashing, and the single-writer journal appender in
   `bot_bottle/store/` (reusing `sha256_hex`); redaction wired to the
   existing `gateway/egress/dlp_detectors` (`scan_token_patterns` /
   `redact_tokens`); unit tests for determinism, chain-break detection,
   `epoch`/`seq` continuity across a simulated restart, and redaction of
   both a deny-listed key and a token-shaped value.
3. **SQLite index + `audit rebuild` / `audit verify` CLI.** New
   `audit_events` migration; replay-from-journal; offline chain verifier;
   local query commands (by bottled-agent / type / time / producer).
4. **Host controller as first producer (#468).** Wire
   `lifecycle.bottled_agent_*` emission into the host controller's
   start/stop/crash paths; establish the `epoch` bump + chain-head carry on
   writer restart here (the host controller owns the single writer).
5. **Migrate existing producers.** Re-emit supervise `decision.*` (retiring
   the standalone `supervise_audit_entries` shape behind the index), egress
   `egress.*`, git-gate `forge.*`, control-plane `auth.*`.
6. **CloudEvents / OTel export adapters.** A projection layer emitting each
   event as a CloudEvents JSON envelope and/or an OTel LogRecord (field
   re-map per *Export / interoperability*); feeds #324's webhook delivery.
7. **(follow-up.)** Cross-host merge transport (#324); per-writer signing +
   external anchoring on the chain head; retention/rotation policy.

## Resolved in review (#495)

- **Single writer per host — decided.** The host controller owns the sole
  appender; per-producer chains are a future scaling path only. (Design →
  *Single writer; ordering across restarts*.)
- **Restarts — decided.** An `epoch` counter (bumped per writer boot) plus
  carrying the last chain head as the next `prev` gives a total order of
  `(epoch, seq)` that survives restarts; `ts_mono` orders only within an
  epoch. (Design → *ordering across restarts*.)
- **Flatten to one `untrusted` region — decided.** Everything outside
  `untrusted` (chain metadata, `producer`/`host`, `bottled_agent`, `ts_*`)
  is trusted by construction, so the separate `trusted` sub-block is
  removed; a field is trusted unless deliberately placed under `untrusted`.
  (Design → *The envelope*.)
- **Subject term is `bottled_agent` everywhere** — the top-level field and
  the `lifecycle.bottled_agent_*` leaf names. (Design → *The envelope* /
  *Event registry*.)
- **Retention head-carry — yes.** When a journal segment is rotated out,
  the new segment's genesis `prev` is the rotated-out head, so the verifier
  still trusts the current head across a rotation. (Folds into the
  retention follow-up.)
- **Redaction reuses the egress detectors — yes, scoped.** Reuse the
  deterministic `dlp_detectors` (`scan_token_patterns` / `redact_tokens` /
  `scan_known_secrets`); exclude `scan_entropy` as brittle on the short,
  high-entropy structured values audit records carry. (Design → *Redaction
  rule*.)

## Open questions

- **Value-scan cost on the hot path.** The single writer runs the reused
  detectors on every event's `untrusted` block inline. Is that cheap enough
  at lifecycle-event volume, or should the value scan move to index-build
  time (journal stays raw, index stores the redacted view)? Leaning inline
  so the raw journal never contains a leaked value in the first place.
- **`epoch` persistence location.** Store the per-writer `epoch` + chain
  head in the SQLite index (rebuildable, but then the writer needs the DB
  at boot) or in a tiny sidecar file next to the journal (independent of
  the index)? Leaning sidecar, so the writer can start and append without
  the index present.