bot-bottle/docs/prds/prd-new-audit-event-schema.md

PRD prd-new: Canonical tamper-evident audit-event schema and local query contract

• Status: Draft
• Author: didericis-claude
• Created: 2026-07-26
• Issue: #487

Summary

bot-bottle already emits security- and provenance-relevant events from several producers — supervise operator decisions (PRD 0013's AuditStore), egress allow/block enforcement, git-gate push decisions, control-plane token minting, and (next) host-controller lifecycle transitions — but each writes its own shape to its own sink. There is no shared envelope, no tamper-evidence, and no single place to search. Local incident reconstruction means grepping several stores that don't agree on field names, timestamps, or how a bottled agent is identified.

This PRD defines one canonical audit-event contract every producer emits into:

1. A versioned envelope — schema version, event id, event type, monotonic + wall-clock timestamps, and an explicit trust boundary between host-supplied and agent-claimed fields.
2. Canonical JSON serialization + a per-writer hash chain, so any deletion or edit of a past record breaks the chain and is detectable offline.
3. An append-only JSONL journal as the source of truth, with a rebuildable SQLite index for local search — no paid platform, no network dependency.
4. An initial event registry covering lifecycle, decision, egress, auth, and forge events.

It is explicitly scheduled to land immediately after the host controller (#468) so the host controller's lifecycle transitions are the first producer wired onto the new contract (per the directive on #487).

Problem

Audit infrastructure is fragmented across #468, #324, and #480 with no shared schema. Concretely:

• No shared envelope. supervise_audit_entries (PRD 0013) has timestamp, bottle_slug, component, operator_action, .... The egress proxy and git-gate log their own ad-hoc lines. There is no common event_id, event_type, or version, so cross-producer correlation ("what did bottled agent X do between its start and this rejected push?") is manual and lossy.
• No tamper-evidence. The audit store is a plain SQLite table. Anyone who can write the DB can delete or rewrite a row and leave no trace. Audit that an attacker (or a buggy agent) can silently rewrite is not audit.
• Trusted and untrusted data are mixed. A bottled agent is attributed by source IP → slug at the gateway (host-supplied, trustworthy). An agent can also claim things about itself in a tool call (agent-claimed, adversarial). Today nothing in the record marks which is which, so a reader can be misled by an agent-supplied field that looks authoritative.
• No local search. Reconstructing an incident means reading multiple sinks with different schemas. There is no query contract and no promise that the index can be rebuilt from the journal if it drifts or is lost.
• No redaction rule. Nothing prohibits a producer from writing a raw token or secret into an audit record, which would turn the audit log itself into a credential store.

Goals / Success Criteria

• A single AuditEvent envelope type, versioned, that every producer emits. Fields are split into a trusted block (host-supplied: bottled-agent slug from source-IP attribution, host wall-clock, producer identity) and an untrusted block (anything the agent or a remote claimed), and the split is structural, not a convention.
• Canonical serialization (sort_keys, (",", ":") separators, UTF-8, ensure_ascii=False) is defined once and reused, so the same logical event always hashes identically across producers and hosts.
• Each writer maintains a hash chain: hash = sha256(prev_hash || canonical(event)). Deleting or editing any past record breaks every subsequent link; a standalone verifier detects the break offline with no secret material.
• The JSONL journal is the source of truth; the SQLite index is fully rebuildable from it (audit rebuild reconstructs the DB and re-verifies the chain).
• Local query works with no paid platform and no egress: filter by bottled agent, event type, time range, and producer, and follow a bottled agent's events in order.
• A redaction rule is enforced at the envelope boundary: known credential-shaped fields are rejected/redacted before a record is written; the writer refuses raw secrets rather than storing them.
• The host controller (#468) emits lifecycle.* events through this contract as the first consumer; existing supervise/egress producers are migrated behind the same envelope without changing operator-facing behavior.

Non-goals

• Cross-host aggregation / shipping. This PRD makes each host's journal canonical and correlatable by construction (stable ids, hash chain), but the transport that merges multiple hosts into one timeline is a follow-up (#324). The schema is designed so that merge is a later append, not a reformat.
• Cryptographic signing / external anchoring. Hash-chaining gives tamper-evidence (you can detect edits), not tamper-resistance against an attacker who can rewrite the whole chain. Per-writer signing keys and periodic external anchoring are a follow-up; the chain-head hash is the seam they attach to.
• Real-time alerting / SIEM rules. Query is local and pull-based here.
• Retention / rotation policy. Journal rotation and TTL are operator policy, tracked separately; the format must survive rotation (chain head carried across segments) but this PRD does not set the schedule.
• Replacing PRD 0013's operator queue. The supervise proposal/response queue is unchanged; only its terminal audit record is re-emitted onto the new envelope.

Design

The envelope

One dataclass, AuditEvent, serialized to a JSON object with a small, stable top level:

{
  "v": 1,                          // schema version — bumped only on a breaking change
  "id": "<uuid4>",                 // globally unique event id
  "type": "egress.decision",       // dotted event type from the registry

  // --- chain / ordering (structural, host-owned) ---
  "epoch": 7,                      // writer-boot counter, bumped once per host-controller (writer) start
  "seq": 1287,                     // monotonic sequence within this epoch (gap-detectable)
  "prev": "<hex>",                 // hash of the previous record in the chain ("" for genesis)
  "hash": "<hex>",                 // sha256(prev + canonical(this event with hash=""))

  // --- trusted: everything the *host* established; authoritative ---
  "trusted": {
    "producer": "host-controller",         // which host component wrote this
    "host": "mac-studio-1",
    "bottled_agent": "amber-fox-12",       // slug from source-IP attribution (null for host-level events)
    "ts_wall": "2026-07-26T18:22:04.113Z", // host wall-clock, RFC3339 UTC
    "ts_mono": 90142.55                    // host monotonic secs since this epoch's boot (intra-epoch ordering only)
  },

  // --- untrusted: anything the agent or a remote claimed; never authoritative ---
  "untrusted": {
    "reason": "npm install needs registry.npmjs.org",
    "target": "registry.npmjs.org:443"
  }
}

The trusted / untrusted split is the core invariant. A producer may only place a field in trusted if the host established it: the source-IP → bottled_agent slug attribution, the host's own clock (ts_wall/ts_mono), and the producer's own identity. producer and ts_* live inside trusted on purpose — they are host-supplied, so grouping them there (rather than as loose top-level fields) keeps the "authoritative ⇔ inside trusted" rule structural, with nothing host-established leaking outside it. Everything an agent or a remote said goes in untrusted. A reader (or a future policy engine) can therefore trust trusted.bottled_agent for attribution and treat untrusted.* as adversarial claims — the distinction the current stores lack.

Only the small structural set — v, id, type, epoch, seq, prev, hash — sits at the top level; it is host-owned too, but it is chain metadata rather than event data, so it stays out of the trusted body to keep that body purely about what happened.

Canonical serialization + hash chain

Serialization is defined once (extends the existing sha256_hex / util.py helpers):

def canonical(event: dict) -> str:
    return json.dumps(event, sort_keys=True, separators=(",", ":"),
                      ensure_ascii=False)

The hash field is computed over the canonical form of the event with hash set to "", prefixed by the previous record's hash:

digest = sha256_hex(prev_hash + canonical({**event, "hash": ""}))

prev is the prior record's hash; genesis uses prev = "". This makes the journal an append-only Merkle-style chain: editing or deleting record n changes its hash, so record n+1's prev no longer matches — the break is local and points at the tampered record. Verification needs only the journal itself (no keys), so it runs offline and in CI.

Single writer; ordering across restarts

Decided: one writer per host (reviewed — the host controller owns it). Producers hand events to the host controller, which is the sole appender, so the chain has one well-defined total order and one seq/epoch counter. This ties audit availability to the host controller being up, which is acceptable because the host controller already gates every lifecycle transition; per-producer chains are noted only as a future scaling path, not built now.

Restarts are handled by the chain, not the clock. ts_mono resets to ~0 on every writer start, so it orders events only within one boot. On start the writer:

1. reads the last line of the journal, adopts its hash as the next record's prev (the chain is continuous across the restart), and
2. bumps epoch (persisted alongside the chain head) and resets seq to 0 for the new boot.

Total order is therefore (epoch, seq) — monotonic across restarts by construction — with ts_wall for human reading and ts_mono for sub-second ordering inside an epoch. A crash mid-append truncates at most the last (partial) line; the verifier flags it and replay resumes from the last intact record.

Journal (source of truth) + SQLite index (rebuildable)

• Journal: one append-only JSONL file per host (path from paths.py, alongside host_db_path()), one canonical event per line, opened O_APPEND. This is authoritative.
• Index: a new audit_events table via the existing DbStore / TableMigrations machinery, holding the envelope columns plus JSON blobs, indexed on (bottled_agent, type, ts_wall). It is a derived cache: audit rebuild truncates and replays the journal, re-verifying the chain as it goes. If the DB is deleted or drifts, it is regenerated from the journal with no data loss. (This supersedes the free-standing supervise_audit_entries table, which becomes a view/producer onto the new index.)

Event registry (initial)

Dotted type names, grouped; the registry is a table mapping type → required untrusted keys so producers and the verifier agree on shape:

• lifecycle.* — lifecycle.bottled_agent_start, lifecycle.bottled_agent_stop, lifecycle.bottled_agent_crash (producer: host-controller, #468). Leaf names use bottled_agent to match the trusted.bottled_agent field — one term for the subject everywhere.
• decision.* — decision.proposed, decision.resolved (producer: supervise; carries operator action + justification, replacing PRD 0013's row shape).
• egress.* — egress.decision (allow/block at the proxy), egress.route_added.
• auth.* — auth.token_minted, auth.token_rejected (control-plane; never the token itself — see redaction).
• forge.* — forge.push_accepted, forge.push_rejected (git-gate), forge.pr_opened.

New types are additive; adding one does not bump v. Removing or re-typing a field bumps v.

Redaction rule

Redaction runs at the envelope boundary, before a record is written, in two layers:

1. Key deny-list (structural). A field whose key matches a known credential shape (token, secret, password, authorization, *_key) is refused — the producer must pass a reference (a token id or sha256 fingerprint), never the raw value. auth.token_minted therefore records the token id and role, not the JWT. This is the primary guard: it is cheap, deterministic, and catches the intended mistake (a producer stuffing a credential into a named field).

2. Value scan — reuse the egress DLP detectors. Per review, the value layer reuses the same deterministic credential-shape detectors the egress proxy already ships: bot_bottle/gateway/egress/dlp_detectors.py — scan_token_patterns / redact_tokens (and scan_known_secrets for host-known secret material). They are pure-Python, mitmproxy-free, and already the project's source of truth for "what a leaked credential looks like," so a single detector set governs both what may leave over the wire and what may land in the journal — they can't drift apart.

   Scoped deliberately: only the pattern/known-secret detectors are reused, not scan_entropy. Entropy scoring is tuned for large streamed request bodies; on the short, high-entropy structured values an audit event legitimately carries (hashes, uuids, base64 ids) it would false-positive and start redacting the very fingerprints the log needs. So the shared layer is the deterministic detectors; entropy stays an egress-only concern. (This is the "evaluate how reasonable that is" from review: reuse the deterministic detectors — yes; share the entropy heuristic — no.)

On a value-layer match the default is redact (scrub to a placeholder and keep the event) rather than drop, so a producer bug can never make an audit event vanish; the key deny-list stays a hard refusal because a credential in a named field is always a producer bug worth surfacing.

Implementation chunks

1. (this PR — PRD only.) The contract above. No code; scheduled to land right after #468.
2. Envelope + canonical + chain core. AuditEvent dataclass, canonical(), chain hashing, and the single-writer journal appender in bot_bottle/store/ (reusing sha256_hex); redaction wired to the existing gateway/egress/dlp_detectors (scan_token_patterns / redact_tokens); unit tests for determinism, chain-break detection, epoch/seq continuity across a simulated restart, and redaction of both a deny-listed key and a token-shaped value.
3. SQLite index + audit rebuild / audit verify CLI. New audit_events migration; replay-from-journal; offline chain verifier; local query commands (by bottled-agent / type / time / producer).
4. Host controller as first producer (#468). Wire lifecycle.bottled_agent_* emission into the host controller's start/stop/crash paths; establish the epoch bump + chain-head carry on writer restart here (the host controller owns the single writer).
5. Migrate existing producers. Re-emit supervise decision.* (retiring the standalone supervise_audit_entries shape behind the index), egress egress.*, git-gate forge.*, control-plane auth.*.
6. (follow-up.) Cross-host merge transport (#324); per-writer signing + external anchoring on the chain head; retention/rotation policy.

Resolved in review (#495)

• Single writer per host — decided. The host controller owns the sole appender; per-producer chains are a future scaling path only. (Design → Single writer; ordering across restarts.)
• Restarts — decided. An epoch counter (bumped per writer boot) plus carrying the last chain head as the next prev gives a total order of (epoch, seq) that survives restarts; ts_mono orders only within an epoch. (Design → ordering across restarts.)
• ts_* and producer belong in trusted. They are host-supplied, so they now sit inside the trusted block; only chain metadata stays at the top level. (Design → The envelope.)
• Subject term is bottled_agent everywhere — the trusted field and the lifecycle.bottled_agent_* leaf names. (Design → The envelope / Event registry.)
• Retention head-carry — yes. When a journal segment is rotated out, the new segment's genesis prev is the rotated-out head, so the verifier still trusts the current head across a rotation. (Folds into the retention follow-up.)
• Redaction reuses the egress detectors — yes, scoped. Reuse the deterministic dlp_detectors (scan_token_patterns / redact_tokens / scan_known_secrets); exclude scan_entropy as brittle on the short, high-entropy structured values audit records carry. (Design → Redaction rule.)

Open questions

• Value-scan cost on the hot path. The single writer runs the reused detectors on every event's untrusted block inline. Is that cheap enough at lifecycle-event volume, or should the value scan move to index-build time (journal stays raw, index stores the redacted view)? Leaning inline so the raw journal never contains a leaked value in the first place.
• epoch persistence location. Store the per-writer epoch + chain head in the SQLite index (rebuildable, but then the writer needs the DB at boot) or in a tiny sidecar file next to the journal (independent of the index)? Leaning sidecar, so the writer can start and append without the index present.