diff --git a/docs/prds/prd-new-audit-event-schema.md b/docs/prds/prd-new-audit-event-schema.md new file mode 100644 index 00000000..7fa67c00 --- /dev/null +++ b/docs/prds/prd-new-audit-event-schema.md @@ -0,0 +1,263 @@ +# PRD prd-new: Canonical tamper-evident audit-event schema and local query contract + +- **Status:** Draft +- **Author:** didericis-claude +- **Created:** 2026-07-26 +- **Issue:** #487 + +## Summary + +bot-bottle already emits security- and provenance-relevant events from +several producers — supervise operator decisions (PRD 0013's +`AuditStore`), egress allow/block enforcement, git-gate push decisions, +control-plane token minting, and (next) host-controller lifecycle +transitions — but each writes its own shape to its own sink. There is no +shared envelope, no tamper-evidence, and no single place to search. Local +incident reconstruction means grepping several stores that don't agree on +field names, timestamps, or how a bottle is identified. + +This PRD defines **one canonical audit-event contract** every producer +emits into: + +1. A **versioned envelope** — schema version, event id, event type, + monotonic + wall-clock timestamps, and an explicit **trust boundary** + between host-supplied and agent-claimed fields. +2. **Canonical JSON serialization + a per-writer hash chain**, so any + deletion or edit of a past record breaks the chain and is detectable + offline. +3. An **append-only JSONL journal as the source of truth**, with a + **rebuildable SQLite index** for local search — no paid platform, no + network dependency. +4. An **initial event registry** covering lifecycle, decision, egress, + auth, and forge events. + +It is explicitly scheduled to land **immediately after the host +controller (#468)** so the host controller's lifecycle transitions are the +first producer wired onto the new contract (per the directive on #487). + +## Problem + +Audit infrastructure is fragmented across #468, #324, and #480 with no +shared schema. Concretely: + +- **No shared envelope.** `supervise_audit_entries` (PRD 0013) has + `timestamp, bottle_slug, component, operator_action, ...`. The egress + proxy and git-gate log their own ad-hoc lines. There is no common + `event_id`, `event_type`, or version, so cross-producer correlation + ("what did bottle X do between its start and this rejected push?") is + manual and lossy. +- **No tamper-evidence.** The audit store is a plain SQLite table. Anyone + who can write the DB can delete or rewrite a row and leave no trace. + Audit that an attacker (or a buggy agent) can silently rewrite is not + audit. +- **Trusted and untrusted data are mixed.** A bottle is attributed by + **source IP → slug** at the gateway (host-supplied, trustworthy). An + agent can also *claim* things about itself in a tool call + (agent-claimed, adversarial). Today nothing in the record marks which is + which, so a reader can be misled by an agent-supplied field that looks + authoritative. +- **No local search.** Reconstructing an incident means reading multiple + sinks with different schemas. There is no query contract and no promise + that the index can be rebuilt from the journal if it drifts or is lost. +- **No redaction rule.** Nothing prohibits a producer from writing a raw + token or secret into an audit record, which would turn the audit log + itself into a credential store. + +## Goals / Success Criteria + +- A single `AuditEvent` envelope type, versioned, that every producer + emits. Fields are split into a **`trusted`** block (host-supplied: + bottle slug from source-IP attribution, host wall-clock, producer + identity) and an **`untrusted`** block (anything the agent or a remote + claimed), and the split is structural, not a convention. +- **Canonical serialization** (`sort_keys`, `(",", ":")` separators, + UTF-8, `ensure_ascii=False`) is defined once and reused, so the same + logical event always hashes identically across producers and hosts. +- Each writer maintains a **hash chain**: `hash = sha256(prev_hash || + canonical(event))`. Deleting or editing any past record breaks every + subsequent link; a standalone verifier detects the break offline with no + secret material. +- The **JSONL journal is the source of truth**; the **SQLite index is + fully rebuildable** from it (`audit rebuild` reconstructs the DB and + re-verifies the chain). +- **Local query** works with no paid platform and no egress: filter by + bottle, event type, time range, and producer, and follow a bottle's + events in order. +- A **redaction rule** is enforced at the envelope boundary: known + credential-shaped fields are rejected/redacted before a record is + written; the writer refuses raw secrets rather than storing them. +- The **host controller (#468)** emits `lifecycle.*` events through this + contract as the first consumer; existing supervise/egress producers are + migrated behind the same envelope without changing operator-facing + behavior. + +## Non-goals + +- **Cross-host aggregation / shipping.** This PRD makes each host's journal + canonical and correlatable *by construction* (stable ids, hash chain), + but the transport that merges multiple hosts into one timeline is a + follow-up (#324). The schema is designed so that merge is a later append, + not a reformat. +- **Cryptographic signing / external anchoring.** Hash-chaining gives + tamper-**evidence** (you can detect edits), not tamper-**resistance** + against an attacker who can rewrite the whole chain. Per-writer signing + keys and periodic external anchoring are a follow-up; the chain-head hash + is the seam they attach to. +- **Real-time alerting / SIEM rules.** Query is local and pull-based here. +- **Retention / rotation policy.** Journal rotation and TTL are operator + policy, tracked separately; the format must survive rotation (chain head + carried across segments) but this PRD does not set the schedule. +- **Replacing PRD 0013's operator queue.** The supervise proposal/response + queue is unchanged; only its terminal *audit* record is re-emitted onto + the new envelope. + +## Design + +### The envelope + +One dataclass, `AuditEvent`, serialized to a JSON object with a small, +stable top level: + +``` +{ + "v": 1, // schema version (integer, bumped only on breaking change) + "id": "", // globally unique event id + "type": "egress.decision", // dotted event type from the registry + "seq": 1287, // per-writer monotonic sequence (gap-detectable) + "ts": { + "wall": "2026-07-26T18:22:04.113Z", // host wall-clock, RFC3339 UTC (TRUSTED) + "mono": 90142.55 // host monotonic seconds since writer start (ordering) + }, + "producer": "host-controller", // TRUSTED: which host component wrote this + "trusted": { // host-supplied, authoritative + "bottle": "amber-fox-12", // slug from source-IP attribution (may be null for host-level events) + "host": "mac-studio-1" + }, + "untrusted": { // agent- or remote-claimed; never authoritative + "reason": "npm install needs registry.npmjs.org", + "target": "registry.npmjs.org:443" + }, + "hash": "", // sha256(prev_hash || canonical(this event with hash="" )) + "prev": "" // hash of the previous record in this writer's chain ("" for genesis) +} +``` + +The **`trusted` / `untrusted` split is the core invariant.** A producer may +only place a field in `trusted` if the *host* established it (source-IP → +slug attribution, the host's own clock, the producer's own identity). +Everything an agent or a remote said goes in `untrusted`. A reader (or a +future policy engine) can therefore trust `trusted.bottle` for attribution +and treat `untrusted.*` as adversarial claims — the distinction the current +stores lack. + +### Canonical serialization + hash chain + +Serialization is defined once (extends the existing `sha256_hex` / +`util.py` helpers): + +``` +def canonical(event: dict) -> str: + return json.dumps(event, sort_keys=True, separators=(",", ":"), + ensure_ascii=False) +``` + +The `hash` field is computed over the canonical form of the event **with +`hash` set to `""`**, prefixed by the previous record's hash: + +``` +digest = sha256_hex(prev_hash + canonical({**event, "hash": ""})) +``` + +`prev` is the prior record's `hash`; genesis uses `prev = ""`. This makes +the journal an append-only Merkle-style chain: editing or deleting record +*n* changes its hash, so record *n+1*'s `prev` no longer matches — the +break is local and points at the tampered record. Verification needs only +the journal itself (no keys), so it runs offline and in CI. + +### Journal (source of truth) + SQLite index (rebuildable) + +- **Journal:** one append-only JSONL file per host (path from `paths.py`, + alongside `host_db_path()`), one canonical event per line, opened + `O_APPEND`. This is authoritative. A single writer per host owns appends + (producers hand events to it) so the chain has one well-defined order — + the host controller is the natural owner since it already gates the + lifecycle. +- **Index:** a new `audit_events` table via the existing `DbStore` / + `TableMigrations` machinery, holding the envelope columns plus JSON + blobs, indexed on `(bottle, type, wall_ts)`. It is a **derived cache**: + `audit rebuild` truncates and replays the journal, re-verifying the + chain as it goes. If the DB is deleted or drifts, it is regenerated from + the journal with no data loss. (This supersedes the free-standing + `supervise_audit_entries` table, which becomes a view/producer onto the + new index.) + +### Event registry (initial) + +Dotted `type` names, grouped; the registry is a table mapping type → +required `untrusted` keys so producers and the verifier agree on shape: + +- **lifecycle.*** — `lifecycle.bottle_start`, `lifecycle.bottle_stop`, + `lifecycle.bottle_crash` (producer: host-controller, #468). +- **decision.*** — `decision.proposed`, `decision.resolved` + (producer: supervise; carries operator action + justification, replacing + PRD 0013's row shape). +- **egress.*** — `egress.decision` (allow/block at the proxy), + `egress.route_added`. +- **auth.*** — `auth.token_minted`, `auth.token_rejected` (control-plane; + **never** the token itself — see redaction). +- **forge.*** — `forge.push_accepted`, `forge.push_rejected` (git-gate), + `forge.pr_opened`. + +New types are additive; adding one does not bump `v`. Removing or +re-typing a field bumps `v`. + +### Redaction rule + +The envelope constructor enforces a deny-list at write time: a field whose +key matches known credential shapes (`token`, `secret`, `password`, +`authorization`, `*_key`, JWT-shaped values) is rejected — the producer +must pass a reference (a token *id* or `sha256` fingerprint), never the raw +value. `auth.token_minted` therefore records the token id and role, not the +JWT. Redaction is enforced structurally so a producer *cannot* accidentally +write a secret into the audit log. + +## Implementation chunks + +1. **(this PR — PRD only.)** The contract above. No code; scheduled to land + right after #468. +2. **Envelope + canonical + chain core.** `AuditEvent` dataclass, + `canonical()`, chain hashing, and the single-writer journal appender in + `bot_bottle/store/` (reusing `sha256_hex`); unit tests for + determinism, chain-break detection, and redaction refusal. +3. **SQLite index + `audit rebuild` / `audit verify` CLI.** New + `audit_events` migration; replay-from-journal; offline chain verifier; + local query commands (by bottle / type / time / producer). +4. **Host controller as first producer (#468).** Wire `lifecycle.*` + emission into the host controller's start/stop/crash paths. +5. **Migrate existing producers.** Re-emit supervise `decision.*` (retiring + the standalone `supervise_audit_entries` shape behind the index), egress + `egress.*`, git-gate `forge.*`, control-plane `auth.*`. +6. **(follow-up.)** Cross-host merge transport (#324); per-writer signing + + external anchoring on the chain head; retention/rotation policy. + +## Open questions + +- **One writer per host vs. per producer chains.** A single appender gives + one total order but makes every producer depend on the host controller + being up. Alternative: one chain *per producer* (independent `seq`/chain + head), merged at query time by `(wall, mono)`. Leaning single-writer for + v1 (simpler verification, matches the host controller owning lifecycle), + with per-producer chains noted as the scaling path. — feedback wanted. +- **Monotonic clock across restarts.** `ts.mono` resets when the writer + restarts; is `seq` + `wall` enough for ordering across a restart, or does + the chain need an explicit `epoch` counter bumped per writer boot? + (Leaning: carry the last chain head across restart, so ordering follows + the chain, not `mono`.) +- **Redaction: reject vs. redact.** Should a credential-shaped field hard- + fail the write (surfacing the producer bug loudly) or silently redact to + `""`? Leaning **reject in tests / redact in prod** behind a + flag, so a producer bug can't drop an event entirely in the field. +- **Retention interaction with the chain.** When an old journal segment is + rotated out, the verifier must still trust the current head. Carry the + rotated-out head as the new segment's genesis `prev`? (Tracked with the + retention follow-up.)