didericis/bot-bottle
1
0
Fork 0
Code Issues 33 Pull Requests 8 Actions 3 Packages Releases Wiki Activity

docs(prd): canonical tamper-evident audit-event schema (#487)
prd-number-check / require-numbered-prds (pull_request) Failing after 8s
Details
tracker-policy-pr / check-pr (pull_request) Successful in 6s
Details

Browse Source 
Draft PRD for a unified, versioned audit-event envelope with a
trusted/untrusted field split, canonical JSON + per-writer hash chain,
an append-only JSONL journal as source of truth, a rebuildable SQLite
index for local query, and an initial event registry. Scheduled to land
immediately after the host controller (#468), which becomes its first
producer.

Refs #487

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
didericis (claude)
2026-07-26 07:12:40 +00:00
parent 0fc5457e41
commit bc42836327
1 changed files with 263 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/prds/prd-new-audit-event-schema.md
+263
View File
@@ -0,0 +1,263 @@
# PRD prd-new: Canonical tamper-evident audit-event schema and local query contract
- **Status:** Draft
- **Author:** didericis-claude
- **Created:** 2026-07-26
- **Issue:** #487
## Summary
bot-bottle already emits security- and provenance-relevant events from
several producers — supervise operator decisions (PRD 0013's
`AuditStore`), egress allow/block enforcement, git-gate push decisions,
control-plane token minting, and (next) host-controller lifecycle
transitions — but each writes its own shape to its own sink. There is no
shared envelope, no tamper-evidence, and no single place to search. Local
incident reconstruction means grepping several stores that don't agree on
field names, timestamps, or how a bottle is identified.
This PRD defines **one canonical audit-event contract** every producer
emits into:
1. A **versioned envelope** — schema version, event id, event type,
monotonic + wall-clock timestamps, and an explicit **trust boundary**
between host-supplied and agent-claimed fields.
2. **Canonical JSON serialization + a per-writer hash chain**, so any
deletion or edit of a past record breaks the chain and is detectable
offline.
3. An **append-only JSONL journal as the source of truth**, with a
**rebuildable SQLite index** for local search — no paid platform, no
network dependency.
4. An **initial event registry** covering lifecycle, decision, egress,
auth, and forge events.
It is explicitly scheduled to land **immediately after the host
controller (#468)** so the host controller's lifecycle transitions are the
first producer wired onto the new contract (per the directive on #487).
## Problem
Audit infrastructure is fragmented across #468, #324, and #480 with no
shared schema. Concretely:
- **No shared envelope.** `supervise_audit_entries` (PRD 0013) has
`timestamp, bottle_slug, component, operator_action, ...`. The egress
proxy and git-gate log their own ad-hoc lines. There is no common
`event_id`, `event_type`, or version, so cross-producer correlation
("what did bottle X do between its start and this rejected push?") is
manual and lossy.
- **No tamper-evidence.** The audit store is a plain SQLite table. Anyone
who can write the DB can delete or rewrite a row and leave no trace.
Audit that an attacker (or a buggy agent) can silently rewrite is not
audit.
- **Trusted and untrusted data are mixed.** A bottle is attributed by
**source IP → slug** at the gateway (host-supplied, trustworthy). An
agent can also *claim* things about itself in a tool call
(agent-claimed, adversarial). Today nothing in the record marks which is
which, so a reader can be misled by an agent-supplied field that looks
authoritative.
- **No local search.** Reconstructing an incident means reading multiple
sinks with different schemas. There is no query contract and no promise
that the index can be rebuilt from the journal if it drifts or is lost.
- **No redaction rule.** Nothing prohibits a producer from writing a raw
token or secret into an audit record, which would turn the audit log
itself into a credential store.
## Goals / Success Criteria
- A single `AuditEvent` envelope type, versioned, that every producer
emits. Fields are split into a **`trusted`** block (host-supplied:
bottle slug from source-IP attribution, host wall-clock, producer
identity) and an **`untrusted`** block (anything the agent or a remote
claimed), and the split is structural, not a convention.
- **Canonical serialization** (`sort_keys`, `(",", ":")` separators,
UTF-8, `ensure_ascii=False`) is defined once and reused, so the same
logical event always hashes identically across producers and hosts.
- Each writer maintains a **hash chain**: `hash = sha256(prev_hash ||
canonical(event))`. Deleting or editing any past record breaks every
subsequent link; a standalone verifier detects the break offline with no
secret material.
- The **JSONL journal is the source of truth**; the **SQLite index is
fully rebuildable** from it (`audit rebuild` reconstructs the DB and
re-verifies the chain).
- **Local query** works with no paid platform and no egress: filter by
bottle, event type, time range, and producer, and follow a bottle's
events in order.
- A **redaction rule** is enforced at the envelope boundary: known
credential-shaped fields are rejected/redacted before a record is
written; the writer refuses raw secrets rather than storing them.
- The **host controller (#468)** emits `lifecycle.*` events through this
contract as the first consumer; existing supervise/egress producers are
migrated behind the same envelope without changing operator-facing
behavior.
## Non-goals
- **Cross-host aggregation / shipping.** This PRD makes each host's journal
canonical and correlatable *by construction* (stable ids, hash chain),
but the transport that merges multiple hosts into one timeline is a
follow-up (#324). The schema is designed so that merge is a later append,
not a reformat.
- **Cryptographic signing / external anchoring.** Hash-chaining gives
tamper-**evidence** (you can detect edits), not tamper-**resistance**
against an attacker who can rewrite the whole chain. Per-writer signing
keys and periodic external anchoring are a follow-up; the chain-head hash
is the seam they attach to.
- **Real-time alerting / SIEM rules.** Query is local and pull-based here.
- **Retention / rotation policy.** Journal rotation and TTL are operator
policy, tracked separately; the format must survive rotation (chain head
carried across segments) but this PRD does not set the schedule.
- **Replacing PRD 0013's operator queue.** The supervise proposal/response
queue is unchanged; only its terminal *audit* record is re-emitted onto
the new envelope.
## Design
### The envelope
One dataclass, `AuditEvent`, serialized to a JSON object with a small,
stable top level:
```
{
"v": 1, // schema version (integer, bumped only on breaking change)
"id": "<uuid4>", // globally unique event id
"type": "egress.decision", // dotted event type from the registry
"seq": 1287, // per-writer monotonic sequence (gap-detectable)
"ts": {
"wall": "2026-07-26T18:22:04.113Z", // host wall-clock, RFC3339 UTC (TRUSTED)
"mono": 90142.55 // host monotonic seconds since writer start (ordering)
},
"producer": "host-controller", // TRUSTED: which host component wrote this
"trusted": { // host-supplied, authoritative
"bottle": "amber-fox-12", // slug from source-IP attribution (may be null for host-level events)
"host": "mac-studio-1"
},
"untrusted": { // agent- or remote-claimed; never authoritative
"reason": "npm install needs registry.npmjs.org",
"target": "registry.npmjs.org:443"
},
"hash": "<hex>", // sha256(prev_hash || canonical(this event with hash="" ))
"prev": "<hex>" // hash of the previous record in this writer's chain ("" for genesis)
}
```
The **`trusted` / `untrusted` split is the core invariant.** A producer may
only place a field in `trusted` if the *host* established it (source-IP →
slug attribution, the host's own clock, the producer's own identity).
Everything an agent or a remote said goes in `untrusted`. A reader (or a
future policy engine) can therefore trust `trusted.bottle` for attribution
and treat `untrusted.*` as adversarial claims — the distinction the current
stores lack.
### Canonical serialization + hash chain
Serialization is defined once (extends the existing `sha256_hex` /
`util.py` helpers):
```
def canonical(event: dict) -> str:
return json.dumps(event, sort_keys=True, separators=(",", ":"),
ensure_ascii=False)
```
The `hash` field is computed over the canonical form of the event **with
`hash` set to `""`**, prefixed by the previous record's hash:
```
digest = sha256_hex(prev_hash + canonical({**event, "hash": ""}))
```
`prev` is the prior record's `hash`; genesis uses `prev = ""`. This makes
the journal an append-only Merkle-style chain: editing or deleting record
*n* changes its hash, so record *n+1*'s `prev` no longer matches — the
break is local and points at the tampered record. Verification needs only
the journal itself (no keys), so it runs offline and in CI.
### Journal (source of truth) + SQLite index (rebuildable)
- **Journal:** one append-only JSONL file per host (path from `paths.py`,
alongside `host_db_path()`), one canonical event per line, opened
`O_APPEND`. This is authoritative. A single writer per host owns appends
(producers hand events to it) so the chain has one well-defined order —
the host controller is the natural owner since it already gates the
lifecycle.
- **Index:** a new `audit_events` table via the existing `DbStore` /
`TableMigrations` machinery, holding the envelope columns plus JSON
blobs, indexed on `(bottle, type, wall_ts)`. It is a **derived cache**:
`audit rebuild` truncates and replays the journal, re-verifying the
chain as it goes. If the DB is deleted or drifts, it is regenerated from
the journal with no data loss. (This supersedes the free-standing
`supervise_audit_entries` table, which becomes a view/producer onto the
new index.)
### Event registry (initial)
Dotted `type` names, grouped; the registry is a table mapping type →
required `untrusted` keys so producers and the verifier agree on shape:
- **lifecycle.*** — `lifecycle.bottle_start`, `lifecycle.bottle_stop`,
`lifecycle.bottle_crash` (producer: host-controller, #468).
- **decision.*** — `decision.proposed`, `decision.resolved`
(producer: supervise; carries operator action + justification, replacing
PRD 0013's row shape).
- **egress.*** — `egress.decision` (allow/block at the proxy),
`egress.route_added`.
- **auth.*** — `auth.token_minted`, `auth.token_rejected` (control-plane;
**never** the token itself — see redaction).
- **forge.*** — `forge.push_accepted`, `forge.push_rejected` (git-gate),
`forge.pr_opened`.
New types are additive; adding one does not bump `v`. Removing or
re-typing a field bumps `v`.
### Redaction rule
The envelope constructor enforces a deny-list at write time: a field whose
key matches known credential shapes (`token`, `secret`, `password`,
`authorization`, `*_key`, JWT-shaped values) is rejected — the producer
must pass a reference (a token *id* or `sha256` fingerprint), never the raw
value. `auth.token_minted` therefore records the token id and role, not the
JWT. Redaction is enforced structurally so a producer *cannot* accidentally
write a secret into the audit log.
## Implementation chunks
1. **(this PR — PRD only.)** The contract above. No code; scheduled to land
right after #468.
2. **Envelope + canonical + chain core.** `AuditEvent` dataclass,
`canonical()`, chain hashing, and the single-writer journal appender in
`bot_bottle/store/` (reusing `sha256_hex`); unit tests for
determinism, chain-break detection, and redaction refusal.
3. **SQLite index + `audit rebuild` / `audit verify` CLI.** New
`audit_events` migration; replay-from-journal; offline chain verifier;
local query commands (by bottle / type / time / producer).
4. **Host controller as first producer (#468).** Wire `lifecycle.*`
emission into the host controller's start/stop/crash paths.
5. **Migrate existing producers.** Re-emit supervise `decision.*` (retiring
the standalone `supervise_audit_entries` shape behind the index), egress
`egress.*`, git-gate `forge.*`, control-plane `auth.*`.
6. **(follow-up.)** Cross-host merge transport (#324); per-writer signing +
external anchoring on the chain head; retention/rotation policy.
## Open questions
- **One writer per host vs. per producer chains.** A single appender gives
one total order but makes every producer depend on the host controller
being up. Alternative: one chain *per producer* (independent `seq`/chain
head), merged at query time by `(wall, mono)`. Leaning single-writer for
v1 (simpler verification, matches the host controller owning lifecycle),
with per-producer chains noted as the scaling path. — feedback wanted.
- **Monotonic clock across restarts.** `ts.mono` resets when the writer
restarts; is `seq` + `wall` enough for ordering across a restart, or does
the chain need an explicit `epoch` counter bumped per writer boot?
(Leaning: carry the last chain head across restart, so ordering follows
the chain, not `mono`.)
- **Redaction: reject vs. redact.** Should a credential-shaped field hard-
fail the write (surfacing the producer bug loudly) or silently redact to
`"<redacted>"`? Leaning **reject in tests / redact in prod** behind a
flag, so a producer bug can't drop an event entirely in the field.
- **Retention interaction with the chain.** When an old journal segment is
rotated out, the verifier must still trust the current head. Carry the
rotated-out head as the new segment's genesis `prev`? (Tracked with the
retention follow-up.)