diff --git a/docs/prds/prd-new-audit-event-schema.md b/docs/prds/prd-new-audit-event-schema.md index 7fa67c00..d458ac88 100644 --- a/docs/prds/prd-new-audit-event-schema.md +++ b/docs/prds/prd-new-audit-event-schema.md @@ -14,7 +14,7 @@ control-plane token minting, and (next) host-controller lifecycle transitions — but each writes its own shape to its own sink. There is no shared envelope, no tamper-evidence, and no single place to search. Local incident reconstruction means grepping several stores that don't agree on -field names, timestamps, or how a bottle is identified. +field names, timestamps, or how a bottled agent is identified. This PRD defines **one canonical audit-event contract** every producer emits into: @@ -44,13 +44,13 @@ shared schema. Concretely: `timestamp, bottle_slug, component, operator_action, ...`. The egress proxy and git-gate log their own ad-hoc lines. There is no common `event_id`, `event_type`, or version, so cross-producer correlation - ("what did bottle X do between its start and this rejected push?") is + ("what did bottled agent X do between its start and this rejected push?") is manual and lossy. - **No tamper-evidence.** The audit store is a plain SQLite table. Anyone who can write the DB can delete or rewrite a row and leave no trace. Audit that an attacker (or a buggy agent) can silently rewrite is not audit. -- **Trusted and untrusted data are mixed.** A bottle is attributed by +- **Trusted and untrusted data are mixed.** A bottled agent is attributed by **source IP → slug** at the gateway (host-supplied, trustworthy). An agent can also *claim* things about itself in a tool call (agent-claimed, adversarial). Today nothing in the record marks which is @@ -67,7 +67,7 @@ shared schema. Concretely: - A single `AuditEvent` envelope type, versioned, that every producer emits. Fields are split into a **`trusted`** block (host-supplied: - bottle slug from source-IP attribution, host wall-clock, producer + bottled-agent slug from source-IP attribution, host wall-clock, producer identity) and an **`untrusted`** block (anything the agent or a remote claimed), and the split is structural, not a convention. - **Canonical serialization** (`sort_keys`, `(",", ":")` separators, @@ -81,8 +81,8 @@ shared schema. Concretely: fully rebuildable** from it (`audit rebuild` reconstructs the DB and re-verifies the chain). - **Local query** works with no paid platform and no egress: filter by - bottle, event type, time range, and producer, and follow a bottle's - events in order. + bottled agent, event type, time range, and producer, and follow a bottled + agent's events in order. - A **redaction rule** is enforced at the envelope boundary: known credential-shaped fields are rejected/redacted before a record is written; the writer refuses raw secrets rather than storing them. @@ -120,35 +120,49 @@ stable top level: ``` { - "v": 1, // schema version (integer, bumped only on breaking change) + "v": 1, // schema version — bumped only on a breaking change "id": "", // globally unique event id "type": "egress.decision", // dotted event type from the registry - "seq": 1287, // per-writer monotonic sequence (gap-detectable) - "ts": { - "wall": "2026-07-26T18:22:04.113Z", // host wall-clock, RFC3339 UTC (TRUSTED) - "mono": 90142.55 // host monotonic seconds since writer start (ordering) + + // --- chain / ordering (structural, host-owned) --- + "epoch": 7, // writer-boot counter, bumped once per host-controller (writer) start + "seq": 1287, // monotonic sequence within this epoch (gap-detectable) + "prev": "", // hash of the previous record in the chain ("" for genesis) + "hash": "", // sha256(prev + canonical(this event with hash="")) + + // --- trusted: everything the *host* established; authoritative --- + "trusted": { + "producer": "host-controller", // which host component wrote this + "host": "mac-studio-1", + "bottled_agent": "amber-fox-12", // slug from source-IP attribution (null for host-level events) + "ts_wall": "2026-07-26T18:22:04.113Z", // host wall-clock, RFC3339 UTC + "ts_mono": 90142.55 // host monotonic secs since this epoch's boot (intra-epoch ordering only) }, - "producer": "host-controller", // TRUSTED: which host component wrote this - "trusted": { // host-supplied, authoritative - "bottle": "amber-fox-12", // slug from source-IP attribution (may be null for host-level events) - "host": "mac-studio-1" - }, - "untrusted": { // agent- or remote-claimed; never authoritative + + // --- untrusted: anything the agent or a remote claimed; never authoritative --- + "untrusted": { "reason": "npm install needs registry.npmjs.org", "target": "registry.npmjs.org:443" - }, - "hash": "", // sha256(prev_hash || canonical(this event with hash="" )) - "prev": "" // hash of the previous record in this writer's chain ("" for genesis) + } } ``` The **`trusted` / `untrusted` split is the core invariant.** A producer may -only place a field in `trusted` if the *host* established it (source-IP → -slug attribution, the host's own clock, the producer's own identity). -Everything an agent or a remote said goes in `untrusted`. A reader (or a -future policy engine) can therefore trust `trusted.bottle` for attribution -and treat `untrusted.*` as adversarial claims — the distinction the current -stores lack. +only place a field in `trusted` if the *host* established it: the +source-IP → `bottled_agent` slug attribution, the host's own clock +(`ts_wall`/`ts_mono`), and the producer's own identity. **`producer` and +`ts_*` live inside `trusted` on purpose** — they are host-supplied, so +grouping them there (rather than as loose top-level fields) keeps the +"authoritative ⇔ inside `trusted`" rule structural, with nothing +host-established leaking outside it. Everything an agent or a remote said +goes in `untrusted`. A reader (or a future policy engine) can therefore +trust `trusted.bottled_agent` for attribution and treat `untrusted.*` as +adversarial claims — the distinction the current stores lack. + +Only the small structural set — `v`, `id`, `type`, `epoch`, `seq`, `prev`, +`hash` — sits at the top level; it is host-owned too, but it is chain +metadata rather than event data, so it stays out of the `trusted` body to +keep that body purely about *what happened*. ### Canonical serialization + hash chain @@ -174,20 +188,42 @@ the journal an append-only Merkle-style chain: editing or deleting record break is local and points at the tampered record. Verification needs only the journal itself (no keys), so it runs offline and in CI. +### Single writer; ordering across restarts + +**Decided: one writer per host** (reviewed — the host controller owns it). +Producers hand events to the host controller, which is the sole appender, +so the chain has one well-defined total order and one `seq`/`epoch` +counter. This ties audit availability to the host controller being up, +which is acceptable because the host controller already gates every +lifecycle transition; per-producer chains are noted only as a future +scaling path, not built now. + +**Restarts** are handled by the chain, not the clock. `ts_mono` resets to +~0 on every writer start, so it orders events only *within* one boot. On +start the writer: + +1. reads the last line of the journal, adopts its `hash` as the next + record's `prev` (the chain is continuous across the restart), and +2. bumps `epoch` (persisted alongside the chain head) and resets `seq` to + 0 for the new boot. + +Total order is therefore `(epoch, seq)` — monotonic across restarts by +construction — with `ts_wall` for human reading and `ts_mono` for +sub-second ordering inside an epoch. A crash mid-append truncates at most +the last (partial) line; the verifier flags it and replay resumes from the +last intact record. + ### Journal (source of truth) + SQLite index (rebuildable) - **Journal:** one append-only JSONL file per host (path from `paths.py`, alongside `host_db_path()`), one canonical event per line, opened - `O_APPEND`. This is authoritative. A single writer per host owns appends - (producers hand events to it) so the chain has one well-defined order — - the host controller is the natural owner since it already gates the - lifecycle. + `O_APPEND`. This is authoritative. - **Index:** a new `audit_events` table via the existing `DbStore` / `TableMigrations` machinery, holding the envelope columns plus JSON - blobs, indexed on `(bottle, type, wall_ts)`. It is a **derived cache**: - `audit rebuild` truncates and replays the journal, re-verifying the - chain as it goes. If the DB is deleted or drifts, it is regenerated from - the journal with no data loss. (This supersedes the free-standing + blobs, indexed on `(bottled_agent, type, ts_wall)`. It is a **derived + cache**: `audit rebuild` truncates and replays the journal, re-verifying + the chain as it goes. If the DB is deleted or drifts, it is regenerated + from the journal with no data loss. (This supersedes the free-standing `supervise_audit_entries` table, which becomes a view/producer onto the new index.) @@ -196,8 +232,10 @@ the journal itself (no keys), so it runs offline and in CI. Dotted `type` names, grouped; the registry is a table mapping type → required `untrusted` keys so producers and the verifier agree on shape: -- **lifecycle.*** — `lifecycle.bottle_start`, `lifecycle.bottle_stop`, - `lifecycle.bottle_crash` (producer: host-controller, #468). +- **lifecycle.*** — `lifecycle.bottled_agent_start`, + `lifecycle.bottled_agent_stop`, `lifecycle.bottled_agent_crash` + (producer: host-controller, #468). Leaf names use `bottled_agent` to match + the `trusted.bottled_agent` field — one term for the subject everywhere. - **decision.*** — `decision.proposed`, `decision.resolved` (producer: supervise; carries operator action + justification, replacing PRD 0013's row shape). @@ -213,13 +251,41 @@ re-typing a field bumps `v`. ### Redaction rule -The envelope constructor enforces a deny-list at write time: a field whose -key matches known credential shapes (`token`, `secret`, `password`, -`authorization`, `*_key`, JWT-shaped values) is rejected — the producer -must pass a reference (a token *id* or `sha256` fingerprint), never the raw -value. `auth.token_minted` therefore records the token id and role, not the -JWT. Redaction is enforced structurally so a producer *cannot* accidentally -write a secret into the audit log. +Redaction runs at the envelope boundary, before a record is written, in two +layers: + +1. **Key deny-list (structural).** A field whose *key* matches a known + credential shape (`token`, `secret`, `password`, `authorization`, + `*_key`) is refused — the producer must pass a reference (a token *id* + or `sha256` fingerprint), never the raw value. `auth.token_minted` + therefore records the token id and role, not the JWT. This is the + primary guard: it is cheap, deterministic, and catches the intended + mistake (a producer stuffing a credential into a named field). + +2. **Value scan — reuse the egress DLP detectors.** Per review, the value + layer reuses the *same* deterministic credential-shape detectors the + egress proxy already ships: + `bot_bottle/gateway/egress/dlp_detectors.py` — + `scan_token_patterns` / `redact_tokens` (and `scan_known_secrets` for + host-known secret material). They are pure-Python, mitmproxy-free, and + already the project's source of truth for "what a leaked credential + looks like," so a single detector set governs both what may leave over + the wire and what may land in the journal — they can't drift apart. + + **Scoped deliberately:** only the pattern/known-secret detectors are + reused, **not** `scan_entropy`. Entropy scoring is tuned for large + streamed request bodies; on the short, high-entropy structured values an + audit event legitimately carries (hashes, uuids, base64 ids) it would + false-positive and start redacting the very fingerprints the log needs. + So the shared layer is the deterministic detectors; entropy stays an + egress-only concern. (This is the "evaluate how reasonable that is" from + review: reuse the deterministic detectors — yes; share the entropy + heuristic — no.) + +On a value-layer match the default is **redact** (scrub to a placeholder +and keep the event) rather than drop, so a producer bug can never make an +audit event vanish; the key deny-list stays a hard refusal because a +credential in a named field is always a producer bug worth surfacing. ## Implementation chunks @@ -227,37 +293,58 @@ write a secret into the audit log. right after #468. 2. **Envelope + canonical + chain core.** `AuditEvent` dataclass, `canonical()`, chain hashing, and the single-writer journal appender in - `bot_bottle/store/` (reusing `sha256_hex`); unit tests for - determinism, chain-break detection, and redaction refusal. + `bot_bottle/store/` (reusing `sha256_hex`); redaction wired to the + existing `gateway/egress/dlp_detectors` (`scan_token_patterns` / + `redact_tokens`); unit tests for determinism, chain-break detection, + `epoch`/`seq` continuity across a simulated restart, and redaction of + both a deny-listed key and a token-shaped value. 3. **SQLite index + `audit rebuild` / `audit verify` CLI.** New `audit_events` migration; replay-from-journal; offline chain verifier; - local query commands (by bottle / type / time / producer). -4. **Host controller as first producer (#468).** Wire `lifecycle.*` - emission into the host controller's start/stop/crash paths. + local query commands (by bottled-agent / type / time / producer). +4. **Host controller as first producer (#468).** Wire + `lifecycle.bottled_agent_*` emission into the host controller's + start/stop/crash paths; establish the `epoch` bump + chain-head carry on + writer restart here (the host controller owns the single writer). 5. **Migrate existing producers.** Re-emit supervise `decision.*` (retiring the standalone `supervise_audit_entries` shape behind the index), egress `egress.*`, git-gate `forge.*`, control-plane `auth.*`. 6. **(follow-up.)** Cross-host merge transport (#324); per-writer signing + external anchoring on the chain head; retention/rotation policy. +## Resolved in review (#495) + +- **Single writer per host — decided.** The host controller owns the sole + appender; per-producer chains are a future scaling path only. (Design → + *Single writer; ordering across restarts*.) +- **Restarts — decided.** An `epoch` counter (bumped per writer boot) plus + carrying the last chain head as the next `prev` gives a total order of + `(epoch, seq)` that survives restarts; `ts_mono` orders only within an + epoch. (Design → *ordering across restarts*.) +- **`ts_*` and `producer` belong in `trusted`.** They are host-supplied, so + they now sit inside the `trusted` block; only chain metadata stays at the + top level. (Design → *The envelope*.) +- **Subject term is `bottled_agent` everywhere** — the `trusted` field and + the `lifecycle.bottled_agent_*` leaf names. (Design → *The envelope* / + *Event registry*.) +- **Retention head-carry — yes.** When a journal segment is rotated out, + the new segment's genesis `prev` is the rotated-out head, so the verifier + still trusts the current head across a rotation. (Folds into the + retention follow-up.) +- **Redaction reuses the egress detectors — yes, scoped.** Reuse the + deterministic `dlp_detectors` (`scan_token_patterns` / `redact_tokens` / + `scan_known_secrets`); exclude `scan_entropy` as brittle on the short, + high-entropy structured values audit records carry. (Design → *Redaction + rule*.) + ## Open questions -- **One writer per host vs. per producer chains.** A single appender gives - one total order but makes every producer depend on the host controller - being up. Alternative: one chain *per producer* (independent `seq`/chain - head), merged at query time by `(wall, mono)`. Leaning single-writer for - v1 (simpler verification, matches the host controller owning lifecycle), - with per-producer chains noted as the scaling path. — feedback wanted. -- **Monotonic clock across restarts.** `ts.mono` resets when the writer - restarts; is `seq` + `wall` enough for ordering across a restart, or does - the chain need an explicit `epoch` counter bumped per writer boot? - (Leaning: carry the last chain head across restart, so ordering follows - the chain, not `mono`.) -- **Redaction: reject vs. redact.** Should a credential-shaped field hard- - fail the write (surfacing the producer bug loudly) or silently redact to - `""`? Leaning **reject in tests / redact in prod** behind a - flag, so a producer bug can't drop an event entirely in the field. -- **Retention interaction with the chain.** When an old journal segment is - rotated out, the verifier must still trust the current head. Carry the - rotated-out head as the new segment's genesis `prev`? (Tracked with the - retention follow-up.) +- **Value-scan cost on the hot path.** The single writer runs the reused + detectors on every event's `untrusted` block inline. Is that cheap enough + at lifecycle-event volume, or should the value scan move to index-build + time (journal stays raw, index stores the redacted view)? Leaning inline + so the raw journal never contains a leaked value in the first place. +- **`epoch` persistence location.** Store the per-writer `epoch` + chain + head in the SQLite index (rebuildable, but then the writer needs the DB + at boot) or in a tiny sidecar file next to the journal (independent of + the index)? Leaning sidecar, so the writer can start and append without + the index present.