From bc428363277bee1157a3b824c050fc1bf3343cf0 Mon Sep 17 00:00:00 2001 From: claude Date: Sun, 26 Jul 2026 07:12:40 +0000 Subject: [PATCH 1/6] docs(prd): canonical tamper-evident audit-event schema (#487) Draft PRD for a unified, versioned audit-event envelope with a trusted/untrusted field split, canonical JSON + per-writer hash chain, an append-only JSONL journal as source of truth, a rebuildable SQLite index for local query, and an initial event registry. Scheduled to land immediately after the host controller (#468), which becomes its first producer. Refs #487 Co-Authored-By: Claude Opus 4.8 --- docs/prds/prd-new-audit-event-schema.md | 263 ++++++++++++++++++++++++ 1 file changed, 263 insertions(+) create mode 100644 docs/prds/prd-new-audit-event-schema.md diff --git a/docs/prds/prd-new-audit-event-schema.md b/docs/prds/prd-new-audit-event-schema.md new file mode 100644 index 00000000..7fa67c00 --- /dev/null +++ b/docs/prds/prd-new-audit-event-schema.md @@ -0,0 +1,263 @@ +# PRD prd-new: Canonical tamper-evident audit-event schema and local query contract + +- **Status:** Draft +- **Author:** didericis-claude +- **Created:** 2026-07-26 +- **Issue:** #487 + +## Summary + +bot-bottle already emits security- and provenance-relevant events from +several producers — supervise operator decisions (PRD 0013's +`AuditStore`), egress allow/block enforcement, git-gate push decisions, +control-plane token minting, and (next) host-controller lifecycle +transitions — but each writes its own shape to its own sink. There is no +shared envelope, no tamper-evidence, and no single place to search. Local +incident reconstruction means grepping several stores that don't agree on +field names, timestamps, or how a bottle is identified. + +This PRD defines **one canonical audit-event contract** every producer +emits into: + +1. A **versioned envelope** — schema version, event id, event type, + monotonic + wall-clock timestamps, and an explicit **trust boundary** + between host-supplied and agent-claimed fields. +2. **Canonical JSON serialization + a per-writer hash chain**, so any + deletion or edit of a past record breaks the chain and is detectable + offline. +3. An **append-only JSONL journal as the source of truth**, with a + **rebuildable SQLite index** for local search — no paid platform, no + network dependency. +4. An **initial event registry** covering lifecycle, decision, egress, + auth, and forge events. + +It is explicitly scheduled to land **immediately after the host +controller (#468)** so the host controller's lifecycle transitions are the +first producer wired onto the new contract (per the directive on #487). + +## Problem + +Audit infrastructure is fragmented across #468, #324, and #480 with no +shared schema. Concretely: + +- **No shared envelope.** `supervise_audit_entries` (PRD 0013) has + `timestamp, bottle_slug, component, operator_action, ...`. The egress + proxy and git-gate log their own ad-hoc lines. There is no common + `event_id`, `event_type`, or version, so cross-producer correlation + ("what did bottle X do between its start and this rejected push?") is + manual and lossy. +- **No tamper-evidence.** The audit store is a plain SQLite table. Anyone + who can write the DB can delete or rewrite a row and leave no trace. + Audit that an attacker (or a buggy agent) can silently rewrite is not + audit. +- **Trusted and untrusted data are mixed.** A bottle is attributed by + **source IP → slug** at the gateway (host-supplied, trustworthy). An + agent can also *claim* things about itself in a tool call + (agent-claimed, adversarial). Today nothing in the record marks which is + which, so a reader can be misled by an agent-supplied field that looks + authoritative. +- **No local search.** Reconstructing an incident means reading multiple + sinks with different schemas. There is no query contract and no promise + that the index can be rebuilt from the journal if it drifts or is lost. +- **No redaction rule.** Nothing prohibits a producer from writing a raw + token or secret into an audit record, which would turn the audit log + itself into a credential store. + +## Goals / Success Criteria + +- A single `AuditEvent` envelope type, versioned, that every producer + emits. Fields are split into a **`trusted`** block (host-supplied: + bottle slug from source-IP attribution, host wall-clock, producer + identity) and an **`untrusted`** block (anything the agent or a remote + claimed), and the split is structural, not a convention. +- **Canonical serialization** (`sort_keys`, `(",", ":")` separators, + UTF-8, `ensure_ascii=False`) is defined once and reused, so the same + logical event always hashes identically across producers and hosts. +- Each writer maintains a **hash chain**: `hash = sha256(prev_hash || + canonical(event))`. Deleting or editing any past record breaks every + subsequent link; a standalone verifier detects the break offline with no + secret material. +- The **JSONL journal is the source of truth**; the **SQLite index is + fully rebuildable** from it (`audit rebuild` reconstructs the DB and + re-verifies the chain). +- **Local query** works with no paid platform and no egress: filter by + bottle, event type, time range, and producer, and follow a bottle's + events in order. +- A **redaction rule** is enforced at the envelope boundary: known + credential-shaped fields are rejected/redacted before a record is + written; the writer refuses raw secrets rather than storing them. +- The **host controller (#468)** emits `lifecycle.*` events through this + contract as the first consumer; existing supervise/egress producers are + migrated behind the same envelope without changing operator-facing + behavior. + +## Non-goals + +- **Cross-host aggregation / shipping.** This PRD makes each host's journal + canonical and correlatable *by construction* (stable ids, hash chain), + but the transport that merges multiple hosts into one timeline is a + follow-up (#324). The schema is designed so that merge is a later append, + not a reformat. +- **Cryptographic signing / external anchoring.** Hash-chaining gives + tamper-**evidence** (you can detect edits), not tamper-**resistance** + against an attacker who can rewrite the whole chain. Per-writer signing + keys and periodic external anchoring are a follow-up; the chain-head hash + is the seam they attach to. +- **Real-time alerting / SIEM rules.** Query is local and pull-based here. +- **Retention / rotation policy.** Journal rotation and TTL are operator + policy, tracked separately; the format must survive rotation (chain head + carried across segments) but this PRD does not set the schedule. +- **Replacing PRD 0013's operator queue.** The supervise proposal/response + queue is unchanged; only its terminal *audit* record is re-emitted onto + the new envelope. + +## Design + +### The envelope + +One dataclass, `AuditEvent`, serialized to a JSON object with a small, +stable top level: + +``` +{ + "v": 1, // schema version (integer, bumped only on breaking change) + "id": "", // globally unique event id + "type": "egress.decision", // dotted event type from the registry + "seq": 1287, // per-writer monotonic sequence (gap-detectable) + "ts": { + "wall": "2026-07-26T18:22:04.113Z", // host wall-clock, RFC3339 UTC (TRUSTED) + "mono": 90142.55 // host monotonic seconds since writer start (ordering) + }, + "producer": "host-controller", // TRUSTED: which host component wrote this + "trusted": { // host-supplied, authoritative + "bottle": "amber-fox-12", // slug from source-IP attribution (may be null for host-level events) + "host": "mac-studio-1" + }, + "untrusted": { // agent- or remote-claimed; never authoritative + "reason": "npm install needs registry.npmjs.org", + "target": "registry.npmjs.org:443" + }, + "hash": "", // sha256(prev_hash || canonical(this event with hash="" )) + "prev": "" // hash of the previous record in this writer's chain ("" for genesis) +} +``` + +The **`trusted` / `untrusted` split is the core invariant.** A producer may +only place a field in `trusted` if the *host* established it (source-IP → +slug attribution, the host's own clock, the producer's own identity). +Everything an agent or a remote said goes in `untrusted`. A reader (or a +future policy engine) can therefore trust `trusted.bottle` for attribution +and treat `untrusted.*` as adversarial claims — the distinction the current +stores lack. + +### Canonical serialization + hash chain + +Serialization is defined once (extends the existing `sha256_hex` / +`util.py` helpers): + +``` +def canonical(event: dict) -> str: + return json.dumps(event, sort_keys=True, separators=(",", ":"), + ensure_ascii=False) +``` + +The `hash` field is computed over the canonical form of the event **with +`hash` set to `""`**, prefixed by the previous record's hash: + +``` +digest = sha256_hex(prev_hash + canonical({**event, "hash": ""})) +``` + +`prev` is the prior record's `hash`; genesis uses `prev = ""`. This makes +the journal an append-only Merkle-style chain: editing or deleting record +*n* changes its hash, so record *n+1*'s `prev` no longer matches — the +break is local and points at the tampered record. Verification needs only +the journal itself (no keys), so it runs offline and in CI. + +### Journal (source of truth) + SQLite index (rebuildable) + +- **Journal:** one append-only JSONL file per host (path from `paths.py`, + alongside `host_db_path()`), one canonical event per line, opened + `O_APPEND`. This is authoritative. A single writer per host owns appends + (producers hand events to it) so the chain has one well-defined order — + the host controller is the natural owner since it already gates the + lifecycle. +- **Index:** a new `audit_events` table via the existing `DbStore` / + `TableMigrations` machinery, holding the envelope columns plus JSON + blobs, indexed on `(bottle, type, wall_ts)`. It is a **derived cache**: + `audit rebuild` truncates and replays the journal, re-verifying the + chain as it goes. If the DB is deleted or drifts, it is regenerated from + the journal with no data loss. (This supersedes the free-standing + `supervise_audit_entries` table, which becomes a view/producer onto the + new index.) + +### Event registry (initial) + +Dotted `type` names, grouped; the registry is a table mapping type → +required `untrusted` keys so producers and the verifier agree on shape: + +- **lifecycle.*** — `lifecycle.bottle_start`, `lifecycle.bottle_stop`, + `lifecycle.bottle_crash` (producer: host-controller, #468). +- **decision.*** — `decision.proposed`, `decision.resolved` + (producer: supervise; carries operator action + justification, replacing + PRD 0013's row shape). +- **egress.*** — `egress.decision` (allow/block at the proxy), + `egress.route_added`. +- **auth.*** — `auth.token_minted`, `auth.token_rejected` (control-plane; + **never** the token itself — see redaction). +- **forge.*** — `forge.push_accepted`, `forge.push_rejected` (git-gate), + `forge.pr_opened`. + +New types are additive; adding one does not bump `v`. Removing or +re-typing a field bumps `v`. + +### Redaction rule + +The envelope constructor enforces a deny-list at write time: a field whose +key matches known credential shapes (`token`, `secret`, `password`, +`authorization`, `*_key`, JWT-shaped values) is rejected — the producer +must pass a reference (a token *id* or `sha256` fingerprint), never the raw +value. `auth.token_minted` therefore records the token id and role, not the +JWT. Redaction is enforced structurally so a producer *cannot* accidentally +write a secret into the audit log. + +## Implementation chunks + +1. **(this PR — PRD only.)** The contract above. No code; scheduled to land + right after #468. +2. **Envelope + canonical + chain core.** `AuditEvent` dataclass, + `canonical()`, chain hashing, and the single-writer journal appender in + `bot_bottle/store/` (reusing `sha256_hex`); unit tests for + determinism, chain-break detection, and redaction refusal. +3. **SQLite index + `audit rebuild` / `audit verify` CLI.** New + `audit_events` migration; replay-from-journal; offline chain verifier; + local query commands (by bottle / type / time / producer). +4. **Host controller as first producer (#468).** Wire `lifecycle.*` + emission into the host controller's start/stop/crash paths. +5. **Migrate existing producers.** Re-emit supervise `decision.*` (retiring + the standalone `supervise_audit_entries` shape behind the index), egress + `egress.*`, git-gate `forge.*`, control-plane `auth.*`. +6. **(follow-up.)** Cross-host merge transport (#324); per-writer signing + + external anchoring on the chain head; retention/rotation policy. + +## Open questions + +- **One writer per host vs. per producer chains.** A single appender gives + one total order but makes every producer depend on the host controller + being up. Alternative: one chain *per producer* (independent `seq`/chain + head), merged at query time by `(wall, mono)`. Leaning single-writer for + v1 (simpler verification, matches the host controller owning lifecycle), + with per-producer chains noted as the scaling path. — feedback wanted. +- **Monotonic clock across restarts.** `ts.mono` resets when the writer + restarts; is `seq` + `wall` enough for ordering across a restart, or does + the chain need an explicit `epoch` counter bumped per writer boot? + (Leaning: carry the last chain head across restart, so ordering follows + the chain, not `mono`.) +- **Redaction: reject vs. redact.** Should a credential-shaped field hard- + fail the write (surfacing the producer bug loudly) or silently redact to + `""`? Leaning **reject in tests / redact in prod** behind a + flag, so a producer bug can't drop an event entirely in the field. +- **Retention interaction with the chain.** When an old journal segment is + rotated out, the verifier must still trust the current head. Carry the + rotated-out head as the new segment's genesis `prev`? (Tracked with the + retention follow-up.) -- 2.52.0 From 60039f2eb33ac4c2b47fc236a69ecc2fcf7b052f Mon Sep 17 00:00:00 2001 From: claude Date: Sun, 26 Jul 2026 07:35:16 +0000 Subject: [PATCH 2/6] docs(prd): address review #495 on audit-event schema - Move ts_wall/ts_mono and producer inside the trusted block (host-supplied). - Rename subject to bottled_agent everywhere (field + lifecycle.bottled_agent_* leaves). - Add explicit epoch (writer-boot) counter + chain-head carry for ordering across host-controller restarts. - Commit to a single writer per host (host controller owns it). - Reuse egress dlp_detectors (scan_token_patterns/redact_tokens) for redaction; exclude scan_entropy as brittle on structured audit values. - Retention: carry rotated-out chain head as new segment genesis prev. - Fold resolved points into design; trim open questions. Refs #487 Co-Authored-By: Claude Opus 4.8 --- docs/prds/prd-new-audit-event-schema.md | 221 +++++++++++++++++------- 1 file changed, 154 insertions(+), 67 deletions(-) diff --git a/docs/prds/prd-new-audit-event-schema.md b/docs/prds/prd-new-audit-event-schema.md index 7fa67c00..d458ac88 100644 --- a/docs/prds/prd-new-audit-event-schema.md +++ b/docs/prds/prd-new-audit-event-schema.md @@ -14,7 +14,7 @@ control-plane token minting, and (next) host-controller lifecycle transitions — but each writes its own shape to its own sink. There is no shared envelope, no tamper-evidence, and no single place to search. Local incident reconstruction means grepping several stores that don't agree on -field names, timestamps, or how a bottle is identified. +field names, timestamps, or how a bottled agent is identified. This PRD defines **one canonical audit-event contract** every producer emits into: @@ -44,13 +44,13 @@ shared schema. Concretely: `timestamp, bottle_slug, component, operator_action, ...`. The egress proxy and git-gate log their own ad-hoc lines. There is no common `event_id`, `event_type`, or version, so cross-producer correlation - ("what did bottle X do between its start and this rejected push?") is + ("what did bottled agent X do between its start and this rejected push?") is manual and lossy. - **No tamper-evidence.** The audit store is a plain SQLite table. Anyone who can write the DB can delete or rewrite a row and leave no trace. Audit that an attacker (or a buggy agent) can silently rewrite is not audit. -- **Trusted and untrusted data are mixed.** A bottle is attributed by +- **Trusted and untrusted data are mixed.** A bottled agent is attributed by **source IP → slug** at the gateway (host-supplied, trustworthy). An agent can also *claim* things about itself in a tool call (agent-claimed, adversarial). Today nothing in the record marks which is @@ -67,7 +67,7 @@ shared schema. Concretely: - A single `AuditEvent` envelope type, versioned, that every producer emits. Fields are split into a **`trusted`** block (host-supplied: - bottle slug from source-IP attribution, host wall-clock, producer + bottled-agent slug from source-IP attribution, host wall-clock, producer identity) and an **`untrusted`** block (anything the agent or a remote claimed), and the split is structural, not a convention. - **Canonical serialization** (`sort_keys`, `(",", ":")` separators, @@ -81,8 +81,8 @@ shared schema. Concretely: fully rebuildable** from it (`audit rebuild` reconstructs the DB and re-verifies the chain). - **Local query** works with no paid platform and no egress: filter by - bottle, event type, time range, and producer, and follow a bottle's - events in order. + bottled agent, event type, time range, and producer, and follow a bottled + agent's events in order. - A **redaction rule** is enforced at the envelope boundary: known credential-shaped fields are rejected/redacted before a record is written; the writer refuses raw secrets rather than storing them. @@ -120,35 +120,49 @@ stable top level: ``` { - "v": 1, // schema version (integer, bumped only on breaking change) + "v": 1, // schema version — bumped only on a breaking change "id": "", // globally unique event id "type": "egress.decision", // dotted event type from the registry - "seq": 1287, // per-writer monotonic sequence (gap-detectable) - "ts": { - "wall": "2026-07-26T18:22:04.113Z", // host wall-clock, RFC3339 UTC (TRUSTED) - "mono": 90142.55 // host monotonic seconds since writer start (ordering) + + // --- chain / ordering (structural, host-owned) --- + "epoch": 7, // writer-boot counter, bumped once per host-controller (writer) start + "seq": 1287, // monotonic sequence within this epoch (gap-detectable) + "prev": "", // hash of the previous record in the chain ("" for genesis) + "hash": "", // sha256(prev + canonical(this event with hash="")) + + // --- trusted: everything the *host* established; authoritative --- + "trusted": { + "producer": "host-controller", // which host component wrote this + "host": "mac-studio-1", + "bottled_agent": "amber-fox-12", // slug from source-IP attribution (null for host-level events) + "ts_wall": "2026-07-26T18:22:04.113Z", // host wall-clock, RFC3339 UTC + "ts_mono": 90142.55 // host monotonic secs since this epoch's boot (intra-epoch ordering only) }, - "producer": "host-controller", // TRUSTED: which host component wrote this - "trusted": { // host-supplied, authoritative - "bottle": "amber-fox-12", // slug from source-IP attribution (may be null for host-level events) - "host": "mac-studio-1" - }, - "untrusted": { // agent- or remote-claimed; never authoritative + + // --- untrusted: anything the agent or a remote claimed; never authoritative --- + "untrusted": { "reason": "npm install needs registry.npmjs.org", "target": "registry.npmjs.org:443" - }, - "hash": "", // sha256(prev_hash || canonical(this event with hash="" )) - "prev": "" // hash of the previous record in this writer's chain ("" for genesis) + } } ``` The **`trusted` / `untrusted` split is the core invariant.** A producer may -only place a field in `trusted` if the *host* established it (source-IP → -slug attribution, the host's own clock, the producer's own identity). -Everything an agent or a remote said goes in `untrusted`. A reader (or a -future policy engine) can therefore trust `trusted.bottle` for attribution -and treat `untrusted.*` as adversarial claims — the distinction the current -stores lack. +only place a field in `trusted` if the *host* established it: the +source-IP → `bottled_agent` slug attribution, the host's own clock +(`ts_wall`/`ts_mono`), and the producer's own identity. **`producer` and +`ts_*` live inside `trusted` on purpose** — they are host-supplied, so +grouping them there (rather than as loose top-level fields) keeps the +"authoritative ⇔ inside `trusted`" rule structural, with nothing +host-established leaking outside it. Everything an agent or a remote said +goes in `untrusted`. A reader (or a future policy engine) can therefore +trust `trusted.bottled_agent` for attribution and treat `untrusted.*` as +adversarial claims — the distinction the current stores lack. + +Only the small structural set — `v`, `id`, `type`, `epoch`, `seq`, `prev`, +`hash` — sits at the top level; it is host-owned too, but it is chain +metadata rather than event data, so it stays out of the `trusted` body to +keep that body purely about *what happened*. ### Canonical serialization + hash chain @@ -174,20 +188,42 @@ the journal an append-only Merkle-style chain: editing or deleting record break is local and points at the tampered record. Verification needs only the journal itself (no keys), so it runs offline and in CI. +### Single writer; ordering across restarts + +**Decided: one writer per host** (reviewed — the host controller owns it). +Producers hand events to the host controller, which is the sole appender, +so the chain has one well-defined total order and one `seq`/`epoch` +counter. This ties audit availability to the host controller being up, +which is acceptable because the host controller already gates every +lifecycle transition; per-producer chains are noted only as a future +scaling path, not built now. + +**Restarts** are handled by the chain, not the clock. `ts_mono` resets to +~0 on every writer start, so it orders events only *within* one boot. On +start the writer: + +1. reads the last line of the journal, adopts its `hash` as the next + record's `prev` (the chain is continuous across the restart), and +2. bumps `epoch` (persisted alongside the chain head) and resets `seq` to + 0 for the new boot. + +Total order is therefore `(epoch, seq)` — monotonic across restarts by +construction — with `ts_wall` for human reading and `ts_mono` for +sub-second ordering inside an epoch. A crash mid-append truncates at most +the last (partial) line; the verifier flags it and replay resumes from the +last intact record. + ### Journal (source of truth) + SQLite index (rebuildable) - **Journal:** one append-only JSONL file per host (path from `paths.py`, alongside `host_db_path()`), one canonical event per line, opened - `O_APPEND`. This is authoritative. A single writer per host owns appends - (producers hand events to it) so the chain has one well-defined order — - the host controller is the natural owner since it already gates the - lifecycle. + `O_APPEND`. This is authoritative. - **Index:** a new `audit_events` table via the existing `DbStore` / `TableMigrations` machinery, holding the envelope columns plus JSON - blobs, indexed on `(bottle, type, wall_ts)`. It is a **derived cache**: - `audit rebuild` truncates and replays the journal, re-verifying the - chain as it goes. If the DB is deleted or drifts, it is regenerated from - the journal with no data loss. (This supersedes the free-standing + blobs, indexed on `(bottled_agent, type, ts_wall)`. It is a **derived + cache**: `audit rebuild` truncates and replays the journal, re-verifying + the chain as it goes. If the DB is deleted or drifts, it is regenerated + from the journal with no data loss. (This supersedes the free-standing `supervise_audit_entries` table, which becomes a view/producer onto the new index.) @@ -196,8 +232,10 @@ the journal itself (no keys), so it runs offline and in CI. Dotted `type` names, grouped; the registry is a table mapping type → required `untrusted` keys so producers and the verifier agree on shape: -- **lifecycle.*** — `lifecycle.bottle_start`, `lifecycle.bottle_stop`, - `lifecycle.bottle_crash` (producer: host-controller, #468). +- **lifecycle.*** — `lifecycle.bottled_agent_start`, + `lifecycle.bottled_agent_stop`, `lifecycle.bottled_agent_crash` + (producer: host-controller, #468). Leaf names use `bottled_agent` to match + the `trusted.bottled_agent` field — one term for the subject everywhere. - **decision.*** — `decision.proposed`, `decision.resolved` (producer: supervise; carries operator action + justification, replacing PRD 0013's row shape). @@ -213,13 +251,41 @@ re-typing a field bumps `v`. ### Redaction rule -The envelope constructor enforces a deny-list at write time: a field whose -key matches known credential shapes (`token`, `secret`, `password`, -`authorization`, `*_key`, JWT-shaped values) is rejected — the producer -must pass a reference (a token *id* or `sha256` fingerprint), never the raw -value. `auth.token_minted` therefore records the token id and role, not the -JWT. Redaction is enforced structurally so a producer *cannot* accidentally -write a secret into the audit log. +Redaction runs at the envelope boundary, before a record is written, in two +layers: + +1. **Key deny-list (structural).** A field whose *key* matches a known + credential shape (`token`, `secret`, `password`, `authorization`, + `*_key`) is refused — the producer must pass a reference (a token *id* + or `sha256` fingerprint), never the raw value. `auth.token_minted` + therefore records the token id and role, not the JWT. This is the + primary guard: it is cheap, deterministic, and catches the intended + mistake (a producer stuffing a credential into a named field). + +2. **Value scan — reuse the egress DLP detectors.** Per review, the value + layer reuses the *same* deterministic credential-shape detectors the + egress proxy already ships: + `bot_bottle/gateway/egress/dlp_detectors.py` — + `scan_token_patterns` / `redact_tokens` (and `scan_known_secrets` for + host-known secret material). They are pure-Python, mitmproxy-free, and + already the project's source of truth for "what a leaked credential + looks like," so a single detector set governs both what may leave over + the wire and what may land in the journal — they can't drift apart. + + **Scoped deliberately:** only the pattern/known-secret detectors are + reused, **not** `scan_entropy`. Entropy scoring is tuned for large + streamed request bodies; on the short, high-entropy structured values an + audit event legitimately carries (hashes, uuids, base64 ids) it would + false-positive and start redacting the very fingerprints the log needs. + So the shared layer is the deterministic detectors; entropy stays an + egress-only concern. (This is the "evaluate how reasonable that is" from + review: reuse the deterministic detectors — yes; share the entropy + heuristic — no.) + +On a value-layer match the default is **redact** (scrub to a placeholder +and keep the event) rather than drop, so a producer bug can never make an +audit event vanish; the key deny-list stays a hard refusal because a +credential in a named field is always a producer bug worth surfacing. ## Implementation chunks @@ -227,37 +293,58 @@ write a secret into the audit log. right after #468. 2. **Envelope + canonical + chain core.** `AuditEvent` dataclass, `canonical()`, chain hashing, and the single-writer journal appender in - `bot_bottle/store/` (reusing `sha256_hex`); unit tests for - determinism, chain-break detection, and redaction refusal. + `bot_bottle/store/` (reusing `sha256_hex`); redaction wired to the + existing `gateway/egress/dlp_detectors` (`scan_token_patterns` / + `redact_tokens`); unit tests for determinism, chain-break detection, + `epoch`/`seq` continuity across a simulated restart, and redaction of + both a deny-listed key and a token-shaped value. 3. **SQLite index + `audit rebuild` / `audit verify` CLI.** New `audit_events` migration; replay-from-journal; offline chain verifier; - local query commands (by bottle / type / time / producer). -4. **Host controller as first producer (#468).** Wire `lifecycle.*` - emission into the host controller's start/stop/crash paths. + local query commands (by bottled-agent / type / time / producer). +4. **Host controller as first producer (#468).** Wire + `lifecycle.bottled_agent_*` emission into the host controller's + start/stop/crash paths; establish the `epoch` bump + chain-head carry on + writer restart here (the host controller owns the single writer). 5. **Migrate existing producers.** Re-emit supervise `decision.*` (retiring the standalone `supervise_audit_entries` shape behind the index), egress `egress.*`, git-gate `forge.*`, control-plane `auth.*`. 6. **(follow-up.)** Cross-host merge transport (#324); per-writer signing + external anchoring on the chain head; retention/rotation policy. +## Resolved in review (#495) + +- **Single writer per host — decided.** The host controller owns the sole + appender; per-producer chains are a future scaling path only. (Design → + *Single writer; ordering across restarts*.) +- **Restarts — decided.** An `epoch` counter (bumped per writer boot) plus + carrying the last chain head as the next `prev` gives a total order of + `(epoch, seq)` that survives restarts; `ts_mono` orders only within an + epoch. (Design → *ordering across restarts*.) +- **`ts_*` and `producer` belong in `trusted`.** They are host-supplied, so + they now sit inside the `trusted` block; only chain metadata stays at the + top level. (Design → *The envelope*.) +- **Subject term is `bottled_agent` everywhere** — the `trusted` field and + the `lifecycle.bottled_agent_*` leaf names. (Design → *The envelope* / + *Event registry*.) +- **Retention head-carry — yes.** When a journal segment is rotated out, + the new segment's genesis `prev` is the rotated-out head, so the verifier + still trusts the current head across a rotation. (Folds into the + retention follow-up.) +- **Redaction reuses the egress detectors — yes, scoped.** Reuse the + deterministic `dlp_detectors` (`scan_token_patterns` / `redact_tokens` / + `scan_known_secrets`); exclude `scan_entropy` as brittle on the short, + high-entropy structured values audit records carry. (Design → *Redaction + rule*.) + ## Open questions -- **One writer per host vs. per producer chains.** A single appender gives - one total order but makes every producer depend on the host controller - being up. Alternative: one chain *per producer* (independent `seq`/chain - head), merged at query time by `(wall, mono)`. Leaning single-writer for - v1 (simpler verification, matches the host controller owning lifecycle), - with per-producer chains noted as the scaling path. — feedback wanted. -- **Monotonic clock across restarts.** `ts.mono` resets when the writer - restarts; is `seq` + `wall` enough for ordering across a restart, or does - the chain need an explicit `epoch` counter bumped per writer boot? - (Leaning: carry the last chain head across restart, so ordering follows - the chain, not `mono`.) -- **Redaction: reject vs. redact.** Should a credential-shaped field hard- - fail the write (surfacing the producer bug loudly) or silently redact to - `""`? Leaning **reject in tests / redact in prod** behind a - flag, so a producer bug can't drop an event entirely in the field. -- **Retention interaction with the chain.** When an old journal segment is - rotated out, the verifier must still trust the current head. Carry the - rotated-out head as the new segment's genesis `prev`? (Tracked with the - retention follow-up.) +- **Value-scan cost on the hot path.** The single writer runs the reused + detectors on every event's `untrusted` block inline. Is that cheap enough + at lifecycle-event volume, or should the value scan move to index-build + time (journal stays raw, index stores the redacted view)? Leaning inline + so the raw journal never contains a leaked value in the first place. +- **`epoch` persistence location.** Store the per-writer `epoch` + chain + head in the SQLite index (rebuildable, but then the writer needs the DB + at boot) or in a tiny sidecar file next to the journal (independent of + the index)? Leaning sidecar, so the writer can start and append without + the index present. -- 2.52.0 From 5a9428cc86a1d5d6a8d80af4ae739df327fb95a3 Mon Sep 17 00:00:00 2001 From: claude Date: Sun, 26 Jul 2026 07:45:58 +0000 Subject: [PATCH 3/6] docs(prd): flatten to one untrusted region + address CloudEvents/OTel export - Remove the trusted sub-block: everything outside untrusted (chain metadata, producer/host, bottled_agent, ts_*) is trusted by construction. A field is trusted unless deliberately placed under untrusted (#495). - Add Export/interoperability section: the flattened envelope projects cleanly onto CloudEvents JSON (top-level scalars -> context/extension attributes, untrusted -> data) and the OpenTelemetry Logs data model (ts_wall -> Timestamp, trusted -> botbottle.* attributes, untrusted -> botbottle.untrusted.*). Attribution preserved structurally; integrity fields carried as data with verification always on the native journal. Satisfies #487's export/interop requirement. Export adapters = chunk 6. Refs #487 Co-Authored-By: Claude Opus 4.8 --- docs/prds/prd-new-audit-event-schema.md | 130 +++++++++++++++++------- 1 file changed, 93 insertions(+), 37 deletions(-) diff --git a/docs/prds/prd-new-audit-event-schema.md b/docs/prds/prd-new-audit-event-schema.md index d458ac88..920d5652 100644 --- a/docs/prds/prd-new-audit-event-schema.md +++ b/docs/prds/prd-new-audit-event-schema.md @@ -66,10 +66,11 @@ shared schema. Concretely: ## Goals / Success Criteria - A single `AuditEvent` envelope type, versioned, that every producer - emits. Fields are split into a **`trusted`** block (host-supplied: - bottled-agent slug from source-IP attribution, host wall-clock, producer - identity) and an **`untrusted`** block (anything the agent or a remote - claimed), and the split is structural, not a convention. + emits. The trust boundary is **one `untrusted` region**: everything + outside it is host-established and trusted (source-IP → bottled-agent + attribution, host wall-clock, producer identity, chain metadata); + `untrusted` is the sole place anything an agent or a remote claimed may + go. The boundary is structural, not a convention. - **Canonical serialization** (`sort_keys`, `(",", ":")` separators, UTF-8, `ensure_ascii=False`) is defined once and reused, so the same logical event always hashes identically across producers and hosts. @@ -86,6 +87,11 @@ shared schema. Concretely: - A **redaction rule** is enforced at the envelope boundary: known credential-shaped fields are rejected/redacted before a record is written; the writer refuses raw secrets rather than storing them. +- The envelope **projects onto the OpenTelemetry Logs data model and a + CloudEvents JSON envelope** by field re-mapping alone (no reformat), + preserving the trust boundary and carrying the integrity fields — per + #487's export/interop requirement. (The export adapters are follow-up; + the *schema* must make them a re-map.) - The **host controller (#468)** emits `lifecycle.*` events through this contract as the first consumer; existing supervise/egress producers are migrated behind the same envelope without changing operator-facing @@ -120,26 +126,21 @@ stable top level: ``` { + // --- everything at the top level is host-established (trusted) --- "v": 1, // schema version — bumped only on a breaking change "id": "", // globally unique event id "type": "egress.decision", // dotted event type from the registry - - // --- chain / ordering (structural, host-owned) --- "epoch": 7, // writer-boot counter, bumped once per host-controller (writer) start "seq": 1287, // monotonic sequence within this epoch (gap-detectable) "prev": "", // hash of the previous record in the chain ("" for genesis) "hash": "", // sha256(prev + canonical(this event with hash="")) + "producer": "host-controller", // which host component wrote this + "host": "mac-studio-1", + "bottled_agent": "amber-fox-12", // slug from source-IP attribution (null for host-level events) + "ts_wall": "2026-07-26T18:22:04.113Z", // host wall-clock, RFC3339 UTC + "ts_mono": 90142.55, // host monotonic secs since this epoch's boot (intra-epoch ordering only) - // --- trusted: everything the *host* established; authoritative --- - "trusted": { - "producer": "host-controller", // which host component wrote this - "host": "mac-studio-1", - "bottled_agent": "amber-fox-12", // slug from source-IP attribution (null for host-level events) - "ts_wall": "2026-07-26T18:22:04.113Z", // host wall-clock, RFC3339 UTC - "ts_mono": 90142.55 // host monotonic secs since this epoch's boot (intra-epoch ordering only) - }, - - // --- untrusted: anything the agent or a remote claimed; never authoritative --- + // --- the ONLY untrusted region: anything the agent or a remote claimed --- "untrusted": { "reason": "npm install needs registry.npmjs.org", "target": "registry.npmjs.org:443" @@ -147,22 +148,22 @@ stable top level: } ``` -The **`trusted` / `untrusted` split is the core invariant.** A producer may -only place a field in `trusted` if the *host* established it: the -source-IP → `bottled_agent` slug attribution, the host's own clock -(`ts_wall`/`ts_mono`), and the producer's own identity. **`producer` and -`ts_*` live inside `trusted` on purpose** — they are host-supplied, so -grouping them there (rather than as loose top-level fields) keeps the -"authoritative ⇔ inside `trusted`" rule structural, with nothing -host-established leaking outside it. Everything an agent or a remote said -goes in `untrusted`. A reader (or a future policy engine) can therefore -trust `trusted.bottled_agent` for attribution and treat `untrusted.*` as -adversarial claims — the distinction the current stores lack. +The **trust boundary is a single region, not a split.** Everything outside +`untrusted` is trusted by construction — the host established it: the +schema/chain metadata (`v`, `id`, `type`, `epoch`, `seq`, `prev`, `hash`), +the producer and host identity, the source-IP → `bottled_agent` slug +attribution, and the host clock (`ts_wall`/`ts_mono`). `untrusted` is the +**one** place anything an agent or a remote claimed may go. A reader (or a +future policy engine) trusts every top-level field for attribution and +treats `untrusted.*` — and only `untrusted.*` — as adversarial claims. -Only the small structural set — `v`, `id`, `type`, `epoch`, `seq`, `prev`, -`hash` — sits at the top level; it is host-owned too, but it is chain -metadata rather than event data, so it stays out of the `trusted` body to -keep that body purely about *what happened*. +Framing it as "one untrusted region, everything else trusted" (rather than +two parallel `trusted`/`untrusted` blocks) removes the mistake where a +producer forgets to nest a host field under `trusted`: a field is trusted +unless it is deliberately placed inside `untrusted`. The construction API +enforces this — producers pass trusted fields positionally and hand all +agent/remote-claimed data as the single `untrusted` mapping, so there is no +way to emit a top-level field that *looks* authoritative but isn't. ### Canonical serialization + hash chain @@ -235,7 +236,7 @@ required `untrusted` keys so producers and the verifier agree on shape: - **lifecycle.*** — `lifecycle.bottled_agent_start`, `lifecycle.bottled_agent_stop`, `lifecycle.bottled_agent_crash` (producer: host-controller, #468). Leaf names use `bottled_agent` to match - the `trusted.bottled_agent` field — one term for the subject everywhere. + the top-level `bottled_agent` field — one term for the subject everywhere. - **decision.*** — `decision.proposed`, `decision.resolved` (producer: supervise; carries operator action + justification, replacing PRD 0013's row shape). @@ -287,6 +288,56 @@ and keep the event) rather than drop, so a producer bug can never make an audit event vanish; the key deny-list stays a hard refusal because a credential in a named field is always a producer bug worth surfacing. +### Export / interoperability (CloudEvents, OpenTelemetry Logs) + +#487 requires the envelope to map onto the **OpenTelemetry Logs data +model** and/or a **CloudEvents JSON** envelope *without losing integrity or +attribution semantics*. The flattened shape (one `untrusted` region, +everything else trusted at top level) does **not** conflict with either — it +maps *more* cleanly than a nested `trusted`/`untrusted` pair would, because +both target models expect a flat set of top-level fields plus one payload +subtree. + +**CloudEvents.** Context attributes MUST be scalar simple types — a map +cannot be a context attribute — so a nested `trusted` block would have had +to be flattened for CloudEvents anyway. Our flat top level maps directly: +`id`→`id`, `type`→`type`, `producer`+`host`→`source`, +`bottled_agent`→`subject`, `ts_wall`→`time`; the integrity/chain fields +(`epoch`, `seq`, `prev`, `hash`, `v`) ride as **extension attributes** +(scalars — legal). The `untrusted` map goes in `data`. Only mechanical +transform needed: extension attribute names must be lowercase-alphanumeric, +so `bottled_agent`/`ts_mono`/etc. are renamed at export (e.g. a +`botbottle`-prefixed form) — a naming rule, not a schema conflict. + +**OpenTelemetry Logs.** `ts_wall`→`Timestamp`; `type`→the `event.name` +attribute; the flat trusted fields → `Attributes` under a `botbottle.*` +namespace (`botbottle.bottled_agent`, `botbottle.producer`, +`botbottle.chain.hash`, …); `untrusted.*` → `Attributes` under +`botbottle.untrusted.*` (or `Body`). OTel attributes are a dotted map that +happily carries the nested subtree. + +**Attribution is preserved** precisely because the boundary is now +structural: on export, top-level fields become trusted context/attributes +and the `untrusted` subtree stays a single, clearly-named region — so a +downstream consumer still sees exactly which fields an agent claimed. +Nothing agent-claimed is promoted to a trusted-looking position. + +**Integrity has one deliberate caveat.** CloudEvents/OTel are +representation envelopes with their own (or no) canonicalization; `hash` +and `prev` are computed over **our** canonical JSON, not over the exported +form. So the chain fields travel *as data* for reference, but +tamper-evidence is always verified against the **native journal** (the +source of truth) — never re-derived from an exported CloudEvents/OTel +record, whose key ordering / number formatting the exporter may change. +Export is thus a lossless-for-attribution **projection** that carries the +integrity fields along; verification stays on the canonical journal. This +satisfies "without losing integrity or attribution semantics": both are +carried, neither is *relied upon* in the foreign format. + +The export adapters themselves (and #324's webhook delivery / causal +ordering) are follow-up implementation — this PRD fixes the *schema* so +that projection is a field re-map, never a reformat. + ## Implementation chunks 1. **(this PR — PRD only.)** The contract above. No code; scheduled to land @@ -308,7 +359,10 @@ credential in a named field is always a producer bug worth surfacing. 5. **Migrate existing producers.** Re-emit supervise `decision.*` (retiring the standalone `supervise_audit_entries` shape behind the index), egress `egress.*`, git-gate `forge.*`, control-plane `auth.*`. -6. **(follow-up.)** Cross-host merge transport (#324); per-writer signing + +6. **CloudEvents / OTel export adapters.** A projection layer emitting each + event as a CloudEvents JSON envelope and/or an OTel LogRecord (field + re-map per *Export / interoperability*); feeds #324's webhook delivery. +7. **(follow-up.)** Cross-host merge transport (#324); per-writer signing + external anchoring on the chain head; retention/rotation policy. ## Resolved in review (#495) @@ -320,10 +374,12 @@ credential in a named field is always a producer bug worth surfacing. carrying the last chain head as the next `prev` gives a total order of `(epoch, seq)` that survives restarts; `ts_mono` orders only within an epoch. (Design → *ordering across restarts*.) -- **`ts_*` and `producer` belong in `trusted`.** They are host-supplied, so - they now sit inside the `trusted` block; only chain metadata stays at the - top level. (Design → *The envelope*.) -- **Subject term is `bottled_agent` everywhere** — the `trusted` field and +- **Flatten to one `untrusted` region — decided.** Everything outside + `untrusted` (chain metadata, `producer`/`host`, `bottled_agent`, `ts_*`) + is trusted by construction, so the separate `trusted` sub-block is + removed; a field is trusted unless deliberately placed under `untrusted`. + (Design → *The envelope*.) +- **Subject term is `bottled_agent` everywhere** — the top-level field and the `lifecycle.bottled_agent_*` leaf names. (Design → *The envelope* / *Event registry*.) - **Retention head-carry — yes.** When a journal segment is rotated out, -- 2.52.0 From 88b82a169eea2fb77828fd282181e4f9af3a64e4 Mon Sep 17 00:00:00 2001 From: claude Date: Sun, 26 Jul 2026 08:08:22 +0000 Subject: [PATCH 4/6] docs(prd): complete audit-event contract to #487 acceptance checklist Expand the PRD from a schema sketch to the full contract the issue mandates (issue is spec-only: 'defines the contract; implementation may be split into follow-up PRs'): - Envelope: add observed vs event timestamps, bottle/activation ids, manifest_digest + policy_version, actor/action/resource/outcome, correlation_id/causation_id, sensitivity class, typed payload, segment id. - Add a per-field trust-provenance table (trusted vs claimed for every common field); per-type trusted/claimed in the registry. - Canonicalization: normative, reproducible hash-chain test vectors; idempotency (id key, UPSERT), ordering guarantees, and behavior across rotation/restart/import/truncation (truncated-tail vs gap). - Storage: indexable fields + local audit query/verify/rebuild/import CLI. - Registry: cover all mandated groups incl hostctl.*, egress request/decision/cutoff/anomaly, commit.signed (#480), auth/authz, and audit.* self-events; schema-evolution + backward-compatible reader rules. - Export: #324 delivery contract (payload, (epoch,seq) cursor, dedup, backpressure, retention ordering); #480 mapping preserving its byte-to-activation-key guarantee. - No raw prompt/response/body capture by default. - Add an acceptance-criteria coverage table mapping each #487 checkbox to a section. Refs #487 Co-Authored-By: Claude Opus 4.8 --- docs/prds/prd-new-audit-event-schema.md | 447 +++++++++++++++++++----- 1 file changed, 365 insertions(+), 82 deletions(-) diff --git a/docs/prds/prd-new-audit-event-schema.md b/docs/prds/prd-new-audit-event-schema.md index 920d5652..f8b4c7b0 100644 --- a/docs/prds/prd-new-audit-event-schema.md +++ b/docs/prds/prd-new-audit-event-schema.md @@ -19,17 +19,25 @@ field names, timestamps, or how a bottled agent is identified. This PRD defines **one canonical audit-event contract** every producer emits into: -1. A **versioned envelope** — schema version, event id, event type, - monotonic + wall-clock timestamps, and an explicit **trust boundary** - between host-supplied and agent-claimed fields. -2. **Canonical JSON serialization + a per-writer hash chain**, so any - deletion or edit of a past record breaks the chain and is detectable - offline. +1. A **versioned envelope** — schema version, event id, event/observed + timestamps, host-attributed identity (`bottle`/`bottled_agent`/ + `activation`), provenance (`manifest_digest`, `policy_version`), + host-observed `actor`/`action`/`resource`/`outcome`, correlation/ + causation ids, a sensitivity class, a typed payload, and an explicit + **trust boundary** between host-supplied and agent-claimed fields. +2. **Canonical JSON serialization + a per-writer hash chain** (with + normative test vectors), so any deletion, edit, or reorder of a past + record breaks the chain and is detectable offline. 3. An **append-only JSONL journal as the source of truth**, with a - **rebuildable SQLite index** for local search — no paid platform, no - network dependency. -4. An **initial event registry** covering lifecycle, decision, egress, - auth, and forge events. + **rebuildable SQLite index** and a local `audit query`/`verify` surface + — no paid platform, no network dependency. +4. An **initial event registry** covering lifecycle, host-controller, + supervise decision, egress (request/decision/cutoff/anomaly), git-gate + and signed-commit (#480), auth/authz, and audit self-events — each with + its trusted-vs-claimed fields and redaction rules. +5. A **stable export projection** (CloudEvents / OpenTelemetry Logs) and + the **#324 delivery contract** (payload, `(epoch, seq)` cursor, dedup, + backpressure, retention ordering). It is explicitly scheduled to land **immediately after the host controller (#468)** so the host controller's lifecycle transitions are the @@ -126,44 +134,108 @@ stable top level: ``` { - // --- everything at the top level is host-established (trusted) --- + // ---- schema + integrity (host-owned) ---- "v": 1, // schema version — bumped only on a breaking change - "id": "", // globally unique event id + "id": "", // globally unique event id; stable across export/replay (dedup key) "type": "egress.decision", // dotted event type from the registry "epoch": 7, // writer-boot counter, bumped once per host-controller (writer) start "seq": 1287, // monotonic sequence within this epoch (gap-detectable) - "prev": "", // hash of the previous record in the chain ("" for genesis) + "segment": "20260726T000000Z", // journal segment id (rotation boundary); chain continues across segments + "prev": "", // hash of the previous record in the chain ("" for a segment genesis) "hash": "", // sha256(prev + canonical(this event with hash="")) - "producer": "host-controller", // which host component wrote this - "host": "mac-studio-1", - "bottled_agent": "amber-fox-12", // slug from source-IP attribution (null for host-level events) - "ts_wall": "2026-07-26T18:22:04.113Z", // host wall-clock, RFC3339 UTC - "ts_mono": 90142.55, // host monotonic secs since this epoch's boot (intra-epoch ordering only) - // --- the ONLY untrusted region: anything the agent or a remote claimed --- + // ---- timestamps (host-owned) ---- + "ts_event": "2026-07-26T18:22:04.061Z", // when the underlying event occurred at the boundary + "ts_recorded": "2026-07-26T18:22:04.113Z", // when the single writer appended it (authoritative) + "ts_mono": 90142.55, // monotonic secs since this epoch's boot (intra-epoch ordering only) + + // ---- attribution + provenance (host-established) ---- + "producer": "egress", // host component that emitted the event + "host": "mac-studio-1", + "bottle": "amber-fox", // bottle (container/VM) identity + "bottled_agent": "amber-fox-12", // bottled-agent slug from source-IP attribution (null for host-level events) + "activation": "01J8Z...", // activation id: one run/session of the bottled agent (null if n/a) + "manifest_digest": "sha256:9f2…", // digest of the manifest the bottled agent is running (null if n/a) + "policy_version": "egress-2026-07-20", // policy/config version in force at decision time (null if n/a) + + // ---- semantics: host-observed facts of what happened ---- + "actor": "bottled-agent:amber-fox-12", // who acted, as a host-attributed identity + "action": "egress.connect", // what was attempted / done + "resource": "registry.npmjs.org:443", // what it acted on, as observed at the boundary + "outcome": "blocked", // host-decided result: allowed|blocked|deferred|success|failure + "sensitivity": "security", // classification: normal|security|restricted (drives redaction + export) + + // ---- correlation (host-assigned) ---- + "correlation_id": "flow-9c2a…", // groups a related flow (request → decision → cutoff) + "causation_id": "", // the event that directly caused this one ("" if root) + + // ---- typed, trusted, event-specific payload (shape fixed per type in the registry) ---- + "payload": { + "route_id": 4, + "detector": "token_patterns" + }, + + // ---- the ONLY untrusted region: agent- or remote-claimed data ---- "untrusted": { - "reason": "npm install needs registry.npmjs.org", - "target": "registry.npmjs.org:443" + "reason": "npm install needs registry.npmjs.org" // the agent's stated justification } } ``` The **trust boundary is a single region, not a split.** Everything outside -`untrusted` is trusted by construction — the host established it: the -schema/chain metadata (`v`, `id`, `type`, `epoch`, `seq`, `prev`, `hash`), -the producer and host identity, the source-IP → `bottled_agent` slug -attribution, and the host clock (`ts_wall`/`ts_mono`). `untrusted` is the -**one** place anything an agent or a remote claimed may go. A reader (or a -future policy engine) trusts every top-level field for attribution and -treats `untrusted.*` — and only `untrusted.*` — as adversarial claims. +`untrusted` is trusted by construction — the host established it: schema and +chain metadata, both timestamps, the attribution/provenance fields +(source-IP → `bottled_agent`/`bottle`/`activation`, `manifest_digest`, +`policy_version`), the host-observed semantics +(`actor`/`action`/`resource`/`outcome`), the host-assigned correlation ids, +and the typed `payload`. `untrusted` is the **one** place anything an agent +or a remote claimed may go (e.g. the agent's free-text `reason`). A reader +(or a future policy engine) trusts every field outside `untrusted` for +attribution and treats `untrusted.*` — and only `untrusted.*` — as +adversarial claims. -Framing it as "one untrusted region, everything else trusted" (rather than -two parallel `trusted`/`untrusted` blocks) removes the mistake where a -producer forgets to nest a host field under `trusted`: a field is trusted -unless it is deliberately placed inside `untrusted`. The construction API -enforces this — producers pass trusted fields positionally and hand all -agent/remote-claimed data as the single `untrusted` mapping, so there is no -way to emit a top-level field that *looks* authoritative but isn't. +Framing it as "one untrusted region, everything else trusted" removes the +mistake where a producer forgets to mark a claimed field: a field is +trusted unless it is deliberately placed inside `untrusted`. The +construction API enforces this — producers pass trusted fields explicitly +and hand all agent/remote-claimed data as the single `untrusted` mapping, +so there is no way to emit a top-level field that *looks* authoritative but +isn't. + +**Two timestamps** because they answer different questions and can diverge +under backpressure: `ts_event` is when the thing happened at the boundary +(the proxy saw the connect, the gate saw the push); `ts_recorded` is when +the single writer durably appended it. Ordering and the chain use +`(epoch, seq)`, never either wall clock. Both are host-set — a bottled +agent never supplies a timestamp. + +**Optionality.** `bottle`/`bottled_agent`/`activation`, `manifest_digest`, +`policy_version`, and `payload`/`untrusted` are `null`/absent for events +that have no such subject (a host-level `hostctl.*` or `audit.*` event has +no bottled agent). Absent ≠ empty: a reader distinguishes "no subject" from +"unknown". `id`, `type`, the chain fields, both timestamps, `producer`, +`host`, `actor`, `action`, `outcome`, and `sensitivity` are always present. + +#### Trust provenance of every common field + +| Field | Trust | Set by | +|---|---|---| +| `v` `id` `type` `epoch` `seq` `segment` `prev` `hash` | trusted | the single writer | +| `ts_event` | trusted | emitting host component (boundary) | +| `ts_recorded` `ts_mono` | trusted | the single writer | +| `producer` `host` | trusted | the single writer | +| `bottle` `bottled_agent` `activation` | trusted | gateway source-IP → slug attribution | +| `manifest_digest` `policy_version` | trusted | control plane (what was in force) | +| `actor` `action` `resource` `outcome` | trusted | host component that observed/decided it | +| `sensitivity` | trusted | registry default for `type`, overridable up (never down) by the producer | +| `correlation_id` `causation_id` | trusted | the single writer (assigned as it threads the flow) | +| `payload.*` | trusted | emitting host component (shape fixed per `type`) | +| `untrusted.*` | **claimed** | copied verbatim from a bottle / gateway / forge / remote | + +Every registry entry (below) restates, per event type, which `payload` +keys are required and names any `untrusted` keys it carries — so "trusted +vs claimed" is explicit for every event-specific attribute, not just the +common ones. ### Canonical serialization + hash chain @@ -183,11 +255,92 @@ The `hash` field is computed over the canonical form of the event **with digest = sha256_hex(prev_hash + canonical({**event, "hash": ""})) ``` -`prev` is the prior record's `hash`; genesis uses `prev = ""`. This makes -the journal an append-only Merkle-style chain: editing or deleting record -*n* changes its hash, so record *n+1*'s `prev` no longer matches — the -break is local and points at the tampered record. Verification needs only -the journal itself (no keys), so it runs offline and in CI. +`prev` is the prior record's `hash`; a segment genesis uses `prev = ""`. +Two exact rules pin the bytes so the chain is reproducible anywhere: + +1. **Serialize the record with its own `hash` field set to `""`** (present, + empty), never omitted — the key set is identical before and after + hashing. +2. **Digest = `sha256_hex(prev + canonical(record_with_empty_hash))`**, + where `prev` is the previous record's `hash` string (`""` at genesis), + `+` is string concatenation, and `canonical` is the function above. + `ts_mono`, being a float, is serialized by Python's shortest-round-trip + `repr` via `json.dumps`; producers therefore emit it as a JSON number + they do not post-process. (All other fields are strings/ints/objects, + which serialize unambiguously.) + +Editing or deleting record *n* changes its hash, so record *n+1*'s `prev` +no longer matches — the break is local and names the tampered record. +Verification needs only the journal itself (no keys), so it runs offline +and in CI. + +#### Test vectors (normative) + +Two records, reduced to the chain-relevant fields, demonstrate the exact +serialization and linkage. An implementation is conformant iff it +reproduces these bytes and hashes. + +``` +# Record 0 — segment genesis (prev = "") +canonical(record0, hash=""): +{"hash":"","id":"11111111-1111-4111-8111-111111111111","prev":"","seq":0,"type":"audit.segment_open"} +hash0 = sha256("" + canonical) = + 942ea5729bcac6efdbdea942396bfa574ab0d6ebf5615402595359422f2aeb83 + +# Record 1 — chains onto record 0 (prev = hash0) +canonical(record1, hash=""): +{"hash":"","id":"22222222-2222-4222-8222-222222222222","prev":"942ea5729bcac6efdbdea942396bfa574ab0d6ebf5615402595359422f2aeb83","seq":1,"type":"lifecycle.bottled_agent_start"} +hash1 = sha256(hash0 + canonical) = + bc082347680405fee50b60a9c304611aa026950b15d869b7e3ae56e1c451b856 + +# Tamper check: flip record0.type → recompute → +# 4553eda647f33f0c608cfea44be28efbbaca45ed30b873fcbd4405fa5ce737ed +# which no longer equals record1.prev (942ea5…) — the break is detected at record1. +``` + +The implementation PR ships these plus full-envelope vectors (every field +populated, and a redaction case) as committed fixtures, so a schema-version +bump that changes the bytes fails a golden test loudly. + +### Ordering, idempotency, and duplicate handling + +- **Ordering.** `(epoch, seq)` is a strict total order per host and, because + the writer is single, a strict order per bottle/activation within that + host — satisfying "at least strict causal order per activation/bottle". + `causation_id` records the explicit cause edges (a DAG) on top of the + total order, so a consumer can reconstruct request → decision → cutoff + even if unrelated events interleave between them. +- **Idempotency.** `id` is the idempotency key. A producer that retries an + emit (e.g. after a writer restart mid-handoff) **reuses the same `id`**; + the writer drops a second append bearing an `id` already present in the + current segment's in-memory set, and the index `UPSERT`s by `id`, so a + duplicate never double-counts or forks the chain. +- **Deduplication downstream.** Because `id` is stable across export and + replay, #324's cursor replay and any cross-host merge dedup on `id` — no + consumer needs to invent a second identity. + +### Behavior across rotation, restart, import, truncation + +- **Rotation.** At a segment boundary the writer opens a new segment file, + sets its `segment` id, and carries the rotated-out segment's head as the + new genesis `prev` — so the chain is continuous *across* segments while + each file stays independently openable. `verify` walks segments in order + and checks the head-to-genesis link at each seam. +- **Restart.** Covered above: read last line → adopt its `hash` as `prev`, + bump `epoch`, reset `seq`. The chain never restarts even though the + counters do. +- **Import.** `audit import ` appends an externally supplied + segment (e.g. recovered from another host or a backup). Import verifies + the incoming chain in isolation first, then links it only if its genesis + `prev` matches a known head or is explicitly grafted; imported records + keep their original `id` (dedup) and are marked with their origin host so + attribution is not laundered. +- **Truncation.** A crash can leave a partial final line; `verify` reports + it as `truncated-tail` (recoverable — replay resumes from the last intact + record). A chain that ends before a persisted head, or a missing interior + `seq`, is reported as `gap`/`missing-suffix` (evidence of deletion, not a + clean crash). The two are distinguished so an operator can tell "power + loss" from "someone trimmed the log". ### Single writer; ordering across restarts @@ -209,7 +362,7 @@ start the writer: 0 for the new boot. Total order is therefore `(epoch, seq)` — monotonic across restarts by -construction — with `ts_wall` for human reading and `ts_mono` for +construction — with `ts_event`/`ts_recorded` for human reading and `ts_mono` for sub-second ordering inside an epoch. A crash mid-append truncates at most the last (partial) line; the verifier flags it and replay resumes from the last intact record. @@ -220,35 +373,91 @@ last intact record. alongside `host_db_path()`), one canonical event per line, opened `O_APPEND`. This is authoritative. - **Index:** a new `audit_events` table via the existing `DbStore` / - `TableMigrations` machinery, holding the envelope columns plus JSON - blobs, indexed on `(bottled_agent, type, ts_wall)`. It is a **derived - cache**: `audit rebuild` truncates and replays the journal, re-verifying - the chain as it goes. If the DB is deleted or drifts, it is regenerated - from the journal with no data loss. (This supersedes the free-standing - `supervise_audit_entries` table, which becomes a view/producer onto the - new index.) + `TableMigrations` machinery. It is a **derived cache, not a second source + of truth**: `audit rebuild` truncates and replays the journal, + re-verifying the chain as it goes, so a deleted or drifted DB is + regenerated from the journal with no data loss. On a `verify` failure + during rebuild it stops and reports rather than indexing past a break. + (This supersedes the free-standing `supervise_audit_entries` table, which + becomes a producer onto the new index.) + +**Indexable fields** (columns + indices): `ts_event`, `ts_recorded`, +`type`, `host`, `bottle`, `bottled_agent`, `activation`, `actor`, +`outcome`, `sensitivity`, `correlation_id`, `causation_id`, plus two +event-specific projections promoted out of `payload` for query — +`repository` and `commit_sha` (populated for `forge.*`/`commit.*`, null +otherwise). The full canonical record is stored verbatim in a `raw` column +so the index never loses fidelity to the journal. + +**Local query surface** — `audit query`, no egress, no paid platform: + +``` +audit query \ + [--since T] [--until T] [--type egress.*] [--host H] [--bottle B] \ + [--activation A] [--agent SLUG] [--actor ID] [--outcome blocked] \ + [--repository R] [--correlation-id C] [--commit SHA] \ + [--follow BOTTLE] # a bottled agent's events in (epoch, seq) order + [--json | --table] +audit verify [--segment S] # offline chain check; exit non-zero on any break +audit rebuild # drop + replay journal → index +audit import # graft an external segment (see above) +``` + +Type filters accept a `group.*` glob. A read-only local HTTP endpoint +mirrors the same filters for the future review console; both are pure reads +over the index and can never mutate the journal. ### Event registry (initial) -Dotted `type` names, grouped; the registry is a table mapping type → -required `untrusted` keys so producers and the verifier agree on shape: +Dotted `type` names, grouped. The registry is a table mapping each type to +its required `payload` keys, its `untrusted` keys (if any), a default +`sensitivity`, and its correlation behavior — so producers and the verifier +agree on shape and "trusted vs claimed" is pinned per type. Initial +coverage (the issue's mandated set): -- **lifecycle.*** — `lifecycle.bottled_agent_start`, - `lifecycle.bottled_agent_stop`, `lifecycle.bottled_agent_crash` - (producer: host-controller, #468). Leaf names use `bottled_agent` to match - the top-level `bottled_agent` field — one term for the subject everywhere. -- **decision.*** — `decision.proposed`, `decision.resolved` - (producer: supervise; carries operator action + justification, replacing - PRD 0013's row shape). -- **egress.*** — `egress.decision` (allow/block at the proxy), - `egress.route_added`. -- **auth.*** — `auth.token_minted`, `auth.token_rejected` (control-plane; - **never** the token itself — see redaction). -- **forge.*** — `forge.push_accepted`, `forge.push_rejected` (git-gate), - `forge.pr_opened`. +| Group / type | Producer | Required `payload` (trusted) | `untrusted` | Default sensitivity | +|---|---|---|---|---| +| **lifecycle.*** — `bottled_agent_start` / `_stop` / `_crash` | host-controller (#468) | `manifest_digest`, `exit` (for stop/crash) | — | normal | +| **hostctl.*** — `broker_launch`, `broker_teardown`, `broker_reject` | host-controller (#468) | `op`, `request_digest` | — | security | +| **decision.*** — `proposed`, `resolved` | supervise | `tool`, `operator_action`, `justification`, `diff_digest` | `agent_rationale` | security | +| **egress.*** — `request`, `decision`, `cutoff`, `anomaly` | egress proxy | `route_id`, `detector` (on match), `bytes` (cutoff) | `reason`, `target_claimed` | security | +| **forge.*** — `push_accepted`, `push_rejected`, `pr_opened` | git-gate | `repository`, `ref`, `gitleaks_result` | `title`, `description` | security | +| **commit.signed** (#480) | git-gate | `repository`, `commit_sha`, `activation_key_id`, `signature_ref` | `commit_message` | security | +| **auth.*** — `token_minted`, `token_rejected`, `authz_denied` | control plane | `role`, `token_id`, `reason_code` | — | security | +| **audit.*** — `segment_open`, `verify_failed`, `truncation_detected`, `export_failed` | audit writer/verifier | `segment`, `detail` | — | security | -New types are additive; adding one does not bump `v`. Removing or -re-typing a field bumps `v`. +Notes: +- **`egress.request` vs `egress.decision`** share a `correlation_id`; the + `decision`'s `causation_id` points at the `request`, and a later `cutoff` + chains onto the `decision` — so a flow is reconstructable. +- **`audit.*` self-events** make the audit subsystem audit itself: a failed + verification, a detected truncation, or a dropped export is itself a + chained, tamper-evident record — you cannot silence the alarm without + breaking the chain that carries it. +- **Free-text and remote-echoed fields are always `untrusted`** (`reason`, + `agent_rationale`, PR `title`/`description`, `target_claimed`), because + they originate in the bottle or a remote response; the host-observed + counterpart (`resource`, `outcome`, `gitleaks_result`) is the trusted + fact. + +**Sensitivity + redaction per type.** Every type's default `sensitivity` +is listed above; a producer may raise it (never lower it). `restricted` +events keep their `payload` in the journal but the export projection ships +only the envelope + a payload digest unless the consumer is authorized — +so a `security`/`restricted` record is still counted and correlated +downstream without leaking its body. The credential-shape redaction rules +(next) apply to **every** type regardless of sensitivity. + +**Schema evolution & backward-compatible readers.** The registry is +append-only: **adding** a type, an optional `payload` key, or an +`untrusted` key does **not** bump `v`; readers ignore unknown fields +(forward-compatible) and treat absent optional fields as `null`. +**Removing** or **re-typing** a field, or making an optional field +required, bumps `v`. A reader declares the max `v` it understands and +refuses to *interpret* a higher-`v` record, but the **verifier is +version-agnostic** — the hash covers whatever fields exist, so chain +integrity is checkable across versions without understanding semantics. +Every `v` bump ships a migration note and updated golden vectors. ### Redaction rule @@ -288,6 +497,14 @@ and keep the event) rather than drop, so a producer bug can never make an audit event vanish; the key deny-list stays a hard refusal because a credential in a named field is always a producer bug worth surfacing. +**No raw-payload capture by default.** The envelope carries *decisions and +metadata*, not traffic. Prompts, model responses, request/response bodies, +and file contents are **not** recorded unless a producer opts a specific, +reviewed field in — and such a field is `untrusted` and subject to both +redaction layers. This keeps the audit log from becoming a covert copy of +the very data the sandbox exists to contain (the issue's "unsafe payload +capture" non-goal). + ### Export / interoperability (CloudEvents, OpenTelemetry Logs) #487 requires the envelope to map onto the **OpenTelemetry Logs data @@ -302,14 +519,14 @@ subtree. cannot be a context attribute — so a nested `trusted` block would have had to be flattened for CloudEvents anyway. Our flat top level maps directly: `id`→`id`, `type`→`type`, `producer`+`host`→`source`, -`bottled_agent`→`subject`, `ts_wall`→`time`; the integrity/chain fields +`bottled_agent`→`subject`, `ts_event`→`time` (`ts_recorded` as an extension); the integrity/chain fields (`epoch`, `seq`, `prev`, `hash`, `v`) ride as **extension attributes** (scalars — legal). The `untrusted` map goes in `data`. Only mechanical transform needed: extension attribute names must be lowercase-alphanumeric, so `bottled_agent`/`ts_mono`/etc. are renamed at export (e.g. a `botbottle`-prefixed form) — a naming rule, not a schema conflict. -**OpenTelemetry Logs.** `ts_wall`→`Timestamp`; `type`→the `event.name` +**OpenTelemetry Logs.** `ts_event`→`Timestamp`, `ts_recorded`→`ObservedTimestamp`; `type`→the `event.name` attribute; the flat trusted fields → `Attributes` under a `botbottle.*` namespace (`botbottle.bottled_agent`, `botbottle.producer`, `botbottle.chain.hash`, …); `untrusted.*` → `Attributes` under @@ -334,9 +551,47 @@ integrity fields along; verification stays on the canonical journal. This satisfies "without losing integrity or attribution semantics": both are carried, neither is *relied upon* in the foreign format. -The export adapters themselves (and #324's webhook delivery / causal -ordering) are follow-up implementation — this PRD fixes the *schema* so -that projection is a field re-map, never a reformat. +The export adapters themselves are follow-up implementation — this PRD +fixes the *schema* so that projection is a field re-map, never a reformat. + +#### The #324 delivery contract (payload, cursor, backpressure) + +#324 transports events off-box; it must not invent a second envelope. This +PRD fixes the contract it depends on: + +- **Payload.** #324 ships the **native canonical record verbatim** (the + exact bytes the hash covers), optionally wrapped in the CloudEvents + projection whose `data` *is* that record. Either way the integrity fields + travel intact and the receiver can verify against the same bytes. +- **Cursor.** The export cursor is `(epoch, seq)` (equivalently the last + exported `hash`). It advances **only on acknowledgement**, so delivery is + at-least-once and gap-free; a crash re-sends from the last acked cursor. +- **Idempotency / replay.** Dedup is on `id` (stable across replay), so + at-least-once delivery is safe — the receiver collapses re-sends. +- **Backpressure.** The outbox is the journal itself plus a cursor; when + the endpoint is slow the cursor simply lags — the writer never blocks on + export, and audit never applies backpressure to the data plane it + records. +- **Retention interaction.** Retention/rotation **must not** prune a + segment whose records are still behind the export cursor; the reaper + honors `min(cursor)` across all configured consumers. (The schedule + itself stays the retention follow-up; this is the *ordering* constraint + that follow-up must respect.) + +#### #480 signed-commit attribution maps in without weakening it + +#480 binds a commit's bytes to a per-activation signing key. It maps to the +`commit.signed` event: `payload` carries `repository`, `commit_sha`, +`activation_key_id`, and a `signature_ref` (the detached-signature +location or its digest) — **not** the private key and not a re-derived +signature. The audit event therefore *references and timestamps* #480's +existing byte-to-activation-key proof inside the tamper-evident chain; it +does not re-implement or replace it, so #480's guarantee is unweakened — +the signature still verifies against the commit bytes independently, and +the audit record adds only "this binding was observed at this point in the +chain". The trusted `actor`/`activation` fields and the `commit_sha` +payload are host-observed at the gate, so attribution cannot be forged by +the committing agent. ## Implementation chunks @@ -349,21 +604,49 @@ that projection is a field re-map, never a reformat. `redact_tokens`); unit tests for determinism, chain-break detection, `epoch`/`seq` continuity across a simulated restart, and redaction of both a deny-listed key and a token-shaped value. -3. **SQLite index + `audit rebuild` / `audit verify` CLI.** New - `audit_events` migration; replay-from-journal; offline chain verifier; - local query commands (by bottled-agent / type / time / producer). +3. **SQLite index + `audit` CLI.** New `audit_events` migration (indexable + fields above); replay-from-journal; offline chain verifier + (`truncated-tail` vs `gap`); `query` / `verify` / `rebuild` / `import`; + idempotent `UPSERT` by `id`. 4. **Host controller as first producer (#468).** Wire - `lifecycle.bottled_agent_*` emission into the host controller's - start/stop/crash paths; establish the `epoch` bump + chain-head carry on - writer restart here (the host controller owns the single writer). + `lifecycle.bottled_agent_*` and `hostctl.*` emission into the host + controller; establish the `epoch` bump + chain-head carry + segment + rotation on writer restart here (it owns the single writer). 5. **Migrate existing producers.** Re-emit supervise `decision.*` (retiring the standalone `supervise_audit_entries` shape behind the index), egress - `egress.*`, git-gate `forge.*`, control-plane `auth.*`. -6. **CloudEvents / OTel export adapters.** A projection layer emitting each - event as a CloudEvents JSON envelope and/or an OTel LogRecord (field - re-map per *Export / interoperability*); feeds #324's webhook delivery. -7. **(follow-up.)** Cross-host merge transport (#324); per-writer signing + - external anchoring on the chain head; retention/rotation policy. + `egress.*`, git-gate `forge.*` + `commit.signed` (#480), control-plane + `auth.*`; add the `audit.*` self-events (verify/truncation/export + failure). +6. **CloudEvents / OTel export adapters + #324 delivery.** Projection layer + (field re-map per *Export / interoperability*) plus the outbox cursor, + ack-driven advance, and retention-ordering guard the #324 contract + specifies. +7. **(follow-up.)** Cross-host merge transport; per-writer signing + + external anchoring on the chain head; retention/rotation *schedule*. + +## Acceptance-criteria coverage (#487) + +The issue defines the contract; implementation is explicitly split into +follow-up PRs. This PRD is the durable decision record; each acceptance box +maps to a section: + +| #487 acceptance criterion | Where | +|---|---| +| Durable PRD defines versioned envelope + initial registry | *The envelope*, *Event registry* | +| Canonical JSON + hash-chain rules, unambiguous, with test vectors | *Canonical serialization + hash chain* → *Test vectors* | +| Trust provenance explicit for every common + event-specific field | *Trust provenance of every common field*; per-type `untrusted` in *Event registry* | +| Redaction prohibits credentials / raw secrets / unsafe capture by default | *Redaction rule*; `untrusted`-only claims; sensitivity classes | +| JSONL journal canonical; SQLite index fully rebuildable | *Journal + SQLite index* (`audit rebuild`) | +| Minimum local search/query contract | *Journal + SQLite index* → *Local query surface* | +| #324 can transport/replay without a second envelope | *The #324 delivery contract* | +| #480 maps in without weakening its byte-to-activation-key guarantee | *#480 signed-commit attribution maps in…* | +| Schema evolution + backward-compatible readers | *Schema evolution & backward-compatible readers* | +| Integrity detects modification / deletion / reorder / bad continuation | *Test vectors* (tamper), *Behavior across rotation…truncation*, `audit verify` | + +Two acceptance items are **specified here, implemented later** by design +(the issue permits this): the concrete test-vector *fixtures* and the +`audit` CLI land in impl chunks 2–3; the #324 outbox lands in chunk 6. +Nothing in the contract is left undefined — only its code is deferred. ## Resolved in review (#495) -- 2.52.0 From f3664dea9fab99f4186c5383c8b59c6545256a5d Mon Sep 17 00:00:00 2001 From: claude Date: Sun, 26 Jul 2026 08:28:10 +0000 Subject: [PATCH 5/6] docs(prd): drop policy_version (manifest is the policy); add engine field MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Per review: bot-bottle has no separate policy artifact — a bottled agent's egress routes etc. are declared in its manifest (manifest/egress.py), so manifest_digest already pins the policy in force. Remove the redundant policy_version. Given a fixed manifest, the only other axis that changes an outcome is the enforcing code, so add 'engine' (bot-bottle version + git SHA) as a trusted field. Runtime operator overrides (supervise egress-allow) are themselves audit events, so effective policy = manifest_digest + logged deltas, reconstructable from the chain. Note the build must stamp the git SHA (only version=0.1.0 exists today). Refs #487 Co-Authored-By: Claude Opus 4.8 --- docs/prds/prd-new-audit-event-schema.md | 41 +++++++++++++++++-------- 1 file changed, 29 insertions(+), 12 deletions(-) diff --git a/docs/prds/prd-new-audit-event-schema.md b/docs/prds/prd-new-audit-event-schema.md index f8b4c7b0..9b781ea2 100644 --- a/docs/prds/prd-new-audit-event-schema.md +++ b/docs/prds/prd-new-audit-event-schema.md @@ -21,7 +21,8 @@ emits into: 1. A **versioned envelope** — schema version, event id, event/observed timestamps, host-attributed identity (`bottle`/`bottled_agent`/ - `activation`), provenance (`manifest_digest`, `policy_version`), + `activation`), provenance (`manifest_digest` — the manifest is the + policy — and `engine` = bot-bottle version/SHA), host-observed `actor`/`action`/`resource`/`outcome`, correlation/ causation ids, a sensitivity class, a typed payload, and an explicit **trust boundary** between host-supplied and agent-claimed fields. @@ -152,11 +153,11 @@ stable top level: // ---- attribution + provenance (host-established) ---- "producer": "egress", // host component that emitted the event "host": "mac-studio-1", + "engine": "bot-bottle/0.1.0+abc1234", // bot-bottle version + git SHA of the enforcing host code "bottle": "amber-fox", // bottle (container/VM) identity "bottled_agent": "amber-fox-12", // bottled-agent slug from source-IP attribution (null for host-level events) "activation": "01J8Z...", // activation id: one run/session of the bottled agent (null if n/a) - "manifest_digest": "sha256:9f2…", // digest of the manifest the bottled agent is running (null if n/a) - "policy_version": "egress-2026-07-20", // policy/config version in force at decision time (null if n/a) + "manifest_digest": "sha256:9f2…", // digest of the manifest — which IS the policy (egress routes etc.); null if n/a // ---- semantics: host-observed facts of what happened ---- "actor": "bottled-agent:amber-fox-12", // who acted, as a host-attributed identity @@ -186,7 +187,7 @@ The **trust boundary is a single region, not a split.** Everything outside `untrusted` is trusted by construction — the host established it: schema and chain metadata, both timestamps, the attribution/provenance fields (source-IP → `bottled_agent`/`bottle`/`activation`, `manifest_digest`, -`policy_version`), the host-observed semantics +`engine`), the host-observed semantics (`actor`/`action`/`resource`/`outcome`), the host-assigned correlation ids, and the typed `payload`. `untrusted` is the **one** place anything an agent or a remote claimed may go (e.g. the agent's free-text `reason`). A reader @@ -209,12 +210,26 @@ the single writer durably appended it. Ordering and the chain use `(epoch, seq)`, never either wall clock. Both are host-set — a bottled agent never supplies a timestamp. +**The manifest *is* the policy.** bot-bottle has no separate policy +artifact — a bottled agent's egress routes and other constraints are +declared in its manifest (`bot_bottle/manifest/egress.py`), so +`manifest_digest` already pins the ruleset in force; there is no distinct +`policy_version`. Given a fixed manifest, the only other thing that can +change a decision's outcome is the enforcing code — captured by `engine` +(bot-bottle version + git SHA). So two `egress.decision` records with the +same `resource` but different `outcome` are explained by exactly one of: +different `manifest_digest` (the rules changed) or different `engine` (the +enforcer changed). Runtime operator overrides (a supervise `egress-allow`) +are themselves audit events, so the effective ruleset at any instant is +`manifest_digest` plus the logged, approved deltas — reconstructable from +the chain, not from a version stamp. + **Optionality.** `bottle`/`bottled_agent`/`activation`, `manifest_digest`, -`policy_version`, and `payload`/`untrusted` are `null`/absent for events -that have no such subject (a host-level `hostctl.*` or `audit.*` event has -no bottled agent). Absent ≠ empty: a reader distinguishes "no subject" from -"unknown". `id`, `type`, the chain fields, both timestamps, `producer`, -`host`, `actor`, `action`, `outcome`, and `sensitivity` are always present. +and `payload`/`untrusted` are `null`/absent for events that have no such +subject (a host-level `hostctl.*` or `audit.*` event has no bottled agent). +Absent ≠ empty: a reader distinguishes "no subject" from "unknown". `id`, +`type`, the chain fields, both timestamps, `producer`, `host`, `engine`, +`actor`, `action`, `outcome`, and `sensitivity` are always present. #### Trust provenance of every common field @@ -223,9 +238,9 @@ no bottled agent). Absent ≠ empty: a reader distinguishes "no subject" from | `v` `id` `type` `epoch` `seq` `segment` `prev` `hash` | trusted | the single writer | | `ts_event` | trusted | emitting host component (boundary) | | `ts_recorded` `ts_mono` | trusted | the single writer | -| `producer` `host` | trusted | the single writer | +| `producer` `host` `engine` | trusted | the single writer | | `bottle` `bottled_agent` `activation` | trusted | gateway source-IP → slug attribution | -| `manifest_digest` `policy_version` | trusted | control plane (what was in force) | +| `manifest_digest` | trusted | control plane (the manifest = the policy in force) | | `actor` `action` `resource` `outcome` | trusted | host component that observed/decided it | | `sensitivity` | trusted | registry default for `type`, overridable up (never down) by the producer | | `correlation_id` `causation_id` | trusted | the single writer (assigned as it threads the flow) | @@ -601,7 +616,9 @@ the committing agent. `canonical()`, chain hashing, and the single-writer journal appender in `bot_bottle/store/` (reusing `sha256_hex`); redaction wired to the existing `gateway/egress/dlp_detectors` (`scan_token_patterns` / - `redact_tokens`); unit tests for determinism, chain-break detection, + `redact_tokens`); embed the git SHA at build so `engine` is populated + (only `version = "0.1.0"` exists in `pyproject.toml` today — the build + must stamp the SHA); unit tests for determinism, chain-break detection, `epoch`/`seq` continuity across a simulated restart, and redaction of both a deny-listed key and a token-shaped value. 3. **SQLite index + `audit` CLI.** New `audit_events` migration (indexable -- 2.52.0 From 4cb7bbbd115280a952ad05e03db07914b2b65d94 Mon Sep 17 00:00:00 2001 From: codex Date: Sun, 26 Jul 2026 17:38:43 +0000 Subject: [PATCH 6/6] docs(prd): tighten audit ordering and chain guarantees --- docs/prds/prd-new-audit-event-schema.md | 82 ++++++++++++++++--------- 1 file changed, 54 insertions(+), 28 deletions(-) diff --git a/docs/prds/prd-new-audit-event-schema.md b/docs/prds/prd-new-audit-event-schema.md index 9b781ea2..073d2ac7 100644 --- a/docs/prds/prd-new-audit-event-schema.md +++ b/docs/prds/prd-new-audit-event-schema.md @@ -37,7 +37,8 @@ emits into: and signed-commit (#480), auth/authz, and audit self-events — each with its trusted-vs-claimed fields and redaction rules. 5. A **stable export projection** (CloudEvents / OpenTelemetry Logs) and - the **#324 delivery contract** (payload, `(epoch, seq)` cursor, dedup, + the **#324 delivery contract** (payload, per-chain + `(host, epoch, seq)` cursor, dedup, backpressure, retention ordering). It is explicitly scheduled to land **immediately after the host @@ -142,7 +143,7 @@ stable top level: "epoch": 7, // writer-boot counter, bumped once per host-controller (writer) start "seq": 1287, // monotonic sequence within this epoch (gap-detectable) "segment": "20260726T000000Z", // journal segment id (rotation boundary); chain continues across segments - "prev": "", // hash of the previous record in the chain ("" for a segment genesis) + "prev": "", // prior record hash ("" only for the first record of a new chain) "hash": "", // sha256(prev + canonical(this event with hash="")) // ---- timestamps (host-owned) ---- @@ -270,7 +271,9 @@ The `hash` field is computed over the canonical form of the event **with digest = sha256_hex(prev_hash + canonical({**event, "hash": ""})) ``` -`prev` is the prior record's `hash`; a segment genesis uses `prev = ""`. +`prev` is the prior record's `hash`; only the first record of a brand-new +chain uses `prev = ""`. The first record of a rotated segment carries the +preceding segment's head, as specified under *Rotation* below. Two exact rules pin the bytes so the chain is reproducible anywhere: 1. **Serialize the record with its own `hash` field set to `""`** (present, @@ -296,7 +299,7 @@ serialization and linkage. An implementation is conformant iff it reproduces these bytes and hashes. ``` -# Record 0 — segment genesis (prev = "") +# Record 0 — brand-new chain genesis (prev = "") canonical(record0, hash=""): {"hash":"","id":"11111111-1111-4111-8111-111111111111","prev":"","seq":0,"type":"audit.segment_open"} hash0 = sha256("" + canonical) = @@ -319,17 +322,28 @@ bump that changes the bytes fails a golden test loudly. ### Ordering, idempotency, and duplicate handling -- **Ordering.** `(epoch, seq)` is a strict total order per host and, because +- **Ordering.** `(epoch, seq)` is a strict total order within one host's + native chain and, because the writer is single, a strict order per bottle/activation within that host — satisfying "at least strict causal order per activation/bottle". `causation_id` records the explicit cause edges (a DAG) on top of the total order, so a consumer can reconstruct request → decision → cutoff - even if unrelated events interleave between them. + even if unrelated events interleave between them. There is deliberately + no invented total order across imported host chains; a cross-host key is + `(host, epoch, seq)`, and consumers use correlation/causation edges where + causal ordering across hosts is known. - **Idempotency.** `id` is the idempotency key. A producer that retries an emit (e.g. after a writer restart mid-handoff) **reuses the same `id`**; - the writer drops a second append bearing an `id` already present in the - current segment's in-memory set, and the index `UPSERT`s by `id`, so a - duplicate never double-counts or forks the chain. + before appending, the writer checks a durable, host-wide id ledger that + spans every segment and survives restart/retention, and drops an id + already present. The ledger is operational metadata, not an audit source + of truth: after a crash it is reconciled from the journal before appends + resume, and retention preserves id tombstones after journal segments are + pruned. An in-memory set may cache the ledger but is never the authority. + The index additionally has a unique key on `id`; its `UPSERT` is + defensive and does not substitute for the pre-append check. Thus a + duplicate never enters the source-of-truth journal, double-counts, or + forks the chain. - **Deduplication downstream.** Because `id` is stable across export and replay, #324's cursor replay and any cross-host merge dedup on `id` — no consumer needs to invent a second identity. @@ -338,18 +352,22 @@ bump that changes the bytes fails a golden test loudly. - **Rotation.** At a segment boundary the writer opens a new segment file, sets its `segment` id, and carries the rotated-out segment's head as the - new genesis `prev` — so the chain is continuous *across* segments while + new segment's first `prev` — so the chain is continuous *across* segments + (`prev = ""` is reserved for the first record of a brand-new chain) while each file stays independently openable. `verify` walks segments in order - and checks the head-to-genesis link at each seam. + and checks the preceding-head-to-first-record link at each seam. - **Restart.** Covered above: read last line → adopt its `hash` as `prev`, bump `epoch`, reset `seq`. The chain never restarts even though the counters do. - **Import.** `audit import ` appends an externally supplied segment (e.g. recovered from another host or a backup). Import verifies - the incoming chain in isolation first, then links it only if its genesis - `prev` matches a known head or is explicitly grafted; imported records - keep their original `id` (dedup) and are marked with their origin host so - attribution is not laundered. + the incoming chain in isolation first. A continuation whose first `prev` + matches a known head extends that chain. A foreign chain is registered as + a separate immutable chain namespace rather than rewriting or grafting + its records (which would invalidate their hashes). Imported records keep + their original `host`, `id`, `epoch`, and `seq`; the index keys their + native order by `(host, epoch, seq)` so attribution is not laundered and + tuples from different hosts cannot collide. - **Truncation.** A crash can leave a partial final line; `verify` reports it as `truncated-tail` (recoverable — replay resumes from the last intact record). A chain that ends before a persisted head, or a missing interior @@ -376,11 +394,13 @@ start the writer: 2. bumps `epoch` (persisted alongside the chain head) and resets `seq` to 0 for the new boot. -Total order is therefore `(epoch, seq)` — monotonic across restarts by -construction — with `ts_event`/`ts_recorded` for human reading and `ts_mono` for -sub-second ordering inside an epoch. A crash mid-append truncates at most -the last (partial) line; the verifier flags it and replay resumes from the -last intact record. +Total order within this host's native chain is therefore `(epoch, seq)` — +monotonic across restarts by construction — with +`ts_event`/`ts_recorded` for human reading and `ts_mono` for sub-second +ordering inside an epoch. Imported chains retain their own +`(host, epoch, seq)` order and do not acquire a fictional order relative to +the local chain. A crash mid-append truncates at most the last (partial) +line; the verifier flags it and replay resumes from the last intact record. ### Journal (source of truth) + SQLite index (rebuildable) @@ -401,8 +421,11 @@ last intact record. `outcome`, `sensitivity`, `correlation_id`, `causation_id`, plus two event-specific projections promoted out of `payload` for query — `repository` and `commit_sha` (populated for `forge.*`/`commit.*`, null -otherwise). The full canonical record is stored verbatim in a `raw` column -so the index never loses fidelity to the journal. +otherwise). The unique event id and native-order index are respectively +`id` and `(host, epoch, seq)`; a local-only `ingest_seq` provides stable +display order when a query intentionally mixes chains without pretending +that it is causal order. The full canonical record is stored verbatim in a +`raw` column so the index never loses fidelity to the journal. **Local query surface** — `audit query`, no egress, no paid platform: @@ -411,7 +434,7 @@ audit query \ [--since T] [--until T] [--type egress.*] [--host H] [--bottle B] \ [--activation A] [--agent SLUG] [--actor ID] [--outcome blocked] \ [--repository R] [--correlation-id C] [--commit SHA] \ - [--follow BOTTLE] # a bottled agent's events in (epoch, seq) order + [--follow BOTTLE] # one host chain's events in (epoch, seq) order [--json | --table] audit verify [--segment S] # offline chain check; exit non-zero on any break audit rebuild # drop + replay journal → index @@ -578,9 +601,12 @@ PRD fixes the contract it depends on: exact bytes the hash covers), optionally wrapped in the CloudEvents projection whose `data` *is* that record. Either way the integrity fields travel intact and the receiver can verify against the same bytes. -- **Cursor.** The export cursor is `(epoch, seq)` (equivalently the last - exported `hash`). It advances **only on acknowledgement**, so delivery is - at-least-once and gap-free; a crash re-sends from the last acked cursor. +- **Cursor.** Export maintains one cursor per native host chain: + `(host, epoch, seq)` (equivalently that chain's last exported `hash`). + It advances **only on acknowledgement**, so delivery is at-least-once and + gap-free; a crash re-sends from the last acked cursor. Imported foreign + chains use independent cursors and never share a bare `(epoch, seq)` + namespace with the local chain. - **Idempotency / replay.** Dedup is on `id` (stable across replay), so at-least-once delivery is safe — the receiver collapses re-sends. - **Backpressure.** The outbox is the journal itself plus a cursor; when @@ -683,8 +709,8 @@ Nothing in the contract is left undefined — only its code is deferred. the `lifecycle.bottled_agent_*` leaf names. (Design → *The envelope* / *Event registry*.) - **Retention head-carry — yes.** When a journal segment is rotated out, - the new segment's genesis `prev` is the rotated-out head, so the verifier - still trusts the current head across a rotation. (Folds into the + the new segment's first record carries the rotated-out head as `prev`, so + the verifier still trusts the current head across a rotation. (Folds into the retention follow-up.) - **Redaction reuses the egress detectors — yes, scoped.** Reuse the deterministic `dlp_detectors` (`scan_token_patterns` / `redact_tokens` / -- 2.52.0