docs(prd): address review #495 on audit-event schema
prd-number-check / require-numbered-prds (pull_request) Failing after 7s
tracker-policy-pr / check-pr (pull_request) Successful in 8s

- Move ts_wall/ts_mono and producer inside the trusted block (host-supplied).
- Rename subject to bottled_agent everywhere (field + lifecycle.bottled_agent_* leaves).
- Add explicit epoch (writer-boot) counter + chain-head carry for ordering across host-controller restarts.
- Commit to a single writer per host (host controller owns it).
- Reuse egress dlp_detectors (scan_token_patterns/redact_tokens) for redaction; exclude scan_entropy as brittle on structured audit values.
- Retention: carry rotated-out chain head as new segment genesis prev.
- Fold resolved points into design; trim open questions.

Refs #487

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
2026-07-26 07:35:16 +00:00
parent bc42836327
commit 60039f2eb3
+154 -67
View File
@@ -14,7 +14,7 @@ control-plane token minting, and (next) host-controller lifecycle
transitions — but each writes its own shape to its own sink. There is no transitions — but each writes its own shape to its own sink. There is no
shared envelope, no tamper-evidence, and no single place to search. Local shared envelope, no tamper-evidence, and no single place to search. Local
incident reconstruction means grepping several stores that don't agree on incident reconstruction means grepping several stores that don't agree on
field names, timestamps, or how a bottle is identified. field names, timestamps, or how a bottled agent is identified.
This PRD defines **one canonical audit-event contract** every producer This PRD defines **one canonical audit-event contract** every producer
emits into: emits into:
@@ -44,13 +44,13 @@ shared schema. Concretely:
`timestamp, bottle_slug, component, operator_action, ...`. The egress `timestamp, bottle_slug, component, operator_action, ...`. The egress
proxy and git-gate log their own ad-hoc lines. There is no common proxy and git-gate log their own ad-hoc lines. There is no common
`event_id`, `event_type`, or version, so cross-producer correlation `event_id`, `event_type`, or version, so cross-producer correlation
("what did bottle X do between its start and this rejected push?") is ("what did bottled agent X do between its start and this rejected push?") is
manual and lossy. manual and lossy.
- **No tamper-evidence.** The audit store is a plain SQLite table. Anyone - **No tamper-evidence.** The audit store is a plain SQLite table. Anyone
who can write the DB can delete or rewrite a row and leave no trace. who can write the DB can delete or rewrite a row and leave no trace.
Audit that an attacker (or a buggy agent) can silently rewrite is not Audit that an attacker (or a buggy agent) can silently rewrite is not
audit. audit.
- **Trusted and untrusted data are mixed.** A bottle is attributed by - **Trusted and untrusted data are mixed.** A bottled agent is attributed by
**source IP → slug** at the gateway (host-supplied, trustworthy). An **source IP → slug** at the gateway (host-supplied, trustworthy). An
agent can also *claim* things about itself in a tool call agent can also *claim* things about itself in a tool call
(agent-claimed, adversarial). Today nothing in the record marks which is (agent-claimed, adversarial). Today nothing in the record marks which is
@@ -67,7 +67,7 @@ shared schema. Concretely:
- A single `AuditEvent` envelope type, versioned, that every producer - A single `AuditEvent` envelope type, versioned, that every producer
emits. Fields are split into a **`trusted`** block (host-supplied: emits. Fields are split into a **`trusted`** block (host-supplied:
bottle slug from source-IP attribution, host wall-clock, producer bottled-agent slug from source-IP attribution, host wall-clock, producer
identity) and an **`untrusted`** block (anything the agent or a remote identity) and an **`untrusted`** block (anything the agent or a remote
claimed), and the split is structural, not a convention. claimed), and the split is structural, not a convention.
- **Canonical serialization** (`sort_keys`, `(",", ":")` separators, - **Canonical serialization** (`sort_keys`, `(",", ":")` separators,
@@ -81,8 +81,8 @@ shared schema. Concretely:
fully rebuildable** from it (`audit rebuild` reconstructs the DB and fully rebuildable** from it (`audit rebuild` reconstructs the DB and
re-verifies the chain). re-verifies the chain).
- **Local query** works with no paid platform and no egress: filter by - **Local query** works with no paid platform and no egress: filter by
bottle, event type, time range, and producer, and follow a bottle's bottled agent, event type, time range, and producer, and follow a bottled
events in order. agent's events in order.
- A **redaction rule** is enforced at the envelope boundary: known - A **redaction rule** is enforced at the envelope boundary: known
credential-shaped fields are rejected/redacted before a record is credential-shaped fields are rejected/redacted before a record is
written; the writer refuses raw secrets rather than storing them. written; the writer refuses raw secrets rather than storing them.
@@ -120,35 +120,49 @@ stable top level:
``` ```
{ {
"v": 1, // schema version (integer, bumped only on breaking change) "v": 1, // schema version bumped only on a breaking change
"id": "<uuid4>", // globally unique event id "id": "<uuid4>", // globally unique event id
"type": "egress.decision", // dotted event type from the registry "type": "egress.decision", // dotted event type from the registry
"seq": 1287, // per-writer monotonic sequence (gap-detectable)
"ts": { // --- chain / ordering (structural, host-owned) ---
"wall": "2026-07-26T18:22:04.113Z", // host wall-clock, RFC3339 UTC (TRUSTED) "epoch": 7, // writer-boot counter, bumped once per host-controller (writer) start
"mono": 90142.55 // host monotonic seconds since writer start (ordering) "seq": 1287, // monotonic sequence within this epoch (gap-detectable)
"prev": "<hex>", // hash of the previous record in the chain ("" for genesis)
"hash": "<hex>", // sha256(prev + canonical(this event with hash=""))
// --- trusted: everything the *host* established; authoritative ---
"trusted": {
"producer": "host-controller", // which host component wrote this
"host": "mac-studio-1",
"bottled_agent": "amber-fox-12", // slug from source-IP attribution (null for host-level events)
"ts_wall": "2026-07-26T18:22:04.113Z", // host wall-clock, RFC3339 UTC
"ts_mono": 90142.55 // host monotonic secs since this epoch's boot (intra-epoch ordering only)
}, },
"producer": "host-controller", // TRUSTED: which host component wrote this
"trusted": { // host-supplied, authoritative // --- untrusted: anything the agent or a remote claimed; never authoritative ---
"bottle": "amber-fox-12", // slug from source-IP attribution (may be null for host-level events) "untrusted": {
"host": "mac-studio-1"
},
"untrusted": { // agent- or remote-claimed; never authoritative
"reason": "npm install needs registry.npmjs.org", "reason": "npm install needs registry.npmjs.org",
"target": "registry.npmjs.org:443" "target": "registry.npmjs.org:443"
}, }
"hash": "<hex>", // sha256(prev_hash || canonical(this event with hash="" ))
"prev": "<hex>" // hash of the previous record in this writer's chain ("" for genesis)
} }
``` ```
The **`trusted` / `untrusted` split is the core invariant.** A producer may The **`trusted` / `untrusted` split is the core invariant.** A producer may
only place a field in `trusted` if the *host* established it (source-IP → only place a field in `trusted` if the *host* established it: the
slug attribution, the host's own clock, the producer's own identity). source-IP → `bottled_agent` slug attribution, the host's own clock
Everything an agent or a remote said goes in `untrusted`. A reader (or a (`ts_wall`/`ts_mono`), and the producer's own identity. **`producer` and
future policy engine) can therefore trust `trusted.bottle` for attribution `ts_*` live inside `trusted` on purpose** — they are host-supplied, so
and treat `untrusted.*` as adversarial claims — the distinction the current grouping them there (rather than as loose top-level fields) keeps the
stores lack. "authoritative ⇔ inside `trusted`" rule structural, with nothing
host-established leaking outside it. Everything an agent or a remote said
goes in `untrusted`. A reader (or a future policy engine) can therefore
trust `trusted.bottled_agent` for attribution and treat `untrusted.*` as
adversarial claims — the distinction the current stores lack.
Only the small structural set — `v`, `id`, `type`, `epoch`, `seq`, `prev`,
`hash` — sits at the top level; it is host-owned too, but it is chain
metadata rather than event data, so it stays out of the `trusted` body to
keep that body purely about *what happened*.
### Canonical serialization + hash chain ### Canonical serialization + hash chain
@@ -174,20 +188,42 @@ the journal an append-only Merkle-style chain: editing or deleting record
break is local and points at the tampered record. Verification needs only break is local and points at the tampered record. Verification needs only
the journal itself (no keys), so it runs offline and in CI. the journal itself (no keys), so it runs offline and in CI.
### Single writer; ordering across restarts
**Decided: one writer per host** (reviewed — the host controller owns it).
Producers hand events to the host controller, which is the sole appender,
so the chain has one well-defined total order and one `seq`/`epoch`
counter. This ties audit availability to the host controller being up,
which is acceptable because the host controller already gates every
lifecycle transition; per-producer chains are noted only as a future
scaling path, not built now.
**Restarts** are handled by the chain, not the clock. `ts_mono` resets to
~0 on every writer start, so it orders events only *within* one boot. On
start the writer:
1. reads the last line of the journal, adopts its `hash` as the next
record's `prev` (the chain is continuous across the restart), and
2. bumps `epoch` (persisted alongside the chain head) and resets `seq` to
0 for the new boot.
Total order is therefore `(epoch, seq)` — monotonic across restarts by
construction — with `ts_wall` for human reading and `ts_mono` for
sub-second ordering inside an epoch. A crash mid-append truncates at most
the last (partial) line; the verifier flags it and replay resumes from the
last intact record.
### Journal (source of truth) + SQLite index (rebuildable) ### Journal (source of truth) + SQLite index (rebuildable)
- **Journal:** one append-only JSONL file per host (path from `paths.py`, - **Journal:** one append-only JSONL file per host (path from `paths.py`,
alongside `host_db_path()`), one canonical event per line, opened alongside `host_db_path()`), one canonical event per line, opened
`O_APPEND`. This is authoritative. A single writer per host owns appends `O_APPEND`. This is authoritative.
(producers hand events to it) so the chain has one well-defined order —
the host controller is the natural owner since it already gates the
lifecycle.
- **Index:** a new `audit_events` table via the existing `DbStore` / - **Index:** a new `audit_events` table via the existing `DbStore` /
`TableMigrations` machinery, holding the envelope columns plus JSON `TableMigrations` machinery, holding the envelope columns plus JSON
blobs, indexed on `(bottle, type, wall_ts)`. It is a **derived cache**: blobs, indexed on `(bottled_agent, type, ts_wall)`. It is a **derived
`audit rebuild` truncates and replays the journal, re-verifying the cache**: `audit rebuild` truncates and replays the journal, re-verifying
chain as it goes. If the DB is deleted or drifts, it is regenerated from the chain as it goes. If the DB is deleted or drifts, it is regenerated
the journal with no data loss. (This supersedes the free-standing from the journal with no data loss. (This supersedes the free-standing
`supervise_audit_entries` table, which becomes a view/producer onto the `supervise_audit_entries` table, which becomes a view/producer onto the
new index.) new index.)
@@ -196,8 +232,10 @@ the journal itself (no keys), so it runs offline and in CI.
Dotted `type` names, grouped; the registry is a table mapping type → Dotted `type` names, grouped; the registry is a table mapping type →
required `untrusted` keys so producers and the verifier agree on shape: required `untrusted` keys so producers and the verifier agree on shape:
- **lifecycle.*** — `lifecycle.bottle_start`, `lifecycle.bottle_stop`, - **lifecycle.*** — `lifecycle.bottled_agent_start`,
`lifecycle.bottle_crash` (producer: host-controller, #468). `lifecycle.bottled_agent_stop`, `lifecycle.bottled_agent_crash`
(producer: host-controller, #468). Leaf names use `bottled_agent` to match
the `trusted.bottled_agent` field — one term for the subject everywhere.
- **decision.*** — `decision.proposed`, `decision.resolved` - **decision.*** — `decision.proposed`, `decision.resolved`
(producer: supervise; carries operator action + justification, replacing (producer: supervise; carries operator action + justification, replacing
PRD 0013's row shape). PRD 0013's row shape).
@@ -213,13 +251,41 @@ re-typing a field bumps `v`.
### Redaction rule ### Redaction rule
The envelope constructor enforces a deny-list at write time: a field whose Redaction runs at the envelope boundary, before a record is written, in two
key matches known credential shapes (`token`, `secret`, `password`, layers:
`authorization`, `*_key`, JWT-shaped values) is rejected — the producer
must pass a reference (a token *id* or `sha256` fingerprint), never the raw 1. **Key deny-list (structural).** A field whose *key* matches a known
value. `auth.token_minted` therefore records the token id and role, not the credential shape (`token`, `secret`, `password`, `authorization`,
JWT. Redaction is enforced structurally so a producer *cannot* accidentally `*_key`) is refused — the producer must pass a reference (a token *id*
write a secret into the audit log. or `sha256` fingerprint), never the raw value. `auth.token_minted`
therefore records the token id and role, not the JWT. This is the
primary guard: it is cheap, deterministic, and catches the intended
mistake (a producer stuffing a credential into a named field).
2. **Value scan — reuse the egress DLP detectors.** Per review, the value
layer reuses the *same* deterministic credential-shape detectors the
egress proxy already ships:
`bot_bottle/gateway/egress/dlp_detectors.py` —
`scan_token_patterns` / `redact_tokens` (and `scan_known_secrets` for
host-known secret material). They are pure-Python, mitmproxy-free, and
already the project's source of truth for "what a leaked credential
looks like," so a single detector set governs both what may leave over
the wire and what may land in the journal — they can't drift apart.
**Scoped deliberately:** only the pattern/known-secret detectors are
reused, **not** `scan_entropy`. Entropy scoring is tuned for large
streamed request bodies; on the short, high-entropy structured values an
audit event legitimately carries (hashes, uuids, base64 ids) it would
false-positive and start redacting the very fingerprints the log needs.
So the shared layer is the deterministic detectors; entropy stays an
egress-only concern. (This is the "evaluate how reasonable that is" from
review: reuse the deterministic detectors — yes; share the entropy
heuristic — no.)
On a value-layer match the default is **redact** (scrub to a placeholder
and keep the event) rather than drop, so a producer bug can never make an
audit event vanish; the key deny-list stays a hard refusal because a
credential in a named field is always a producer bug worth surfacing.
## Implementation chunks ## Implementation chunks
@@ -227,37 +293,58 @@ write a secret into the audit log.
right after #468. right after #468.
2. **Envelope + canonical + chain core.** `AuditEvent` dataclass, 2. **Envelope + canonical + chain core.** `AuditEvent` dataclass,
`canonical()`, chain hashing, and the single-writer journal appender in `canonical()`, chain hashing, and the single-writer journal appender in
`bot_bottle/store/` (reusing `sha256_hex`); unit tests for `bot_bottle/store/` (reusing `sha256_hex`); redaction wired to the
determinism, chain-break detection, and redaction refusal. existing `gateway/egress/dlp_detectors` (`scan_token_patterns` /
`redact_tokens`); unit tests for determinism, chain-break detection,
`epoch`/`seq` continuity across a simulated restart, and redaction of
both a deny-listed key and a token-shaped value.
3. **SQLite index + `audit rebuild` / `audit verify` CLI.** New 3. **SQLite index + `audit rebuild` / `audit verify` CLI.** New
`audit_events` migration; replay-from-journal; offline chain verifier; `audit_events` migration; replay-from-journal; offline chain verifier;
local query commands (by bottle / type / time / producer). local query commands (by bottled-agent / type / time / producer).
4. **Host controller as first producer (#468).** Wire `lifecycle.*` 4. **Host controller as first producer (#468).** Wire
emission into the host controller's start/stop/crash paths. `lifecycle.bottled_agent_*` emission into the host controller's
start/stop/crash paths; establish the `epoch` bump + chain-head carry on
writer restart here (the host controller owns the single writer).
5. **Migrate existing producers.** Re-emit supervise `decision.*` (retiring 5. **Migrate existing producers.** Re-emit supervise `decision.*` (retiring
the standalone `supervise_audit_entries` shape behind the index), egress the standalone `supervise_audit_entries` shape behind the index), egress
`egress.*`, git-gate `forge.*`, control-plane `auth.*`. `egress.*`, git-gate `forge.*`, control-plane `auth.*`.
6. **(follow-up.)** Cross-host merge transport (#324); per-writer signing + 6. **(follow-up.)** Cross-host merge transport (#324); per-writer signing +
external anchoring on the chain head; retention/rotation policy. external anchoring on the chain head; retention/rotation policy.
## Resolved in review (#495)
- **Single writer per host — decided.** The host controller owns the sole
appender; per-producer chains are a future scaling path only. (Design →
*Single writer; ordering across restarts*.)
- **Restarts — decided.** An `epoch` counter (bumped per writer boot) plus
carrying the last chain head as the next `prev` gives a total order of
`(epoch, seq)` that survives restarts; `ts_mono` orders only within an
epoch. (Design → *ordering across restarts*.)
- **`ts_*` and `producer` belong in `trusted`.** They are host-supplied, so
they now sit inside the `trusted` block; only chain metadata stays at the
top level. (Design → *The envelope*.)
- **Subject term is `bottled_agent` everywhere** — the `trusted` field and
the `lifecycle.bottled_agent_*` leaf names. (Design → *The envelope* /
*Event registry*.)
- **Retention head-carry — yes.** When a journal segment is rotated out,
the new segment's genesis `prev` is the rotated-out head, so the verifier
still trusts the current head across a rotation. (Folds into the
retention follow-up.)
- **Redaction reuses the egress detectors — yes, scoped.** Reuse the
deterministic `dlp_detectors` (`scan_token_patterns` / `redact_tokens` /
`scan_known_secrets`); exclude `scan_entropy` as brittle on the short,
high-entropy structured values audit records carry. (Design → *Redaction
rule*.)
## Open questions ## Open questions
- **One writer per host vs. per producer chains.** A single appender gives - **Value-scan cost on the hot path.** The single writer runs the reused
one total order but makes every producer depend on the host controller detectors on every event's `untrusted` block inline. Is that cheap enough
being up. Alternative: one chain *per producer* (independent `seq`/chain at lifecycle-event volume, or should the value scan move to index-build
head), merged at query time by `(wall, mono)`. Leaning single-writer for time (journal stays raw, index stores the redacted view)? Leaning inline
v1 (simpler verification, matches the host controller owning lifecycle), so the raw journal never contains a leaked value in the first place.
with per-producer chains noted as the scaling path. — feedback wanted. - **`epoch` persistence location.** Store the per-writer `epoch` + chain
- **Monotonic clock across restarts.** `ts.mono` resets when the writer head in the SQLite index (rebuildable, but then the writer needs the DB
restarts; is `seq` + `wall` enough for ordering across a restart, or does at boot) or in a tiny sidecar file next to the journal (independent of
the chain need an explicit `epoch` counter bumped per writer boot? the index)? Leaning sidecar, so the writer can start and append without
(Leaning: carry the last chain head across restart, so ordering follows the index present.
the chain, not `mono`.)
- **Redaction: reject vs. redact.** Should a credential-shaped field hard-
fail the write (surfacing the producer bug loudly) or silently redact to
`"<redacted>"`? Leaning **reject in tests / redact in prod** behind a
flag, so a producer bug can't drop an event entirely in the field.
- **Retention interaction with the chain.** When an old journal segment is
rotated out, the verifier must still trust the current head. Carry the
rotated-out head as the new segment's genesis `prev`? (Tracked with the
retention follow-up.)