docs(prd): address review #495 on audit-event schema
- Move ts_wall/ts_mono and producer inside the trusted block (host-supplied). - Rename subject to bottled_agent everywhere (field + lifecycle.bottled_agent_* leaves). - Add explicit epoch (writer-boot) counter + chain-head carry for ordering across host-controller restarts. - Commit to a single writer per host (host controller owns it). - Reuse egress dlp_detectors (scan_token_patterns/redact_tokens) for redaction; exclude scan_entropy as brittle on structured audit values. - Retention: carry rotated-out chain head as new segment genesis prev. - Fold resolved points into design; trim open questions. Refs #487 Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
@@ -14,7 +14,7 @@ control-plane token minting, and (next) host-controller lifecycle
|
|||||||
transitions — but each writes its own shape to its own sink. There is no
|
transitions — but each writes its own shape to its own sink. There is no
|
||||||
shared envelope, no tamper-evidence, and no single place to search. Local
|
shared envelope, no tamper-evidence, and no single place to search. Local
|
||||||
incident reconstruction means grepping several stores that don't agree on
|
incident reconstruction means grepping several stores that don't agree on
|
||||||
field names, timestamps, or how a bottle is identified.
|
field names, timestamps, or how a bottled agent is identified.
|
||||||
|
|
||||||
This PRD defines **one canonical audit-event contract** every producer
|
This PRD defines **one canonical audit-event contract** every producer
|
||||||
emits into:
|
emits into:
|
||||||
@@ -44,13 +44,13 @@ shared schema. Concretely:
|
|||||||
`timestamp, bottle_slug, component, operator_action, ...`. The egress
|
`timestamp, bottle_slug, component, operator_action, ...`. The egress
|
||||||
proxy and git-gate log their own ad-hoc lines. There is no common
|
proxy and git-gate log their own ad-hoc lines. There is no common
|
||||||
`event_id`, `event_type`, or version, so cross-producer correlation
|
`event_id`, `event_type`, or version, so cross-producer correlation
|
||||||
("what did bottle X do between its start and this rejected push?") is
|
("what did bottled agent X do between its start and this rejected push?") is
|
||||||
manual and lossy.
|
manual and lossy.
|
||||||
- **No tamper-evidence.** The audit store is a plain SQLite table. Anyone
|
- **No tamper-evidence.** The audit store is a plain SQLite table. Anyone
|
||||||
who can write the DB can delete or rewrite a row and leave no trace.
|
who can write the DB can delete or rewrite a row and leave no trace.
|
||||||
Audit that an attacker (or a buggy agent) can silently rewrite is not
|
Audit that an attacker (or a buggy agent) can silently rewrite is not
|
||||||
audit.
|
audit.
|
||||||
- **Trusted and untrusted data are mixed.** A bottle is attributed by
|
- **Trusted and untrusted data are mixed.** A bottled agent is attributed by
|
||||||
**source IP → slug** at the gateway (host-supplied, trustworthy). An
|
**source IP → slug** at the gateway (host-supplied, trustworthy). An
|
||||||
agent can also *claim* things about itself in a tool call
|
agent can also *claim* things about itself in a tool call
|
||||||
(agent-claimed, adversarial). Today nothing in the record marks which is
|
(agent-claimed, adversarial). Today nothing in the record marks which is
|
||||||
@@ -67,7 +67,7 @@ shared schema. Concretely:
|
|||||||
|
|
||||||
- A single `AuditEvent` envelope type, versioned, that every producer
|
- A single `AuditEvent` envelope type, versioned, that every producer
|
||||||
emits. Fields are split into a **`trusted`** block (host-supplied:
|
emits. Fields are split into a **`trusted`** block (host-supplied:
|
||||||
bottle slug from source-IP attribution, host wall-clock, producer
|
bottled-agent slug from source-IP attribution, host wall-clock, producer
|
||||||
identity) and an **`untrusted`** block (anything the agent or a remote
|
identity) and an **`untrusted`** block (anything the agent or a remote
|
||||||
claimed), and the split is structural, not a convention.
|
claimed), and the split is structural, not a convention.
|
||||||
- **Canonical serialization** (`sort_keys`, `(",", ":")` separators,
|
- **Canonical serialization** (`sort_keys`, `(",", ":")` separators,
|
||||||
@@ -81,8 +81,8 @@ shared schema. Concretely:
|
|||||||
fully rebuildable** from it (`audit rebuild` reconstructs the DB and
|
fully rebuildable** from it (`audit rebuild` reconstructs the DB and
|
||||||
re-verifies the chain).
|
re-verifies the chain).
|
||||||
- **Local query** works with no paid platform and no egress: filter by
|
- **Local query** works with no paid platform and no egress: filter by
|
||||||
bottle, event type, time range, and producer, and follow a bottle's
|
bottled agent, event type, time range, and producer, and follow a bottled
|
||||||
events in order.
|
agent's events in order.
|
||||||
- A **redaction rule** is enforced at the envelope boundary: known
|
- A **redaction rule** is enforced at the envelope boundary: known
|
||||||
credential-shaped fields are rejected/redacted before a record is
|
credential-shaped fields are rejected/redacted before a record is
|
||||||
written; the writer refuses raw secrets rather than storing them.
|
written; the writer refuses raw secrets rather than storing them.
|
||||||
@@ -120,35 +120,49 @@ stable top level:
|
|||||||
|
|
||||||
```
|
```
|
||||||
{
|
{
|
||||||
"v": 1, // schema version (integer, bumped only on breaking change)
|
"v": 1, // schema version — bumped only on a breaking change
|
||||||
"id": "<uuid4>", // globally unique event id
|
"id": "<uuid4>", // globally unique event id
|
||||||
"type": "egress.decision", // dotted event type from the registry
|
"type": "egress.decision", // dotted event type from the registry
|
||||||
"seq": 1287, // per-writer monotonic sequence (gap-detectable)
|
|
||||||
"ts": {
|
// --- chain / ordering (structural, host-owned) ---
|
||||||
"wall": "2026-07-26T18:22:04.113Z", // host wall-clock, RFC3339 UTC (TRUSTED)
|
"epoch": 7, // writer-boot counter, bumped once per host-controller (writer) start
|
||||||
"mono": 90142.55 // host monotonic seconds since writer start (ordering)
|
"seq": 1287, // monotonic sequence within this epoch (gap-detectable)
|
||||||
|
"prev": "<hex>", // hash of the previous record in the chain ("" for genesis)
|
||||||
|
"hash": "<hex>", // sha256(prev + canonical(this event with hash=""))
|
||||||
|
|
||||||
|
// --- trusted: everything the *host* established; authoritative ---
|
||||||
|
"trusted": {
|
||||||
|
"producer": "host-controller", // which host component wrote this
|
||||||
|
"host": "mac-studio-1",
|
||||||
|
"bottled_agent": "amber-fox-12", // slug from source-IP attribution (null for host-level events)
|
||||||
|
"ts_wall": "2026-07-26T18:22:04.113Z", // host wall-clock, RFC3339 UTC
|
||||||
|
"ts_mono": 90142.55 // host monotonic secs since this epoch's boot (intra-epoch ordering only)
|
||||||
},
|
},
|
||||||
"producer": "host-controller", // TRUSTED: which host component wrote this
|
|
||||||
"trusted": { // host-supplied, authoritative
|
// --- untrusted: anything the agent or a remote claimed; never authoritative ---
|
||||||
"bottle": "amber-fox-12", // slug from source-IP attribution (may be null for host-level events)
|
"untrusted": {
|
||||||
"host": "mac-studio-1"
|
|
||||||
},
|
|
||||||
"untrusted": { // agent- or remote-claimed; never authoritative
|
|
||||||
"reason": "npm install needs registry.npmjs.org",
|
"reason": "npm install needs registry.npmjs.org",
|
||||||
"target": "registry.npmjs.org:443"
|
"target": "registry.npmjs.org:443"
|
||||||
},
|
}
|
||||||
"hash": "<hex>", // sha256(prev_hash || canonical(this event with hash="" ))
|
|
||||||
"prev": "<hex>" // hash of the previous record in this writer's chain ("" for genesis)
|
|
||||||
}
|
}
|
||||||
```
|
```
|
||||||
|
|
||||||
The **`trusted` / `untrusted` split is the core invariant.** A producer may
|
The **`trusted` / `untrusted` split is the core invariant.** A producer may
|
||||||
only place a field in `trusted` if the *host* established it (source-IP →
|
only place a field in `trusted` if the *host* established it: the
|
||||||
slug attribution, the host's own clock, the producer's own identity).
|
source-IP → `bottled_agent` slug attribution, the host's own clock
|
||||||
Everything an agent or a remote said goes in `untrusted`. A reader (or a
|
(`ts_wall`/`ts_mono`), and the producer's own identity. **`producer` and
|
||||||
future policy engine) can therefore trust `trusted.bottle` for attribution
|
`ts_*` live inside `trusted` on purpose** — they are host-supplied, so
|
||||||
and treat `untrusted.*` as adversarial claims — the distinction the current
|
grouping them there (rather than as loose top-level fields) keeps the
|
||||||
stores lack.
|
"authoritative ⇔ inside `trusted`" rule structural, with nothing
|
||||||
|
host-established leaking outside it. Everything an agent or a remote said
|
||||||
|
goes in `untrusted`. A reader (or a future policy engine) can therefore
|
||||||
|
trust `trusted.bottled_agent` for attribution and treat `untrusted.*` as
|
||||||
|
adversarial claims — the distinction the current stores lack.
|
||||||
|
|
||||||
|
Only the small structural set — `v`, `id`, `type`, `epoch`, `seq`, `prev`,
|
||||||
|
`hash` — sits at the top level; it is host-owned too, but it is chain
|
||||||
|
metadata rather than event data, so it stays out of the `trusted` body to
|
||||||
|
keep that body purely about *what happened*.
|
||||||
|
|
||||||
### Canonical serialization + hash chain
|
### Canonical serialization + hash chain
|
||||||
|
|
||||||
@@ -174,20 +188,42 @@ the journal an append-only Merkle-style chain: editing or deleting record
|
|||||||
break is local and points at the tampered record. Verification needs only
|
break is local and points at the tampered record. Verification needs only
|
||||||
the journal itself (no keys), so it runs offline and in CI.
|
the journal itself (no keys), so it runs offline and in CI.
|
||||||
|
|
||||||
|
### Single writer; ordering across restarts
|
||||||
|
|
||||||
|
**Decided: one writer per host** (reviewed — the host controller owns it).
|
||||||
|
Producers hand events to the host controller, which is the sole appender,
|
||||||
|
so the chain has one well-defined total order and one `seq`/`epoch`
|
||||||
|
counter. This ties audit availability to the host controller being up,
|
||||||
|
which is acceptable because the host controller already gates every
|
||||||
|
lifecycle transition; per-producer chains are noted only as a future
|
||||||
|
scaling path, not built now.
|
||||||
|
|
||||||
|
**Restarts** are handled by the chain, not the clock. `ts_mono` resets to
|
||||||
|
~0 on every writer start, so it orders events only *within* one boot. On
|
||||||
|
start the writer:
|
||||||
|
|
||||||
|
1. reads the last line of the journal, adopts its `hash` as the next
|
||||||
|
record's `prev` (the chain is continuous across the restart), and
|
||||||
|
2. bumps `epoch` (persisted alongside the chain head) and resets `seq` to
|
||||||
|
0 for the new boot.
|
||||||
|
|
||||||
|
Total order is therefore `(epoch, seq)` — monotonic across restarts by
|
||||||
|
construction — with `ts_wall` for human reading and `ts_mono` for
|
||||||
|
sub-second ordering inside an epoch. A crash mid-append truncates at most
|
||||||
|
the last (partial) line; the verifier flags it and replay resumes from the
|
||||||
|
last intact record.
|
||||||
|
|
||||||
### Journal (source of truth) + SQLite index (rebuildable)
|
### Journal (source of truth) + SQLite index (rebuildable)
|
||||||
|
|
||||||
- **Journal:** one append-only JSONL file per host (path from `paths.py`,
|
- **Journal:** one append-only JSONL file per host (path from `paths.py`,
|
||||||
alongside `host_db_path()`), one canonical event per line, opened
|
alongside `host_db_path()`), one canonical event per line, opened
|
||||||
`O_APPEND`. This is authoritative. A single writer per host owns appends
|
`O_APPEND`. This is authoritative.
|
||||||
(producers hand events to it) so the chain has one well-defined order —
|
|
||||||
the host controller is the natural owner since it already gates the
|
|
||||||
lifecycle.
|
|
||||||
- **Index:** a new `audit_events` table via the existing `DbStore` /
|
- **Index:** a new `audit_events` table via the existing `DbStore` /
|
||||||
`TableMigrations` machinery, holding the envelope columns plus JSON
|
`TableMigrations` machinery, holding the envelope columns plus JSON
|
||||||
blobs, indexed on `(bottle, type, wall_ts)`. It is a **derived cache**:
|
blobs, indexed on `(bottled_agent, type, ts_wall)`. It is a **derived
|
||||||
`audit rebuild` truncates and replays the journal, re-verifying the
|
cache**: `audit rebuild` truncates and replays the journal, re-verifying
|
||||||
chain as it goes. If the DB is deleted or drifts, it is regenerated from
|
the chain as it goes. If the DB is deleted or drifts, it is regenerated
|
||||||
the journal with no data loss. (This supersedes the free-standing
|
from the journal with no data loss. (This supersedes the free-standing
|
||||||
`supervise_audit_entries` table, which becomes a view/producer onto the
|
`supervise_audit_entries` table, which becomes a view/producer onto the
|
||||||
new index.)
|
new index.)
|
||||||
|
|
||||||
@@ -196,8 +232,10 @@ the journal itself (no keys), so it runs offline and in CI.
|
|||||||
Dotted `type` names, grouped; the registry is a table mapping type →
|
Dotted `type` names, grouped; the registry is a table mapping type →
|
||||||
required `untrusted` keys so producers and the verifier agree on shape:
|
required `untrusted` keys so producers and the verifier agree on shape:
|
||||||
|
|
||||||
- **lifecycle.*** — `lifecycle.bottle_start`, `lifecycle.bottle_stop`,
|
- **lifecycle.*** — `lifecycle.bottled_agent_start`,
|
||||||
`lifecycle.bottle_crash` (producer: host-controller, #468).
|
`lifecycle.bottled_agent_stop`, `lifecycle.bottled_agent_crash`
|
||||||
|
(producer: host-controller, #468). Leaf names use `bottled_agent` to match
|
||||||
|
the `trusted.bottled_agent` field — one term for the subject everywhere.
|
||||||
- **decision.*** — `decision.proposed`, `decision.resolved`
|
- **decision.*** — `decision.proposed`, `decision.resolved`
|
||||||
(producer: supervise; carries operator action + justification, replacing
|
(producer: supervise; carries operator action + justification, replacing
|
||||||
PRD 0013's row shape).
|
PRD 0013's row shape).
|
||||||
@@ -213,13 +251,41 @@ re-typing a field bumps `v`.
|
|||||||
|
|
||||||
### Redaction rule
|
### Redaction rule
|
||||||
|
|
||||||
The envelope constructor enforces a deny-list at write time: a field whose
|
Redaction runs at the envelope boundary, before a record is written, in two
|
||||||
key matches known credential shapes (`token`, `secret`, `password`,
|
layers:
|
||||||
`authorization`, `*_key`, JWT-shaped values) is rejected — the producer
|
|
||||||
must pass a reference (a token *id* or `sha256` fingerprint), never the raw
|
1. **Key deny-list (structural).** A field whose *key* matches a known
|
||||||
value. `auth.token_minted` therefore records the token id and role, not the
|
credential shape (`token`, `secret`, `password`, `authorization`,
|
||||||
JWT. Redaction is enforced structurally so a producer *cannot* accidentally
|
`*_key`) is refused — the producer must pass a reference (a token *id*
|
||||||
write a secret into the audit log.
|
or `sha256` fingerprint), never the raw value. `auth.token_minted`
|
||||||
|
therefore records the token id and role, not the JWT. This is the
|
||||||
|
primary guard: it is cheap, deterministic, and catches the intended
|
||||||
|
mistake (a producer stuffing a credential into a named field).
|
||||||
|
|
||||||
|
2. **Value scan — reuse the egress DLP detectors.** Per review, the value
|
||||||
|
layer reuses the *same* deterministic credential-shape detectors the
|
||||||
|
egress proxy already ships:
|
||||||
|
`bot_bottle/gateway/egress/dlp_detectors.py` —
|
||||||
|
`scan_token_patterns` / `redact_tokens` (and `scan_known_secrets` for
|
||||||
|
host-known secret material). They are pure-Python, mitmproxy-free, and
|
||||||
|
already the project's source of truth for "what a leaked credential
|
||||||
|
looks like," so a single detector set governs both what may leave over
|
||||||
|
the wire and what may land in the journal — they can't drift apart.
|
||||||
|
|
||||||
|
**Scoped deliberately:** only the pattern/known-secret detectors are
|
||||||
|
reused, **not** `scan_entropy`. Entropy scoring is tuned for large
|
||||||
|
streamed request bodies; on the short, high-entropy structured values an
|
||||||
|
audit event legitimately carries (hashes, uuids, base64 ids) it would
|
||||||
|
false-positive and start redacting the very fingerprints the log needs.
|
||||||
|
So the shared layer is the deterministic detectors; entropy stays an
|
||||||
|
egress-only concern. (This is the "evaluate how reasonable that is" from
|
||||||
|
review: reuse the deterministic detectors — yes; share the entropy
|
||||||
|
heuristic — no.)
|
||||||
|
|
||||||
|
On a value-layer match the default is **redact** (scrub to a placeholder
|
||||||
|
and keep the event) rather than drop, so a producer bug can never make an
|
||||||
|
audit event vanish; the key deny-list stays a hard refusal because a
|
||||||
|
credential in a named field is always a producer bug worth surfacing.
|
||||||
|
|
||||||
## Implementation chunks
|
## Implementation chunks
|
||||||
|
|
||||||
@@ -227,37 +293,58 @@ write a secret into the audit log.
|
|||||||
right after #468.
|
right after #468.
|
||||||
2. **Envelope + canonical + chain core.** `AuditEvent` dataclass,
|
2. **Envelope + canonical + chain core.** `AuditEvent` dataclass,
|
||||||
`canonical()`, chain hashing, and the single-writer journal appender in
|
`canonical()`, chain hashing, and the single-writer journal appender in
|
||||||
`bot_bottle/store/` (reusing `sha256_hex`); unit tests for
|
`bot_bottle/store/` (reusing `sha256_hex`); redaction wired to the
|
||||||
determinism, chain-break detection, and redaction refusal.
|
existing `gateway/egress/dlp_detectors` (`scan_token_patterns` /
|
||||||
|
`redact_tokens`); unit tests for determinism, chain-break detection,
|
||||||
|
`epoch`/`seq` continuity across a simulated restart, and redaction of
|
||||||
|
both a deny-listed key and a token-shaped value.
|
||||||
3. **SQLite index + `audit rebuild` / `audit verify` CLI.** New
|
3. **SQLite index + `audit rebuild` / `audit verify` CLI.** New
|
||||||
`audit_events` migration; replay-from-journal; offline chain verifier;
|
`audit_events` migration; replay-from-journal; offline chain verifier;
|
||||||
local query commands (by bottle / type / time / producer).
|
local query commands (by bottled-agent / type / time / producer).
|
||||||
4. **Host controller as first producer (#468).** Wire `lifecycle.*`
|
4. **Host controller as first producer (#468).** Wire
|
||||||
emission into the host controller's start/stop/crash paths.
|
`lifecycle.bottled_agent_*` emission into the host controller's
|
||||||
|
start/stop/crash paths; establish the `epoch` bump + chain-head carry on
|
||||||
|
writer restart here (the host controller owns the single writer).
|
||||||
5. **Migrate existing producers.** Re-emit supervise `decision.*` (retiring
|
5. **Migrate existing producers.** Re-emit supervise `decision.*` (retiring
|
||||||
the standalone `supervise_audit_entries` shape behind the index), egress
|
the standalone `supervise_audit_entries` shape behind the index), egress
|
||||||
`egress.*`, git-gate `forge.*`, control-plane `auth.*`.
|
`egress.*`, git-gate `forge.*`, control-plane `auth.*`.
|
||||||
6. **(follow-up.)** Cross-host merge transport (#324); per-writer signing +
|
6. **(follow-up.)** Cross-host merge transport (#324); per-writer signing +
|
||||||
external anchoring on the chain head; retention/rotation policy.
|
external anchoring on the chain head; retention/rotation policy.
|
||||||
|
|
||||||
|
## Resolved in review (#495)
|
||||||
|
|
||||||
|
- **Single writer per host — decided.** The host controller owns the sole
|
||||||
|
appender; per-producer chains are a future scaling path only. (Design →
|
||||||
|
*Single writer; ordering across restarts*.)
|
||||||
|
- **Restarts — decided.** An `epoch` counter (bumped per writer boot) plus
|
||||||
|
carrying the last chain head as the next `prev` gives a total order of
|
||||||
|
`(epoch, seq)` that survives restarts; `ts_mono` orders only within an
|
||||||
|
epoch. (Design → *ordering across restarts*.)
|
||||||
|
- **`ts_*` and `producer` belong in `trusted`.** They are host-supplied, so
|
||||||
|
they now sit inside the `trusted` block; only chain metadata stays at the
|
||||||
|
top level. (Design → *The envelope*.)
|
||||||
|
- **Subject term is `bottled_agent` everywhere** — the `trusted` field and
|
||||||
|
the `lifecycle.bottled_agent_*` leaf names. (Design → *The envelope* /
|
||||||
|
*Event registry*.)
|
||||||
|
- **Retention head-carry — yes.** When a journal segment is rotated out,
|
||||||
|
the new segment's genesis `prev` is the rotated-out head, so the verifier
|
||||||
|
still trusts the current head across a rotation. (Folds into the
|
||||||
|
retention follow-up.)
|
||||||
|
- **Redaction reuses the egress detectors — yes, scoped.** Reuse the
|
||||||
|
deterministic `dlp_detectors` (`scan_token_patterns` / `redact_tokens` /
|
||||||
|
`scan_known_secrets`); exclude `scan_entropy` as brittle on the short,
|
||||||
|
high-entropy structured values audit records carry. (Design → *Redaction
|
||||||
|
rule*.)
|
||||||
|
|
||||||
## Open questions
|
## Open questions
|
||||||
|
|
||||||
- **One writer per host vs. per producer chains.** A single appender gives
|
- **Value-scan cost on the hot path.** The single writer runs the reused
|
||||||
one total order but makes every producer depend on the host controller
|
detectors on every event's `untrusted` block inline. Is that cheap enough
|
||||||
being up. Alternative: one chain *per producer* (independent `seq`/chain
|
at lifecycle-event volume, or should the value scan move to index-build
|
||||||
head), merged at query time by `(wall, mono)`. Leaning single-writer for
|
time (journal stays raw, index stores the redacted view)? Leaning inline
|
||||||
v1 (simpler verification, matches the host controller owning lifecycle),
|
so the raw journal never contains a leaked value in the first place.
|
||||||
with per-producer chains noted as the scaling path. — feedback wanted.
|
- **`epoch` persistence location.** Store the per-writer `epoch` + chain
|
||||||
- **Monotonic clock across restarts.** `ts.mono` resets when the writer
|
head in the SQLite index (rebuildable, but then the writer needs the DB
|
||||||
restarts; is `seq` + `wall` enough for ordering across a restart, or does
|
at boot) or in a tiny sidecar file next to the journal (independent of
|
||||||
the chain need an explicit `epoch` counter bumped per writer boot?
|
the index)? Leaning sidecar, so the writer can start and append without
|
||||||
(Leaning: carry the last chain head across restart, so ordering follows
|
the index present.
|
||||||
the chain, not `mono`.)
|
|
||||||
- **Redaction: reject vs. redact.** Should a credential-shaped field hard-
|
|
||||||
fail the write (surfacing the producer bug loudly) or silently redact to
|
|
||||||
`"<redacted>"`? Leaning **reject in tests / redact in prod** behind a
|
|
||||||
flag, so a producer bug can't drop an event entirely in the field.
|
|
||||||
- **Retention interaction with the chain.** When an old journal segment is
|
|
||||||
rotated out, the verifier must still trust the current head. Carry the
|
|
||||||
rotated-out head as the new segment's genesis `prev`? (Tracked with the
|
|
||||||
retention follow-up.)
|
|
||||||
|
|||||||
Reference in New Issue
Block a user