PRD: canonical tamper-evident audit-event schema (#487) #495

Closed
didericis wants to merge 5 commits from didericis/prd-audit-event-schema AGit into main
Showing only changes of commit f3664dea9f - Show all commits
+29 -12
View File
@@ -21,7 +21,8 @@ emits into:
1. A **versioned envelope** — schema version, event id, event/observed
timestamps, host-attributed identity (`bottle`/`bottled_agent`/
`activation`), provenance (`manifest_digest`, `policy_version`),
`activation`), provenance (`manifest_digest` — the manifest is the
policy — and `engine` = bot-bottle version/SHA),
host-observed `actor`/`action`/`resource`/`outcome`, correlation/
causation ids, a sensitivity class, a typed payload, and an explicit
**trust boundary** between host-supplied and agent-claimed fields.
8
@@ -152,11 +153,11 @@ stable top level:
// ---- attribution + provenance (host-established) ----
"producer": "egress", // host component that emitted the event
"host": "mac-studio-1",
"engine": "bot-bottle/0.1.0+abc1234", // bot-bottle version + git SHA of the enforcing host code
"bottle": "amber-fox", // bottle (container/VM) identity
"bottled_agent": "amber-fox-12", // bottled-agent slug from source-IP attribution (null for host-level events)
"activation": "01J8Z...", // activation id: one run/session of the bottled agent (null if n/a)
"manifest_digest": "sha256:9f2…", // digest of the manifest the bottled agent is running (null if n/a)
"policy_version": "egress-2026-07-20", // policy/config version in force at decision time (null if n/a)
"manifest_digest": "sha256:9f2…", // digest of the manifest — which IS the policy (egress routes etc.); null if n/a
// ---- semantics: host-observed facts of what happened ----
"actor": "bottled-agent:amber-fox-12", // who acted, as a host-attributed identity
@@ -186,7 +187,7 @@ The **trust boundary is a single region, not a split.** Everything outside
`untrusted` is trusted by construction — the host established it: schema and
chain metadata, both timestamps, the attribution/provenance fields
(source-IP → `bottled_agent`/`bottle`/`activation`, `manifest_digest`,
`policy_version`), the host-observed semantics
`engine`), the host-observed semantics
(`actor`/`action`/`resource`/`outcome`), the host-assigned correlation ids,
and the typed `payload`. `untrusted` is the **one** place anything an agent
or a remote claimed may go (e.g. the agent's free-text `reason`). A reader
2
@@ -209,12 +210,26 @@ the single writer durably appended it. Ordering and the chain use
`(epoch, seq)`, never either wall clock. Both are host-set — a bottled
agent never supplies a timestamp.
**The manifest *is* the policy.** bot-bottle has no separate policy
artifact — a bottled agent's egress routes and other constraints are
declared in its manifest (`bot_bottle/manifest/egress.py`), so
`manifest_digest` already pins the ruleset in force; there is no distinct
`policy_version`. Given a fixed manifest, the only other thing that can
change a decision's outcome is the enforcing code — captured by `engine`
didericis marked this conversation as resolved
Review

Would be nice if we use the same credential shape detectors we use on egress, but not sure whether it's worth it or if that would be brittle/concerns would diverge. Evaluate how reasonable that is.

Would be nice if we use the same credential shape detectors we use on egress, but not sure whether it's worth it or if that would be brittle/concerns would diverge. Evaluate how reasonable that is.
Review

Evaluated: reuse the deterministic detectors in gateway/egress/dlp_detectors.py (scan_token_patterns / redact_tokens, plus scan_known_secrets) — pure-Python and already the project's source of truth for credential shapes, so wire-egress and journal-redaction can't drift. Excluded scan_entropy: it's tuned for large streamed bodies and would false-positive on the short high-entropy values audit records legitimately carry (hashes, uuids, base64 ids), redacting the very fingerprints we want to keep. So: share the deterministic detectors, keep entropy egress-only.

Evaluated: reuse the deterministic detectors in `gateway/egress/dlp_detectors.py` (`scan_token_patterns` / `redact_tokens`, plus `scan_known_secrets`) — pure-Python and already the project's source of truth for credential shapes, so wire-egress and journal-redaction can't drift. **Excluded `scan_entropy`**: it's tuned for large streamed bodies and would false-positive on the short high-entropy values audit records legitimately carry (hashes, uuids, base64 ids), redacting the very fingerprints we want to keep. So: share the deterministic detectors, keep entropy egress-only.
(bot-bottle version + git SHA). So two `egress.decision` records with the
same `resource` but different `outcome` are explained by exactly one of:
different `manifest_digest` (the rules changed) or different `engine` (the
enforcer changed). Runtime operator overrides (a supervise `egress-allow`)
are themselves audit events, so the effective ruleset at any instant is
`manifest_digest` plus the logged, approved deltas — reconstructable from
the chain, not from a version stamp.
**Optionality.** `bottle`/`bottled_agent`/`activation`, `manifest_digest`,
`policy_version`, and `payload`/`untrusted` are `null`/absent for events
that have no such subject (a host-level `hostctl.*` or `audit.*` event has
no bottled agent). Absent ≠ empty: a reader distinguishes "no subject" from
"unknown". `id`, `type`, the chain fields, both timestamps, `producer`,
`host`, `actor`, `action`, `outcome`, and `sensitivity` are always present.
and `payload`/`untrusted` are `null`/absent for events that have no such
subject (a host-level `hostctl.*` or `audit.*` event has no bottled agent).
Absent ≠ empty: a reader distinguishes "no subject" from "unknown". `id`,
`type`, the chain fields, both timestamps, `producer`, `host`, `engine`,
`actor`, `action`, `outcome`, and `sensitivity` are always present.
#### Trust provenance of every common field
@@ -223,9 +238,9 @@ no bottled agent). Absent ≠ empty: a reader distinguishes "no subject" from
| `v` `id` `type` `epoch` `seq` `segment` `prev` `hash` | trusted | the single writer |
| `ts_event` | trusted | emitting host component (boundary) |
| `ts_recorded` `ts_mono` | trusted | the single writer |
| `producer` `host` | trusted | the single writer |
| `producer` `host` `engine` | trusted | the single writer |
| `bottle` `bottled_agent` `activation` | trusted | gateway source-IP → slug attribution |
| `manifest_digest` `policy_version` | trusted | control plane (what was in force) |
| `manifest_digest` | trusted | control plane (the manifest = the policy in force) |
| `actor` `action` `resource` `outcome` | trusted | host component that observed/decided it |
| `sensitivity` | trusted | registry default for `type`, overridable up (never down) by the producer |
| `correlation_id` `causation_id` | trusted | the single writer (assigned as it threads the flow) |
4
@@ -601,7 +616,9 @@ the committing agent.
`canonical()`, chain hashing, and the single-writer journal appender in
`bot_bottle/store/` (reusing `sha256_hex`); redaction wired to the
existing `gateway/egress/dlp_detectors` (`scan_token_patterns` /
`redact_tokens`); unit tests for determinism, chain-break detection,
`redact_tokens`); embed the git SHA at build so `engine` is populated
(only `version = "0.1.0"` exists in `pyproject.toml` today — the build
must stamp the SHA); unit tests for determinism, chain-break detection,
`epoch`/`seq` continuity across a simulated restart, and redaction of
both a deny-listed key and a token-shaped value.
3. **SQLite index + `audit` CLI.** New `audit_events` migration (indexable