Compare commits

...

4 Commits

Author SHA1 Message Date
didericis-codex 4cb7bbbd11 docs(prd): tighten audit ordering and chain guarantees
prd-number-check / require-numbered-prds (pull_request) Failing after 6s
tracker-policy-pr / check-pr (pull_request) Successful in 6s
2026-07-26 17:38:43 +00:00
didericis-claude f3664dea9f docs(prd): drop policy_version (manifest is the policy); add engine field
prd-number-check / require-numbered-prds (pull_request) Failing after 6s
tracker-policy-pr / check-pr (pull_request) Successful in 5s
Per review: bot-bottle has no separate policy artifact — a bottled agent's
egress routes etc. are declared in its manifest (manifest/egress.py), so
manifest_digest already pins the policy in force. Remove the redundant
policy_version. Given a fixed manifest, the only other axis that changes an
outcome is the enforcing code, so add 'engine' (bot-bottle version + git
SHA) as a trusted field. Runtime operator overrides (supervise egress-allow)
are themselves audit events, so effective policy = manifest_digest + logged
deltas, reconstructable from the chain. Note the build must stamp the git
SHA (only version=0.1.0 exists today).

Refs #487

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-26 08:28:10 +00:00
didericis-claude 88b82a169e docs(prd): complete audit-event contract to #487 acceptance checklist
prd-number-check / require-numbered-prds (pull_request) Failing after 10s
tracker-policy-pr / check-pr (pull_request) Successful in 11s
Expand the PRD from a schema sketch to the full contract the issue mandates
(issue is spec-only: 'defines the contract; implementation may be split
into follow-up PRs'):

- Envelope: add observed vs event timestamps, bottle/activation ids,
  manifest_digest + policy_version, actor/action/resource/outcome,
  correlation_id/causation_id, sensitivity class, typed payload, segment id.
- Add a per-field trust-provenance table (trusted vs claimed for every
  common field); per-type trusted/claimed in the registry.
- Canonicalization: normative, reproducible hash-chain test vectors;
  idempotency (id key, UPSERT), ordering guarantees, and behavior across
  rotation/restart/import/truncation (truncated-tail vs gap).
- Storage: indexable fields + local audit query/verify/rebuild/import CLI.
- Registry: cover all mandated groups incl hostctl.*, egress
  request/decision/cutoff/anomaly, commit.signed (#480), auth/authz, and
  audit.* self-events; schema-evolution + backward-compatible reader rules.
- Export: #324 delivery contract (payload, (epoch,seq) cursor, dedup,
  backpressure, retention ordering); #480 mapping preserving its
  byte-to-activation-key guarantee.
- No raw prompt/response/body capture by default.
- Add an acceptance-criteria coverage table mapping each #487 checkbox to a
  section.

Refs #487

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-26 08:08:22 +00:00
didericis-claude 5a9428cc86 docs(prd): flatten to one untrusted region + address CloudEvents/OTel export
prd-number-check / require-numbered-prds (pull_request) Failing after 7s
tracker-policy-pr / check-pr (pull_request) Successful in 19s
- Remove the trusted sub-block: everything outside untrusted (chain
  metadata, producer/host, bottled_agent, ts_*) is trusted by construction.
  A field is trusted unless deliberately placed under untrusted (#495).
- Add Export/interoperability section: the flattened envelope projects
  cleanly onto CloudEvents JSON (top-level scalars -> context/extension
  attributes, untrusted -> data) and the OpenTelemetry Logs data model
  (ts_wall -> Timestamp, trusted -> botbottle.* attributes, untrusted ->
  botbottle.untrusted.*). Attribution preserved structurally; integrity
  fields carried as data with verification always on the native journal.
  Satisfies #487's export/interop requirement. Export adapters = chunk 6.

Refs #487

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-26 07:45:58 +00:00
+475 -93
View File
@@ -19,17 +19,27 @@ field names, timestamps, or how a bottled agent is identified.
This PRD defines **one canonical audit-event contract** every producer
emits into:
1. A **versioned envelope** — schema version, event id, event type,
monotonic + wall-clock timestamps, and an explicit **trust boundary**
between host-supplied and agent-claimed fields.
2. **Canonical JSON serialization + a per-writer hash chain**, so any
deletion or edit of a past record breaks the chain and is detectable
offline.
1. A **versioned envelope** — schema version, event id, event/observed
timestamps, host-attributed identity (`bottle`/`bottled_agent`/
`activation`), provenance (`manifest_digest` — the manifest is the
policy — and `engine` = bot-bottle version/SHA),
host-observed `actor`/`action`/`resource`/`outcome`, correlation/
causation ids, a sensitivity class, a typed payload, and an explicit
**trust boundary** between host-supplied and agent-claimed fields.
2. **Canonical JSON serialization + a per-writer hash chain** (with
normative test vectors), so any deletion, edit, or reorder of a past
record breaks the chain and is detectable offline.
3. An **append-only JSONL journal as the source of truth**, with a
**rebuildable SQLite index** for local search — no paid platform, no
network dependency.
4. An **initial event registry** covering lifecycle, decision, egress,
auth, and forge events.
**rebuildable SQLite index** and a local `audit query`/`verify` surface
— no paid platform, no network dependency.
4. An **initial event registry** covering lifecycle, host-controller,
supervise decision, egress (request/decision/cutoff/anomaly), git-gate
and signed-commit (#480), auth/authz, and audit self-events — each with
its trusted-vs-claimed fields and redaction rules.
5. A **stable export projection** (CloudEvents / OpenTelemetry Logs) and
the **#324 delivery contract** (payload, per-chain
`(host, epoch, seq)` cursor, dedup,
backpressure, retention ordering).
It is explicitly scheduled to land **immediately after the host
controller (#468)** so the host controller's lifecycle transitions are the
@@ -66,10 +76,11 @@ shared schema. Concretely:
## Goals / Success Criteria
- A single `AuditEvent` envelope type, versioned, that every producer
emits. Fields are split into a **`trusted`** block (host-supplied:
bottled-agent slug from source-IP attribution, host wall-clock, producer
identity) and an **`untrusted`** block (anything the agent or a remote
claimed), and the split is structural, not a convention.
emits. The trust boundary is **one `untrusted` region**: everything
outside it is host-established and trusted (source-IP → bottled-agent
attribution, host wall-clock, producer identity, chain metadata);
`untrusted` is the sole place anything an agent or a remote claimed may
go. The boundary is structural, not a convention.
- **Canonical serialization** (`sort_keys`, `(",", ":")` separators,
UTF-8, `ensure_ascii=False`) is defined once and reused, so the same
logical event always hashes identically across producers and hosts.
@@ -86,6 +97,11 @@ shared schema. Concretely:
- A **redaction rule** is enforced at the envelope boundary: known
credential-shaped fields are rejected/redacted before a record is
written; the writer refuses raw secrets rather than storing them.
- The envelope **projects onto the OpenTelemetry Logs data model and a
CloudEvents JSON envelope** by field re-mapping alone (no reformat),
preserving the trust boundary and carrying the integrity fields — per
#487's export/interop requirement. (The export adapters are follow-up;
the *schema* must make them a re-map.)
- The **host controller (#468)** emits `lifecycle.*` events through this
contract as the first consumer; existing supervise/egress producers are
migrated behind the same envelope without changing operator-facing
@@ -120,49 +136,122 @@ stable top level:
```
{
// ---- schema + integrity (host-owned) ----
"v": 1, // schema version — bumped only on a breaking change
"id": "<uuid4>", // globally unique event id
"id": "<uuid4>", // globally unique event id; stable across export/replay (dedup key)
"type": "egress.decision", // dotted event type from the registry
// --- chain / ordering (structural, host-owned) ---
"epoch": 7, // writer-boot counter, bumped once per host-controller (writer) start
"seq": 1287, // monotonic sequence within this epoch (gap-detectable)
"prev": "<hex>", // hash of the previous record in the chain ("" for genesis)
"segment": "20260726T000000Z", // journal segment id (rotation boundary); chain continues across segments
"prev": "<hex>", // prior record hash ("" only for the first record of a new chain)
"hash": "<hex>", // sha256(prev + canonical(this event with hash=""))
// --- trusted: everything the *host* established; authoritative ---
"trusted": {
"producer": "host-controller", // which host component wrote this
"host": "mac-studio-1",
"bottled_agent": "amber-fox-12", // slug from source-IP attribution (null for host-level events)
"ts_wall": "2026-07-26T18:22:04.113Z", // host wall-clock, RFC3339 UTC
"ts_mono": 90142.55 // host monotonic secs since this epoch's boot (intra-epoch ordering only)
// ---- timestamps (host-owned) ----
"ts_event": "2026-07-26T18:22:04.061Z", // when the underlying event occurred at the boundary
"ts_recorded": "2026-07-26T18:22:04.113Z", // when the single writer appended it (authoritative)
"ts_mono": 90142.55, // monotonic secs since this epoch's boot (intra-epoch ordering only)
// ---- attribution + provenance (host-established) ----
"producer": "egress", // host component that emitted the event
"host": "mac-studio-1",
"engine": "bot-bottle/0.1.0+abc1234", // bot-bottle version + git SHA of the enforcing host code
"bottle": "amber-fox", // bottle (container/VM) identity
"bottled_agent": "amber-fox-12", // bottled-agent slug from source-IP attribution (null for host-level events)
"activation": "01J8Z...", // activation id: one run/session of the bottled agent (null if n/a)
"manifest_digest": "sha256:9f2…", // digest of the manifest — which IS the policy (egress routes etc.); null if n/a
// ---- semantics: host-observed facts of what happened ----
"actor": "bottled-agent:amber-fox-12", // who acted, as a host-attributed identity
"action": "egress.connect", // what was attempted / done
"resource": "registry.npmjs.org:443", // what it acted on, as observed at the boundary
"outcome": "blocked", // host-decided result: allowed|blocked|deferred|success|failure
"sensitivity": "security", // classification: normal|security|restricted (drives redaction + export)
// ---- correlation (host-assigned) ----
"correlation_id": "flow-9c2a…", // groups a related flow (request → decision → cutoff)
"causation_id": "<event id>", // the event that directly caused this one ("" if root)
// ---- typed, trusted, event-specific payload (shape fixed per type in the registry) ----
"payload": {
"route_id": 4,
"detector": "token_patterns"
},
// --- untrusted: anything the agent or a remote claimed; never authoritative ---
// ---- the ONLY untrusted region: agent- or remote-claimed data ----
"untrusted": {
"reason": "npm install needs registry.npmjs.org",
"target": "registry.npmjs.org:443"
"reason": "npm install needs registry.npmjs.org" // the agent's stated justification
}
}
```
The **`trusted` / `untrusted` split is the core invariant.** A producer may
only place a field in `trusted` if the *host* established it: the
source-IP → `bottled_agent` slug attribution, the host's own clock
(`ts_wall`/`ts_mono`), and the producer's own identity. **`producer` and
`ts_*` live inside `trusted` on purpose** — they are host-supplied, so
grouping them there (rather than as loose top-level fields) keeps the
"authoritative ⇔ inside `trusted`" rule structural, with nothing
host-established leaking outside it. Everything an agent or a remote said
goes in `untrusted`. A reader (or a future policy engine) can therefore
trust `trusted.bottled_agent` for attribution and treat `untrusted.*` as
adversarial claims — the distinction the current stores lack.
The **trust boundary is a single region, not a split.** Everything outside
`untrusted` is trusted by construction — the host established it: schema and
chain metadata, both timestamps, the attribution/provenance fields
(source-IP → `bottled_agent`/`bottle`/`activation`, `manifest_digest`,
`engine`), the host-observed semantics
(`actor`/`action`/`resource`/`outcome`), the host-assigned correlation ids,
and the typed `payload`. `untrusted` is the **one** place anything an agent
or a remote claimed may go (e.g. the agent's free-text `reason`). A reader
(or a future policy engine) trusts every field outside `untrusted` for
attribution and treats `untrusted.*` — and only `untrusted.*` as
adversarial claims.
Only the small structural set — `v`, `id`, `type`, `epoch`, `seq`, `prev`,
`hash` — sits at the top level; it is host-owned too, but it is chain
metadata rather than event data, so it stays out of the `trusted` body to
keep that body purely about *what happened*.
Framing it as "one untrusted region, everything else trusted" removes the
mistake where a producer forgets to mark a claimed field: a field is
trusted unless it is deliberately placed inside `untrusted`. The
construction API enforces this — producers pass trusted fields explicitly
and hand all agent/remote-claimed data as the single `untrusted` mapping,
so there is no way to emit a top-level field that *looks* authoritative but
isn't.
**Two timestamps** because they answer different questions and can diverge
under backpressure: `ts_event` is when the thing happened at the boundary
(the proxy saw the connect, the gate saw the push); `ts_recorded` is when
the single writer durably appended it. Ordering and the chain use
`(epoch, seq)`, never either wall clock. Both are host-set — a bottled
agent never supplies a timestamp.
**The manifest *is* the policy.** bot-bottle has no separate policy
artifact — a bottled agent's egress routes and other constraints are
declared in its manifest (`bot_bottle/manifest/egress.py`), so
`manifest_digest` already pins the ruleset in force; there is no distinct
`policy_version`. Given a fixed manifest, the only other thing that can
change a decision's outcome is the enforcing code — captured by `engine`
(bot-bottle version + git SHA). So two `egress.decision` records with the
same `resource` but different `outcome` are explained by exactly one of:
different `manifest_digest` (the rules changed) or different `engine` (the
enforcer changed). Runtime operator overrides (a supervise `egress-allow`)
are themselves audit events, so the effective ruleset at any instant is
`manifest_digest` plus the logged, approved deltas — reconstructable from
the chain, not from a version stamp.
**Optionality.** `bottle`/`bottled_agent`/`activation`, `manifest_digest`,
and `payload`/`untrusted` are `null`/absent for events that have no such
subject (a host-level `hostctl.*` or `audit.*` event has no bottled agent).
Absent ≠ empty: a reader distinguishes "no subject" from "unknown". `id`,
`type`, the chain fields, both timestamps, `producer`, `host`, `engine`,
`actor`, `action`, `outcome`, and `sensitivity` are always present.
#### Trust provenance of every common field
| Field | Trust | Set by |
|---|---|---|
| `v` `id` `type` `epoch` `seq` `segment` `prev` `hash` | trusted | the single writer |
| `ts_event` | trusted | emitting host component (boundary) |
| `ts_recorded` `ts_mono` | trusted | the single writer |
| `producer` `host` `engine` | trusted | the single writer |
| `bottle` `bottled_agent` `activation` | trusted | gateway source-IP → slug attribution |
| `manifest_digest` | trusted | control plane (the manifest = the policy in force) |
| `actor` `action` `resource` `outcome` | trusted | host component that observed/decided it |
| `sensitivity` | trusted | registry default for `type`, overridable up (never down) by the producer |
| `correlation_id` `causation_id` | trusted | the single writer (assigned as it threads the flow) |
| `payload.*` | trusted | emitting host component (shape fixed per `type`) |
| `untrusted.*` | **claimed** | copied verbatim from a bottle / gateway / forge / remote |
Every registry entry (below) restates, per event type, which `payload`
keys are required and names any `untrusted` keys it carries — so "trusted
vs claimed" is explicit for every event-specific attribute, not just the
common ones.
### Canonical serialization + hash chain
@@ -182,11 +271,109 @@ The `hash` field is computed over the canonical form of the event **with
digest = sha256_hex(prev_hash + canonical({**event, "hash": ""}))
```
`prev` is the prior record's `hash`; genesis uses `prev = ""`. This makes
the journal an append-only Merkle-style chain: editing or deleting record
*n* changes its hash, so record *n+1*'s `prev` no longer matches — the
break is local and points at the tampered record. Verification needs only
the journal itself (no keys), so it runs offline and in CI.
`prev` is the prior record's `hash`; only the first record of a brand-new
chain uses `prev = ""`. The first record of a rotated segment carries the
preceding segment's head, as specified under *Rotation* below.
Two exact rules pin the bytes so the chain is reproducible anywhere:
1. **Serialize the record with its own `hash` field set to `""`** (present,
empty), never omitted — the key set is identical before and after
hashing.
2. **Digest = `sha256_hex(prev + canonical(record_with_empty_hash))`**,
where `prev` is the previous record's `hash` string (`""` at genesis),
`+` is string concatenation, and `canonical` is the function above.
`ts_mono`, being a float, is serialized by Python's shortest-round-trip
`repr` via `json.dumps`; producers therefore emit it as a JSON number
they do not post-process. (All other fields are strings/ints/objects,
which serialize unambiguously.)
Editing or deleting record *n* changes its hash, so record *n+1*'s `prev`
no longer matches — the break is local and names the tampered record.
Verification needs only the journal itself (no keys), so it runs offline
and in CI.
#### Test vectors (normative)
Two records, reduced to the chain-relevant fields, demonstrate the exact
serialization and linkage. An implementation is conformant iff it
reproduces these bytes and hashes.
```
# Record 0 — brand-new chain genesis (prev = "")
canonical(record0, hash=""):
{"hash":"","id":"11111111-1111-4111-8111-111111111111","prev":"","seq":0,"type":"audit.segment_open"}
hash0 = sha256("" + canonical) =
942ea5729bcac6efdbdea942396bfa574ab0d6ebf5615402595359422f2aeb83
# Record 1 — chains onto record 0 (prev = hash0)
canonical(record1, hash=""):
{"hash":"","id":"22222222-2222-4222-8222-222222222222","prev":"942ea5729bcac6efdbdea942396bfa574ab0d6ebf5615402595359422f2aeb83","seq":1,"type":"lifecycle.bottled_agent_start"}
hash1 = sha256(hash0 + canonical) =
bc082347680405fee50b60a9c304611aa026950b15d869b7e3ae56e1c451b856
# Tamper check: flip record0.type → recompute →
# 4553eda647f33f0c608cfea44be28efbbaca45ed30b873fcbd4405fa5ce737ed
# which no longer equals record1.prev (942ea5…) — the break is detected at record1.
```
The implementation PR ships these plus full-envelope vectors (every field
populated, and a redaction case) as committed fixtures, so a schema-version
bump that changes the bytes fails a golden test loudly.
### Ordering, idempotency, and duplicate handling
- **Ordering.** `(epoch, seq)` is a strict total order within one host's
native chain and, because
the writer is single, a strict order per bottle/activation within that
host — satisfying "at least strict causal order per activation/bottle".
`causation_id` records the explicit cause edges (a DAG) on top of the
total order, so a consumer can reconstruct request → decision → cutoff
even if unrelated events interleave between them. There is deliberately
no invented total order across imported host chains; a cross-host key is
`(host, epoch, seq)`, and consumers use correlation/causation edges where
causal ordering across hosts is known.
- **Idempotency.** `id` is the idempotency key. A producer that retries an
emit (e.g. after a writer restart mid-handoff) **reuses the same `id`**;
before appending, the writer checks a durable, host-wide id ledger that
spans every segment and survives restart/retention, and drops an id
already present. The ledger is operational metadata, not an audit source
of truth: after a crash it is reconciled from the journal before appends
resume, and retention preserves id tombstones after journal segments are
pruned. An in-memory set may cache the ledger but is never the authority.
The index additionally has a unique key on `id`; its `UPSERT` is
defensive and does not substitute for the pre-append check. Thus a
duplicate never enters the source-of-truth journal, double-counts, or
forks the chain.
- **Deduplication downstream.** Because `id` is stable across export and
replay, #324's cursor replay and any cross-host merge dedup on `id` — no
consumer needs to invent a second identity.
### Behavior across rotation, restart, import, truncation
- **Rotation.** At a segment boundary the writer opens a new segment file,
sets its `segment` id, and carries the rotated-out segment's head as the
new segment's first `prev` — so the chain is continuous *across* segments
(`prev = ""` is reserved for the first record of a brand-new chain) while
each file stays independently openable. `verify` walks segments in order
and checks the preceding-head-to-first-record link at each seam.
- **Restart.** Covered above: read last line → adopt its `hash` as `prev`,
bump `epoch`, reset `seq`. The chain never restarts even though the
counters do.
- **Import.** `audit import <segment>` appends an externally supplied
segment (e.g. recovered from another host or a backup). Import verifies
the incoming chain in isolation first. A continuation whose first `prev`
matches a known head extends that chain. A foreign chain is registered as
a separate immutable chain namespace rather than rewriting or grafting
its records (which would invalidate their hashes). Imported records keep
their original `host`, `id`, `epoch`, and `seq`; the index keys their
native order by `(host, epoch, seq)` so attribution is not laundered and
tuples from different hosts cannot collide.
- **Truncation.** A crash can leave a partial final line; `verify` reports
it as `truncated-tail` (recoverable — replay resumes from the last intact
record). A chain that ends before a persisted head, or a missing interior
`seq`, is reported as `gap`/`missing-suffix` (evidence of deletion, not a
clean crash). The two are distinguished so an operator can tell "power
loss" from "someone trimmed the log".
### Single writer; ordering across restarts
@@ -207,11 +394,13 @@ start the writer:
2. bumps `epoch` (persisted alongside the chain head) and resets `seq` to
0 for the new boot.
Total order is therefore `(epoch, seq)` — monotonic across restarts by
construction — with `ts_wall` for human reading and `ts_mono` for
sub-second ordering inside an epoch. A crash mid-append truncates at most
the last (partial) line; the verifier flags it and replay resumes from the
last intact record.
Total order within this host's native chain is therefore `(epoch, seq)` —
monotonic across restarts by construction — with
`ts_event`/`ts_recorded` for human reading and `ts_mono` for sub-second
ordering inside an epoch. Imported chains retain their own
`(host, epoch, seq)` order and do not acquire a fictional order relative to
the local chain. A crash mid-append truncates at most the last (partial)
line; the verifier flags it and replay resumes from the last intact record.
### Journal (source of truth) + SQLite index (rebuildable)
@@ -219,35 +408,94 @@ last intact record.
alongside `host_db_path()`), one canonical event per line, opened
`O_APPEND`. This is authoritative.
- **Index:** a new `audit_events` table via the existing `DbStore` /
`TableMigrations` machinery, holding the envelope columns plus JSON
blobs, indexed on `(bottled_agent, type, ts_wall)`. It is a **derived
cache**: `audit rebuild` truncates and replays the journal, re-verifying
the chain as it goes. If the DB is deleted or drifts, it is regenerated
from the journal with no data loss. (This supersedes the free-standing
`supervise_audit_entries` table, which becomes a view/producer onto the
new index.)
`TableMigrations` machinery. It is a **derived cache, not a second source
of truth**: `audit rebuild` truncates and replays the journal,
re-verifying the chain as it goes, so a deleted or drifted DB is
regenerated from the journal with no data loss. On a `verify` failure
during rebuild it stops and reports rather than indexing past a break.
(This supersedes the free-standing `supervise_audit_entries` table, which
becomes a producer onto the new index.)
**Indexable fields** (columns + indices): `ts_event`, `ts_recorded`,
`type`, `host`, `bottle`, `bottled_agent`, `activation`, `actor`,
`outcome`, `sensitivity`, `correlation_id`, `causation_id`, plus two
event-specific projections promoted out of `payload` for query —
`repository` and `commit_sha` (populated for `forge.*`/`commit.*`, null
otherwise). The unique event id and native-order index are respectively
`id` and `(host, epoch, seq)`; a local-only `ingest_seq` provides stable
display order when a query intentionally mixes chains without pretending
that it is causal order. The full canonical record is stored verbatim in a
`raw` column so the index never loses fidelity to the journal.
**Local query surface** — `audit query`, no egress, no paid platform:
```
audit query \
[--since T] [--until T] [--type egress.*] [--host H] [--bottle B] \
[--activation A] [--agent SLUG] [--actor ID] [--outcome blocked] \
[--repository R] [--correlation-id C] [--commit SHA] \
[--follow BOTTLE] # one host chain's events in (epoch, seq) order
[--json | --table]
audit verify [--segment S] # offline chain check; exit non-zero on any break
audit rebuild # drop + replay journal → index
audit import <segment> # graft an external segment (see above)
```
Type filters accept a `group.*` glob. A read-only local HTTP endpoint
mirrors the same filters for the future review console; both are pure reads
over the index and can never mutate the journal.
### Event registry (initial)
Dotted `type` names, grouped; the registry is a table mapping type
required `untrusted` keys so producers and the verifier agree on shape:
Dotted `type` names, grouped. The registry is a table mapping each type to
its required `payload` keys, its `untrusted` keys (if any), a default
`sensitivity`, and its correlation behavior — so producers and the verifier
agree on shape and "trusted vs claimed" is pinned per type. Initial
coverage (the issue's mandated set):
- **lifecycle.*** — `lifecycle.bottled_agent_start`,
`lifecycle.bottled_agent_stop`, `lifecycle.bottled_agent_crash`
(producer: host-controller, #468). Leaf names use `bottled_agent` to match
the `trusted.bottled_agent` field — one term for the subject everywhere.
- **decision.*** — `decision.proposed`, `decision.resolved`
(producer: supervise; carries operator action + justification, replacing
PRD 0013's row shape).
- **egress.*** — `egress.decision` (allow/block at the proxy),
`egress.route_added`.
- **auth.*** — `auth.token_minted`, `auth.token_rejected` (control-plane;
**never** the token itself — see redaction).
- **forge.*** — `forge.push_accepted`, `forge.push_rejected` (git-gate),
`forge.pr_opened`.
| Group / type | Producer | Required `payload` (trusted) | `untrusted` | Default sensitivity |
|---|---|---|---|---|
| **lifecycle.*** — `bottled_agent_start` / `_stop` / `_crash` | host-controller (#468) | `manifest_digest`, `exit` (for stop/crash) | — | normal |
| **hostctl.*** — `broker_launch`, `broker_teardown`, `broker_reject` | host-controller (#468) | `op`, `request_digest` | — | security |
| **decision.*** — `proposed`, `resolved` | supervise | `tool`, `operator_action`, `justification`, `diff_digest` | `agent_rationale` | security |
| **egress.*** — `request`, `decision`, `cutoff`, `anomaly` | egress proxy | `route_id`, `detector` (on match), `bytes` (cutoff) | `reason`, `target_claimed` | security |
| **forge.*** — `push_accepted`, `push_rejected`, `pr_opened` | git-gate | `repository`, `ref`, `gitleaks_result` | `title`, `description` | security |
| **commit.signed** (#480) | git-gate | `repository`, `commit_sha`, `activation_key_id`, `signature_ref` | `commit_message` | security |
| **auth.*** — `token_minted`, `token_rejected`, `authz_denied` | control plane | `role`, `token_id`, `reason_code` | — | security |
| **audit.*** — `segment_open`, `verify_failed`, `truncation_detected`, `export_failed` | audit writer/verifier | `segment`, `detail` | — | security |
New types are additive; adding one does not bump `v`. Removing or
re-typing a field bumps `v`.
Notes:
- **`egress.request` vs `egress.decision`** share a `correlation_id`; the
`decision`'s `causation_id` points at the `request`, and a later `cutoff`
chains onto the `decision` — so a flow is reconstructable.
- **`audit.*` self-events** make the audit subsystem audit itself: a failed
verification, a detected truncation, or a dropped export is itself a
chained, tamper-evident record — you cannot silence the alarm without
breaking the chain that carries it.
- **Free-text and remote-echoed fields are always `untrusted`** (`reason`,
`agent_rationale`, PR `title`/`description`, `target_claimed`), because
they originate in the bottle or a remote response; the host-observed
counterpart (`resource`, `outcome`, `gitleaks_result`) is the trusted
fact.
**Sensitivity + redaction per type.** Every type's default `sensitivity`
is listed above; a producer may raise it (never lower it). `restricted`
events keep their `payload` in the journal but the export projection ships
only the envelope + a payload digest unless the consumer is authorized —
so a `security`/`restricted` record is still counted and correlated
downstream without leaking its body. The credential-shape redaction rules
(next) apply to **every** type regardless of sensitivity.
**Schema evolution & backward-compatible readers.** The registry is
append-only: **adding** a type, an optional `payload` key, or an
`untrusted` key does **not** bump `v`; readers ignore unknown fields
(forward-compatible) and treat absent optional fields as `null`.
**Removing** or **re-typing** a field, or making an optional field
required, bumps `v`. A reader declares the max `v` it understands and
refuses to *interpret* a higher-`v` record, but the **verifier is
version-agnostic** — the hash covers whatever fields exist, so chain
integrity is checkable across versions without understanding semantics.
Every `v` bump ships a migration note and updated golden vectors.
### Redaction rule
@@ -287,6 +535,105 @@ and keep the event) rather than drop, so a producer bug can never make an
audit event vanish; the key deny-list stays a hard refusal because a
credential in a named field is always a producer bug worth surfacing.
**No raw-payload capture by default.** The envelope carries *decisions and
metadata*, not traffic. Prompts, model responses, request/response bodies,
and file contents are **not** recorded unless a producer opts a specific,
reviewed field in — and such a field is `untrusted` and subject to both
redaction layers. This keeps the audit log from becoming a covert copy of
the very data the sandbox exists to contain (the issue's "unsafe payload
capture" non-goal).
### Export / interoperability (CloudEvents, OpenTelemetry Logs)
#487 requires the envelope to map onto the **OpenTelemetry Logs data
model** and/or a **CloudEvents JSON** envelope *without losing integrity or
attribution semantics*. The flattened shape (one `untrusted` region,
everything else trusted at top level) does **not** conflict with either — it
maps *more* cleanly than a nested `trusted`/`untrusted` pair would, because
both target models expect a flat set of top-level fields plus one payload
subtree.
**CloudEvents.** Context attributes MUST be scalar simple types — a map
cannot be a context attribute — so a nested `trusted` block would have had
to be flattened for CloudEvents anyway. Our flat top level maps directly:
`id`→`id`, `type`→`type`, `producer`+`host`→`source`,
`bottled_agent`→`subject`, `ts_event`→`time` (`ts_recorded` as an extension); the integrity/chain fields
(`epoch`, `seq`, `prev`, `hash`, `v`) ride as **extension attributes**
(scalars — legal). The `untrusted` map goes in `data`. Only mechanical
transform needed: extension attribute names must be lowercase-alphanumeric,
so `bottled_agent`/`ts_mono`/etc. are renamed at export (e.g. a
`botbottle`-prefixed form) — a naming rule, not a schema conflict.
**OpenTelemetry Logs.** `ts_event`→`Timestamp`, `ts_recorded`→`ObservedTimestamp`; `type`→the `event.name`
attribute; the flat trusted fields → `Attributes` under a `botbottle.*`
namespace (`botbottle.bottled_agent`, `botbottle.producer`,
`botbottle.chain.hash`, …); `untrusted.*` → `Attributes` under
`botbottle.untrusted.*` (or `Body`). OTel attributes are a dotted map that
happily carries the nested subtree.
**Attribution is preserved** precisely because the boundary is now
structural: on export, top-level fields become trusted context/attributes
and the `untrusted` subtree stays a single, clearly-named region — so a
downstream consumer still sees exactly which fields an agent claimed.
Nothing agent-claimed is promoted to a trusted-looking position.
**Integrity has one deliberate caveat.** CloudEvents/OTel are
representation envelopes with their own (or no) canonicalization; `hash`
and `prev` are computed over **our** canonical JSON, not over the exported
form. So the chain fields travel *as data* for reference, but
tamper-evidence is always verified against the **native journal** (the
source of truth) — never re-derived from an exported CloudEvents/OTel
record, whose key ordering / number formatting the exporter may change.
Export is thus a lossless-for-attribution **projection** that carries the
integrity fields along; verification stays on the canonical journal. This
satisfies "without losing integrity or attribution semantics": both are
carried, neither is *relied upon* in the foreign format.
The export adapters themselves are follow-up implementation — this PRD
fixes the *schema* so that projection is a field re-map, never a reformat.
#### The #324 delivery contract (payload, cursor, backpressure)
#324 transports events off-box; it must not invent a second envelope. This
PRD fixes the contract it depends on:
- **Payload.** #324 ships the **native canonical record verbatim** (the
exact bytes the hash covers), optionally wrapped in the CloudEvents
projection whose `data` *is* that record. Either way the integrity fields
travel intact and the receiver can verify against the same bytes.
- **Cursor.** Export maintains one cursor per native host chain:
`(host, epoch, seq)` (equivalently that chain's last exported `hash`).
It advances **only on acknowledgement**, so delivery is at-least-once and
gap-free; a crash re-sends from the last acked cursor. Imported foreign
chains use independent cursors and never share a bare `(epoch, seq)`
namespace with the local chain.
- **Idempotency / replay.** Dedup is on `id` (stable across replay), so
at-least-once delivery is safe — the receiver collapses re-sends.
- **Backpressure.** The outbox is the journal itself plus a cursor; when
the endpoint is slow the cursor simply lags — the writer never blocks on
export, and audit never applies backpressure to the data plane it
records.
- **Retention interaction.** Retention/rotation **must not** prune a
segment whose records are still behind the export cursor; the reaper
honors `min(cursor)` across all configured consumers. (The schedule
itself stays the retention follow-up; this is the *ordering* constraint
that follow-up must respect.)
#### #480 signed-commit attribution maps in without weakening it
#480 binds a commit's bytes to a per-activation signing key. It maps to the
`commit.signed` event: `payload` carries `repository`, `commit_sha`,
`activation_key_id`, and a `signature_ref` (the detached-signature
location or its digest) — **not** the private key and not a re-derived
signature. The audit event therefore *references and timestamps* #480's
existing byte-to-activation-key proof inside the tamper-evident chain; it
does not re-implement or replace it, so #480's guarantee is unweakened —
the signature still verifies against the commit bytes independently, and
the audit record adds only "this binding was observed at this point in the
chain". The trusted `actor`/`activation` fields and the `commit_sha`
payload are host-observed at the gate, so attribution cannot be forged by
the committing agent.
## Implementation chunks
1. **(this PR — PRD only.)** The contract above. No code; scheduled to land
@@ -295,21 +642,54 @@ credential in a named field is always a producer bug worth surfacing.
`canonical()`, chain hashing, and the single-writer journal appender in
`bot_bottle/store/` (reusing `sha256_hex`); redaction wired to the
existing `gateway/egress/dlp_detectors` (`scan_token_patterns` /
`redact_tokens`); unit tests for determinism, chain-break detection,
`redact_tokens`); embed the git SHA at build so `engine` is populated
(only `version = "0.1.0"` exists in `pyproject.toml` today — the build
must stamp the SHA); unit tests for determinism, chain-break detection,
`epoch`/`seq` continuity across a simulated restart, and redaction of
both a deny-listed key and a token-shaped value.
3. **SQLite index + `audit rebuild` / `audit verify` CLI.** New
`audit_events` migration; replay-from-journal; offline chain verifier;
local query commands (by bottled-agent / type / time / producer).
3. **SQLite index + `audit` CLI.** New `audit_events` migration (indexable
fields above); replay-from-journal; offline chain verifier
(`truncated-tail` vs `gap`); `query` / `verify` / `rebuild` / `import`;
idempotent `UPSERT` by `id`.
4. **Host controller as first producer (#468).** Wire
`lifecycle.bottled_agent_*` emission into the host controller's
start/stop/crash paths; establish the `epoch` bump + chain-head carry on
writer restart here (the host controller owns the single writer).
`lifecycle.bottled_agent_*` and `hostctl.*` emission into the host
controller; establish the `epoch` bump + chain-head carry + segment
rotation on writer restart here (it owns the single writer).
5. **Migrate existing producers.** Re-emit supervise `decision.*` (retiring
the standalone `supervise_audit_entries` shape behind the index), egress
`egress.*`, git-gate `forge.*`, control-plane `auth.*`.
6. **(follow-up.)** Cross-host merge transport (#324); per-writer signing +
external anchoring on the chain head; retention/rotation policy.
`egress.*`, git-gate `forge.*` + `commit.signed` (#480), control-plane
`auth.*`; add the `audit.*` self-events (verify/truncation/export
failure).
6. **CloudEvents / OTel export adapters + #324 delivery.** Projection layer
(field re-map per *Export / interoperability*) plus the outbox cursor,
ack-driven advance, and retention-ordering guard the #324 contract
specifies.
7. **(follow-up.)** Cross-host merge transport; per-writer signing +
external anchoring on the chain head; retention/rotation *schedule*.
## Acceptance-criteria coverage (#487)
The issue defines the contract; implementation is explicitly split into
follow-up PRs. This PRD is the durable decision record; each acceptance box
maps to a section:
| #487 acceptance criterion | Where |
|---|---|
| Durable PRD defines versioned envelope + initial registry | *The envelope*, *Event registry* |
| Canonical JSON + hash-chain rules, unambiguous, with test vectors | *Canonical serialization + hash chain* → *Test vectors* |
| Trust provenance explicit for every common + event-specific field | *Trust provenance of every common field*; per-type `untrusted` in *Event registry* |
| Redaction prohibits credentials / raw secrets / unsafe capture by default | *Redaction rule*; `untrusted`-only claims; sensitivity classes |
| JSONL journal canonical; SQLite index fully rebuildable | *Journal + SQLite index* (`audit rebuild`) |
| Minimum local search/query contract | *Journal + SQLite index* → *Local query surface* |
| #324 can transport/replay without a second envelope | *The #324 delivery contract* |
| #480 maps in without weakening its byte-to-activation-key guarantee | *#480 signed-commit attribution maps in…* |
| Schema evolution + backward-compatible readers | *Schema evolution & backward-compatible readers* |
| Integrity detects modification / deletion / reorder / bad continuation | *Test vectors* (tamper), *Behavior across rotation…truncation*, `audit verify` |
Two acceptance items are **specified here, implemented later** by design
(the issue permits this): the concrete test-vector *fixtures* and the
`audit` CLI land in impl chunks 23; the #324 outbox lands in chunk 6.
Nothing in the contract is left undefined — only its code is deferred.
## Resolved in review (#495)
@@ -320,15 +700,17 @@ credential in a named field is always a producer bug worth surfacing.
carrying the last chain head as the next `prev` gives a total order of
`(epoch, seq)` that survives restarts; `ts_mono` orders only within an
epoch. (Design → *ordering across restarts*.)
- **`ts_*` and `producer` belong in `trusted`.** They are host-supplied, so
they now sit inside the `trusted` block; only chain metadata stays at the
top level. (Design → *The envelope*.)
- **Subject term is `bottled_agent` everywhere** — the `trusted` field and
- **Flatten to one `untrusted` region — decided.** Everything outside
`untrusted` (chain metadata, `producer`/`host`, `bottled_agent`, `ts_*`)
is trusted by construction, so the separate `trusted` sub-block is
removed; a field is trusted unless deliberately placed under `untrusted`.
(Design → *The envelope*.)
- **Subject term is `bottled_agent` everywhere** — the top-level field and
the `lifecycle.bottled_agent_*` leaf names. (Design → *The envelope* /
*Event registry*.)
- **Retention head-carry — yes.** When a journal segment is rotated out,
the new segment's genesis `prev` is the rotated-out head, so the verifier
still trusts the current head across a rotation. (Folds into the
the new segment's first record carries the rotated-out head as `prev`, so
the verifier still trusts the current head across a rotation. (Folds into the
retention follow-up.)
- **Redaction reuses the egress detectors — yes, scoped.** Reuse the
deterministic `dlp_detectors` (`scan_token_patterns` / `redact_tokens` /