Compare commits

...

2 Commits

Author SHA1 Message Date
didericis-codex 4cb7bbbd11 docs(prd): tighten audit ordering and chain guarantees
prd-number-check / require-numbered-prds (pull_request) Failing after 6s
tracker-policy-pr / check-pr (pull_request) Successful in 6s
2026-07-26 17:38:43 +00:00
didericis-claude f3664dea9f docs(prd): drop policy_version (manifest is the policy); add engine field
prd-number-check / require-numbered-prds (pull_request) Failing after 6s
tracker-policy-pr / check-pr (pull_request) Successful in 5s
Per review: bot-bottle has no separate policy artifact — a bottled agent's
egress routes etc. are declared in its manifest (manifest/egress.py), so
manifest_digest already pins the policy in force. Remove the redundant
policy_version. Given a fixed manifest, the only other axis that changes an
outcome is the enforcing code, so add 'engine' (bot-bottle version + git
SHA) as a trusted field. Runtime operator overrides (supervise egress-allow)
are themselves audit events, so effective policy = manifest_digest + logged
deltas, reconstructable from the chain. Note the build must stamp the git
SHA (only version=0.1.0 exists today).

Refs #487

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-26 08:28:10 +00:00
+83 -40
View File
@@ -21,7 +21,8 @@ emits into:
1. A **versioned envelope** — schema version, event id, event/observed
timestamps, host-attributed identity (`bottle`/`bottled_agent`/
`activation`), provenance (`manifest_digest`, `policy_version`),
`activation`), provenance (`manifest_digest` — the manifest is the
policy — and `engine` = bot-bottle version/SHA),
host-observed `actor`/`action`/`resource`/`outcome`, correlation/
causation ids, a sensitivity class, a typed payload, and an explicit
**trust boundary** between host-supplied and agent-claimed fields.
@@ -36,7 +37,8 @@ emits into:
and signed-commit (#480), auth/authz, and audit self-events — each with
its trusted-vs-claimed fields and redaction rules.
5. A **stable export projection** (CloudEvents / OpenTelemetry Logs) and
the **#324 delivery contract** (payload, `(epoch, seq)` cursor, dedup,
the **#324 delivery contract** (payload, per-chain
`(host, epoch, seq)` cursor, dedup,
backpressure, retention ordering).
It is explicitly scheduled to land **immediately after the host
@@ -141,7 +143,7 @@ stable top level:
"epoch": 7, // writer-boot counter, bumped once per host-controller (writer) start
"seq": 1287, // monotonic sequence within this epoch (gap-detectable)
"segment": "20260726T000000Z", // journal segment id (rotation boundary); chain continues across segments
"prev": "<hex>", // hash of the previous record in the chain ("" for a segment genesis)
"prev": "<hex>", // prior record hash ("" only for the first record of a new chain)
"hash": "<hex>", // sha256(prev + canonical(this event with hash=""))
// ---- timestamps (host-owned) ----
@@ -152,11 +154,11 @@ stable top level:
// ---- attribution + provenance (host-established) ----
"producer": "egress", // host component that emitted the event
"host": "mac-studio-1",
"engine": "bot-bottle/0.1.0+abc1234", // bot-bottle version + git SHA of the enforcing host code
"bottle": "amber-fox", // bottle (container/VM) identity
"bottled_agent": "amber-fox-12", // bottled-agent slug from source-IP attribution (null for host-level events)
"activation": "01J8Z...", // activation id: one run/session of the bottled agent (null if n/a)
"manifest_digest": "sha256:9f2…", // digest of the manifest the bottled agent is running (null if n/a)
"policy_version": "egress-2026-07-20", // policy/config version in force at decision time (null if n/a)
"manifest_digest": "sha256:9f2…", // digest of the manifest — which IS the policy (egress routes etc.); null if n/a
// ---- semantics: host-observed facts of what happened ----
"actor": "bottled-agent:amber-fox-12", // who acted, as a host-attributed identity
@@ -186,7 +188,7 @@ The **trust boundary is a single region, not a split.** Everything outside
`untrusted` is trusted by construction — the host established it: schema and
chain metadata, both timestamps, the attribution/provenance fields
(source-IP → `bottled_agent`/`bottle`/`activation`, `manifest_digest`,
`policy_version`), the host-observed semantics
`engine`), the host-observed semantics
(`actor`/`action`/`resource`/`outcome`), the host-assigned correlation ids,
and the typed `payload`. `untrusted` is the **one** place anything an agent
or a remote claimed may go (e.g. the agent's free-text `reason`). A reader
@@ -209,12 +211,26 @@ the single writer durably appended it. Ordering and the chain use
`(epoch, seq)`, never either wall clock. Both are host-set — a bottled
agent never supplies a timestamp.
**The manifest *is* the policy.** bot-bottle has no separate policy
artifact — a bottled agent's egress routes and other constraints are
declared in its manifest (`bot_bottle/manifest/egress.py`), so
`manifest_digest` already pins the ruleset in force; there is no distinct
`policy_version`. Given a fixed manifest, the only other thing that can
change a decision's outcome is the enforcing code — captured by `engine`
(bot-bottle version + git SHA). So two `egress.decision` records with the
same `resource` but different `outcome` are explained by exactly one of:
different `manifest_digest` (the rules changed) or different `engine` (the
enforcer changed). Runtime operator overrides (a supervise `egress-allow`)
are themselves audit events, so the effective ruleset at any instant is
`manifest_digest` plus the logged, approved deltas — reconstructable from
the chain, not from a version stamp.
**Optionality.** `bottle`/`bottled_agent`/`activation`, `manifest_digest`,
`policy_version`, and `payload`/`untrusted` are `null`/absent for events
that have no such subject (a host-level `hostctl.*` or `audit.*` event has
no bottled agent). Absent ≠ empty: a reader distinguishes "no subject" from
"unknown". `id`, `type`, the chain fields, both timestamps, `producer`,
`host`, `actor`, `action`, `outcome`, and `sensitivity` are always present.
and `payload`/`untrusted` are `null`/absent for events that have no such
subject (a host-level `hostctl.*` or `audit.*` event has no bottled agent).
Absent ≠ empty: a reader distinguishes "no subject" from "unknown". `id`,
`type`, the chain fields, both timestamps, `producer`, `host`, `engine`,
`actor`, `action`, `outcome`, and `sensitivity` are always present.
#### Trust provenance of every common field
@@ -223,9 +239,9 @@ no bottled agent). Absent ≠ empty: a reader distinguishes "no subject" from
| `v` `id` `type` `epoch` `seq` `segment` `prev` `hash` | trusted | the single writer |
| `ts_event` | trusted | emitting host component (boundary) |
| `ts_recorded` `ts_mono` | trusted | the single writer |
| `producer` `host` | trusted | the single writer |
| `producer` `host` `engine` | trusted | the single writer |
| `bottle` `bottled_agent` `activation` | trusted | gateway source-IP → slug attribution |
| `manifest_digest` `policy_version` | trusted | control plane (what was in force) |
| `manifest_digest` | trusted | control plane (the manifest = the policy in force) |
| `actor` `action` `resource` `outcome` | trusted | host component that observed/decided it |
| `sensitivity` | trusted | registry default for `type`, overridable up (never down) by the producer |
| `correlation_id` `causation_id` | trusted | the single writer (assigned as it threads the flow) |
@@ -255,7 +271,9 @@ The `hash` field is computed over the canonical form of the event **with
digest = sha256_hex(prev_hash + canonical({**event, "hash": ""}))
```
`prev` is the prior record's `hash`; a segment genesis uses `prev = ""`.
`prev` is the prior record's `hash`; only the first record of a brand-new
chain uses `prev = ""`. The first record of a rotated segment carries the
preceding segment's head, as specified under *Rotation* below.
Two exact rules pin the bytes so the chain is reproducible anywhere:
1. **Serialize the record with its own `hash` field set to `""`** (present,
@@ -281,7 +299,7 @@ serialization and linkage. An implementation is conformant iff it
reproduces these bytes and hashes.
```
# Record 0 — segment genesis (prev = "")
# Record 0 — brand-new chain genesis (prev = "")
canonical(record0, hash=""):
{"hash":"","id":"11111111-1111-4111-8111-111111111111","prev":"","seq":0,"type":"audit.segment_open"}
hash0 = sha256("" + canonical) =
@@ -304,17 +322,28 @@ bump that changes the bytes fails a golden test loudly.
### Ordering, idempotency, and duplicate handling
- **Ordering.** `(epoch, seq)` is a strict total order per host and, because
- **Ordering.** `(epoch, seq)` is a strict total order within one host's
native chain and, because
the writer is single, a strict order per bottle/activation within that
host — satisfying "at least strict causal order per activation/bottle".
`causation_id` records the explicit cause edges (a DAG) on top of the
total order, so a consumer can reconstruct request → decision → cutoff
even if unrelated events interleave between them.
even if unrelated events interleave between them. There is deliberately
no invented total order across imported host chains; a cross-host key is
`(host, epoch, seq)`, and consumers use correlation/causation edges where
causal ordering across hosts is known.
- **Idempotency.** `id` is the idempotency key. A producer that retries an
emit (e.g. after a writer restart mid-handoff) **reuses the same `id`**;
the writer drops a second append bearing an `id` already present in the
current segment's in-memory set, and the index `UPSERT`s by `id`, so a
duplicate never double-counts or forks the chain.
before appending, the writer checks a durable, host-wide id ledger that
spans every segment and survives restart/retention, and drops an id
already present. The ledger is operational metadata, not an audit source
of truth: after a crash it is reconciled from the journal before appends
resume, and retention preserves id tombstones after journal segments are
pruned. An in-memory set may cache the ledger but is never the authority.
The index additionally has a unique key on `id`; its `UPSERT` is
defensive and does not substitute for the pre-append check. Thus a
duplicate never enters the source-of-truth journal, double-counts, or
forks the chain.
- **Deduplication downstream.** Because `id` is stable across export and
replay, #324's cursor replay and any cross-host merge dedup on `id` — no
consumer needs to invent a second identity.
@@ -323,18 +352,22 @@ bump that changes the bytes fails a golden test loudly.
- **Rotation.** At a segment boundary the writer opens a new segment file,
sets its `segment` id, and carries the rotated-out segment's head as the
new genesis `prev` — so the chain is continuous *across* segments while
new segment's first `prev` — so the chain is continuous *across* segments
(`prev = ""` is reserved for the first record of a brand-new chain) while
each file stays independently openable. `verify` walks segments in order
and checks the head-to-genesis link at each seam.
and checks the preceding-head-to-first-record link at each seam.
- **Restart.** Covered above: read last line → adopt its `hash` as `prev`,
bump `epoch`, reset `seq`. The chain never restarts even though the
counters do.
- **Import.** `audit import <segment>` appends an externally supplied
segment (e.g. recovered from another host or a backup). Import verifies
the incoming chain in isolation first, then links it only if its genesis
`prev` matches a known head or is explicitly grafted; imported records
keep their original `id` (dedup) and are marked with their origin host so
attribution is not laundered.
the incoming chain in isolation first. A continuation whose first `prev`
matches a known head extends that chain. A foreign chain is registered as
a separate immutable chain namespace rather than rewriting or grafting
its records (which would invalidate their hashes). Imported records keep
their original `host`, `id`, `epoch`, and `seq`; the index keys their
native order by `(host, epoch, seq)` so attribution is not laundered and
tuples from different hosts cannot collide.
- **Truncation.** A crash can leave a partial final line; `verify` reports
it as `truncated-tail` (recoverable — replay resumes from the last intact
record). A chain that ends before a persisted head, or a missing interior
@@ -361,11 +394,13 @@ start the writer:
2. bumps `epoch` (persisted alongside the chain head) and resets `seq` to
0 for the new boot.
Total order is therefore `(epoch, seq)` — monotonic across restarts by
construction — with `ts_event`/`ts_recorded` for human reading and `ts_mono` for
sub-second ordering inside an epoch. A crash mid-append truncates at most
the last (partial) line; the verifier flags it and replay resumes from the
last intact record.
Total order within this host's native chain is therefore `(epoch, seq)` —
monotonic across restarts by construction — with
`ts_event`/`ts_recorded` for human reading and `ts_mono` for sub-second
ordering inside an epoch. Imported chains retain their own
`(host, epoch, seq)` order and do not acquire a fictional order relative to
the local chain. A crash mid-append truncates at most the last (partial)
line; the verifier flags it and replay resumes from the last intact record.
### Journal (source of truth) + SQLite index (rebuildable)
@@ -386,8 +421,11 @@ last intact record.
`outcome`, `sensitivity`, `correlation_id`, `causation_id`, plus two
event-specific projections promoted out of `payload` for query —
`repository` and `commit_sha` (populated for `forge.*`/`commit.*`, null
otherwise). The full canonical record is stored verbatim in a `raw` column
so the index never loses fidelity to the journal.
otherwise). The unique event id and native-order index are respectively
`id` and `(host, epoch, seq)`; a local-only `ingest_seq` provides stable
display order when a query intentionally mixes chains without pretending
that it is causal order. The full canonical record is stored verbatim in a
`raw` column so the index never loses fidelity to the journal.
**Local query surface** — `audit query`, no egress, no paid platform:
@@ -396,7 +434,7 @@ audit query \
[--since T] [--until T] [--type egress.*] [--host H] [--bottle B] \
[--activation A] [--agent SLUG] [--actor ID] [--outcome blocked] \
[--repository R] [--correlation-id C] [--commit SHA] \
[--follow BOTTLE] # a bottled agent's events in (epoch, seq) order
[--follow BOTTLE] # one host chain's events in (epoch, seq) order
[--json | --table]
audit verify [--segment S] # offline chain check; exit non-zero on any break
audit rebuild # drop + replay journal → index
@@ -563,9 +601,12 @@ PRD fixes the contract it depends on:
exact bytes the hash covers), optionally wrapped in the CloudEvents
projection whose `data` *is* that record. Either way the integrity fields
travel intact and the receiver can verify against the same bytes.
- **Cursor.** The export cursor is `(epoch, seq)` (equivalently the last
exported `hash`). It advances **only on acknowledgement**, so delivery is
at-least-once and gap-free; a crash re-sends from the last acked cursor.
- **Cursor.** Export maintains one cursor per native host chain:
`(host, epoch, seq)` (equivalently that chain's last exported `hash`).
It advances **only on acknowledgement**, so delivery is at-least-once and
gap-free; a crash re-sends from the last acked cursor. Imported foreign
chains use independent cursors and never share a bare `(epoch, seq)`
namespace with the local chain.
- **Idempotency / replay.** Dedup is on `id` (stable across replay), so
at-least-once delivery is safe — the receiver collapses re-sends.
- **Backpressure.** The outbox is the journal itself plus a cursor; when
@@ -601,7 +642,9 @@ the committing agent.
`canonical()`, chain hashing, and the single-writer journal appender in
`bot_bottle/store/` (reusing `sha256_hex`); redaction wired to the
existing `gateway/egress/dlp_detectors` (`scan_token_patterns` /
`redact_tokens`); unit tests for determinism, chain-break detection,
`redact_tokens`); embed the git SHA at build so `engine` is populated
(only `version = "0.1.0"` exists in `pyproject.toml` today — the build
must stamp the SHA); unit tests for determinism, chain-break detection,
`epoch`/`seq` continuity across a simulated restart, and redaction of
both a deny-listed key and a token-shaped value.
3. **SQLite index + `audit` CLI.** New `audit_events` migration (indexable
@@ -666,8 +709,8 @@ Nothing in the contract is left undefined — only its code is deferred.
the `lifecycle.bottled_agent_*` leaf names. (Design → *The envelope* /
*Event registry*.)
- **Retention head-carry — yes.** When a journal segment is rotated out,
the new segment's genesis `prev` is the rotated-out head, so the verifier
still trusts the current head across a rotation. (Folds into the
the new segment's first record carries the rotated-out head as `prev`, so
the verifier still trusts the current head across a rotation. (Folds into the
retention follow-up.)
- **Redaction reuses the egress detectors — yes, scoped.** Reuse the
deterministic `dlp_detectors` (`scan_token_patterns` / `redact_tokens` /