Compare commits
2 Commits
| Author | SHA1 | Date | |
|---|---|---|---|
| 4cb7bbbd11 | |||
| f3664dea9f |
@@ -21,7 +21,8 @@ emits into:
|
||||
|
||||
1. A **versioned envelope** — schema version, event id, event/observed
|
||||
timestamps, host-attributed identity (`bottle`/`bottled_agent`/
|
||||
`activation`), provenance (`manifest_digest`, `policy_version`),
|
||||
`activation`), provenance (`manifest_digest` — the manifest is the
|
||||
policy — and `engine` = bot-bottle version/SHA),
|
||||
host-observed `actor`/`action`/`resource`/`outcome`, correlation/
|
||||
causation ids, a sensitivity class, a typed payload, and an explicit
|
||||
**trust boundary** between host-supplied and agent-claimed fields.
|
||||
@@ -36,7 +37,8 @@ emits into:
|
||||
and signed-commit (#480), auth/authz, and audit self-events — each with
|
||||
its trusted-vs-claimed fields and redaction rules.
|
||||
5. A **stable export projection** (CloudEvents / OpenTelemetry Logs) and
|
||||
the **#324 delivery contract** (payload, `(epoch, seq)` cursor, dedup,
|
||||
the **#324 delivery contract** (payload, per-chain
|
||||
`(host, epoch, seq)` cursor, dedup,
|
||||
backpressure, retention ordering).
|
||||
|
||||
It is explicitly scheduled to land **immediately after the host
|
||||
@@ -141,7 +143,7 @@ stable top level:
|
||||
"epoch": 7, // writer-boot counter, bumped once per host-controller (writer) start
|
||||
"seq": 1287, // monotonic sequence within this epoch (gap-detectable)
|
||||
"segment": "20260726T000000Z", // journal segment id (rotation boundary); chain continues across segments
|
||||
"prev": "<hex>", // hash of the previous record in the chain ("" for a segment genesis)
|
||||
"prev": "<hex>", // prior record hash ("" only for the first record of a new chain)
|
||||
"hash": "<hex>", // sha256(prev + canonical(this event with hash=""))
|
||||
|
||||
// ---- timestamps (host-owned) ----
|
||||
@@ -152,11 +154,11 @@ stable top level:
|
||||
// ---- attribution + provenance (host-established) ----
|
||||
"producer": "egress", // host component that emitted the event
|
||||
"host": "mac-studio-1",
|
||||
"engine": "bot-bottle/0.1.0+abc1234", // bot-bottle version + git SHA of the enforcing host code
|
||||
"bottle": "amber-fox", // bottle (container/VM) identity
|
||||
"bottled_agent": "amber-fox-12", // bottled-agent slug from source-IP attribution (null for host-level events)
|
||||
"activation": "01J8Z...", // activation id: one run/session of the bottled agent (null if n/a)
|
||||
"manifest_digest": "sha256:9f2…", // digest of the manifest the bottled agent is running (null if n/a)
|
||||
"policy_version": "egress-2026-07-20", // policy/config version in force at decision time (null if n/a)
|
||||
"manifest_digest": "sha256:9f2…", // digest of the manifest — which IS the policy (egress routes etc.); null if n/a
|
||||
|
||||
// ---- semantics: host-observed facts of what happened ----
|
||||
"actor": "bottled-agent:amber-fox-12", // who acted, as a host-attributed identity
|
||||
@@ -186,7 +188,7 @@ The **trust boundary is a single region, not a split.** Everything outside
|
||||
`untrusted` is trusted by construction — the host established it: schema and
|
||||
chain metadata, both timestamps, the attribution/provenance fields
|
||||
(source-IP → `bottled_agent`/`bottle`/`activation`, `manifest_digest`,
|
||||
`policy_version`), the host-observed semantics
|
||||
`engine`), the host-observed semantics
|
||||
(`actor`/`action`/`resource`/`outcome`), the host-assigned correlation ids,
|
||||
and the typed `payload`. `untrusted` is the **one** place anything an agent
|
||||
or a remote claimed may go (e.g. the agent's free-text `reason`). A reader
|
||||
@@ -209,12 +211,26 @@ the single writer durably appended it. Ordering and the chain use
|
||||
`(epoch, seq)`, never either wall clock. Both are host-set — a bottled
|
||||
agent never supplies a timestamp.
|
||||
|
||||
**The manifest *is* the policy.** bot-bottle has no separate policy
|
||||
artifact — a bottled agent's egress routes and other constraints are
|
||||
declared in its manifest (`bot_bottle/manifest/egress.py`), so
|
||||
`manifest_digest` already pins the ruleset in force; there is no distinct
|
||||
`policy_version`. Given a fixed manifest, the only other thing that can
|
||||
change a decision's outcome is the enforcing code — captured by `engine`
|
||||
(bot-bottle version + git SHA). So two `egress.decision` records with the
|
||||
same `resource` but different `outcome` are explained by exactly one of:
|
||||
different `manifest_digest` (the rules changed) or different `engine` (the
|
||||
enforcer changed). Runtime operator overrides (a supervise `egress-allow`)
|
||||
are themselves audit events, so the effective ruleset at any instant is
|
||||
`manifest_digest` plus the logged, approved deltas — reconstructable from
|
||||
the chain, not from a version stamp.
|
||||
|
||||
**Optionality.** `bottle`/`bottled_agent`/`activation`, `manifest_digest`,
|
||||
`policy_version`, and `payload`/`untrusted` are `null`/absent for events
|
||||
that have no such subject (a host-level `hostctl.*` or `audit.*` event has
|
||||
no bottled agent). Absent ≠ empty: a reader distinguishes "no subject" from
|
||||
"unknown". `id`, `type`, the chain fields, both timestamps, `producer`,
|
||||
`host`, `actor`, `action`, `outcome`, and `sensitivity` are always present.
|
||||
and `payload`/`untrusted` are `null`/absent for events that have no such
|
||||
subject (a host-level `hostctl.*` or `audit.*` event has no bottled agent).
|
||||
Absent ≠ empty: a reader distinguishes "no subject" from "unknown". `id`,
|
||||
`type`, the chain fields, both timestamps, `producer`, `host`, `engine`,
|
||||
`actor`, `action`, `outcome`, and `sensitivity` are always present.
|
||||
|
||||
#### Trust provenance of every common field
|
||||
|
||||
@@ -223,9 +239,9 @@ no bottled agent). Absent ≠ empty: a reader distinguishes "no subject" from
|
||||
| `v` `id` `type` `epoch` `seq` `segment` `prev` `hash` | trusted | the single writer |
|
||||
| `ts_event` | trusted | emitting host component (boundary) |
|
||||
| `ts_recorded` `ts_mono` | trusted | the single writer |
|
||||
| `producer` `host` | trusted | the single writer |
|
||||
| `producer` `host` `engine` | trusted | the single writer |
|
||||
| `bottle` `bottled_agent` `activation` | trusted | gateway source-IP → slug attribution |
|
||||
| `manifest_digest` `policy_version` | trusted | control plane (what was in force) |
|
||||
| `manifest_digest` | trusted | control plane (the manifest = the policy in force) |
|
||||
| `actor` `action` `resource` `outcome` | trusted | host component that observed/decided it |
|
||||
| `sensitivity` | trusted | registry default for `type`, overridable up (never down) by the producer |
|
||||
| `correlation_id` `causation_id` | trusted | the single writer (assigned as it threads the flow) |
|
||||
@@ -255,7 +271,9 @@ The `hash` field is computed over the canonical form of the event **with
|
||||
digest = sha256_hex(prev_hash + canonical({**event, "hash": ""}))
|
||||
```
|
||||
|
||||
`prev` is the prior record's `hash`; a segment genesis uses `prev = ""`.
|
||||
`prev` is the prior record's `hash`; only the first record of a brand-new
|
||||
chain uses `prev = ""`. The first record of a rotated segment carries the
|
||||
preceding segment's head, as specified under *Rotation* below.
|
||||
Two exact rules pin the bytes so the chain is reproducible anywhere:
|
||||
|
||||
1. **Serialize the record with its own `hash` field set to `""`** (present,
|
||||
@@ -281,7 +299,7 @@ serialization and linkage. An implementation is conformant iff it
|
||||
reproduces these bytes and hashes.
|
||||
|
||||
```
|
||||
# Record 0 — segment genesis (prev = "")
|
||||
# Record 0 — brand-new chain genesis (prev = "")
|
||||
canonical(record0, hash=""):
|
||||
{"hash":"","id":"11111111-1111-4111-8111-111111111111","prev":"","seq":0,"type":"audit.segment_open"}
|
||||
hash0 = sha256("" + canonical) =
|
||||
@@ -304,17 +322,28 @@ bump that changes the bytes fails a golden test loudly.
|
||||
|
||||
### Ordering, idempotency, and duplicate handling
|
||||
|
||||
- **Ordering.** `(epoch, seq)` is a strict total order per host and, because
|
||||
- **Ordering.** `(epoch, seq)` is a strict total order within one host's
|
||||
native chain and, because
|
||||
the writer is single, a strict order per bottle/activation within that
|
||||
host — satisfying "at least strict causal order per activation/bottle".
|
||||
`causation_id` records the explicit cause edges (a DAG) on top of the
|
||||
total order, so a consumer can reconstruct request → decision → cutoff
|
||||
even if unrelated events interleave between them.
|
||||
even if unrelated events interleave between them. There is deliberately
|
||||
no invented total order across imported host chains; a cross-host key is
|
||||
`(host, epoch, seq)`, and consumers use correlation/causation edges where
|
||||
causal ordering across hosts is known.
|
||||
- **Idempotency.** `id` is the idempotency key. A producer that retries an
|
||||
emit (e.g. after a writer restart mid-handoff) **reuses the same `id`**;
|
||||
the writer drops a second append bearing an `id` already present in the
|
||||
current segment's in-memory set, and the index `UPSERT`s by `id`, so a
|
||||
duplicate never double-counts or forks the chain.
|
||||
before appending, the writer checks a durable, host-wide id ledger that
|
||||
spans every segment and survives restart/retention, and drops an id
|
||||
already present. The ledger is operational metadata, not an audit source
|
||||
of truth: after a crash it is reconciled from the journal before appends
|
||||
resume, and retention preserves id tombstones after journal segments are
|
||||
pruned. An in-memory set may cache the ledger but is never the authority.
|
||||
The index additionally has a unique key on `id`; its `UPSERT` is
|
||||
defensive and does not substitute for the pre-append check. Thus a
|
||||
duplicate never enters the source-of-truth journal, double-counts, or
|
||||
forks the chain.
|
||||
- **Deduplication downstream.** Because `id` is stable across export and
|
||||
replay, #324's cursor replay and any cross-host merge dedup on `id` — no
|
||||
consumer needs to invent a second identity.
|
||||
@@ -323,18 +352,22 @@ bump that changes the bytes fails a golden test loudly.
|
||||
|
||||
- **Rotation.** At a segment boundary the writer opens a new segment file,
|
||||
sets its `segment` id, and carries the rotated-out segment's head as the
|
||||
new genesis `prev` — so the chain is continuous *across* segments while
|
||||
new segment's first `prev` — so the chain is continuous *across* segments
|
||||
(`prev = ""` is reserved for the first record of a brand-new chain) while
|
||||
each file stays independently openable. `verify` walks segments in order
|
||||
and checks the head-to-genesis link at each seam.
|
||||
and checks the preceding-head-to-first-record link at each seam.
|
||||
- **Restart.** Covered above: read last line → adopt its `hash` as `prev`,
|
||||
bump `epoch`, reset `seq`. The chain never restarts even though the
|
||||
counters do.
|
||||
- **Import.** `audit import <segment>` appends an externally supplied
|
||||
segment (e.g. recovered from another host or a backup). Import verifies
|
||||
the incoming chain in isolation first, then links it only if its genesis
|
||||
`prev` matches a known head or is explicitly grafted; imported records
|
||||
keep their original `id` (dedup) and are marked with their origin host so
|
||||
attribution is not laundered.
|
||||
the incoming chain in isolation first. A continuation whose first `prev`
|
||||
matches a known head extends that chain. A foreign chain is registered as
|
||||
a separate immutable chain namespace rather than rewriting or grafting
|
||||
its records (which would invalidate their hashes). Imported records keep
|
||||
their original `host`, `id`, `epoch`, and `seq`; the index keys their
|
||||
native order by `(host, epoch, seq)` so attribution is not laundered and
|
||||
tuples from different hosts cannot collide.
|
||||
- **Truncation.** A crash can leave a partial final line; `verify` reports
|
||||
it as `truncated-tail` (recoverable — replay resumes from the last intact
|
||||
record). A chain that ends before a persisted head, or a missing interior
|
||||
@@ -361,11 +394,13 @@ start the writer:
|
||||
2. bumps `epoch` (persisted alongside the chain head) and resets `seq` to
|
||||
0 for the new boot.
|
||||
|
||||
Total order is therefore `(epoch, seq)` — monotonic across restarts by
|
||||
construction — with `ts_event`/`ts_recorded` for human reading and `ts_mono` for
|
||||
sub-second ordering inside an epoch. A crash mid-append truncates at most
|
||||
the last (partial) line; the verifier flags it and replay resumes from the
|
||||
last intact record.
|
||||
Total order within this host's native chain is therefore `(epoch, seq)` —
|
||||
monotonic across restarts by construction — with
|
||||
`ts_event`/`ts_recorded` for human reading and `ts_mono` for sub-second
|
||||
ordering inside an epoch. Imported chains retain their own
|
||||
`(host, epoch, seq)` order and do not acquire a fictional order relative to
|
||||
the local chain. A crash mid-append truncates at most the last (partial)
|
||||
line; the verifier flags it and replay resumes from the last intact record.
|
||||
|
||||
### Journal (source of truth) + SQLite index (rebuildable)
|
||||
|
||||
@@ -386,8 +421,11 @@ last intact record.
|
||||
`outcome`, `sensitivity`, `correlation_id`, `causation_id`, plus two
|
||||
event-specific projections promoted out of `payload` for query —
|
||||
`repository` and `commit_sha` (populated for `forge.*`/`commit.*`, null
|
||||
otherwise). The full canonical record is stored verbatim in a `raw` column
|
||||
so the index never loses fidelity to the journal.
|
||||
otherwise). The unique event id and native-order index are respectively
|
||||
`id` and `(host, epoch, seq)`; a local-only `ingest_seq` provides stable
|
||||
display order when a query intentionally mixes chains without pretending
|
||||
that it is causal order. The full canonical record is stored verbatim in a
|
||||
`raw` column so the index never loses fidelity to the journal.
|
||||
|
||||
**Local query surface** — `audit query`, no egress, no paid platform:
|
||||
|
||||
@@ -396,7 +434,7 @@ audit query \
|
||||
[--since T] [--until T] [--type egress.*] [--host H] [--bottle B] \
|
||||
[--activation A] [--agent SLUG] [--actor ID] [--outcome blocked] \
|
||||
[--repository R] [--correlation-id C] [--commit SHA] \
|
||||
[--follow BOTTLE] # a bottled agent's events in (epoch, seq) order
|
||||
[--follow BOTTLE] # one host chain's events in (epoch, seq) order
|
||||
[--json | --table]
|
||||
audit verify [--segment S] # offline chain check; exit non-zero on any break
|
||||
audit rebuild # drop + replay journal → index
|
||||
@@ -563,9 +601,12 @@ PRD fixes the contract it depends on:
|
||||
exact bytes the hash covers), optionally wrapped in the CloudEvents
|
||||
projection whose `data` *is* that record. Either way the integrity fields
|
||||
travel intact and the receiver can verify against the same bytes.
|
||||
- **Cursor.** The export cursor is `(epoch, seq)` (equivalently the last
|
||||
exported `hash`). It advances **only on acknowledgement**, so delivery is
|
||||
at-least-once and gap-free; a crash re-sends from the last acked cursor.
|
||||
- **Cursor.** Export maintains one cursor per native host chain:
|
||||
`(host, epoch, seq)` (equivalently that chain's last exported `hash`).
|
||||
It advances **only on acknowledgement**, so delivery is at-least-once and
|
||||
gap-free; a crash re-sends from the last acked cursor. Imported foreign
|
||||
chains use independent cursors and never share a bare `(epoch, seq)`
|
||||
namespace with the local chain.
|
||||
- **Idempotency / replay.** Dedup is on `id` (stable across replay), so
|
||||
at-least-once delivery is safe — the receiver collapses re-sends.
|
||||
- **Backpressure.** The outbox is the journal itself plus a cursor; when
|
||||
@@ -601,7 +642,9 @@ the committing agent.
|
||||
`canonical()`, chain hashing, and the single-writer journal appender in
|
||||
`bot_bottle/store/` (reusing `sha256_hex`); redaction wired to the
|
||||
existing `gateway/egress/dlp_detectors` (`scan_token_patterns` /
|
||||
`redact_tokens`); unit tests for determinism, chain-break detection,
|
||||
`redact_tokens`); embed the git SHA at build so `engine` is populated
|
||||
(only `version = "0.1.0"` exists in `pyproject.toml` today — the build
|
||||
must stamp the SHA); unit tests for determinism, chain-break detection,
|
||||
`epoch`/`seq` continuity across a simulated restart, and redaction of
|
||||
both a deny-listed key and a token-shaped value.
|
||||
3. **SQLite index + `audit` CLI.** New `audit_events` migration (indexable
|
||||
@@ -666,8 +709,8 @@ Nothing in the contract is left undefined — only its code is deferred.
|
||||
the `lifecycle.bottled_agent_*` leaf names. (Design → *The envelope* /
|
||||
*Event registry*.)
|
||||
- **Retention head-carry — yes.** When a journal segment is rotated out,
|
||||
the new segment's genesis `prev` is the rotated-out head, so the verifier
|
||||
still trusts the current head across a rotation. (Folds into the
|
||||
the new segment's first record carries the rotated-out head as `prev`, so
|
||||
the verifier still trusts the current head across a rotation. (Folds into the
|
||||
retention follow-up.)
|
||||
- **Redaction reuses the egress detectors — yes, scoped.** Reuse the
|
||||
deterministic `dlp_detectors` (`scan_token_patterns` / `redact_tokens` /
|
||||
|
||||
Reference in New Issue
Block a user