docs/prds/prd-new-audit-event-schema.md

@@ -14,7 +14,7 @@ control-plane token minting, and (next) host-controller lifecycle
 transitions — but each writes its own shape to its own sink. There is no
 shared envelope, no tamper-evidence, and no single place to search. Local
 incident reconstruction means grepping several stores that don't agree on
-field names, timestamps, or how a bottle is identified.
+field names, timestamps, or how a bottled agent is identified.
 
 This PRD defines **one canonical audit-event contract** every producer
 emits into:
@@ -44,13 +44,13 @@ shared schema. Concretely:
   `timestamp, bottle_slug, component, operator_action, ...`. The egress
   proxy and git-gate log their own ad-hoc lines. There is no common
   `event_id`, `event_type`, or version, so cross-producer correlation
-  ("what did bottle X do between its start and this rejected push?") is
+  ("what did bottled agent X do between its start and this rejected push?") is
   manual and lossy.
 - **No tamper-evidence.** The audit store is a plain SQLite table. Anyone
   who can write the DB can delete or rewrite a row and leave no trace.
   Audit that an attacker (or a buggy agent) can silently rewrite is not
   audit.
-- **Trusted and untrusted data are mixed.** A bottle is attributed by
+- **Trusted and untrusted data are mixed.** A bottled agent is attributed by
   **source IP → slug** at the gateway (host-supplied, trustworthy). An
   agent can also *claim* things about itself in a tool call
   (agent-claimed, adversarial). Today nothing in the record marks which is
@@ -67,7 +67,7 @@ shared schema. Concretely:
 
 - A single `AuditEvent` envelope type, versioned, that every producer
   emits. Fields are split into a **`trusted`** block (host-supplied:
-  bottle slug from source-IP attribution, host wall-clock, producer
+  bottled-agent slug from source-IP attribution, host wall-clock, producer
   identity) and an **`untrusted`** block (anything the agent or a remote
   claimed), and the split is structural, not a convention.
 - **Canonical serialization** (`sort_keys`, `(",", ":")` separators,
@@ -81,8 +81,8 @@ shared schema. Concretely:
   fully rebuildable** from it (`audit rebuild` reconstructs the DB and
   re-verifies the chain).
 - **Local query** works with no paid platform and no egress: filter by
-  bottle, event type, time range, and producer, and follow a bottle's
-  events in order.
+  bottled agent, event type, time range, and producer, and follow a bottled
+  agent's events in order.
 - A **redaction rule** is enforced at the envelope boundary: known
   credential-shaped fields are rejected/redacted before a record is
   written; the writer refuses raw secrets rather than storing them.
@@ -120,35 +120,49 @@ stable top level:
 
 ```
 {
-  "v": 1,                          // schema version (integer, bumped only on breaking change)
+  "v": 1,                          // schema version — bumped only on a breaking change
   "id": "<uuid4>",                 // globally unique event id
   "type": "egress.decision",       // dotted event type from the registry
-  "seq": 1287,                     // per-writer monotonic sequence (gap-detectable)
-  "ts": {
-    "wall": "2026-07-26T18:22:04.113Z",  // host wall-clock, RFC3339 UTC (TRUSTED)
-    "mono": 90142.55                      // host monotonic seconds since writer start (ordering)
+
+  // --- chain / ordering (structural, host-owned) ---
+  "epoch": 7,                      // writer-boot counter, bumped once per host-controller (writer) start
+  "seq": 1287,                     // monotonic sequence within this epoch (gap-detectable)
+  "prev": "<hex>",                 // hash of the previous record in the chain ("" for genesis)
+  "hash": "<hex>",                 // sha256(prev + canonical(this event with hash=""))
+
+  // --- trusted: everything the *host* established; authoritative ---
+  "trusted": {
+    "producer": "host-controller",         // which host component wrote this
+    "host": "mac-studio-1",
+    "bottled_agent": "amber-fox-12",       // slug from source-IP attribution (null for host-level events)
+    "ts_wall": "2026-07-26T18:22:04.113Z", // host wall-clock, RFC3339 UTC
+    "ts_mono": 90142.55                    // host monotonic secs since this epoch's boot (intra-epoch ordering only)
   },
-  "producer": "host-controller",   // TRUSTED: which host component wrote this
-  "trusted": {                     // host-supplied, authoritative
-    "bottle": "amber-fox-12",      // slug from source-IP attribution (may be null for host-level events)
-    "host": "mac-studio-1"
-  },
-  "untrusted": {                   // agent- or remote-claimed; never authoritative
+
+  // --- untrusted: anything the agent or a remote claimed; never authoritative ---
+  "untrusted": {
     "reason": "npm install needs registry.npmjs.org",
     "target": "registry.npmjs.org:443"
-  },
-  "hash": "<hex>",                 // sha256(prev_hash || canonical(this event with hash="" ))
-  "prev": "<hex>"                  // hash of the previous record in this writer's chain ("" for genesis)
+  }
 }
 ```
 
 The **`trusted` / `untrusted` split is the core invariant.** A producer may
-only place a field in `trusted` if the *host* established it (source-IP →
-slug attribution, the host's own clock, the producer's own identity).
-Everything an agent or a remote said goes in `untrusted`. A reader (or a
-future policy engine) can therefore trust `trusted.bottle` for attribution
-and treat `untrusted.*` as adversarial claims — the distinction the current
-stores lack.
+only place a field in `trusted` if the *host* established it: the
+source-IP → `bottled_agent` slug attribution, the host's own clock
+(`ts_wall`/`ts_mono`), and the producer's own identity. **`producer` and
+`ts_*` live inside `trusted` on purpose** — they are host-supplied, so
+grouping them there (rather than as loose top-level fields) keeps the
+"authoritative ⇔ inside `trusted`" rule structural, with nothing
+host-established leaking outside it. Everything an agent or a remote said
+goes in `untrusted`. A reader (or a future policy engine) can therefore
+trust `trusted.bottled_agent` for attribution and treat `untrusted.*` as
+adversarial claims — the distinction the current stores lack.
+
+Only the small structural set — `v`, `id`, `type`, `epoch`, `seq`, `prev`,
+`hash` — sits at the top level; it is host-owned too, but it is chain
+metadata rather than event data, so it stays out of the `trusted` body to
+keep that body purely about *what happened*.
 
 ### Canonical serialization + hash chain
 
@@ -174,20 +188,42 @@ the journal an append-only Merkle-style chain: editing or deleting record
 break is local and points at the tampered record. Verification needs only
 the journal itself (no keys), so it runs offline and in CI.
 
+### Single writer; ordering across restarts
+
+**Decided: one writer per host** (reviewed — the host controller owns it).
+Producers hand events to the host controller, which is the sole appender,
+so the chain has one well-defined total order and one `seq`/`epoch`
+counter. This ties audit availability to the host controller being up,
+which is acceptable because the host controller already gates every
+lifecycle transition; per-producer chains are noted only as a future
+scaling path, not built now.
+
+**Restarts** are handled by the chain, not the clock. `ts_mono` resets to
+~0 on every writer start, so it orders events only *within* one boot. On
+start the writer:
+
+1. reads the last line of the journal, adopts its `hash` as the next
+   record's `prev` (the chain is continuous across the restart), and
+2. bumps `epoch` (persisted alongside the chain head) and resets `seq` to
+   0 for the new boot.
+
+Total order is therefore `(epoch, seq)` — monotonic across restarts by
+construction — with `ts_wall` for human reading and `ts_mono` for
+sub-second ordering inside an epoch. A crash mid-append truncates at most
+the last (partial) line; the verifier flags it and replay resumes from the
+last intact record.
+
 ### Journal (source of truth) + SQLite index (rebuildable)
 
 - **Journal:** one append-only JSONL file per host (path from `paths.py`,
   alongside `host_db_path()`), one canonical event per line, opened
-  `O_APPEND`. This is authoritative. A single writer per host owns appends
-  (producers hand events to it) so the chain has one well-defined order —
-  the host controller is the natural owner since it already gates the
-  lifecycle.
+  `O_APPEND`. This is authoritative.
 - **Index:** a new `audit_events` table via the existing `DbStore` /
   `TableMigrations` machinery, holding the envelope columns plus JSON
-  blobs, indexed on `(bottle, type, wall_ts)`. It is a **derived cache**:
-  `audit rebuild` truncates and replays the journal, re-verifying the
-  chain as it goes. If the DB is deleted or drifts, it is regenerated from
-  the journal with no data loss. (This supersedes the free-standing
+  blobs, indexed on `(bottled_agent, type, ts_wall)`. It is a **derived
+  cache**: `audit rebuild` truncates and replays the journal, re-verifying
+  the chain as it goes. If the DB is deleted or drifts, it is regenerated
+  from the journal with no data loss. (This supersedes the free-standing
   `supervise_audit_entries` table, which becomes a view/producer onto the
   new index.)
 
@@ -196,8 +232,10 @@ the journal itself (no keys), so it runs offline and in CI.
 Dotted `type` names, grouped; the registry is a table mapping type →
 required `untrusted` keys so producers and the verifier agree on shape:
 
-- **lifecycle.*** — `lifecycle.bottle_start`, `lifecycle.bottle_stop`,
-  `lifecycle.bottle_crash` (producer: host-controller, #468).
+- **lifecycle.*** — `lifecycle.bottled_agent_start`,
+  `lifecycle.bottled_agent_stop`, `lifecycle.bottled_agent_crash`
+  (producer: host-controller, #468). Leaf names use `bottled_agent` to match
+  the `trusted.bottled_agent` field — one term for the subject everywhere.
 - **decision.*** — `decision.proposed`, `decision.resolved`
   (producer: supervise; carries operator action + justification, replacing
   PRD 0013's row shape).
@@ -213,13 +251,41 @@ re-typing a field bumps `v`.
 
 ### Redaction rule
 
-The envelope constructor enforces a deny-list at write time: a field whose
-key matches known credential shapes (`token`, `secret`, `password`,
-`authorization`, `*_key`, JWT-shaped values) is rejected — the producer
-must pass a reference (a token *id* or `sha256` fingerprint), never the raw
-value. `auth.token_minted` therefore records the token id and role, not the
-JWT. Redaction is enforced structurally so a producer *cannot* accidentally
-write a secret into the audit log.
+Redaction runs at the envelope boundary, before a record is written, in two
+layers:
+
+1. **Key deny-list (structural).** A field whose *key* matches a known
+   credential shape (`token`, `secret`, `password`, `authorization`,
+   `*_key`) is refused — the producer must pass a reference (a token *id*
+   or `sha256` fingerprint), never the raw value. `auth.token_minted`
+   therefore records the token id and role, not the JWT. This is the
+   primary guard: it is cheap, deterministic, and catches the intended
+   mistake (a producer stuffing a credential into a named field).
+
+2. **Value scan — reuse the egress DLP detectors.** Per review, the value
+   layer reuses the *same* deterministic credential-shape detectors the
+   egress proxy already ships:
+   `bot_bottle/gateway/egress/dlp_detectors.py` —
+   `scan_token_patterns` / `redact_tokens` (and `scan_known_secrets` for
+   host-known secret material). They are pure-Python, mitmproxy-free, and
+   already the project's source of truth for "what a leaked credential
+   looks like," so a single detector set governs both what may leave over
+   the wire and what may land in the journal — they can't drift apart.
+
+   **Scoped deliberately:** only the pattern/known-secret detectors are
+   reused, **not** `scan_entropy`. Entropy scoring is tuned for large
+   streamed request bodies; on the short, high-entropy structured values an
+   audit event legitimately carries (hashes, uuids, base64 ids) it would
+   false-positive and start redacting the very fingerprints the log needs.
+   So the shared layer is the deterministic detectors; entropy stays an
+   egress-only concern. (This is the "evaluate how reasonable that is" from
+   review: reuse the deterministic detectors — yes; share the entropy
+   heuristic — no.)
+
+On a value-layer match the default is **redact** (scrub to a placeholder
+and keep the event) rather than drop, so a producer bug can never make an
+audit event vanish; the key deny-list stays a hard refusal because a
+credential in a named field is always a producer bug worth surfacing.
 
 ## Implementation chunks
 
@@ -227,37 +293,58 @@ write a secret into the audit log.
    right after #468.
 2. **Envelope + canonical + chain core.** `AuditEvent` dataclass,
    `canonical()`, chain hashing, and the single-writer journal appender in
-   `bot_bottle/store/` (reusing `sha256_hex`); unit tests for
-   determinism, chain-break detection, and redaction refusal.
+   `bot_bottle/store/` (reusing `sha256_hex`); redaction wired to the
+   existing `gateway/egress/dlp_detectors` (`scan_token_patterns` /
+   `redact_tokens`); unit tests for determinism, chain-break detection,
+   `epoch`/`seq` continuity across a simulated restart, and redaction of
+   both a deny-listed key and a token-shaped value.
 3. **SQLite index + `audit rebuild` / `audit verify` CLI.** New
    `audit_events` migration; replay-from-journal; offline chain verifier;
-   local query commands (by bottle / type / time / producer).
-4. **Host controller as first producer (#468).** Wire `lifecycle.*`
-   emission into the host controller's start/stop/crash paths.
+   local query commands (by bottled-agent / type / time / producer).
+4. **Host controller as first producer (#468).** Wire
+   `lifecycle.bottled_agent_*` emission into the host controller's
+   start/stop/crash paths; establish the `epoch` bump + chain-head carry on
+   writer restart here (the host controller owns the single writer).
 5. **Migrate existing producers.** Re-emit supervise `decision.*` (retiring
    the standalone `supervise_audit_entries` shape behind the index), egress
    `egress.*`, git-gate `forge.*`, control-plane `auth.*`.
 6. **(follow-up.)** Cross-host merge transport (#324); per-writer signing +
    external anchoring on the chain head; retention/rotation policy.
 
+## Resolved in review (#495)
+
+- **Single writer per host — decided.** The host controller owns the sole
+  appender; per-producer chains are a future scaling path only. (Design →
+  *Single writer; ordering across restarts*.)
+- **Restarts — decided.** An `epoch` counter (bumped per writer boot) plus
+  carrying the last chain head as the next `prev` gives a total order of
+  `(epoch, seq)` that survives restarts; `ts_mono` orders only within an
+  epoch. (Design → *ordering across restarts*.)
+- **`ts_*` and `producer` belong in `trusted`.** They are host-supplied, so
+  they now sit inside the `trusted` block; only chain metadata stays at the
+  top level. (Design → *The envelope*.)
+- **Subject term is `bottled_agent` everywhere** — the `trusted` field and
+  the `lifecycle.bottled_agent_*` leaf names. (Design → *The envelope* /
+  *Event registry*.)
+- **Retention head-carry — yes.** When a journal segment is rotated out,
+  the new segment's genesis `prev` is the rotated-out head, so the verifier
+  still trusts the current head across a rotation. (Folds into the
+  retention follow-up.)
+- **Redaction reuses the egress detectors — yes, scoped.** Reuse the
+  deterministic `dlp_detectors` (`scan_token_patterns` / `redact_tokens` /
+  `scan_known_secrets`); exclude `scan_entropy` as brittle on the short,
+  high-entropy structured values audit records carry. (Design → *Redaction
+  rule*.)
+
 ## Open questions
 
-- **One writer per host vs. per producer chains.** A single appender gives
-  one total order but makes every producer depend on the host controller
-  being up. Alternative: one chain *per producer* (independent `seq`/chain
-  head), merged at query time by `(wall, mono)`. Leaning single-writer for
-  v1 (simpler verification, matches the host controller owning lifecycle),
-  with per-producer chains noted as the scaling path. — feedback wanted.
-- **Monotonic clock across restarts.** `ts.mono` resets when the writer
-  restarts; is `seq` + `wall` enough for ordering across a restart, or does
-  the chain need an explicit `epoch` counter bumped per writer boot?
-  (Leaning: carry the last chain head across restart, so ordering follows
-  the chain, not `mono`.)
-- **Redaction: reject vs. redact.** Should a credential-shaped field hard-
-  fail the write (surfacing the producer bug loudly) or silently redact to
-  `"<redacted>"`? Leaning **reject in tests / redact in prod** behind a
-  flag, so a producer bug can't drop an event entirely in the field.
-- **Retention interaction with the chain.** When an old journal segment is
-  rotated out, the verifier must still trust the current head. Carry the
-  rotated-out head as the new segment's genesis `prev`? (Tracked with the
-  retention follow-up.)
+- **Value-scan cost on the hot path.** The single writer runs the reused
+  detectors on every event's `untrusted` block inline. Is that cheap enough
+  at lifecycle-event volume, or should the value scan move to index-build
+  time (journal stays raw, index stores the redacted view)? Leaning inline
+  so the raw journal never contains a leaked value in the first place.
+- **`epoch` persistence location.** Store the per-writer `epoch` + chain
+  head in the SQLite index (rebuildable, but then the writer needs the DB
+  at boot) or in a tiny sidecar file next to the journal (independent of
+  the index)? Leaning sidecar, so the writer can start and append without
+  the index present.