PRD: canonical tamper-evident audit-event schema (#487) #495
Closed
didericis
wants to merge 5 commits from
didericis/prd-audit-event-schema AGit into main
pull from: didericis/prd-audit-event-schema
merge into: didericis:main
didericis:main
didericis:refactor/orchestrator-fastapi
didericis:chore/misc-quality-cleanup
didericis:prd-gateway-reprovision-on-bringup
didericis:host-control-server-2-secret
didericis:host-control-server-3-list-live
didericis:host-control-server-1-transport
didericis:prd-new-host-control-server
didericis:fix/gitgate-persist-512
didericis:fix/firecracker-gateway-ca-persist-510
didericis:prd-forge-subroles
didericis:didericis/disable-agit-refs
didericis:fix/pr-495-audit-contract
didericis:prd-egress-control-plane
didericis:prd-macos-container-ci-runner
didericis:refactor/control-plane-auth-provisioning-476
didericis:fix/db-off-data-plane-469
didericis:feat/encrypted-egress-secrets
didericis:spike/rootless-docker-macos
didericis:fix/ci-coverage-artifact-paths
didericis:claude-forward-host-credentials-rebased
didericis:fix-gateway-gitleaks-arch
didericis:fix/websocket-response-dlp-multitenant
didericis:orchestrator-agent-compose
didericis:orchestrator-gateway-ca
didericis:orchestrator-consolidated-launch
didericis:orchestrator-gateway-provision
didericis:orchestrator-gateway-network
didericis:orchestrator-client
didericis:orchestrator-gateway-net
didericis:orchestrator-gitgate-provision
didericis:orchestrator-registration
didericis:orchestrator-lifecycle
didericis:orchestrator-supervise-writers
didericis:orchestrator-supervise-multitenant
didericis:orchestrator-gitgate-multitenant
didericis:orchestrator-rename-gateway
didericis:orchestrator-slice8
didericis:orchestrator-slice7
didericis:orchestrator-slice6
didericis:prd-orchestrator
didericis:orchestrator-slice5
didericis:orchestrator-slice4
didericis:orchestrator-slice3
didericis:orchestrator-slice2
didericis:firecracker-backend
didericis:forge-native-integration
didericis:prd-smolmachines-linux
didericis:manifest-break-import-cycle
didericis:dlp-supervise-quality-fixes
didericis:table-drive-dlp-tests
didericis:fix-integration-test-failures
didericis:fix/macos-container-relative-dockerfile
didericis:prd-0054-install-script
didericis:commit-bottle-state
didericis:pr-211
didericis:move-codex-auth-to-contrib
didericis:feat/pipelock-skip-scan-extensions
didericis:prd-0049-named-labelled-agents
didericis:harden-git-gate-shell-rendering
Dismiss Review
Are you sure you want to dismiss this review?
Labels
Clear labels
Compat/Breaking
Kind/Bug
Kind/Documentation
Kind/Enhancement
Kind/Feature
Kind/Security
Kind/Testing
Status/Needs Triage
Breaking change that won't be backward compatible
Something is not working
Documentation changes
Improve existing functionality
New functionality
This is security issue
Issue or pull request related to testing
Priority
Critical
1
The priority is critical
Priority
High
2
The priority is high
Priority
Low
4
The priority is low
Priority
Medium
3
The priority is medium
Status
Abandoned
3
Somebody has started to work on this but abandoned work
Status
Blocked
1
Something is blocking this issue or pull request
Status
Need More Info
2
Feedback is required to reproduce issue or to continue work
Awaiting initial classification
No Label
Milestone
No items
No Milestone
No Assignees
Notifications
Due Date
No due date set.
Dependencies
No dependencies set.
Reference: didericis/bot-bottle#495
Reference in New Issue
Block a user
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
Delete Branch "didericis/prd-audit-event-schema"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
closes #487
Highlights
AuditEventenvelope with a structural trusted-vs-untrusted field split (host-supplied attribution vs agent-claimed data — the distinction today's stores lack).sort_keys, tight separators) + a per-writer hash chain (sha256(prev || canonical(event))) for offline tamper-evidence.lifecycle.*,decision.*,egress.*,auth.*,forge.*.Sequenced per the directive on this issue: lands immediately after the host controller (#468), which becomes the first producer (
lifecycle.*); existing supervise (PRD 0013AuditStore), egress, git-gate, and control-plane producers migrate behind the same envelope. This PR is PRD-only and stays Draft.Consolidates the audit threads in #468 / #324 / #480 onto one contract.
docs(prd): canonical tamper-evident audit-event schema (#487)to PRD: canonical tamper-evident audit-event schema (#487)@@ -0,0 +125,4 @@"type": "egress.decision", // dotted event type from the registry"seq": 1287, // per-writer monotonic sequence (gap-detectable)"ts": {"wall": "2026-07-26T18:22:04.113Z", // host wall-clock, RFC3339 UTC (TRUSTED)why not put this in trusted?
@@ -0,0 +126,4 @@"seq": 1287, // per-writer monotonic sequence (gap-detectable)"ts": {"wall": "2026-07-26T18:22:04.113Z", // host wall-clock, RFC3339 UTC (TRUSTED)"mono": 90142.55 // host monotonic seconds since writer start (ordering)How does this account for host controller restarts?
@@ -0,0 +130,4 @@},"producer": "host-controller", // TRUSTED: which host component wrote this"trusted": { // host-supplied, authoritative"bottle": "amber-fox-12", // slug from source-IP attribution (may be null for host-level events)should be precise/label this a "bottled-agent"
@@ -0,0 +197,4 @@required `untrusted` keys so producers and the verifier agree on shape:- **lifecycle.*** — `lifecycle.bottle_start`, `lifecycle.bottle_stop`,`lifecycle.bottle_crash` (producer: host-controller, #468).annoying, but should be consistent: bottled_agent_start, bottled_agent_stop, bottled_agent_crash
@@ -0,0 +215,4 @@The envelope constructor enforces a deny-list at write time: a field whosekey matches known credential shapes (`token`, `secret`, `password`,`authorization`, `*_key`, JWT-shaped values) is rejected — the producerWould be nice if we use the same credential shape detectors we use on egress, but not sure whether it's worth it or if that would be brittle/concerns would diverge. Evaluate how reasonable that is.
@@ -0,0 +247,4 @@being up. Alternative: one chain *per producer* (independent `seq`/chainhead), merged at query time by `(wall, mono)`. Leaning single-writer forv1 (simpler verification, matches the host controller owning lifecycle),with per-producer chains noted as the scaling path. — feedback wanted.Single writer, definitely
@@ -0,0 +259,4 @@flag, so a producer bug can't drop an event entirely in the field.- **Retention interaction with the chain.** When an old journal segment isrotated out, the verifier must still trust the current head. Carry therotated-out head as the new segment's genesis `prev`? (Tracked with theYes
bc42836327to60039f2eb3@@ -0,0 +197,4 @@which is acceptable because the host controller already gates everylifecycle transition; per-producer chains are noted only as a futurescaling path, not built now.Done:
lifecycle.bottled_agent_start/_stop/_crash, matching thebottled_agentfield name.@@ -0,0 +247,4 @@`forge.pr_opened`.New types are additive; adding one does not bump `v`. Removing orre-typing a field bumps `v`.Locked in as decided — single writer, host controller owns it. Removed the equivocation; per-producer chains are noted only as a future scaling path.
@@ -0,0 +259,4 @@`*_key`) is refused — the producer must pass a reference (a token *id*or `sha256` fingerprint), never the raw value. `auth.token_minted`therefore records the token id and role, not the JWT. This is theprimary guard: it is cheap, deterministic, and catches the intendedAdopted — the rotated-out head becomes the new segment's genesis
prev, so the verifier still trusts the current head across a rotation. Folded into the retention follow-up.@@ -0,0 +215,4 @@### Journal (source of truth) + SQLite index (rebuildable)- **Journal:** one append-only JSONL file per host (path from `paths.py`,Evaluated: reuse the deterministic detectors in
gateway/egress/dlp_detectors.py(scan_token_patterns/redact_tokens, plusscan_known_secrets) — pure-Python and already the project's source of truth for credential shapes, so wire-egress and journal-redaction can't drift. Excludedscan_entropy: it's tuned for large streamed bodies and would false-positive on the short high-entropy values audit records legitimately carry (hashes, uuids, base64 ids), redacting the very fingerprints we want to keep. So: share the deterministic detectors, keep entropy egress-only.@@ -0,0 +126,4 @@// --- chain / ordering (structural, host-owned) ---"epoch": 7, // writer-boot counter, bumped once per host-controller (writer) start"seq": 1287, // monotonic sequence within this epoch (gap-detectable)Handled explicitly now. Ordering is
(epoch, seq), not the clock: on restart the writer reads the last journal line, adopts itshashas the nextprev(chain stays continuous), bumpsepoch, and resetsseq.ts_monoorders only within an epoch. See the new Single writer; ordering across restarts section.@@ -0,0 +130,4 @@"prev": "<hex>", // hash of the previous record in the chain ("" for genesis)"hash": "<hex>", // sha256(prev + canonical(this event with hash=""))// --- trusted: everything the *host* established; authoritative ---Renamed to
bottled_agent— both thetrusted.bottled_agentfield and thelifecycle.bottled_agent_*leaves, so one term for the subject everywhere.@@ -0,0 +125,4 @@"type": "egress.decision", // dotted event type from the registry// --- chain / ordering (structural, host-owned) ---"epoch": 7, // writer-boot counter, bumped once per host-controller (writer) startAgreed — it's host-supplied, so it doesn't belong outside
trusted. Movedts_wall/ts_monoandproducerinto thetrustedblock; only chain metadata (v/id/type/epoch/seq/prev/hash) stays at the top level.Actually let's change this a bit: everything outside of "untrusted" is trusted, (including epoc, seq) so we should just remove the "trusted" subsection.
Thanks — all seven addressed and pushed (head now
60039f2). Summary:(epoch, seq), not the clock.ts_monois intra-epoch only.bottled_agenteverywhere (field + prose).ts_wallnot intrusted?ts_wall/ts_monoandproducerintotrusted; only chain metadata stays top-level.lifecycle.bottled_agent_start/_stop/_crash.prev.dlp_detectorsscan_token_patterns/redact_tokens/scan_known_secrets; excludescan_entropy(brittle on short structured audit values).Resolved points are folded into the design under a new Resolved in review (#495) block; two smaller open questions remain (inline-vs-index-time value scan, and where
epochis persisted). Still PRD-only / Draft.60039f2eb3to5a9428cc86@@ -0,0 +125,4 @@stable top level:```{Done — removed the
trustedsub-block. The envelope is now flat:v/id/type/epoch/seq/prev/hash/producer/host/bottled_agent/ts_wall/ts_monoall sit at the top level and are trusted by construction;untrustedis the one region for agent/remote-claimed data. The rule is now "a field is trusted unless it's deliberately placed underuntrusted," and the construction API takes trusted fields positionally + a singleuntrustedmapping, so a producer can't accidentally emit a top-level field that looks authoritative but isn't. (Head5a9428c.)RE export/interop (CloudEvents & OpenTelemetry Logs) — does the flatten conflict? No, it helps.
Good prompt — #487's Export/interoperability line ("map the envelope to the OTel Logs data model and/or a CloudEvents JSON envelope without losing integrity or attribution semantics") was under-addressed, so I added a dedicated Export / interoperability section. Analysis:
trustedblock would have had to be flattened for CloudEvents anyway. The now-flat top level maps directly:id→id,type→type,producer+host→source,bottled_agent→subject,ts_wall→time; the chain/integrity fields (epoch/seq/prev/hash/v) ride as extension attributes (scalars — legal);untrusted→data. Only mechanical transform: extension names must be lowercase-alnum, sobottled_agent/ts_mono/etc. get abotbottle-prefixed rename at export. Naming rule, not a conflict.ts_wall→Timestamp,type→event.name, flat trusted fields →Attributesunderbotbottle.*,untrusted.*→botbottle.untrusted.*(orBody). OTel attributes are a dotted map that carries the subtree fine.untrustedstays one clearly-named region, so a downstream consumer still sees exactly which fields the agent claimed. Nothing agent-claimed is promoted to a trusted-looking slot.hash/prevare computed over our canonical JSON, not the CloudEvents/OTel form. So the chain fields travel as data for reference, but tamper-evidence is always verified against the native journal — never re-derived from an exported record (whose key order / number formatting the exporter may change). Export is a lossless-for-attribution projection; verification stays on the source of truth.Net: the flatten makes the mapping a field re-map rather than a reformat. Export adapters are broken out as impl chunk 6 (feeding #324's webhook delivery); this PRD just fixes the schema so they stay a re-map. All in
5a9428c.5a9428cc86to88b82a169eFilled the gaps against #487's acceptance checklist (head
88b82a1). The issue is spec-only ("defines the contract; implementation may be split into follow-up PRs"), so this PR is now the complete decision record — no wiring needed here; the wiring is #468/#324/#480.What I added:
bottle/activationids (alongsidebottled_agent),manifest_digest+policy_version, host-observedactor/action/resource/outcome,correlation_id/causation_id, asensitivityclass, typedpayload, andsegmentid. Plus a per-field trust-provenance table.idkey +UPSERT), ordering guarantees, and behavior across rotation/restart/import/truncation (truncated-tailvsgap).audit query/verify/rebuild/importCLI surface.hostctl.*, egressrequest/decision/cutoff/anomaly,commit.signed(#480), auth/authz, andaudit.*self-events (verify/truncation/export failure); each with trusted-vs-claimed fields, default sensitivity, redaction, correlation. Plus schema-evolution + backward-compatible reader rules.(epoch,seq)cursor, dedup onid, backpressure via lagging cursor, retention-must-not-prune-ahead-of-cursor), and the #480 mapping that references (doesn't replace) its byte-to-activation-key signature.auditCLI in chunks 2-3, #324 outbox in chunk 6) — which the issue explicitly permits.I believe this now satisfies the full checklist. Since it's spec-only, my suggestion: merge with
Closes #487when you're happy (it staysStatus: Draft— PRD-only — and the implementation lands as the follow-up chunks). Happy to trim scope to a leaner v1 with named follow-up decision records instead, if you'd prefer.88b82a169etof3664dea9fDropped
policy_version, addedengine(headf3664de).You're right — bot-bottle has no separate policy artifact; the egress routes and other constraints are declared in the manifest (
manifest/egress.py), somanifest_digestalready pins the policy in force. A separatepolicy_versionwas redundant, so it's removed.Given a fixed manifest, the only other axis that can change a decision's outcome is the enforcing code — so I added
engine(bot-bottle version + git SHA) as a trusted field. Now two same-resourcedecisions with differentoutcomeare explained by exactly one of: differentmanifest_digest(rules changed) or differentengine(enforcer changed). Runtime operator overrides (a superviseegress-allow) are themselves audit events, so effective policy at any instant =manifest_digest+ logged approved deltas — reconstructable from the chain, no version stamp needed.Caveat:
engineneeds the build to stamp the git SHA — onlyversion = "0.1.0"exists inpyproject.tomltoday — noted in impl chunk 2.Fixed in #503 (commit
4cb7bbb). This PR was created from a syntheticrefs/pull/495/headwith no writable source branch, so Gitea rejects direct updates to it. #503 preserves this PR verbatim and adds the durable deduplication, imported-chain ordering/cursor, and chain-genesis corrections from review.Replaced by #504, which is backed by the normal branch
didericis/prd-audit-event-schemaand includes the review fixes.Pull request closed