Restructure PRD 0062 to the init-prd template
test / unit (pull_request) Successful in 39s
test / integration (pull_request) Successful in 25s
lint / lint (push) Successful in 2m8s
test / unit (push) Successful in 41s
test / integration (push) Successful in 25s
Update Quality Badges / update-badges (push) Successful in 1m32s
test / unit (pull_request) Successful in 39s
test / integration (pull_request) Successful in 25s
lint / lint (push) Successful in 2m8s
test / unit (push) Successful in 41s
test / integration (push) Successful in 25s
Update Quality Badges / update-badges (push) Successful in 1m32s
Conform the PRD to the standard PRD-new skeleton: add a Scope section (In scope / Out of scope), rename Design -> Proposed Design and split its prose into New services / Existing code touched / Data model changes / External dependencies, fold the old Implementation chunks into In scope, and add a References section. No change in substance. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit was merged in pull request #262.
This commit is contained in:
@@ -76,25 +76,37 @@ fine-grained way to say "this specific value is fine."
|
|||||||
corrupts legitimate data, so it is opt-in per route; `supervise` (human in
|
corrupts legitimate data, so it is opt-in per route; `supervise` (human in
|
||||||
the loop) stays the default.
|
the loop) stays the default.
|
||||||
|
|
||||||
## Design
|
## Scope
|
||||||
|
|
||||||
### On-match policy
|
### In scope
|
||||||
|
|
||||||
`dlp.outbound_on_match` is a per-route enum threaded from the bottle manifest
|
The minimum cut that ships, in build order:
|
||||||
(`manifest_egress`) through the resolved route (`egress.EgressRoute`), the
|
|
||||||
rendered `routes.yaml` (`egress_render_routes`), and the addon's `Route`
|
|
||||||
(`egress_addon_core`). Unset renders nothing and resolves to `supervise` at
|
|
||||||
request time. The `list-egress-routes` introspection endpoint round-trips it so
|
|
||||||
the agent's proposals preserve it.
|
|
||||||
|
|
||||||
**Provider routes default to `redact`.** Agent-provider routes (the agent
|
1. **Core** — `ScanResult.matched`; thread `safe_tokens` through
|
||||||
talking to its own LLM API — `api.anthropic.com`, the Codex backend, etc.) are
|
`scan_outbound` / the token detectors; `build_token_allow_payload`.
|
||||||
the worst source of token-shaped false positives because the whole
|
2. **Supervise + TUI** — `TOOL_EGRESS_TOKEN_ALLOW`; TUI suffix, modify guard,
|
||||||
conversation payload flows through them. `egress_routes_for_bottle` fills
|
required approval reason.
|
||||||
`outbound_on_match=redact` on any provider route that doesn't set it
|
3. **Addon glue** — async `request`, safe-tokens set, proposal write + async
|
||||||
explicitly, so a match there is scrubbed and forwarded rather than blocked or
|
poll, allow/block decision; pass `safe_tokens` into the WebSocket path.
|
||||||
queued. A provider that sets the policy keeps its choice; manifest routes are
|
4. **On-match policy** — `dlp.outbound_on_match` through manifest → render →
|
||||||
unaffected (they default to `supervise`).
|
addon; `redact` surface scrub with fail-closed re-scan; policy dispatch in
|
||||||
|
the addon's outbound handler.
|
||||||
|
5. **Tests + docs** — core/supervise/TUI/manifest/render unit tests; README
|
||||||
|
egress + supervisor notes.
|
||||||
|
|
||||||
|
### Out of scope
|
||||||
|
|
||||||
|
The deferrals enumerated under **Non-goals** — restart persistence, inbound /
|
||||||
|
WebSocket-frame supervision, cross-encoding generalisation, replacing
|
||||||
|
`dlp.outbound_detectors`, and making `redact` the default.
|
||||||
|
|
||||||
|
## Proposed Design
|
||||||
|
|
||||||
|
### New services / components
|
||||||
|
|
||||||
|
A new proposal tool constant `egress-token-allow` (`TOOL_EGRESS_TOKEN_ALLOW`)
|
||||||
|
is added to `supervise.TOOLS`, and the egress addon gains an in-memory
|
||||||
|
safe-tokens set plus the policy-dispatch path that drives it.
|
||||||
|
|
||||||
On an outbound block the addon dispatches on the resolved policy:
|
On an outbound block the addon dispatches on the resolved policy:
|
||||||
|
|
||||||
@@ -107,30 +119,14 @@ On an outbound block the addon dispatches on the resolved policy:
|
|||||||
hostname, or a unicode-evasion token redaction can't reach) it fails closed
|
hostname, or a unicode-evasion token redaction can't reach) it fails closed
|
||||||
with a `403`.
|
with a `403`.
|
||||||
- **`block`** writes the `403` immediately.
|
- **`block`** writes the `403` immediately.
|
||||||
- **`supervise`** runs the queue-and-wait loop below, falling back to `block`
|
- **`supervise`** runs the queue-and-wait loop, falling back to `block` when
|
||||||
when supervise isn't wired for the bottle.
|
supervise isn't wired for the bottle.
|
||||||
|
|
||||||
### Detected-value plumbing
|
For `supervise`, the addon writes the proposal directly to
|
||||||
|
|
||||||
`ScanResult` gains a `matched: str = ""` field carrying the raw substring the
|
|
||||||
detector matched. The token detectors (`scan_token_patterns`,
|
|
||||||
`scan_known_secrets`) populate it; the structural CRLF detector leaves it
|
|
||||||
empty. The value stays inside the egress sidecar process — it is never written
|
|
||||||
to a log line (logs already use the redacted `context`) nor to the proposal
|
|
||||||
file.
|
|
||||||
|
|
||||||
`scan_outbound` (and the token detectors it calls) accept a `safe_tokens`
|
|
||||||
set. A match whose value is in `safe_tokens` is skipped, so an approved token
|
|
||||||
no longer blocks. The scanners keep searching past a safelisted match so a
|
|
||||||
second, un-approved secret in the same request is still caught.
|
|
||||||
|
|
||||||
### Supervisor proposal
|
|
||||||
|
|
||||||
A new proposal tool constant `egress-token-allow` is added to
|
|
||||||
`supervise.TOOLS`. The egress addon writes the proposal directly to
|
|
||||||
`SUPERVISE_QUEUE_DIR` (the queue is bind-mounted into the sidecar bundle and
|
`SUPERVISE_QUEUE_DIR` (the queue is bind-mounted into the sidecar bundle and
|
||||||
shared by every daemon, exactly as git-gate's `gitleaks-allow` proposal in PRD
|
shared by every daemon, exactly as git-gate's `gitleaks-allow` proposal in PRD
|
||||||
0061 does). The proposal's `proposed_file` is a human-readable text payload:
|
0061 does). The proposal's `proposed_file` is a human-readable text payload
|
||||||
|
built by `build_token_allow_payload`:
|
||||||
|
|
||||||
```
|
```
|
||||||
egress blocked an outbound request carrying a detected token
|
egress blocked an outbound request carrying a detected token
|
||||||
@@ -142,46 +138,63 @@ context: ...before ******** after...
|
|||||||
```
|
```
|
||||||
|
|
||||||
The justification tells the operator to approve only if the value is a false
|
The justification tells the operator to approve only if the value is a false
|
||||||
positive or a credential the request legitimately needs.
|
positive or a credential the request legitimately needs. The addon then polls
|
||||||
|
`<proposal-id>.response.json` for `EGRESS_TOKEN_ALLOW_TIMEOUT_SECONDS` (default
|
||||||
|
300). `approved`/`modified` allow the request and add the value to the
|
||||||
|
safe-tokens set; `rejected`, malformed responses, and timeout fail the request
|
||||||
|
closed. The proposal + response are archived to `processed/` after a decision.
|
||||||
|
Because the wait happens inside mitmproxy's asyncio loop, the addon's `request`
|
||||||
|
hook is async and polls with `asyncio.sleep`, so concurrent flows are
|
||||||
|
unaffected.
|
||||||
|
|
||||||
The addon then polls `<proposal-id>.response.json` for
|
### Existing code touched
|
||||||
`EGRESS_TOKEN_ALLOW_TIMEOUT_SECONDS` (default 300). `approved`/`modified`
|
|
||||||
allow the request and add the value to the safe-tokens set; `rejected`,
|
|
||||||
malformed responses, and timeout fail the request closed. The proposal +
|
|
||||||
response are archived to `processed/` after a decision.
|
|
||||||
|
|
||||||
Because the wait happens inside mitmproxy's asyncio loop, the addon's
|
- **Policy threading.** `dlp.outbound_on_match` is a per-route enum threaded
|
||||||
`request` hook is async and polls with `asyncio.sleep`, so concurrent flows
|
from the bottle manifest (`manifest_egress`) through the resolved route
|
||||||
are unaffected.
|
(`egress.EgressRoute`), the rendered `routes.yaml` (`egress_render_routes`),
|
||||||
|
and the addon's `Route` (`egress_addon_core`). Unset renders nothing and
|
||||||
|
resolves to `supervise` at request time. The `list-egress-routes`
|
||||||
|
introspection endpoint round-trips it so the agent's proposals preserve it.
|
||||||
|
- **Provider-route default.** Agent-provider routes (the agent talking to its
|
||||||
|
own LLM API — `api.anthropic.com`, the Codex backend, etc.) are the worst
|
||||||
|
source of token-shaped false positives because the whole conversation payload
|
||||||
|
flows through them. `egress_routes_for_bottle` fills `outbound_on_match=redact`
|
||||||
|
on any provider route that doesn't set it explicitly; a provider that sets the
|
||||||
|
policy keeps its choice, and manifest routes are unaffected (they default to
|
||||||
|
`supervise`).
|
||||||
|
- **Scanners.** `scan_outbound` (and the token detectors `scan_token_patterns`
|
||||||
|
/ `scan_known_secrets` it calls) accept a `safe_tokens` set. A match whose
|
||||||
|
value is in `safe_tokens` is skipped, so an approved token no longer blocks;
|
||||||
|
the scanners keep searching past a safelisted match so a second, un-approved
|
||||||
|
secret in the same request is still caught. The WebSocket path is passed the
|
||||||
|
same `safe_tokens` set.
|
||||||
|
- **Supervisor UI.** `cli/supervise.py` renders `egress-token-allow` like
|
||||||
|
`gitleaks-allow`: the text payload is shown, modify is unavailable (there is
|
||||||
|
no file patch to edit), and approval prompts for a non-empty reason recorded
|
||||||
|
in the response notes. There is no on-disk config diff, so — like
|
||||||
|
`gitleaks-allow` and `capability-block` — it writes no egress audit-log entry.
|
||||||
|
- **Failure handling.** If `SUPERVISE_QUEUE_DIR` / `SUPERVISE_BOTTLE_SLUG` are
|
||||||
|
unset (supervise disabled for the bottle), the addon skips the queue and
|
||||||
|
returns the existing `403`. Any error writing the proposal or reading the
|
||||||
|
response also fails closed.
|
||||||
|
|
||||||
### Supervisor UI
|
### Data model changes
|
||||||
|
|
||||||
`cli/supervise.py` renders `egress-token-allow` like `gitleaks-allow`: the
|
- New per-route manifest field `dlp.outbound_on_match: block | redact |
|
||||||
text payload is shown, modify is unavailable (there is no file patch to edit),
|
supervise`, rendered into `routes.yaml` (omitted when unset).
|
||||||
and approval prompts for a non-empty reason that is recorded in the response
|
- `ScanResult` gains a `matched: str = ""` field carrying the raw substring the
|
||||||
notes. There is no on-disk config diff, so — like `gitleaks-allow` and
|
detector matched. The token detectors populate it; the structural CRLF
|
||||||
`capability-block` — it writes no egress audit-log entry.
|
detector leaves it empty. The value stays inside the egress sidecar process —
|
||||||
|
never written to a log line (logs use the redacted `context`) nor to the
|
||||||
|
proposal file.
|
||||||
|
- Proposal text payload (above) plus `<proposal-id>.response.json` in
|
||||||
|
`SUPERVISE_QUEUE_DIR`, archived to `processed/` after a decision.
|
||||||
|
- New env var `EGRESS_TOKEN_ALLOW_TIMEOUT_SECONDS` (default 300).
|
||||||
|
|
||||||
### Failure handling
|
### External dependencies
|
||||||
|
|
||||||
If `SUPERVISE_QUEUE_DIR` / `SUPERVISE_BOTTLE_SLUG` are unset (supervise
|
None. Reuses the existing supervisor queue (`SUPERVISE_QUEUE_DIR`) and the
|
||||||
disabled for the bottle), the addon skips the queue and returns the existing
|
mitmproxy addon framework already in the egress sidecar.
|
||||||
`403`. Any error writing the proposal or reading the response also fails
|
|
||||||
closed.
|
|
||||||
|
|
||||||
## Implementation chunks
|
|
||||||
|
|
||||||
1. **Core** — `ScanResult.matched`; thread `safe_tokens` through
|
|
||||||
`scan_outbound` / token detectors; `build_token_allow_payload`.
|
|
||||||
2. **Supervise + TUI** — `TOOL_EGRESS_TOKEN_ALLOW`; TUI suffix, modify guard,
|
|
||||||
required approval reason.
|
|
||||||
3. **Addon glue** — async `request`, safe-tokens set, proposal write + async
|
|
||||||
poll, allow/block decision; pass `safe_tokens` into the WebSocket path.
|
|
||||||
4. **On-match policy** — `dlp.outbound_on_match` through manifest → render →
|
|
||||||
addon; `redact` surface scrub with fail-closed re-scan; policy dispatch in
|
|
||||||
the addon's outbound handler.
|
|
||||||
5. **Tests + docs** — core/supervise/TUI/manifest/render unit tests; README
|
|
||||||
egress + supervisor notes.
|
|
||||||
|
|
||||||
## Open questions
|
## Open questions
|
||||||
|
|
||||||
@@ -190,3 +203,8 @@ closed.
|
|||||||
approval is an explicit operator decision and the safe-tokens set matches the
|
approval is an explicit operator decision and the safe-tokens set matches the
|
||||||
exact found value — but a future revision could restrict `known_secrets` to
|
exact found value — but a future revision could restrict `known_secrets` to
|
||||||
reject-only.
|
reject-only.
|
||||||
|
|
||||||
|
## References
|
||||||
|
|
||||||
|
- Issue #261
|
||||||
|
- PRD 0061 — `gitleaks-allow` supervisor proposal pattern this reuses.
|
||||||
|
|||||||
Reference in New Issue
Block a user