PRD: Egress token-block policy (supervise / redact / block) #262
@@ -76,25 +76,37 @@ fine-grained way to say "this specific value is fine."
|
||||
corrupts legitimate data, so it is opt-in per route; `supervise` (human in
|
||||
the loop) stays the default.
|
||||
|
||||
## Design
|
||||
## Scope
|
||||
|
||||
### On-match policy
|
||||
### In scope
|
||||
|
||||
`dlp.outbound_on_match` is a per-route enum threaded from the bottle manifest
|
||||
(`manifest_egress`) through the resolved route (`egress.EgressRoute`), the
|
||||
rendered `routes.yaml` (`egress_render_routes`), and the addon's `Route`
|
||||
(`egress_addon_core`). Unset renders nothing and resolves to `supervise` at
|
||||
request time. The `list-egress-routes` introspection endpoint round-trips it so
|
||||
the agent's proposals preserve it.
|
||||
The minimum cut that ships, in build order:
|
||||
|
||||
**Provider routes default to `redact`.** Agent-provider routes (the agent
|
||||
talking to its own LLM API — `api.anthropic.com`, the Codex backend, etc.) are
|
||||
the worst source of token-shaped false positives because the whole
|
||||
conversation payload flows through them. `egress_routes_for_bottle` fills
|
||||
`outbound_on_match=redact` on any provider route that doesn't set it
|
||||
explicitly, so a match there is scrubbed and forwarded rather than blocked or
|
||||
queued. A provider that sets the policy keeps its choice; manifest routes are
|
||||
unaffected (they default to `supervise`).
|
||||
1. **Core** — `ScanResult.matched`; thread `safe_tokens` through
|
||||
`scan_outbound` / the token detectors; `build_token_allow_payload`.
|
||||
2. **Supervise + TUI** — `TOOL_EGRESS_TOKEN_ALLOW`; TUI suffix, modify guard,
|
||||
required approval reason.
|
||||
3. **Addon glue** — async `request`, safe-tokens set, proposal write + async
|
||||
poll, allow/block decision; pass `safe_tokens` into the WebSocket path.
|
||||
4. **On-match policy** — `dlp.outbound_on_match` through manifest → render →
|
||||
addon; `redact` surface scrub with fail-closed re-scan; policy dispatch in
|
||||
the addon's outbound handler.
|
||||
5. **Tests + docs** — core/supervise/TUI/manifest/render unit tests; README
|
||||
egress + supervisor notes.
|
||||
|
||||
### Out of scope
|
||||
|
||||
The deferrals enumerated under **Non-goals** — restart persistence, inbound /
|
||||
WebSocket-frame supervision, cross-encoding generalisation, replacing
|
||||
`dlp.outbound_detectors`, and making `redact` the default.
|
||||
|
||||
## Proposed Design
|
||||
|
||||
### New services / components
|
||||
|
||||
A new proposal tool constant `egress-token-allow` (`TOOL_EGRESS_TOKEN_ALLOW`)
|
||||
is added to `supervise.TOOLS`, and the egress addon gains an in-memory
|
||||
safe-tokens set plus the policy-dispatch path that drives it.
|
||||
|
||||
On an outbound block the addon dispatches on the resolved policy:
|
||||
|
||||
@@ -107,30 +119,14 @@ On an outbound block the addon dispatches on the resolved policy:
|
||||
hostname, or a unicode-evasion token redaction can't reach) it fails closed
|
||||
with a `403`.
|
||||
- **`block`** writes the `403` immediately.
|
||||
- **`supervise`** runs the queue-and-wait loop below, falling back to `block`
|
||||
when supervise isn't wired for the bottle.
|
||||
- **`supervise`** runs the queue-and-wait loop, falling back to `block` when
|
||||
supervise isn't wired for the bottle.
|
||||
|
||||
### Detected-value plumbing
|
||||
|
||||
`ScanResult` gains a `matched: str = ""` field carrying the raw substring the
|
||||
detector matched. The token detectors (`scan_token_patterns`,
|
||||
`scan_known_secrets`) populate it; the structural CRLF detector leaves it
|
||||
empty. The value stays inside the egress sidecar process — it is never written
|
||||
to a log line (logs already use the redacted `context`) nor to the proposal
|
||||
file.
|
||||
|
||||
`scan_outbound` (and the token detectors it calls) accept a `safe_tokens`
|
||||
set. A match whose value is in `safe_tokens` is skipped, so an approved token
|
||||
no longer blocks. The scanners keep searching past a safelisted match so a
|
||||
second, un-approved secret in the same request is still caught.
|
||||
|
||||
### Supervisor proposal
|
||||
|
||||
A new proposal tool constant `egress-token-allow` is added to
|
||||
`supervise.TOOLS`. The egress addon writes the proposal directly to
|
||||
For `supervise`, the addon writes the proposal directly to
|
||||
`SUPERVISE_QUEUE_DIR` (the queue is bind-mounted into the sidecar bundle and
|
||||
shared by every daemon, exactly as git-gate's `gitleaks-allow` proposal in PRD
|
||||
0061 does). The proposal's `proposed_file` is a human-readable text payload:
|
||||
0061 does). The proposal's `proposed_file` is a human-readable text payload
|
||||
built by `build_token_allow_payload`:
|
||||
|
||||
```
|
||||
egress blocked an outbound request carrying a detected token
|
||||
@@ -142,46 +138,63 @@ context: ...before ******** after...
|
||||
```
|
||||
|
||||
The justification tells the operator to approve only if the value is a false
|
||||
positive or a credential the request legitimately needs.
|
||||
positive or a credential the request legitimately needs. The addon then polls
|
||||
`<proposal-id>.response.json` for `EGRESS_TOKEN_ALLOW_TIMEOUT_SECONDS` (default
|
||||
300). `approved`/`modified` allow the request and add the value to the
|
||||
safe-tokens set; `rejected`, malformed responses, and timeout fail the request
|
||||
closed. The proposal + response are archived to `processed/` after a decision.
|
||||
Because the wait happens inside mitmproxy's asyncio loop, the addon's `request`
|
||||
hook is async and polls with `asyncio.sleep`, so concurrent flows are
|
||||
unaffected.
|
||||
|
||||
The addon then polls `<proposal-id>.response.json` for
|
||||
`EGRESS_TOKEN_ALLOW_TIMEOUT_SECONDS` (default 300). `approved`/`modified`
|
||||
allow the request and add the value to the safe-tokens set; `rejected`,
|
||||
malformed responses, and timeout fail the request closed. The proposal +
|
||||
response are archived to `processed/` after a decision.
|
||||
### Existing code touched
|
||||
|
||||
Because the wait happens inside mitmproxy's asyncio loop, the addon's
|
||||
`request` hook is async and polls with `asyncio.sleep`, so concurrent flows
|
||||
are unaffected.
|
||||
- **Policy threading.** `dlp.outbound_on_match` is a per-route enum threaded
|
||||
from the bottle manifest (`manifest_egress`) through the resolved route
|
||||
(`egress.EgressRoute`), the rendered `routes.yaml` (`egress_render_routes`),
|
||||
and the addon's `Route` (`egress_addon_core`). Unset renders nothing and
|
||||
resolves to `supervise` at request time. The `list-egress-routes`
|
||||
introspection endpoint round-trips it so the agent's proposals preserve it.
|
||||
- **Provider-route default.** Agent-provider routes (the agent talking to its
|
||||
own LLM API — `api.anthropic.com`, the Codex backend, etc.) are the worst
|
||||
source of token-shaped false positives because the whole conversation payload
|
||||
flows through them. `egress_routes_for_bottle` fills `outbound_on_match=redact`
|
||||
on any provider route that doesn't set it explicitly; a provider that sets the
|
||||
policy keeps its choice, and manifest routes are unaffected (they default to
|
||||
`supervise`).
|
||||
- **Scanners.** `scan_outbound` (and the token detectors `scan_token_patterns`
|
||||
/ `scan_known_secrets` it calls) accept a `safe_tokens` set. A match whose
|
||||
value is in `safe_tokens` is skipped, so an approved token no longer blocks;
|
||||
the scanners keep searching past a safelisted match so a second, un-approved
|
||||
secret in the same request is still caught. The WebSocket path is passed the
|
||||
same `safe_tokens` set.
|
||||
- **Supervisor UI.** `cli/supervise.py` renders `egress-token-allow` like
|
||||
`gitleaks-allow`: the text payload is shown, modify is unavailable (there is
|
||||
no file patch to edit), and approval prompts for a non-empty reason recorded
|
||||
in the response notes. There is no on-disk config diff, so — like
|
||||
`gitleaks-allow` and `capability-block` — it writes no egress audit-log entry.
|
||||
- **Failure handling.** If `SUPERVISE_QUEUE_DIR` / `SUPERVISE_BOTTLE_SLUG` are
|
||||
unset (supervise disabled for the bottle), the addon skips the queue and
|
||||
returns the existing `403`. Any error writing the proposal or reading the
|
||||
response also fails closed.
|
||||
|
||||
### Supervisor UI
|
||||
### Data model changes
|
||||
|
||||
`cli/supervise.py` renders `egress-token-allow` like `gitleaks-allow`: the
|
||||
text payload is shown, modify is unavailable (there is no file patch to edit),
|
||||
and approval prompts for a non-empty reason that is recorded in the response
|
||||
notes. There is no on-disk config diff, so — like `gitleaks-allow` and
|
||||
`capability-block` — it writes no egress audit-log entry.
|
||||
- New per-route manifest field `dlp.outbound_on_match: block | redact |
|
||||
supervise`, rendered into `routes.yaml` (omitted when unset).
|
||||
- `ScanResult` gains a `matched: str = ""` field carrying the raw substring the
|
||||
detector matched. The token detectors populate it; the structural CRLF
|
||||
detector leaves it empty. The value stays inside the egress sidecar process —
|
||||
never written to a log line (logs use the redacted `context`) nor to the
|
||||
proposal file.
|
||||
- Proposal text payload (above) plus `<proposal-id>.response.json` in
|
||||
`SUPERVISE_QUEUE_DIR`, archived to `processed/` after a decision.
|
||||
- New env var `EGRESS_TOKEN_ALLOW_TIMEOUT_SECONDS` (default 300).
|
||||
|
||||
### Failure handling
|
||||
### External dependencies
|
||||
|
||||
If `SUPERVISE_QUEUE_DIR` / `SUPERVISE_BOTTLE_SLUG` are unset (supervise
|
||||
disabled for the bottle), the addon skips the queue and returns the existing
|
||||
`403`. Any error writing the proposal or reading the response also fails
|
||||
closed.
|
||||
|
||||
## Implementation chunks
|
||||
|
||||
1. **Core** — `ScanResult.matched`; thread `safe_tokens` through
|
||||
`scan_outbound` / token detectors; `build_token_allow_payload`.
|
||||
2. **Supervise + TUI** — `TOOL_EGRESS_TOKEN_ALLOW`; TUI suffix, modify guard,
|
||||
required approval reason.
|
||||
3. **Addon glue** — async `request`, safe-tokens set, proposal write + async
|
||||
poll, allow/block decision; pass `safe_tokens` into the WebSocket path.
|
||||
4. **On-match policy** — `dlp.outbound_on_match` through manifest → render →
|
||||
addon; `redact` surface scrub with fail-closed re-scan; policy dispatch in
|
||||
the addon's outbound handler.
|
||||
5. **Tests + docs** — core/supervise/TUI/manifest/render unit tests; README
|
||||
egress + supervisor notes.
|
||||
None. Reuses the existing supervisor queue (`SUPERVISE_QUEUE_DIR`) and the
|
||||
mitmproxy addon framework already in the egress sidecar.
|
||||
|
||||
## Open questions
|
||||
|
||||
@@ -190,3 +203,8 @@ closed.
|
||||
approval is an explicit operator decision and the safe-tokens set matches the
|
||||
exact found value — but a future revision could restrict `known_secrets` to
|
||||
reject-only.
|
||||
|
||||
## References
|
||||
|
||||
- Issue #261
|
||||
- PRD 0061 — `gitleaks-allow` supervisor proposal pattern this reuses.
|
||||
|
||||
Reference in New Issue
Block a user