Add dlp.outbound_on_match policy (block | redact | supervise)
Give each egress route a policy for what the proxy does when an outbound DLP detector matches a token, defaulting to the supervise flow added in the previous commit. The goal is cutting false-positive friction without weakening default-deny. - redact: scrub the matched value(s) from the body, non-host headers, and path/query via redact_tokens, then re-scan. Forward if clean; fail closed with a 403 if a match remains on a surface redaction can't rewrite (the hostname, or a unicode-evasion token). For routes where a token-shaped value is noise the upstream doesn't need. - block: the original hard 403, never overridable. - supervise (default, unset): hold the request for operator approval. Structural blocks (CRLF, no safelist-able value) stay hard 403s under every policy. Threads outbound_on_match from the bottle manifest (manifest_egress) through the resolved EgressRoute and rendered routes.yaml (egress.py) to the addon's Route (egress_addon_core), and round-trips it via the list-egress-routes introspection endpoint. The allow/egress-block tool descriptions document the new key. Tests: manifest parse/validation, core parse/validation, full manifest->render->addon round-trip for redact. README + PRD 0062 updated. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HnvBjPZC5V7qeQpFbQdDmS
This commit is contained in:
@@ -15,7 +15,7 @@
|
||||
## Features
|
||||
|
||||
- **Per-bottle egress allowlist** — TLS-bumped HTTP/HTTPS chokepoint with a per-manifest host allowlist; per-route path/method/header `matches` filtering; outbound DLP scanning for known tokens and secrets, inbound DLP scanning for prompt-injection attempts; DoH and arbitrary hosts blocked by default.
|
||||
- **Supervisor override for token blocks** — when the outbound DLP catches a token, the request is held and surfaced in `./cli.py supervise` instead of failing outright. The operator approves or rejects; an approved value is remembered for the life of the egress proxy so the request — and later ones carrying it — flow through. Fails closed on rejection or timeout.
|
||||
- **Per-route token-match policy** — each egress route picks what happens when the outbound DLP catches a token via `dlp.outbound_on_match`: `supervise` (default) holds the request and surfaces it in `./cli.py supervise` for approval (an approved value is remembered for the life of the proxy); `redact` scrubs the value and forwards; `block` is a hard `403`. Cuts false-positive friction without weakening default-deny.
|
||||
- **Tokens the agent never sees** — host secrets live in a sidecar; the agent dials `http://sidecar:9099/<path>` and the proxy strips inbound `Authorization` and injects the real token before forwarding. `printenv` in the agent shows proxy URLs only.
|
||||
- **Gitleaks-scanned push (git-gate)** — `bottle.git` remotes route through a per-bottle `git daemon` that gitleaks-scans incoming refs pre-receive and forwards clean refs upstream over SSH. The agent never holds the upstream credential.
|
||||
- **Manifest-scoped skills + secrets** — each bottle declares its skills, env, git identity, remotes, and egress routes; unknown keys die at load.
|
||||
@@ -149,9 +149,10 @@ You help maintain Gitea-hosted projects.
|
||||
| `dlp` | no | Per-route DLP overrides. Omit to use defaults (all detectors on). |
|
||||
| `dlp.outbound_detectors` | no | `false` disables outbound scanning; list restricts to named detectors (`token_patterns`, `known_secrets`). |
|
||||
| `dlp.inbound_detectors` | no | `false` disables inbound scanning; list restricts to named detectors (`naive_injection_detection`). |
|
||||
| `dlp.outbound_on_match` | no | What to do when an outbound token is detected: `supervise` (default — hold for operator approval), `redact` (scrub the value and forward), or `block` (hard 403). |
|
||||
| `git.fetch` | no | `true` permits smart HTTP clone/fetch (`git-upload-pack`) for this host. Push (`git-receive-pack`) remains blocked. |
|
||||
|
||||
When an outbound request is blocked because a DLP detector matched a token, the proxy queues an `egress-token-allow` proposal for the operator's `./cli.py supervise` TUI and holds the request open until it is answered (or `EGRESS_TOKEN_ALLOW_TIMEOUT_SECONDS`, default 300s, elapses — after which it fails closed). The operator never sees the raw token, only the host, method, path, and a redacted snippet. Approving adds the value to an in-memory safelist for the life of the egress proxy. Structural blocks (CRLF injection) and not-in-allowlist host blocks stay hard `403`s.
|
||||
When an outbound DLP detector matches a token, the route's `dlp.outbound_on_match` policy decides what happens. Under the default `supervise`, the proxy queues an `egress-token-allow` proposal for the operator's `./cli.py supervise` TUI and holds the request open until it is answered (or `EGRESS_TOKEN_ALLOW_TIMEOUT_SECONDS`, default 300s, elapses — after which it fails closed). The operator never sees the raw token, only the host, method, path, and a redacted snippet; approving adds the value to an in-memory safelist for the life of the egress proxy. Under `redact`, the matched value is scrubbed from the body, headers, and path and the request is forwarded (failing closed if a match lands somewhere unredactable, like the hostname). Under `block` it stays a hard `403`. Structural blocks (CRLF injection) and not-in-allowlist host blocks are always hard `403`s regardless of policy.
|
||||
|
||||
More examples in `examples/`. Full design lives under `docs/prds/`; the trust-boundary rationale is in `docs/prds/0011-per-file-md-manifest.md`.
|
||||
|
||||
|
||||
+10
-1
@@ -95,6 +95,7 @@ def egress_manifest_routes(
|
||||
git_fetch=r.GitFetch,
|
||||
outbound_detectors=r.OutboundDetectors,
|
||||
inbound_detectors=r.InboundDetectors,
|
||||
outbound_on_match=r.OutboundOnMatch,
|
||||
))
|
||||
return tuple(out)
|
||||
|
||||
@@ -177,7 +178,11 @@ def _route_to_yaml_fields(r: Route) -> dict[str, object]:
|
||||
fields["matches"] = matches_data
|
||||
if r.git_fetch:
|
||||
fields["git"] = {"fetch": True}
|
||||
if r.outbound_detectors is not None or r.inbound_detectors is not None:
|
||||
if (
|
||||
r.outbound_detectors is not None
|
||||
or r.inbound_detectors is not None
|
||||
or r.outbound_on_match
|
||||
):
|
||||
dlp: dict[str, object] = {}
|
||||
if r.outbound_detectors is not None:
|
||||
dlp["outbound_detectors"] = (
|
||||
@@ -189,6 +194,8 @@ def _route_to_yaml_fields(r: Route) -> dict[str, object]:
|
||||
False if not r.inbound_detectors
|
||||
else list(r.inbound_detectors)
|
||||
)
|
||||
if r.outbound_on_match:
|
||||
dlp["outbound_on_match"] = r.outbound_on_match
|
||||
fields["dlp"] = dlp
|
||||
return fields
|
||||
|
||||
@@ -260,6 +267,8 @@ def egress_render_routes(
|
||||
elif isinstance(dv, list):
|
||||
items_str = ", ".join(f'"{x}"' for x in dv)
|
||||
lines.append(f" {dk}: [{items_str}]")
|
||||
elif isinstance(dv, str):
|
||||
lines.append(f' {dk}: "{dv}"')
|
||||
return "\n".join(lines) + "\n"
|
||||
|
||||
|
||||
|
||||
+112
-30
@@ -17,7 +17,11 @@ from mitmproxy import http # type: ignore[import-not-found] # pylint: disable=
|
||||
from egress_addon_core import ( # type: ignore[import-not-found] # pylint: disable=import-error
|
||||
LOG_BLOCKS,
|
||||
LOG_FULL,
|
||||
DEFAULT_OUTBOUND_ON_MATCH,
|
||||
ON_MATCH_BLOCK,
|
||||
ON_MATCH_REDACT,
|
||||
Config,
|
||||
Route,
|
||||
ScanResult,
|
||||
build_inbound_scan_text,
|
||||
build_outbound_scan_text,
|
||||
@@ -189,37 +193,11 @@ class EgressAddon:
|
||||
# Hostname is included to catch DNS-tunnelling exfiltration attempts.
|
||||
route = match_route(self.config.routes, flow.request.pretty_host)
|
||||
if route is not None:
|
||||
body = flow.request.get_text(strict=False) or ""
|
||||
# Re-scan after each operator approval so a second, un-approved
|
||||
# token in the same request is still caught (PRD 0062).
|
||||
while True:
|
||||
scan_text = build_outbound_scan_text(
|
||||
flow.request.pretty_host,
|
||||
request_path,
|
||||
query,
|
||||
outbound_scan_headers(route, dict(flow.request.headers)),
|
||||
body,
|
||||
)
|
||||
dlp_result = scan_outbound(
|
||||
route, scan_text, os.environ, safe_tokens=self.safe_tokens,
|
||||
)
|
||||
if dlp_result is None or dlp_result.severity != "block":
|
||||
break
|
||||
# Token blocks (a match with a safelist-able value) can be
|
||||
# routed to the operator; structural blocks (CRLF, matched="")
|
||||
# and any block when supervise is disabled stay hard 403s.
|
||||
if dlp_result.matched and self._supervise_available():
|
||||
approved = await self._supervise_token_block(
|
||||
flow, request_path, dlp_result,
|
||||
)
|
||||
if approved:
|
||||
continue # re-scan; matched value now in safe_tokens
|
||||
return # _supervise_token_block wrote the 403 response
|
||||
ctx = self._req_ctx(flow)
|
||||
if dlp_result.context:
|
||||
ctx = {**ctx, "context": dlp_result.context}
|
||||
self._block(flow, f"egress DLP: {dlp_result.reason}", ctx=ctx)
|
||||
if not await self._handle_outbound_dlp(flow, route):
|
||||
return
|
||||
# The redact policy may have rewritten the request line; recompute
|
||||
# the path/query the git checks below rely on.
|
||||
request_path, _, query = flow.request.path.partition("?")
|
||||
|
||||
if is_git_push_request(request_path, query):
|
||||
self._block(
|
||||
@@ -269,6 +247,110 @@ class EgressAddon:
|
||||
if self.config.log >= LOG_FULL:
|
||||
self._log_request(flow)
|
||||
|
||||
def _block_dlp(self, flow: http.HTTPFlow, result: ScanResult) -> None:
|
||||
ctx = self._req_ctx(flow)
|
||||
if result.context:
|
||||
ctx = {**ctx, "context": result.context}
|
||||
self._block(flow, f"egress DLP: {result.reason}", ctx=ctx)
|
||||
|
||||
async def _handle_outbound_dlp(
|
||||
self,
|
||||
flow: http.HTTPFlow,
|
||||
route: Route,
|
||||
) -> bool:
|
||||
"""Scan the outbound request and apply the route's on-match policy
|
||||
(PRD 0062). Returns True if the request may be forwarded, False if a
|
||||
403 response has been written to `flow`.
|
||||
|
||||
Loops so the supervise policy can re-scan after each approval — a
|
||||
second, un-approved token in the same request is still caught."""
|
||||
while True:
|
||||
request_path, _, query = flow.request.path.partition("?")
|
||||
body = flow.request.get_text(strict=False) or ""
|
||||
scan_text = build_outbound_scan_text(
|
||||
flow.request.pretty_host,
|
||||
request_path,
|
||||
query,
|
||||
outbound_scan_headers(route, dict(flow.request.headers)),
|
||||
body,
|
||||
)
|
||||
result = scan_outbound(
|
||||
route, scan_text, os.environ, safe_tokens=self.safe_tokens,
|
||||
)
|
||||
if result is None or result.severity != "block":
|
||||
return True
|
||||
|
||||
# Structural blocks (CRLF, no safelist-able value) are always a
|
||||
# hard 403, regardless of the route's on-match policy.
|
||||
if not result.matched:
|
||||
self._block_dlp(flow, result)
|
||||
return False
|
||||
|
||||
policy = route.outbound_on_match or DEFAULT_OUTBOUND_ON_MATCH
|
||||
|
||||
if policy == ON_MATCH_REDACT:
|
||||
if self._redact_outbound(flow, route):
|
||||
if self.config.log >= LOG_BLOCKS:
|
||||
sys.stderr.write(json.dumps({
|
||||
"event": "egress_redacted",
|
||||
"reason": f"egress DLP: {result.reason}",
|
||||
**self._req_ctx(flow),
|
||||
}) + "\n")
|
||||
return True
|
||||
self._block(
|
||||
flow,
|
||||
f"egress DLP: {result.reason}; redaction could not remove "
|
||||
"all matches (e.g. a match in the hostname)",
|
||||
ctx=self._req_ctx(flow),
|
||||
)
|
||||
return False
|
||||
|
||||
if policy == ON_MATCH_BLOCK:
|
||||
self._block_dlp(flow, result)
|
||||
return False
|
||||
|
||||
# supervise (default): hold the request for operator approval.
|
||||
# Fall back to a hard 403 when supervise isn't wired for the bottle.
|
||||
if not self._supervise_available():
|
||||
self._block_dlp(flow, result)
|
||||
return False
|
||||
approved = await self._supervise_token_block(flow, request_path, result)
|
||||
if not approved:
|
||||
return False # _supervise_token_block wrote the 403 response
|
||||
# loop: the approved value is now in safe_tokens; re-scan.
|
||||
|
||||
def _redact_outbound(self, flow: http.HTTPFlow, route: Route) -> bool:
|
||||
"""Scrub detected tokens from the mutable request surfaces (body,
|
||||
headers, path/query) and re-scan. Returns True if the request is now
|
||||
clean; False if a block-severity match remains on a surface redaction
|
||||
cannot rewrite (the hostname) so the caller fails closed."""
|
||||
body = flow.request.get_text(strict=False)
|
||||
if body:
|
||||
redacted_body = redact_tokens(body, env=os.environ)
|
||||
if redacted_body != body:
|
||||
flow.request.text = redacted_body
|
||||
for name, value in list(flow.request.headers.items()):
|
||||
if name.lower() == "host":
|
||||
continue # routing-critical; never a legitimate token
|
||||
redacted = redact_tokens(value, env=os.environ)
|
||||
if redacted != value:
|
||||
flow.request.headers[name] = redacted
|
||||
redacted_path = redact_tokens(flow.request.path, env=os.environ)
|
||||
if redacted_path != flow.request.path:
|
||||
flow.request.path = redacted_path
|
||||
|
||||
request_path, _, query = flow.request.path.partition("?")
|
||||
new_body = flow.request.get_text(strict=False) or ""
|
||||
scan_text = build_outbound_scan_text(
|
||||
flow.request.pretty_host,
|
||||
request_path,
|
||||
query,
|
||||
outbound_scan_headers(route, dict(flow.request.headers)),
|
||||
new_body,
|
||||
)
|
||||
result = scan_outbound(route, scan_text, os.environ)
|
||||
return result is None or result.severity != "block"
|
||||
|
||||
async def _supervise_token_block(
|
||||
self,
|
||||
flow: http.HTTPFlow,
|
||||
|
||||
@@ -37,6 +37,15 @@ VALID_METHODS = frozenset({
|
||||
OUTBOUND_DETECTOR_NAMES = frozenset({"token_patterns", "known_secrets"})
|
||||
INBOUND_DETECTOR_NAMES = frozenset({"naive_injection_detection"})
|
||||
|
||||
# Per-route policy for what the proxy does when an outbound DLP detector
|
||||
# matches a token (PRD 0062).
|
||||
ON_MATCH_BLOCK = "block" # hard 403, never overridable
|
||||
ON_MATCH_REDACT = "redact" # scrub the matched value, forward the request
|
||||
ON_MATCH_SUPERVISE = "supervise" # queue for operator approval, hold the request
|
||||
OUTBOUND_ON_MATCH_VALUES = (ON_MATCH_BLOCK, ON_MATCH_REDACT, ON_MATCH_SUPERVISE)
|
||||
# Unset resolves to supervise (fall back to block when supervise is not wired).
|
||||
DEFAULT_OUTBOUND_ON_MATCH = ON_MATCH_SUPERVISE
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class PathMatch:
|
||||
@@ -69,6 +78,8 @@ class Route:
|
||||
git_fetch: bool = False
|
||||
outbound_detectors: tuple[str, ...] | None = None
|
||||
inbound_detectors: tuple[str, ...] | None = None
|
||||
# "" means unset → DEFAULT_OUTBOUND_ON_MATCH. See OUTBOUND_ON_MATCH_VALUES.
|
||||
outbound_on_match: str = ""
|
||||
|
||||
|
||||
LOG_OFF = 0 # no logging
|
||||
@@ -223,12 +234,12 @@ def _parse_detectors(
|
||||
idx: int,
|
||||
host: str,
|
||||
raw_dict: dict[str, object],
|
||||
) -> tuple[tuple[str, ...] | None, tuple[str, ...] | None]:
|
||||
) -> tuple[tuple[str, ...] | None, tuple[str, ...] | None, str]:
|
||||
"""Parse the optional `dlp` block on a route, returning
|
||||
(outbound_detectors, inbound_detectors)."""
|
||||
(outbound_detectors, inbound_detectors, outbound_on_match)."""
|
||||
dlp_raw = raw_dict.get("dlp")
|
||||
if dlp_raw is None:
|
||||
return None, None
|
||||
return None, None, ""
|
||||
label = f"route[{idx}] ({host})"
|
||||
if not isinstance(dlp_raw, dict):
|
||||
raise ValueError(f"{label}: 'dlp' must be an object")
|
||||
@@ -265,13 +276,24 @@ def _parse_detectors(
|
||||
outbound = _parse_detector_field("outbound_detectors", OUTBOUND_DETECTOR_NAMES)
|
||||
inbound = _parse_detector_field("inbound_detectors", INBOUND_DETECTOR_NAMES)
|
||||
|
||||
on_match = ""
|
||||
on_match_raw = dlp.get("outbound_on_match")
|
||||
if on_match_raw is not None:
|
||||
if not isinstance(on_match_raw, str) or on_match_raw not in OUTBOUND_ON_MATCH_VALUES:
|
||||
raise ValueError(
|
||||
f"{label}: dlp.outbound_on_match must be one of "
|
||||
f"{', '.join(OUTBOUND_ON_MATCH_VALUES)} (got {on_match_raw!r})"
|
||||
)
|
||||
on_match = on_match_raw
|
||||
|
||||
for k in dlp:
|
||||
if k not in ("outbound_detectors", "inbound_detectors"):
|
||||
if k not in ("outbound_detectors", "inbound_detectors", "outbound_on_match"):
|
||||
raise ValueError(
|
||||
f"{label}: dlp has unknown key {k!r}; accepted keys "
|
||||
f"are 'outbound_detectors', 'inbound_detectors'"
|
||||
f"are 'outbound_detectors', 'inbound_detectors', "
|
||||
f"'outbound_on_match'"
|
||||
)
|
||||
return outbound, inbound
|
||||
return outbound, inbound, on_match
|
||||
|
||||
|
||||
def parse_routes(payload: object) -> tuple[Route, ...]:
|
||||
@@ -342,7 +364,7 @@ def _parse_one(idx: int, raw: object) -> Route:
|
||||
)
|
||||
|
||||
# dlp detectors
|
||||
outbound_detectors, inbound_detectors = _parse_detectors(
|
||||
outbound_detectors, inbound_detectors, outbound_on_match = _parse_detectors(
|
||||
idx, host, raw_dict,
|
||||
)
|
||||
|
||||
@@ -361,6 +383,7 @@ def _parse_one(idx: int, raw: object) -> Route:
|
||||
git_fetch=git_fetch,
|
||||
outbound_detectors=outbound_detectors,
|
||||
inbound_detectors=inbound_detectors,
|
||||
outbound_on_match=outbound_on_match,
|
||||
)
|
||||
|
||||
|
||||
@@ -409,6 +432,8 @@ def route_to_yaml_dict(r: Route) -> dict[str, object]:
|
||||
dlp["outbound_detectors"] = list(r.outbound_detectors)
|
||||
if r.inbound_detectors is not None:
|
||||
dlp["inbound_detectors"] = list(r.inbound_detectors)
|
||||
if r.outbound_on_match:
|
||||
dlp["outbound_on_match"] = r.outbound_on_match
|
||||
if dlp:
|
||||
d["dlp"] = dlp
|
||||
return d
|
||||
@@ -781,6 +806,11 @@ __all__ = [
|
||||
"route_to_yaml_dict",
|
||||
"LOG_FULL",
|
||||
"LOG_OFF",
|
||||
"ON_MATCH_BLOCK",
|
||||
"ON_MATCH_REDACT",
|
||||
"ON_MATCH_SUPERVISE",
|
||||
"OUTBOUND_ON_MATCH_VALUES",
|
||||
"DEFAULT_OUTBOUND_ON_MATCH",
|
||||
"Config",
|
||||
"Decision",
|
||||
"HeaderMatch",
|
||||
|
||||
@@ -21,6 +21,9 @@ VALID_METHODS = frozenset({
|
||||
OUTBOUND_DETECTOR_NAMES = frozenset({"token_patterns", "known_secrets"})
|
||||
INBOUND_DETECTOR_NAMES = frozenset({"naive_injection_detection"})
|
||||
|
||||
# What the proxy does on an outbound token match (PRD 0062).
|
||||
OUTBOUND_ON_MATCH_VALUES = ("block", "redact", "supervise")
|
||||
|
||||
|
||||
def validate_egress_routes(
|
||||
bottle_name: str,
|
||||
@@ -67,6 +70,7 @@ class ManifestEgressRoute:
|
||||
GitFetch: bool = False
|
||||
OutboundDetectors: tuple[str, ...] | None = None
|
||||
InboundDetectors: tuple[str, ...] | None = None
|
||||
OutboundOnMatch: str = ""
|
||||
|
||||
@classmethod
|
||||
def from_dict(cls, bottle_name: str, idx: int, raw: object) -> "ManifestEgressRoute":
|
||||
@@ -161,8 +165,9 @@ class ManifestEgressRoute:
|
||||
# --- dlp ---
|
||||
outbound_detectors: tuple[str, ...] | None = None
|
||||
inbound_detectors: tuple[str, ...] | None = None
|
||||
outbound_on_match = ""
|
||||
if "dlp" in d:
|
||||
outbound_detectors, inbound_detectors = _parse_dlp_block(
|
||||
outbound_detectors, inbound_detectors, outbound_on_match = _parse_dlp_block(
|
||||
label, d.get("dlp"),
|
||||
)
|
||||
|
||||
@@ -201,6 +206,7 @@ class ManifestEgressRoute:
|
||||
GitFetch=git_fetch,
|
||||
OutboundDetectors=outbound_detectors,
|
||||
InboundDetectors=inbound_detectors,
|
||||
OutboundOnMatch=outbound_on_match,
|
||||
)
|
||||
|
||||
|
||||
@@ -323,7 +329,7 @@ def _parse_header_match(
|
||||
def _parse_dlp_block(
|
||||
route_label: str,
|
||||
raw: object,
|
||||
) -> tuple[tuple[str, ...] | None, tuple[str, ...] | None]:
|
||||
) -> tuple[tuple[str, ...] | None, tuple[str, ...] | None, str]:
|
||||
label = f"{route_label} dlp"
|
||||
d = as_json_object(raw, label)
|
||||
|
||||
@@ -358,13 +364,24 @@ def _parse_dlp_block(
|
||||
outbound = _parse_field("outbound_detectors", OUTBOUND_DETECTOR_NAMES)
|
||||
inbound = _parse_field("inbound_detectors", INBOUND_DETECTOR_NAMES)
|
||||
|
||||
on_match = ""
|
||||
on_match_raw = d.get("outbound_on_match")
|
||||
if on_match_raw is not None:
|
||||
if not isinstance(on_match_raw, str) or on_match_raw not in OUTBOUND_ON_MATCH_VALUES:
|
||||
raise ManifestError(
|
||||
f"{label} outbound_on_match must be one of "
|
||||
f"{', '.join(OUTBOUND_ON_MATCH_VALUES)} (got {on_match_raw!r})"
|
||||
)
|
||||
on_match = on_match_raw
|
||||
|
||||
for k in d:
|
||||
if k not in ("outbound_detectors", "inbound_detectors"):
|
||||
if k not in ("outbound_detectors", "inbound_detectors", "outbound_on_match"):
|
||||
raise ManifestError(
|
||||
f"{label} has unknown key {k!r}; accepted keys are "
|
||||
f"'outbound_detectors', 'inbound_detectors'"
|
||||
f"'outbound_detectors', 'inbound_detectors', "
|
||||
f"'outbound_on_match'"
|
||||
)
|
||||
return outbound, inbound
|
||||
return outbound, inbound, on_match
|
||||
|
||||
|
||||
LOG_LEVELS = frozenset({0, 1, 2})
|
||||
|
||||
@@ -187,6 +187,7 @@ TOOL_DEFINITIONS: list[dict[str, object]] = [
|
||||
" dlp: (optional DLP scanner overrides)\n"
|
||||
" outbound_detectors: [token_patterns, known_secrets]\n"
|
||||
" inbound_detectors: [naive_injection_detection]\n"
|
||||
" outbound_on_match: block|redact|supervise (default supervise)\n"
|
||||
"Omit any key that should use its default. "
|
||||
"`list-egress-routes` returns routes in this same format."
|
||||
),
|
||||
@@ -228,6 +229,7 @@ TOOL_DEFINITIONS: list[dict[str, object]] = [
|
||||
" dlp: (optional DLP scanner overrides)\n"
|
||||
" outbound_detectors: [token_patterns, known_secrets]\n"
|
||||
" inbound_detectors: [naive_injection_detection]\n"
|
||||
" outbound_on_match: block|redact|supervise (default supervise)\n"
|
||||
"Omit any key that should use its default. "
|
||||
"`list-egress-routes` returns routes in this same format."
|
||||
),
|
||||
|
||||
@@ -7,12 +7,26 @@
|
||||
|
||||
## Summary
|
||||
|
||||
When the egress proxy blocks an outbound request because a DLP detector
|
||||
matched a token/secret, route that block through the existing supervisor
|
||||
Give each egress route a policy for what happens when an outbound DLP detector
|
||||
matches a token, via `dlp.outbound_on_match: block | redact | supervise`
|
||||
(default `supervise`):
|
||||
|
||||
- **`supervise`** (default) — route the block through the existing supervisor
|
||||
approval queue instead of returning `403` immediately. The proxy holds the
|
||||
request open until the operator approves or rejects it. On approval, the
|
||||
request open until the operator approves or rejects it. On approval the
|
||||
matched token is added to an in-memory "safe tokens" set so the request — and
|
||||
any later request carrying the same token — flows through without re-prompting.
|
||||
any later request carrying the same token — flows through without
|
||||
re-prompting.
|
||||
- **`redact`** — scrub the matched value(s) from the request and forward it,
|
||||
no operator in the loop. For routes where a token-shaped value is noise the
|
||||
upstream doesn't need (telemetry/log sinks). Fails closed if a match lands on
|
||||
a surface redaction can't rewrite (the hostname).
|
||||
- **`block`** — the original hard `403`; never overridable. For routes where a
|
||||
detected token must always stop.
|
||||
|
||||
The motivating goal is reducing friction from false positives without weakening
|
||||
the default-deny posture: supervise keeps a human in the loop, redact is an
|
||||
explicit per-route opt-in, and block stays available for sensitive routes.
|
||||
|
||||
## Problem
|
||||
|
||||
@@ -58,9 +72,35 @@ fine-grained way to say "this specific value is fine."
|
||||
the exact value the detector found.
|
||||
- Replacing the per-route `dlp.outbound_detectors` override. That remains the
|
||||
way to turn a detector off wholesale.
|
||||
- Making `redact` the default. Silent redaction of a true false positive
|
||||
corrupts legitimate data, so it is opt-in per route; `supervise` (human in
|
||||
the loop) stays the default.
|
||||
|
||||
## Design
|
||||
|
||||
### On-match policy
|
||||
|
||||
`dlp.outbound_on_match` is a per-route enum threaded from the bottle manifest
|
||||
(`manifest_egress`) through the resolved route (`egress.EgressRoute`), the
|
||||
rendered `routes.yaml` (`egress_render_routes`), and the addon's `Route`
|
||||
(`egress_addon_core`). Unset renders nothing and resolves to `supervise` at
|
||||
request time. The `list-egress-routes` introspection endpoint round-trips it so
|
||||
the agent's proposals preserve it.
|
||||
|
||||
On an outbound block the addon dispatches on the resolved policy:
|
||||
|
||||
- **Structural blocks always 403.** A `ScanResult` with no `matched` value
|
||||
(CRLF injection) is a hard `403` regardless of policy — there is nothing to
|
||||
redact or safelist.
|
||||
- **`redact`** runs `redact_tokens` over the body, non-`host` header values,
|
||||
and path/query, then re-scans. If the re-scan is clean the (rewritten)
|
||||
request is forwarded; if a block-severity match remains (e.g. in the
|
||||
hostname, or a unicode-evasion token redaction can't reach) it fails closed
|
||||
with a `403`.
|
||||
- **`block`** writes the `403` immediately.
|
||||
- **`supervise`** runs the queue-and-wait loop below, falling back to `block`
|
||||
when supervise isn't wired for the bottle.
|
||||
|
||||
### Detected-value plumbing
|
||||
|
||||
`ScanResult` gains a `matched: str = ""` field carrying the raw substring the
|
||||
@@ -128,8 +168,11 @@ closed.
|
||||
required approval reason.
|
||||
3. **Addon glue** — async `request`, safe-tokens set, proposal write + async
|
||||
poll, allow/block decision; pass `safe_tokens` into the WebSocket path.
|
||||
4. **Tests + docs** — core/supervise/TUI unit tests; README egress + supervisor
|
||||
notes.
|
||||
4. **On-match policy** — `dlp.outbound_on_match` through manifest → render →
|
||||
addon; `redact` surface scrub with fail-closed re-scan; policy dispatch in
|
||||
the addon's outbound handler.
|
||||
5. **Tests + docs** — core/supervise/TUI/manifest/render unit tests; README
|
||||
egress + supervisor notes.
|
||||
|
||||
## Open questions
|
||||
|
||||
|
||||
@@ -329,6 +329,23 @@ class TestRenderRoutes(unittest.TestCase):
|
||||
self.assertEqual(("token_patterns",), addon_routes[0].outbound_detectors)
|
||||
self.assertEqual((), addon_routes[0].inbound_detectors)
|
||||
|
||||
def test_outbound_on_match_round_trips(self):
|
||||
from bot_bottle.egress_addon_core import load_routes
|
||||
b = _bottle([{"host": "logs.example", "dlp": {
|
||||
"outbound_on_match": "redact",
|
||||
}}])
|
||||
routes = egress_routes_for_bottle(b)
|
||||
rendered = egress_render_routes(routes)
|
||||
self.assertIn('outbound_on_match: "redact"', rendered)
|
||||
addon_routes = load_routes(rendered)
|
||||
self.assertEqual("redact", addon_routes[0].outbound_on_match)
|
||||
|
||||
def test_outbound_on_match_default_omitted_from_render(self):
|
||||
b = _bottle([{"host": "x.example"}])
|
||||
routes = egress_routes_for_bottle(b)
|
||||
rendered = egress_render_routes(routes)
|
||||
self.assertNotIn("outbound_on_match", rendered)
|
||||
|
||||
def test_git_fetch_policy_round_trips(self):
|
||||
from bot_bottle.egress_addon_core import load_routes
|
||||
b = _bottle([{"host": "github.com", "git": {"fetch": True}}])
|
||||
|
||||
@@ -269,6 +269,25 @@ class TestParseDlp(unittest.TestCase):
|
||||
"dlp": {"wat": True},
|
||||
}]})
|
||||
|
||||
def test_outbound_on_match_default_empty(self):
|
||||
routes = parse_routes({"routes": [{"host": "x.example"}]})
|
||||
self.assertEqual("", routes[0].outbound_on_match)
|
||||
|
||||
def test_outbound_on_match_parsed(self):
|
||||
for policy in ("block", "redact", "supervise"):
|
||||
routes = parse_routes({"routes": [{
|
||||
"host": "x.example",
|
||||
"dlp": {"outbound_on_match": policy},
|
||||
}]})
|
||||
self.assertEqual(policy, routes[0].outbound_on_match)
|
||||
|
||||
def test_outbound_on_match_invalid_rejected(self):
|
||||
with self.assertRaises(ValueError):
|
||||
parse_routes({"routes": [{
|
||||
"host": "x.example",
|
||||
"dlp": {"outbound_on_match": "nope"},
|
||||
}]})
|
||||
|
||||
|
||||
# --- load_routes ---------------------------------------------------------
|
||||
|
||||
|
||||
@@ -302,6 +302,24 @@ class TestDlp(unittest.TestCase):
|
||||
"bogus": True,
|
||||
}}])
|
||||
|
||||
def test_outbound_on_match_omitted_is_empty(self):
|
||||
b = _bottle([{"host": "x.example"}])
|
||||
self.assertEqual("", b.egress.routes[0].OutboundOnMatch)
|
||||
|
||||
def test_outbound_on_match_accepts_policies(self):
|
||||
for policy in ("block", "redact", "supervise"):
|
||||
with self.subTest(policy=policy):
|
||||
b = _bottle([{"host": "x.example", "dlp": {
|
||||
"outbound_on_match": policy,
|
||||
}}])
|
||||
self.assertEqual(policy, b.egress.routes[0].OutboundOnMatch)
|
||||
|
||||
def test_outbound_on_match_rejects_unknown_value(self):
|
||||
with self.assertRaises(ManifestError):
|
||||
_bottle([{"host": "x.example", "dlp": {
|
||||
"outbound_on_match": "allow",
|
||||
}}])
|
||||
|
||||
|
||||
class TestGitPolicy(unittest.TestCase):
|
||||
def test_omitted_means_https_git_fetch_disabled(self):
|
||||
|
||||
Reference in New Issue
Block a user