Add dlp.outbound_on_match policy (block | redact | supervise)
Give each egress route a policy for what the proxy does when an outbound DLP detector matches a token, defaulting to the supervise flow added in the previous commit. The goal is cutting false-positive friction without weakening default-deny. - redact: scrub the matched value(s) from the body, non-host headers, and path/query via redact_tokens, then re-scan. Forward if clean; fail closed with a 403 if a match remains on a surface redaction can't rewrite (the hostname, or a unicode-evasion token). For routes where a token-shaped value is noise the upstream doesn't need. - block: the original hard 403, never overridable. - supervise (default, unset): hold the request for operator approval. Structural blocks (CRLF, no safelist-able value) stay hard 403s under every policy. Threads outbound_on_match from the bottle manifest (manifest_egress) through the resolved EgressRoute and rendered routes.yaml (egress.py) to the addon's Route (egress_addon_core), and round-trips it via the list-egress-routes introspection endpoint. The allow/egress-block tool descriptions document the new key. Tests: manifest parse/validation, core parse/validation, full manifest->render->addon round-trip for redact. README + PRD 0062 updated. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HnvBjPZC5V7qeQpFbQdDmS
This commit is contained in:
@@ -15,7 +15,7 @@
|
|||||||
## Features
|
## Features
|
||||||
|
|
||||||
- **Per-bottle egress allowlist** — TLS-bumped HTTP/HTTPS chokepoint with a per-manifest host allowlist; per-route path/method/header `matches` filtering; outbound DLP scanning for known tokens and secrets, inbound DLP scanning for prompt-injection attempts; DoH and arbitrary hosts blocked by default.
|
- **Per-bottle egress allowlist** — TLS-bumped HTTP/HTTPS chokepoint with a per-manifest host allowlist; per-route path/method/header `matches` filtering; outbound DLP scanning for known tokens and secrets, inbound DLP scanning for prompt-injection attempts; DoH and arbitrary hosts blocked by default.
|
||||||
- **Supervisor override for token blocks** — when the outbound DLP catches a token, the request is held and surfaced in `./cli.py supervise` instead of failing outright. The operator approves or rejects; an approved value is remembered for the life of the egress proxy so the request — and later ones carrying it — flow through. Fails closed on rejection or timeout.
|
- **Per-route token-match policy** — each egress route picks what happens when the outbound DLP catches a token via `dlp.outbound_on_match`: `supervise` (default) holds the request and surfaces it in `./cli.py supervise` for approval (an approved value is remembered for the life of the proxy); `redact` scrubs the value and forwards; `block` is a hard `403`. Cuts false-positive friction without weakening default-deny.
|
||||||
- **Tokens the agent never sees** — host secrets live in a sidecar; the agent dials `http://sidecar:9099/<path>` and the proxy strips inbound `Authorization` and injects the real token before forwarding. `printenv` in the agent shows proxy URLs only.
|
- **Tokens the agent never sees** — host secrets live in a sidecar; the agent dials `http://sidecar:9099/<path>` and the proxy strips inbound `Authorization` and injects the real token before forwarding. `printenv` in the agent shows proxy URLs only.
|
||||||
- **Gitleaks-scanned push (git-gate)** — `bottle.git` remotes route through a per-bottle `git daemon` that gitleaks-scans incoming refs pre-receive and forwards clean refs upstream over SSH. The agent never holds the upstream credential.
|
- **Gitleaks-scanned push (git-gate)** — `bottle.git` remotes route through a per-bottle `git daemon` that gitleaks-scans incoming refs pre-receive and forwards clean refs upstream over SSH. The agent never holds the upstream credential.
|
||||||
- **Manifest-scoped skills + secrets** — each bottle declares its skills, env, git identity, remotes, and egress routes; unknown keys die at load.
|
- **Manifest-scoped skills + secrets** — each bottle declares its skills, env, git identity, remotes, and egress routes; unknown keys die at load.
|
||||||
@@ -149,9 +149,10 @@ You help maintain Gitea-hosted projects.
|
|||||||
| `dlp` | no | Per-route DLP overrides. Omit to use defaults (all detectors on). |
|
| `dlp` | no | Per-route DLP overrides. Omit to use defaults (all detectors on). |
|
||||||
| `dlp.outbound_detectors` | no | `false` disables outbound scanning; list restricts to named detectors (`token_patterns`, `known_secrets`). |
|
| `dlp.outbound_detectors` | no | `false` disables outbound scanning; list restricts to named detectors (`token_patterns`, `known_secrets`). |
|
||||||
| `dlp.inbound_detectors` | no | `false` disables inbound scanning; list restricts to named detectors (`naive_injection_detection`). |
|
| `dlp.inbound_detectors` | no | `false` disables inbound scanning; list restricts to named detectors (`naive_injection_detection`). |
|
||||||
|
| `dlp.outbound_on_match` | no | What to do when an outbound token is detected: `supervise` (default — hold for operator approval), `redact` (scrub the value and forward), or `block` (hard 403). |
|
||||||
| `git.fetch` | no | `true` permits smart HTTP clone/fetch (`git-upload-pack`) for this host. Push (`git-receive-pack`) remains blocked. |
|
| `git.fetch` | no | `true` permits smart HTTP clone/fetch (`git-upload-pack`) for this host. Push (`git-receive-pack`) remains blocked. |
|
||||||
|
|
||||||
When an outbound request is blocked because a DLP detector matched a token, the proxy queues an `egress-token-allow` proposal for the operator's `./cli.py supervise` TUI and holds the request open until it is answered (or `EGRESS_TOKEN_ALLOW_TIMEOUT_SECONDS`, default 300s, elapses — after which it fails closed). The operator never sees the raw token, only the host, method, path, and a redacted snippet. Approving adds the value to an in-memory safelist for the life of the egress proxy. Structural blocks (CRLF injection) and not-in-allowlist host blocks stay hard `403`s.
|
When an outbound DLP detector matches a token, the route's `dlp.outbound_on_match` policy decides what happens. Under the default `supervise`, the proxy queues an `egress-token-allow` proposal for the operator's `./cli.py supervise` TUI and holds the request open until it is answered (or `EGRESS_TOKEN_ALLOW_TIMEOUT_SECONDS`, default 300s, elapses — after which it fails closed). The operator never sees the raw token, only the host, method, path, and a redacted snippet; approving adds the value to an in-memory safelist for the life of the egress proxy. Under `redact`, the matched value is scrubbed from the body, headers, and path and the request is forwarded (failing closed if a match lands somewhere unredactable, like the hostname). Under `block` it stays a hard `403`. Structural blocks (CRLF injection) and not-in-allowlist host blocks are always hard `403`s regardless of policy.
|
||||||
|
|
||||||
More examples in `examples/`. Full design lives under `docs/prds/`; the trust-boundary rationale is in `docs/prds/0011-per-file-md-manifest.md`.
|
More examples in `examples/`. Full design lives under `docs/prds/`; the trust-boundary rationale is in `docs/prds/0011-per-file-md-manifest.md`.
|
||||||
|
|
||||||
|
|||||||
+10
-1
@@ -95,6 +95,7 @@ def egress_manifest_routes(
|
|||||||
git_fetch=r.GitFetch,
|
git_fetch=r.GitFetch,
|
||||||
outbound_detectors=r.OutboundDetectors,
|
outbound_detectors=r.OutboundDetectors,
|
||||||
inbound_detectors=r.InboundDetectors,
|
inbound_detectors=r.InboundDetectors,
|
||||||
|
outbound_on_match=r.OutboundOnMatch,
|
||||||
))
|
))
|
||||||
return tuple(out)
|
return tuple(out)
|
||||||
|
|
||||||
@@ -177,7 +178,11 @@ def _route_to_yaml_fields(r: Route) -> dict[str, object]:
|
|||||||
fields["matches"] = matches_data
|
fields["matches"] = matches_data
|
||||||
if r.git_fetch:
|
if r.git_fetch:
|
||||||
fields["git"] = {"fetch": True}
|
fields["git"] = {"fetch": True}
|
||||||
if r.outbound_detectors is not None or r.inbound_detectors is not None:
|
if (
|
||||||
|
r.outbound_detectors is not None
|
||||||
|
or r.inbound_detectors is not None
|
||||||
|
or r.outbound_on_match
|
||||||
|
):
|
||||||
dlp: dict[str, object] = {}
|
dlp: dict[str, object] = {}
|
||||||
if r.outbound_detectors is not None:
|
if r.outbound_detectors is not None:
|
||||||
dlp["outbound_detectors"] = (
|
dlp["outbound_detectors"] = (
|
||||||
@@ -189,6 +194,8 @@ def _route_to_yaml_fields(r: Route) -> dict[str, object]:
|
|||||||
False if not r.inbound_detectors
|
False if not r.inbound_detectors
|
||||||
else list(r.inbound_detectors)
|
else list(r.inbound_detectors)
|
||||||
)
|
)
|
||||||
|
if r.outbound_on_match:
|
||||||
|
dlp["outbound_on_match"] = r.outbound_on_match
|
||||||
fields["dlp"] = dlp
|
fields["dlp"] = dlp
|
||||||
return fields
|
return fields
|
||||||
|
|
||||||
@@ -260,6 +267,8 @@ def egress_render_routes(
|
|||||||
elif isinstance(dv, list):
|
elif isinstance(dv, list):
|
||||||
items_str = ", ".join(f'"{x}"' for x in dv)
|
items_str = ", ".join(f'"{x}"' for x in dv)
|
||||||
lines.append(f" {dk}: [{items_str}]")
|
lines.append(f" {dk}: [{items_str}]")
|
||||||
|
elif isinstance(dv, str):
|
||||||
|
lines.append(f' {dk}: "{dv}"')
|
||||||
return "\n".join(lines) + "\n"
|
return "\n".join(lines) + "\n"
|
||||||
|
|
||||||
|
|
||||||
|
|||||||
+112
-30
@@ -17,7 +17,11 @@ from mitmproxy import http # type: ignore[import-not-found] # pylint: disable=
|
|||||||
from egress_addon_core import ( # type: ignore[import-not-found] # pylint: disable=import-error
|
from egress_addon_core import ( # type: ignore[import-not-found] # pylint: disable=import-error
|
||||||
LOG_BLOCKS,
|
LOG_BLOCKS,
|
||||||
LOG_FULL,
|
LOG_FULL,
|
||||||
|
DEFAULT_OUTBOUND_ON_MATCH,
|
||||||
|
ON_MATCH_BLOCK,
|
||||||
|
ON_MATCH_REDACT,
|
||||||
Config,
|
Config,
|
||||||
|
Route,
|
||||||
ScanResult,
|
ScanResult,
|
||||||
build_inbound_scan_text,
|
build_inbound_scan_text,
|
||||||
build_outbound_scan_text,
|
build_outbound_scan_text,
|
||||||
@@ -189,37 +193,11 @@ class EgressAddon:
|
|||||||
# Hostname is included to catch DNS-tunnelling exfiltration attempts.
|
# Hostname is included to catch DNS-tunnelling exfiltration attempts.
|
||||||
route = match_route(self.config.routes, flow.request.pretty_host)
|
route = match_route(self.config.routes, flow.request.pretty_host)
|
||||||
if route is not None:
|
if route is not None:
|
||||||
body = flow.request.get_text(strict=False) or ""
|
if not await self._handle_outbound_dlp(flow, route):
|
||||||
# Re-scan after each operator approval so a second, un-approved
|
|
||||||
# token in the same request is still caught (PRD 0062).
|
|
||||||
while True:
|
|
||||||
scan_text = build_outbound_scan_text(
|
|
||||||
flow.request.pretty_host,
|
|
||||||
request_path,
|
|
||||||
query,
|
|
||||||
outbound_scan_headers(route, dict(flow.request.headers)),
|
|
||||||
body,
|
|
||||||
)
|
|
||||||
dlp_result = scan_outbound(
|
|
||||||
route, scan_text, os.environ, safe_tokens=self.safe_tokens,
|
|
||||||
)
|
|
||||||
if dlp_result is None or dlp_result.severity != "block":
|
|
||||||
break
|
|
||||||
# Token blocks (a match with a safelist-able value) can be
|
|
||||||
# routed to the operator; structural blocks (CRLF, matched="")
|
|
||||||
# and any block when supervise is disabled stay hard 403s.
|
|
||||||
if dlp_result.matched and self._supervise_available():
|
|
||||||
approved = await self._supervise_token_block(
|
|
||||||
flow, request_path, dlp_result,
|
|
||||||
)
|
|
||||||
if approved:
|
|
||||||
continue # re-scan; matched value now in safe_tokens
|
|
||||||
return # _supervise_token_block wrote the 403 response
|
|
||||||
ctx = self._req_ctx(flow)
|
|
||||||
if dlp_result.context:
|
|
||||||
ctx = {**ctx, "context": dlp_result.context}
|
|
||||||
self._block(flow, f"egress DLP: {dlp_result.reason}", ctx=ctx)
|
|
||||||
return
|
return
|
||||||
|
# The redact policy may have rewritten the request line; recompute
|
||||||
|
# the path/query the git checks below rely on.
|
||||||
|
request_path, _, query = flow.request.path.partition("?")
|
||||||
|
|
||||||
if is_git_push_request(request_path, query):
|
if is_git_push_request(request_path, query):
|
||||||
self._block(
|
self._block(
|
||||||
@@ -269,6 +247,110 @@ class EgressAddon:
|
|||||||
if self.config.log >= LOG_FULL:
|
if self.config.log >= LOG_FULL:
|
||||||
self._log_request(flow)
|
self._log_request(flow)
|
||||||
|
|
||||||
|
def _block_dlp(self, flow: http.HTTPFlow, result: ScanResult) -> None:
|
||||||
|
ctx = self._req_ctx(flow)
|
||||||
|
if result.context:
|
||||||
|
ctx = {**ctx, "context": result.context}
|
||||||
|
self._block(flow, f"egress DLP: {result.reason}", ctx=ctx)
|
||||||
|
|
||||||
|
async def _handle_outbound_dlp(
|
||||||
|
self,
|
||||||
|
flow: http.HTTPFlow,
|
||||||
|
route: Route,
|
||||||
|
) -> bool:
|
||||||
|
"""Scan the outbound request and apply the route's on-match policy
|
||||||
|
(PRD 0062). Returns True if the request may be forwarded, False if a
|
||||||
|
403 response has been written to `flow`.
|
||||||
|
|
||||||
|
Loops so the supervise policy can re-scan after each approval — a
|
||||||
|
second, un-approved token in the same request is still caught."""
|
||||||
|
while True:
|
||||||
|
request_path, _, query = flow.request.path.partition("?")
|
||||||
|
body = flow.request.get_text(strict=False) or ""
|
||||||
|
scan_text = build_outbound_scan_text(
|
||||||
|
flow.request.pretty_host,
|
||||||
|
request_path,
|
||||||
|
query,
|
||||||
|
outbound_scan_headers(route, dict(flow.request.headers)),
|
||||||
|
body,
|
||||||
|
)
|
||||||
|
result = scan_outbound(
|
||||||
|
route, scan_text, os.environ, safe_tokens=self.safe_tokens,
|
||||||
|
)
|
||||||
|
if result is None or result.severity != "block":
|
||||||
|
return True
|
||||||
|
|
||||||
|
# Structural blocks (CRLF, no safelist-able value) are always a
|
||||||
|
# hard 403, regardless of the route's on-match policy.
|
||||||
|
if not result.matched:
|
||||||
|
self._block_dlp(flow, result)
|
||||||
|
return False
|
||||||
|
|
||||||
|
policy = route.outbound_on_match or DEFAULT_OUTBOUND_ON_MATCH
|
||||||
|
|
||||||
|
if policy == ON_MATCH_REDACT:
|
||||||
|
if self._redact_outbound(flow, route):
|
||||||
|
if self.config.log >= LOG_BLOCKS:
|
||||||
|
sys.stderr.write(json.dumps({
|
||||||
|
"event": "egress_redacted",
|
||||||
|
"reason": f"egress DLP: {result.reason}",
|
||||||
|
**self._req_ctx(flow),
|
||||||
|
}) + "\n")
|
||||||
|
return True
|
||||||
|
self._block(
|
||||||
|
flow,
|
||||||
|
f"egress DLP: {result.reason}; redaction could not remove "
|
||||||
|
"all matches (e.g. a match in the hostname)",
|
||||||
|
ctx=self._req_ctx(flow),
|
||||||
|
)
|
||||||
|
return False
|
||||||
|
|
||||||
|
if policy == ON_MATCH_BLOCK:
|
||||||
|
self._block_dlp(flow, result)
|
||||||
|
return False
|
||||||
|
|
||||||
|
# supervise (default): hold the request for operator approval.
|
||||||
|
# Fall back to a hard 403 when supervise isn't wired for the bottle.
|
||||||
|
if not self._supervise_available():
|
||||||
|
self._block_dlp(flow, result)
|
||||||
|
return False
|
||||||
|
approved = await self._supervise_token_block(flow, request_path, result)
|
||||||
|
if not approved:
|
||||||
|
return False # _supervise_token_block wrote the 403 response
|
||||||
|
# loop: the approved value is now in safe_tokens; re-scan.
|
||||||
|
|
||||||
|
def _redact_outbound(self, flow: http.HTTPFlow, route: Route) -> bool:
|
||||||
|
"""Scrub detected tokens from the mutable request surfaces (body,
|
||||||
|
headers, path/query) and re-scan. Returns True if the request is now
|
||||||
|
clean; False if a block-severity match remains on a surface redaction
|
||||||
|
cannot rewrite (the hostname) so the caller fails closed."""
|
||||||
|
body = flow.request.get_text(strict=False)
|
||||||
|
if body:
|
||||||
|
redacted_body = redact_tokens(body, env=os.environ)
|
||||||
|
if redacted_body != body:
|
||||||
|
flow.request.text = redacted_body
|
||||||
|
for name, value in list(flow.request.headers.items()):
|
||||||
|
if name.lower() == "host":
|
||||||
|
continue # routing-critical; never a legitimate token
|
||||||
|
redacted = redact_tokens(value, env=os.environ)
|
||||||
|
if redacted != value:
|
||||||
|
flow.request.headers[name] = redacted
|
||||||
|
redacted_path = redact_tokens(flow.request.path, env=os.environ)
|
||||||
|
if redacted_path != flow.request.path:
|
||||||
|
flow.request.path = redacted_path
|
||||||
|
|
||||||
|
request_path, _, query = flow.request.path.partition("?")
|
||||||
|
new_body = flow.request.get_text(strict=False) or ""
|
||||||
|
scan_text = build_outbound_scan_text(
|
||||||
|
flow.request.pretty_host,
|
||||||
|
request_path,
|
||||||
|
query,
|
||||||
|
outbound_scan_headers(route, dict(flow.request.headers)),
|
||||||
|
new_body,
|
||||||
|
)
|
||||||
|
result = scan_outbound(route, scan_text, os.environ)
|
||||||
|
return result is None or result.severity != "block"
|
||||||
|
|
||||||
async def _supervise_token_block(
|
async def _supervise_token_block(
|
||||||
self,
|
self,
|
||||||
flow: http.HTTPFlow,
|
flow: http.HTTPFlow,
|
||||||
|
|||||||
@@ -37,6 +37,15 @@ VALID_METHODS = frozenset({
|
|||||||
OUTBOUND_DETECTOR_NAMES = frozenset({"token_patterns", "known_secrets"})
|
OUTBOUND_DETECTOR_NAMES = frozenset({"token_patterns", "known_secrets"})
|
||||||
INBOUND_DETECTOR_NAMES = frozenset({"naive_injection_detection"})
|
INBOUND_DETECTOR_NAMES = frozenset({"naive_injection_detection"})
|
||||||
|
|
||||||
|
# Per-route policy for what the proxy does when an outbound DLP detector
|
||||||
|
# matches a token (PRD 0062).
|
||||||
|
ON_MATCH_BLOCK = "block" # hard 403, never overridable
|
||||||
|
ON_MATCH_REDACT = "redact" # scrub the matched value, forward the request
|
||||||
|
ON_MATCH_SUPERVISE = "supervise" # queue for operator approval, hold the request
|
||||||
|
OUTBOUND_ON_MATCH_VALUES = (ON_MATCH_BLOCK, ON_MATCH_REDACT, ON_MATCH_SUPERVISE)
|
||||||
|
# Unset resolves to supervise (fall back to block when supervise is not wired).
|
||||||
|
DEFAULT_OUTBOUND_ON_MATCH = ON_MATCH_SUPERVISE
|
||||||
|
|
||||||
|
|
||||||
@dataclass(frozen=True)
|
@dataclass(frozen=True)
|
||||||
class PathMatch:
|
class PathMatch:
|
||||||
@@ -69,6 +78,8 @@ class Route:
|
|||||||
git_fetch: bool = False
|
git_fetch: bool = False
|
||||||
outbound_detectors: tuple[str, ...] | None = None
|
outbound_detectors: tuple[str, ...] | None = None
|
||||||
inbound_detectors: tuple[str, ...] | None = None
|
inbound_detectors: tuple[str, ...] | None = None
|
||||||
|
# "" means unset → DEFAULT_OUTBOUND_ON_MATCH. See OUTBOUND_ON_MATCH_VALUES.
|
||||||
|
outbound_on_match: str = ""
|
||||||
|
|
||||||
|
|
||||||
LOG_OFF = 0 # no logging
|
LOG_OFF = 0 # no logging
|
||||||
@@ -223,12 +234,12 @@ def _parse_detectors(
|
|||||||
idx: int,
|
idx: int,
|
||||||
host: str,
|
host: str,
|
||||||
raw_dict: dict[str, object],
|
raw_dict: dict[str, object],
|
||||||
) -> tuple[tuple[str, ...] | None, tuple[str, ...] | None]:
|
) -> tuple[tuple[str, ...] | None, tuple[str, ...] | None, str]:
|
||||||
"""Parse the optional `dlp` block on a route, returning
|
"""Parse the optional `dlp` block on a route, returning
|
||||||
(outbound_detectors, inbound_detectors)."""
|
(outbound_detectors, inbound_detectors, outbound_on_match)."""
|
||||||
dlp_raw = raw_dict.get("dlp")
|
dlp_raw = raw_dict.get("dlp")
|
||||||
if dlp_raw is None:
|
if dlp_raw is None:
|
||||||
return None, None
|
return None, None, ""
|
||||||
label = f"route[{idx}] ({host})"
|
label = f"route[{idx}] ({host})"
|
||||||
if not isinstance(dlp_raw, dict):
|
if not isinstance(dlp_raw, dict):
|
||||||
raise ValueError(f"{label}: 'dlp' must be an object")
|
raise ValueError(f"{label}: 'dlp' must be an object")
|
||||||
@@ -265,13 +276,24 @@ def _parse_detectors(
|
|||||||
outbound = _parse_detector_field("outbound_detectors", OUTBOUND_DETECTOR_NAMES)
|
outbound = _parse_detector_field("outbound_detectors", OUTBOUND_DETECTOR_NAMES)
|
||||||
inbound = _parse_detector_field("inbound_detectors", INBOUND_DETECTOR_NAMES)
|
inbound = _parse_detector_field("inbound_detectors", INBOUND_DETECTOR_NAMES)
|
||||||
|
|
||||||
|
on_match = ""
|
||||||
|
on_match_raw = dlp.get("outbound_on_match")
|
||||||
|
if on_match_raw is not None:
|
||||||
|
if not isinstance(on_match_raw, str) or on_match_raw not in OUTBOUND_ON_MATCH_VALUES:
|
||||||
|
raise ValueError(
|
||||||
|
f"{label}: dlp.outbound_on_match must be one of "
|
||||||
|
f"{', '.join(OUTBOUND_ON_MATCH_VALUES)} (got {on_match_raw!r})"
|
||||||
|
)
|
||||||
|
on_match = on_match_raw
|
||||||
|
|
||||||
for k in dlp:
|
for k in dlp:
|
||||||
if k not in ("outbound_detectors", "inbound_detectors"):
|
if k not in ("outbound_detectors", "inbound_detectors", "outbound_on_match"):
|
||||||
raise ValueError(
|
raise ValueError(
|
||||||
f"{label}: dlp has unknown key {k!r}; accepted keys "
|
f"{label}: dlp has unknown key {k!r}; accepted keys "
|
||||||
f"are 'outbound_detectors', 'inbound_detectors'"
|
f"are 'outbound_detectors', 'inbound_detectors', "
|
||||||
|
f"'outbound_on_match'"
|
||||||
)
|
)
|
||||||
return outbound, inbound
|
return outbound, inbound, on_match
|
||||||
|
|
||||||
|
|
||||||
def parse_routes(payload: object) -> tuple[Route, ...]:
|
def parse_routes(payload: object) -> tuple[Route, ...]:
|
||||||
@@ -342,7 +364,7 @@ def _parse_one(idx: int, raw: object) -> Route:
|
|||||||
)
|
)
|
||||||
|
|
||||||
# dlp detectors
|
# dlp detectors
|
||||||
outbound_detectors, inbound_detectors = _parse_detectors(
|
outbound_detectors, inbound_detectors, outbound_on_match = _parse_detectors(
|
||||||
idx, host, raw_dict,
|
idx, host, raw_dict,
|
||||||
)
|
)
|
||||||
|
|
||||||
@@ -361,6 +383,7 @@ def _parse_one(idx: int, raw: object) -> Route:
|
|||||||
git_fetch=git_fetch,
|
git_fetch=git_fetch,
|
||||||
outbound_detectors=outbound_detectors,
|
outbound_detectors=outbound_detectors,
|
||||||
inbound_detectors=inbound_detectors,
|
inbound_detectors=inbound_detectors,
|
||||||
|
outbound_on_match=outbound_on_match,
|
||||||
)
|
)
|
||||||
|
|
||||||
|
|
||||||
@@ -409,6 +432,8 @@ def route_to_yaml_dict(r: Route) -> dict[str, object]:
|
|||||||
dlp["outbound_detectors"] = list(r.outbound_detectors)
|
dlp["outbound_detectors"] = list(r.outbound_detectors)
|
||||||
if r.inbound_detectors is not None:
|
if r.inbound_detectors is not None:
|
||||||
dlp["inbound_detectors"] = list(r.inbound_detectors)
|
dlp["inbound_detectors"] = list(r.inbound_detectors)
|
||||||
|
if r.outbound_on_match:
|
||||||
|
dlp["outbound_on_match"] = r.outbound_on_match
|
||||||
if dlp:
|
if dlp:
|
||||||
d["dlp"] = dlp
|
d["dlp"] = dlp
|
||||||
return d
|
return d
|
||||||
@@ -781,6 +806,11 @@ __all__ = [
|
|||||||
"route_to_yaml_dict",
|
"route_to_yaml_dict",
|
||||||
"LOG_FULL",
|
"LOG_FULL",
|
||||||
"LOG_OFF",
|
"LOG_OFF",
|
||||||
|
"ON_MATCH_BLOCK",
|
||||||
|
"ON_MATCH_REDACT",
|
||||||
|
"ON_MATCH_SUPERVISE",
|
||||||
|
"OUTBOUND_ON_MATCH_VALUES",
|
||||||
|
"DEFAULT_OUTBOUND_ON_MATCH",
|
||||||
"Config",
|
"Config",
|
||||||
"Decision",
|
"Decision",
|
||||||
"HeaderMatch",
|
"HeaderMatch",
|
||||||
|
|||||||
@@ -21,6 +21,9 @@ VALID_METHODS = frozenset({
|
|||||||
OUTBOUND_DETECTOR_NAMES = frozenset({"token_patterns", "known_secrets"})
|
OUTBOUND_DETECTOR_NAMES = frozenset({"token_patterns", "known_secrets"})
|
||||||
INBOUND_DETECTOR_NAMES = frozenset({"naive_injection_detection"})
|
INBOUND_DETECTOR_NAMES = frozenset({"naive_injection_detection"})
|
||||||
|
|
||||||
|
# What the proxy does on an outbound token match (PRD 0062).
|
||||||
|
OUTBOUND_ON_MATCH_VALUES = ("block", "redact", "supervise")
|
||||||
|
|
||||||
|
|
||||||
def validate_egress_routes(
|
def validate_egress_routes(
|
||||||
bottle_name: str,
|
bottle_name: str,
|
||||||
@@ -67,6 +70,7 @@ class ManifestEgressRoute:
|
|||||||
GitFetch: bool = False
|
GitFetch: bool = False
|
||||||
OutboundDetectors: tuple[str, ...] | None = None
|
OutboundDetectors: tuple[str, ...] | None = None
|
||||||
InboundDetectors: tuple[str, ...] | None = None
|
InboundDetectors: tuple[str, ...] | None = None
|
||||||
|
OutboundOnMatch: str = ""
|
||||||
|
|
||||||
@classmethod
|
@classmethod
|
||||||
def from_dict(cls, bottle_name: str, idx: int, raw: object) -> "ManifestEgressRoute":
|
def from_dict(cls, bottle_name: str, idx: int, raw: object) -> "ManifestEgressRoute":
|
||||||
@@ -161,8 +165,9 @@ class ManifestEgressRoute:
|
|||||||
# --- dlp ---
|
# --- dlp ---
|
||||||
outbound_detectors: tuple[str, ...] | None = None
|
outbound_detectors: tuple[str, ...] | None = None
|
||||||
inbound_detectors: tuple[str, ...] | None = None
|
inbound_detectors: tuple[str, ...] | None = None
|
||||||
|
outbound_on_match = ""
|
||||||
if "dlp" in d:
|
if "dlp" in d:
|
||||||
outbound_detectors, inbound_detectors = _parse_dlp_block(
|
outbound_detectors, inbound_detectors, outbound_on_match = _parse_dlp_block(
|
||||||
label, d.get("dlp"),
|
label, d.get("dlp"),
|
||||||
)
|
)
|
||||||
|
|
||||||
@@ -201,6 +206,7 @@ class ManifestEgressRoute:
|
|||||||
GitFetch=git_fetch,
|
GitFetch=git_fetch,
|
||||||
OutboundDetectors=outbound_detectors,
|
OutboundDetectors=outbound_detectors,
|
||||||
InboundDetectors=inbound_detectors,
|
InboundDetectors=inbound_detectors,
|
||||||
|
OutboundOnMatch=outbound_on_match,
|
||||||
)
|
)
|
||||||
|
|
||||||
|
|
||||||
@@ -323,7 +329,7 @@ def _parse_header_match(
|
|||||||
def _parse_dlp_block(
|
def _parse_dlp_block(
|
||||||
route_label: str,
|
route_label: str,
|
||||||
raw: object,
|
raw: object,
|
||||||
) -> tuple[tuple[str, ...] | None, tuple[str, ...] | None]:
|
) -> tuple[tuple[str, ...] | None, tuple[str, ...] | None, str]:
|
||||||
label = f"{route_label} dlp"
|
label = f"{route_label} dlp"
|
||||||
d = as_json_object(raw, label)
|
d = as_json_object(raw, label)
|
||||||
|
|
||||||
@@ -358,13 +364,24 @@ def _parse_dlp_block(
|
|||||||
outbound = _parse_field("outbound_detectors", OUTBOUND_DETECTOR_NAMES)
|
outbound = _parse_field("outbound_detectors", OUTBOUND_DETECTOR_NAMES)
|
||||||
inbound = _parse_field("inbound_detectors", INBOUND_DETECTOR_NAMES)
|
inbound = _parse_field("inbound_detectors", INBOUND_DETECTOR_NAMES)
|
||||||
|
|
||||||
|
on_match = ""
|
||||||
|
on_match_raw = d.get("outbound_on_match")
|
||||||
|
if on_match_raw is not None:
|
||||||
|
if not isinstance(on_match_raw, str) or on_match_raw not in OUTBOUND_ON_MATCH_VALUES:
|
||||||
|
raise ManifestError(
|
||||||
|
f"{label} outbound_on_match must be one of "
|
||||||
|
f"{', '.join(OUTBOUND_ON_MATCH_VALUES)} (got {on_match_raw!r})"
|
||||||
|
)
|
||||||
|
on_match = on_match_raw
|
||||||
|
|
||||||
for k in d:
|
for k in d:
|
||||||
if k not in ("outbound_detectors", "inbound_detectors"):
|
if k not in ("outbound_detectors", "inbound_detectors", "outbound_on_match"):
|
||||||
raise ManifestError(
|
raise ManifestError(
|
||||||
f"{label} has unknown key {k!r}; accepted keys are "
|
f"{label} has unknown key {k!r}; accepted keys are "
|
||||||
f"'outbound_detectors', 'inbound_detectors'"
|
f"'outbound_detectors', 'inbound_detectors', "
|
||||||
|
f"'outbound_on_match'"
|
||||||
)
|
)
|
||||||
return outbound, inbound
|
return outbound, inbound, on_match
|
||||||
|
|
||||||
|
|
||||||
LOG_LEVELS = frozenset({0, 1, 2})
|
LOG_LEVELS = frozenset({0, 1, 2})
|
||||||
|
|||||||
@@ -187,6 +187,7 @@ TOOL_DEFINITIONS: list[dict[str, object]] = [
|
|||||||
" dlp: (optional DLP scanner overrides)\n"
|
" dlp: (optional DLP scanner overrides)\n"
|
||||||
" outbound_detectors: [token_patterns, known_secrets]\n"
|
" outbound_detectors: [token_patterns, known_secrets]\n"
|
||||||
" inbound_detectors: [naive_injection_detection]\n"
|
" inbound_detectors: [naive_injection_detection]\n"
|
||||||
|
" outbound_on_match: block|redact|supervise (default supervise)\n"
|
||||||
"Omit any key that should use its default. "
|
"Omit any key that should use its default. "
|
||||||
"`list-egress-routes` returns routes in this same format."
|
"`list-egress-routes` returns routes in this same format."
|
||||||
),
|
),
|
||||||
@@ -228,6 +229,7 @@ TOOL_DEFINITIONS: list[dict[str, object]] = [
|
|||||||
" dlp: (optional DLP scanner overrides)\n"
|
" dlp: (optional DLP scanner overrides)\n"
|
||||||
" outbound_detectors: [token_patterns, known_secrets]\n"
|
" outbound_detectors: [token_patterns, known_secrets]\n"
|
||||||
" inbound_detectors: [naive_injection_detection]\n"
|
" inbound_detectors: [naive_injection_detection]\n"
|
||||||
|
" outbound_on_match: block|redact|supervise (default supervise)\n"
|
||||||
"Omit any key that should use its default. "
|
"Omit any key that should use its default. "
|
||||||
"`list-egress-routes` returns routes in this same format."
|
"`list-egress-routes` returns routes in this same format."
|
||||||
),
|
),
|
||||||
|
|||||||
@@ -7,12 +7,26 @@
|
|||||||
|
|
||||||
## Summary
|
## Summary
|
||||||
|
|
||||||
When the egress proxy blocks an outbound request because a DLP detector
|
Give each egress route a policy for what happens when an outbound DLP detector
|
||||||
matched a token/secret, route that block through the existing supervisor
|
matches a token, via `dlp.outbound_on_match: block | redact | supervise`
|
||||||
|
(default `supervise`):
|
||||||
|
|
||||||
|
- **`supervise`** (default) — route the block through the existing supervisor
|
||||||
approval queue instead of returning `403` immediately. The proxy holds the
|
approval queue instead of returning `403` immediately. The proxy holds the
|
||||||
request open until the operator approves or rejects it. On approval, the
|
request open until the operator approves or rejects it. On approval the
|
||||||
matched token is added to an in-memory "safe tokens" set so the request — and
|
matched token is added to an in-memory "safe tokens" set so the request — and
|
||||||
any later request carrying the same token — flows through without re-prompting.
|
any later request carrying the same token — flows through without
|
||||||
|
re-prompting.
|
||||||
|
- **`redact`** — scrub the matched value(s) from the request and forward it,
|
||||||
|
no operator in the loop. For routes where a token-shaped value is noise the
|
||||||
|
upstream doesn't need (telemetry/log sinks). Fails closed if a match lands on
|
||||||
|
a surface redaction can't rewrite (the hostname).
|
||||||
|
- **`block`** — the original hard `403`; never overridable. For routes where a
|
||||||
|
detected token must always stop.
|
||||||
|
|
||||||
|
The motivating goal is reducing friction from false positives without weakening
|
||||||
|
the default-deny posture: supervise keeps a human in the loop, redact is an
|
||||||
|
explicit per-route opt-in, and block stays available for sensitive routes.
|
||||||
|
|
||||||
## Problem
|
## Problem
|
||||||
|
|
||||||
@@ -58,9 +72,35 @@ fine-grained way to say "this specific value is fine."
|
|||||||
the exact value the detector found.
|
the exact value the detector found.
|
||||||
- Replacing the per-route `dlp.outbound_detectors` override. That remains the
|
- Replacing the per-route `dlp.outbound_detectors` override. That remains the
|
||||||
way to turn a detector off wholesale.
|
way to turn a detector off wholesale.
|
||||||
|
- Making `redact` the default. Silent redaction of a true false positive
|
||||||
|
corrupts legitimate data, so it is opt-in per route; `supervise` (human in
|
||||||
|
the loop) stays the default.
|
||||||
|
|
||||||
## Design
|
## Design
|
||||||
|
|
||||||
|
### On-match policy
|
||||||
|
|
||||||
|
`dlp.outbound_on_match` is a per-route enum threaded from the bottle manifest
|
||||||
|
(`manifest_egress`) through the resolved route (`egress.EgressRoute`), the
|
||||||
|
rendered `routes.yaml` (`egress_render_routes`), and the addon's `Route`
|
||||||
|
(`egress_addon_core`). Unset renders nothing and resolves to `supervise` at
|
||||||
|
request time. The `list-egress-routes` introspection endpoint round-trips it so
|
||||||
|
the agent's proposals preserve it.
|
||||||
|
|
||||||
|
On an outbound block the addon dispatches on the resolved policy:
|
||||||
|
|
||||||
|
- **Structural blocks always 403.** A `ScanResult` with no `matched` value
|
||||||
|
(CRLF injection) is a hard `403` regardless of policy — there is nothing to
|
||||||
|
redact or safelist.
|
||||||
|
- **`redact`** runs `redact_tokens` over the body, non-`host` header values,
|
||||||
|
and path/query, then re-scans. If the re-scan is clean the (rewritten)
|
||||||
|
request is forwarded; if a block-severity match remains (e.g. in the
|
||||||
|
hostname, or a unicode-evasion token redaction can't reach) it fails closed
|
||||||
|
with a `403`.
|
||||||
|
- **`block`** writes the `403` immediately.
|
||||||
|
- **`supervise`** runs the queue-and-wait loop below, falling back to `block`
|
||||||
|
when supervise isn't wired for the bottle.
|
||||||
|
|
||||||
### Detected-value plumbing
|
### Detected-value plumbing
|
||||||
|
|
||||||
`ScanResult` gains a `matched: str = ""` field carrying the raw substring the
|
`ScanResult` gains a `matched: str = ""` field carrying the raw substring the
|
||||||
@@ -128,8 +168,11 @@ closed.
|
|||||||
required approval reason.
|
required approval reason.
|
||||||
3. **Addon glue** — async `request`, safe-tokens set, proposal write + async
|
3. **Addon glue** — async `request`, safe-tokens set, proposal write + async
|
||||||
poll, allow/block decision; pass `safe_tokens` into the WebSocket path.
|
poll, allow/block decision; pass `safe_tokens` into the WebSocket path.
|
||||||
4. **Tests + docs** — core/supervise/TUI unit tests; README egress + supervisor
|
4. **On-match policy** — `dlp.outbound_on_match` through manifest → render →
|
||||||
notes.
|
addon; `redact` surface scrub with fail-closed re-scan; policy dispatch in
|
||||||
|
the addon's outbound handler.
|
||||||
|
5. **Tests + docs** — core/supervise/TUI/manifest/render unit tests; README
|
||||||
|
egress + supervisor notes.
|
||||||
|
|
||||||
## Open questions
|
## Open questions
|
||||||
|
|
||||||
|
|||||||
@@ -329,6 +329,23 @@ class TestRenderRoutes(unittest.TestCase):
|
|||||||
self.assertEqual(("token_patterns",), addon_routes[0].outbound_detectors)
|
self.assertEqual(("token_patterns",), addon_routes[0].outbound_detectors)
|
||||||
self.assertEqual((), addon_routes[0].inbound_detectors)
|
self.assertEqual((), addon_routes[0].inbound_detectors)
|
||||||
|
|
||||||
|
def test_outbound_on_match_round_trips(self):
|
||||||
|
from bot_bottle.egress_addon_core import load_routes
|
||||||
|
b = _bottle([{"host": "logs.example", "dlp": {
|
||||||
|
"outbound_on_match": "redact",
|
||||||
|
}}])
|
||||||
|
routes = egress_routes_for_bottle(b)
|
||||||
|
rendered = egress_render_routes(routes)
|
||||||
|
self.assertIn('outbound_on_match: "redact"', rendered)
|
||||||
|
addon_routes = load_routes(rendered)
|
||||||
|
self.assertEqual("redact", addon_routes[0].outbound_on_match)
|
||||||
|
|
||||||
|
def test_outbound_on_match_default_omitted_from_render(self):
|
||||||
|
b = _bottle([{"host": "x.example"}])
|
||||||
|
routes = egress_routes_for_bottle(b)
|
||||||
|
rendered = egress_render_routes(routes)
|
||||||
|
self.assertNotIn("outbound_on_match", rendered)
|
||||||
|
|
||||||
def test_git_fetch_policy_round_trips(self):
|
def test_git_fetch_policy_round_trips(self):
|
||||||
from bot_bottle.egress_addon_core import load_routes
|
from bot_bottle.egress_addon_core import load_routes
|
||||||
b = _bottle([{"host": "github.com", "git": {"fetch": True}}])
|
b = _bottle([{"host": "github.com", "git": {"fetch": True}}])
|
||||||
|
|||||||
@@ -269,6 +269,25 @@ class TestParseDlp(unittest.TestCase):
|
|||||||
"dlp": {"wat": True},
|
"dlp": {"wat": True},
|
||||||
}]})
|
}]})
|
||||||
|
|
||||||
|
def test_outbound_on_match_default_empty(self):
|
||||||
|
routes = parse_routes({"routes": [{"host": "x.example"}]})
|
||||||
|
self.assertEqual("", routes[0].outbound_on_match)
|
||||||
|
|
||||||
|
def test_outbound_on_match_parsed(self):
|
||||||
|
for policy in ("block", "redact", "supervise"):
|
||||||
|
routes = parse_routes({"routes": [{
|
||||||
|
"host": "x.example",
|
||||||
|
"dlp": {"outbound_on_match": policy},
|
||||||
|
}]})
|
||||||
|
self.assertEqual(policy, routes[0].outbound_on_match)
|
||||||
|
|
||||||
|
def test_outbound_on_match_invalid_rejected(self):
|
||||||
|
with self.assertRaises(ValueError):
|
||||||
|
parse_routes({"routes": [{
|
||||||
|
"host": "x.example",
|
||||||
|
"dlp": {"outbound_on_match": "nope"},
|
||||||
|
}]})
|
||||||
|
|
||||||
|
|
||||||
# --- load_routes ---------------------------------------------------------
|
# --- load_routes ---------------------------------------------------------
|
||||||
|
|
||||||
|
|||||||
@@ -302,6 +302,24 @@ class TestDlp(unittest.TestCase):
|
|||||||
"bogus": True,
|
"bogus": True,
|
||||||
}}])
|
}}])
|
||||||
|
|
||||||
|
def test_outbound_on_match_omitted_is_empty(self):
|
||||||
|
b = _bottle([{"host": "x.example"}])
|
||||||
|
self.assertEqual("", b.egress.routes[0].OutboundOnMatch)
|
||||||
|
|
||||||
|
def test_outbound_on_match_accepts_policies(self):
|
||||||
|
for policy in ("block", "redact", "supervise"):
|
||||||
|
with self.subTest(policy=policy):
|
||||||
|
b = _bottle([{"host": "x.example", "dlp": {
|
||||||
|
"outbound_on_match": policy,
|
||||||
|
}}])
|
||||||
|
self.assertEqual(policy, b.egress.routes[0].OutboundOnMatch)
|
||||||
|
|
||||||
|
def test_outbound_on_match_rejects_unknown_value(self):
|
||||||
|
with self.assertRaises(ManifestError):
|
||||||
|
_bottle([{"host": "x.example", "dlp": {
|
||||||
|
"outbound_on_match": "allow",
|
||||||
|
}}])
|
||||||
|
|
||||||
|
|
||||||
class TestGitPolicy(unittest.TestCase):
|
class TestGitPolicy(unittest.TestCase):
|
||||||
def test_omitted_means_https_git_fetch_disabled(self):
|
def test_omitted_means_https_git_fetch_disabled(self):
|
||||||
|
|||||||
Reference in New Issue
Block a user