bot-bottle/docs/prds/prd-new-extended-outbound-scan.md

PRD prd-new: Extended outbound DLP scan surfaces

• Status: Active
• Author: claude
• Created: 2026-06-06
• Issue: #204

Summary

The outbound DLP scan introduced in PRD 0052 covers only the request body and the Authorization header. An agent can exfiltrate a provisioned secret or credential token via several other request surfaces that are currently unscanned: arbitrary HTTP headers, URL query parameters, URL path segments, and DNS-level hostnames (DNS tunnelling). This PRD closes those gaps by extending the scan corpus to cover all four surfaces.

Problem

After PRD 0052, egress_addon.py builds the DLP scan text as:

auth_header = flow.request.headers.get("authorization", "")
scan_text = body
if auth_header:
    scan_text = auth_header + "\n" + body

This leaves the following attack surfaces unscanned:

Surface	Example exfiltration vector
Other request headers	X-Api-Key: <secret>, Cookie: token=<secret>
Query parameters	GET /search?api_key=<secret>
URL path segments	GET /proxy/<base64-secret>/endpoint
DNS hostname	<base64-secret>.attacker.com (DNS tunnelling)

Each surface allows a determined agent to bypass the current scan and leak a provisioned EGRESS_TOKEN_* value or a known token pattern to an upstream attacker.

Goals / Success Criteria

1. All four surfaces (headers, query params, path, hostname) are included in the outbound DLP scan text for every route that has outbound scanning enabled.
2. A pure helper build_outbound_scan_text(host, path, query, headers, body) in egress_addon_core.py assembles the scan corpus so the logic is fully unit-testable without a mitmproxy dependency.
3. Unit tests demonstrate that scan_outbound blocks a request when a known token pattern or provisioned secret appears in each surface independently.
4. No manifest schema changes — the dlp block's outbound_detectors field continues to control which detectors run; all surfaces are scanned by whichever detectors are active.
5. The auth-strip ordering invariant from PRD 0052 is preserved: the outbound scan sees the original Authorization header before the addon strips it.

Non-goals

• Raw UDP/DNS queries — these bypass the HTTP proxy entirely and require a network-level DNS sinkhole (tracked separately in issue #205).
• Structured query-param parsing — scanning the raw query string is sufficient.
• Changes to the dlp block schema or detector names.
• Scanning outbound request bodies for prompt injection (inbound only, per PRD 0052 design).
• LLM-based semantic detection or entropy-based secret scanning (deferred, per PRD 0052 non-goals).

Design

build_outbound_scan_text in egress_addon_core.py

A new pure function assembles all request surfaces into a single newline- delimited string suitable for passing to scan_outbound:

def build_outbound_scan_text(
    host: str,
    path: str,
    query: str,
    headers: typing.Mapping[str, str],
    body: str,
) -> str:
    parts: list[str] = [host, path]
    if query:
        parts.append(query)
    for name, value in headers.items():
        parts.append(f"{name}: {value}")
    if body:
        parts.append(body)
    return "\n".join(parts)

Why hostname in the scan corpus?
DNS tunnelling encodes data into subdomain labels (<base64-secret>.attacker.com). The mitmproxy request hook sees the pretty_host field before the TCP connection is fully established, so scanning it catches this vector. Both the token_patterns and known_secrets detectors handle encoded variants (raw, base64, URL-encoded, hex), so the existing encoding-variant logic in _encoded_variants already covers common DNS-tunnelling encodings.

egress_addon.py update

The narrow scan-text construction is replaced with a call to build_outbound_scan_text, which the addon has already split path and query from flow.request.path at the top of request():

# Build full scan corpus: hostname + path + query + all headers + body
body = flow.request.get_text(strict=False) or ""
scan_text = build_outbound_scan_text(
    flow.request.pretty_host,
    request_path,
    query,
    dict(flow.request.headers),
    body,
)
dlp_result = scan_outbound(route, scan_text, os.environ)

The Authorization header is present in flow.request.headers at this point (the strip happens below on line 115), so the auth-strip ordering invariant is automatically preserved.

build_inbound_scan_text in egress_addon_core.py

An analogous helper assembles the inbound response corpus (all response headers + body) for scan_inbound. The response() hook now passes this combined text instead of the body alone, closing the response-header injection vector.

WebSocket frame scanning

A new websocket_message hook in EgressAddon scans every frame after the HTTP 101 upgrade. Outbound frames (from_client=True) are scanned for credential patterns and known secrets; inbound frames are scanned for prompt injection. On a block the entire WebSocket connection is killed via flow.kill() (there is no HTTP response surface to write to after upgrade).

Extended encoding variants in _encoded_variants

_encoded_variants is extended from 4 to 9 encoding forms:

Added encoding	Rationale
Standard base64 without padding	Common in log lines where = is stripped
URL-safe base64 with padding	JWT / OAuth standard alphabet
URL-safe base64 without padding	Same, padding stripped
Hex uppercase	Complements existing hex-lowercase variant
Base32	TOTP seeds; some DNS-exfil channels use base32 subdomains
gzip + base64	Recognisable by H4sI prefix; naive compression before encode

OpenAI project key pattern

TOKEN_PATTERNS gains sk-proj-[A-Za-z0-9_\-]{48,} covering OpenAI's newer project-scoped API key format.

Implementation

Delivered across three commits on the same branch:

1. Outbound scan surfaces — build_outbound_scan_text, egress_addon.py request() rewrite, TestBuildOutboundScanText, TestScanOutbound.
2. Remaining gaps — extended _encoded_variants, sk-proj- pattern, build_inbound_scan_text, response-header scanning, websocket_message hook, and matching unit tests.
3. PRD flip — Status: Draft → Active (committed with the first implementation commit; updated here to reflect final scope).