PRD: Egress traffic logging #207

Merged

didericis merged 7 commits from feat/egress-log-option into main 2026-06-07 20:32:47 -04:00

Conversation 0 Commits 7 Files Changed 10

+668 -94

didericis-claude commented 2026-06-06 14:00:39 -04:00

Collaborator

Copy Link

Copy Source

Closes #203.

PRD 0056

Summary

• Adds egress.log (integer 0/1/2) as a global log level for the egress proxy
• Level 0 (off, default): no output beyond the boot line — no behaviour change for existing deployments
• Level 1 (blocks): every block and DLP warn emitted as a JSON line with reason (including secret type for DLP hits) and request context (host, method, path; + response status for inbound events)
• Level 2 (full): level-1 events + egress_request / egress_response JSON lines for every forwarded connection (headers, body after auth injection)
• Config dataclass in egress_addon_core wraps routes + log; parse_config() / load_config() are the new entry points (old load_routes kept for backward compat)
• Boot message shows log level label: [log=off], [log=blocks], [log=full]
• 22 new tests across test_egress_addon_core, test_egress, and test_manifest_egress

Closes #203. [PRD 0056](https://gitea.dideric.is/didericis/bot-bottle/src/commit/57514f515102ca88d1aa6f08f54168d9d72c5ba4/docs/prds/0056-egress-traffic-logging.md) ## Summary - Adds `egress.log` (integer 0/1/2) as a global log level for the egress proxy - **Level 0** (off, default): no output beyond the boot line — no behaviour change for existing deployments - **Level 1** (blocks): every block and DLP warn emitted as a JSON line with reason (including secret type for DLP hits) and request context (host, method, path; + response status for inbound events) - **Level 2** (full): level-1 events + `egress_request` / `egress_response` JSON lines for every forwarded connection (headers, body after auth injection) - `Config` dataclass in `egress_addon_core` wraps `routes + log`; `parse_config()` / `load_config()` are the new entry points (old `load_routes` kept for backward compat) - Boot message shows log level label: `[log=off]`, `[log=blocks]`, `[log=full]` - 22 new tests across `test_egress_addon_core`, `test_egress`, and `test_manifest_egress`

didericis-claude changed title from feat(egress): add global log option for full request/response logging to PRD 0056: Egress traffic logging 2026-06-06 16:49:09 -04:00

didericis-claude changed title from PRD 0056: Egress traffic logging to PRD: Egress traffic logging 2026-06-06 22:10:41 -04:00

didericis added 5 commits 2026-06-07 14:41:31 -04:00

feat(egress): add global log option for full request/response logging ... 76dd153760

Adds a top-level `log: true` option to the egress config that logs the
full request (method, path, headers, body) and response (status, headers,
body) for every forwarded connection as JSON lines on stderr.

Wire format: `log: true` at the root of routes.yaml, parsed into the new
`Config` dataclass alongside `routes`. The sidecar addon switches from
`self.routes` to `self.config` and writes `_log_request` / `_log_response`
JSON lines when `self.config.log` is set.

Manifest: `egress.log: true` in bottle YAML flows through `EgressConfig.Log`
→ `Egress.prepare()` → `egress_render_routes(..., log=)` → routes.yaml.
`EgressPlan` also carries the flag for introspection.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

feat(egress): replace log bool with integer log levels (0/1/2) ... 79212481c9

Level 0 (off, default): no stderr output beyond boot line.
Level 1 (blocks): each block/warn emitted as JSON with reason and
request context (host, method, path, response_status for inbound).
Level 2 (full): level-1 events + egress_request and egress_response
JSON lines for every forwarded connection.

Block logging at level 1+ replaces the previous plain-text stderr write.
DLP warn logging is also gated on level 1+. All block call sites now pass
_req_ctx(flow) so the blocked request is visible in the log entry.
Boot message shows log level label (off/blocks/full).

Adds PRD 0053 documenting wire format, manifest format, and all log event
shapes.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

feat(egress): add location, context snippets, and token redaction to DLP logging ... 86b0a4d285

Each DLP block/warn now reports where the match was found (body,
authorization header, response body) and includes a context snippet:
SNIPPET_CONTEXT chars before and after the match, with the matched
value replaced by REDACT ("********").

scan_token_patterns/scan_known_secrets/scan_naive_injection all gain
`location` and `context` fields on their ScanResult returns. The
outbound scanner takes `auth_header` as a separate kwarg so the two
locations are scanned and reported independently.

redact_tokens() is added to dlp_detectors and used in egress_addon.py
to scrub token patterns and provisioned secrets from host/path fields
before they appear in any log output (level 1 and 2).

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

docs(prd): renumber PRD 0053 → 0056 (0053 slot claimed by user-provider-plugins) 205e94f960

ci(prd): rename PRD to prd-new placeholder per new convention 8743299226

didericis force-pushed feat/egress-log-option from 5091552625 to 8743299226 2026-06-07 14:41:31 -04:00 Compare

didericis added 1 commit 2026-06-07 20:10:37 -04:00

fix(lint): resolve pylint and pyright issues on egress-log-option ... 545ff3582f

- egress.py: extract _render_match_entry helper to reduce nesting depth
- egress_addon_core.py: make request_method/request_headers keyword-only
  to satisfy too-many-positional-arguments; wrap long lazy import lines
- egress_addon.py: remove unused Route import; add pylint disable for
  import-error on sidecar-only mitmproxy/egress_addon_core imports
- dlp_detectors.py: remove dead _min_distance function (superseded by
  _closest_pair)

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

didericis added 1 commit 2026-06-07 20:26:04 -04:00

fix(lint): add parse_config tests to satisfy pyright unused-import ... 55cb3429d4

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

didericis merged commit 55cb3429d4 into main 2026-06-07 20:32:47 -04:00

didericis deleted branch feat/egress-log-option 2026-06-07 20:32:47 -04:00

Sign in to join this conversation.

Reviewers

No Reviewers

Labels

Clear labels

Compat/Breaking

Breaking change that won't be backward compatible

Kind/Bug

Something is not working

Kind/Documentation

Documentation changes

Kind/Enhancement

Improve existing functionality

Kind/Feature

New functionality

Kind/Security

This is security issue

Kind/Testing

Issue or pull request related to testing

Priority

Critical

1

The priority is critical

Priority

High

2

The priority is high

Priority

Low

4

The priority is low

Priority

Medium

3

The priority is medium

Reviewed

Confirmed

1

Issue has been confirmed

Reviewed

Duplicate

2

This issue or pull request already exists

Reviewed

Invalid

3

Invalid issue

Reviewed

Won't Fix

3

This issue won't be fixed

Status

Abandoned

3

Somebody has started to work on this but abandoned work

Status

Blocked

1

Something is blocking this issue or pull request

Status

Need More Info

2

Feedback is required to reproduce issue or to continue work

No Label

Milestone

No items

No Milestone

Projects

Clear projects

No project

Assignees

Clear assignees

didericis-claude (didericis (claude)) didericis-codex (didericis (codex)) didericis-gemma (gemma) didericis

No Assignees

2 Participants

Notifications

Subscribe

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: didericis/bot-bottle#207