bot-bottle/docs/prds/0017-egress-proxy-via-mitmproxy.md

# PRD 0017: Egress-proxy — universal MITM with path filtering + auth injection

- **Status:** Active
- **Author:** didericis
- **Created:** 2026-05-25
- **Supersedes:** the cred-proxy sidecar (PRD 0010) — hard cutover.

## Summary

Replace the per-bottle cred-proxy sidecar with a new `egress-proxy`
sidecar built on mitmproxy. The egress-proxy is the agent's
`HTTP_PROXY` / `HTTPS_PROXY` — every agent HTTP/HTTPS request flows
through it before reaching pipelock. It owns three jobs that today
are split between cred-proxy and pipelock:

1. **MITM the agent's HTTPS.** Uses the per-bottle CA today held by
   pipelock; that key moves to the egress-proxy.
2. **Path-level allow/deny.** Manifest-declared `path_allowlist`
   per route. Universal coverage — any HTTPS path the agent reaches
   for is inspected here, not just traffic that voluntarily dials
   the cred-proxy URL.
3. **Credential injection.** Continues cred-proxy's existing role:
   match by hostname (or hostname + path), strip inbound
   Authorization, inject one based on the route's optional `auth:
   { scheme, token_ref }` block.

Pipelock's role narrows to hostname allowlist + DLP body scanning
on the egress-proxy → upstream leg. Pipelock no longer holds the
CA private key; no longer the agent's direct proxy.

## Problem

PR #25's pipelock-block flow exposed an honest gap: pipelock's
`api_allowlist` is hostname-only (verified by probing the binary's
strict preset and the `pipelock check --url` output). Approving a
proposed `pipelock-block` opens the entire host, not the URL's
path. For shared platforms (github.com, gitlab.com, public
registries) operators routinely want narrower-than-host granularity
— allow github.com/didericis but block github.com/somebody-else.

Cred-proxy already does path-prefix routing for credentialed APIs,
but it only sees the requests the agent voluntarily routes to it
(via `ANTHROPIC_BASE_URL`, `~/.gitconfig` insteadOf, npmrc
`registry=`). A raw `curl https://github.com/anyone` from the agent
goes to `HTTPS_PROXY=pipelock` directly and bypasses cred-proxy
entirely. So extending cred-proxy with `path_allowlist` (the earlier
PRD 0017 draft) buys *opt-in* path filtering, not enforcement.

For enforcement we need a layer that sits on the agent's
`HTTPS_PROXY` path — universal coverage of agent egress.

## Goals / Success Criteria

A bottle manifest declares an egress-proxy route with a
`path_allowlist`. From inside the bottle, `curl
https://github.com/didericis/foo` succeeds; `curl
https://github.com/somebody-else/secret` gets a 403 from
egress-proxy, never reaches pipelock or the real github. The same
holds for any tool inside the bottle that respects
`HTTPS_PROXY` — claude-code, git over HTTPS, npm, raw curl, random
Python `requests`. No tool-specific rewrite is required for path
enforcement.

Existing cred-proxy responsibilities continue to work after the
cutover: Anthropic OAuth injection for claude-code (via the
proxy-side header injection rather than the dotfile rewrite),
git-insteadof routing into the proxy stays useful for hostname
canonicalisation but is no longer load-bearing for credential
delivery.

## Non-goals

- Replacing pipelock. Pipelock keeps doing hostname allowlist +
  DLP body scanning on the egress-proxy → upstream leg.
- Building our own MITM stack. mitmproxy already does it; we ship
  addons.
- Backward compatibility with `bottle.cred_proxy.routes[]`. Hard
  cutover (see Migration).
- Path-level rules in pipelock. Upstream feature request is a
  separate track (file independently); this PRD doesn't depend on
  it.

## Scope

### In scope

- A new `egress-proxy` sidecar replacing the cred-proxy sidecar.
  mitmproxy image, pinned by digest. Addons in Python.
- Per-bottle CA generation **moves from pipelock to egress-proxy**.
  The agent's trust store is rebuilt against the egress-proxy CA
  (was pipelock's CA).
- Manifest rename: `bottle.cred_proxy.routes[]` →
  `bottle.egress_proxy.routes[]`. The route shape gains optional
  `path_allowlist: [<prefix>, ...]` and a nested optional `auth:
  { scheme, token_ref }` block (presence/absence of `auth` is the
  authenticated vs unauthenticated signal — replaces the old
  `auth_scheme: "none"` pattern).
- Agent's `HTTP_PROXY` / `HTTPS_PROXY` env vars repointed at the
  egress-proxy (was pipelock).
- Pipelock retains its sidecar slot and its own DLP + hostname
  scanner. The agent never dials it directly anymore; egress-proxy
  uses `HTTPS_PROXY=pipelock` for its outbound leg, matching the
  current cred-proxy → pipelock pattern.
- Existing PRDs that depend on cred-proxy:
  - PRD 0014 (cred-proxy-block remediation) → renames + retargets
    apply path. SIGHUP reload semantics carry over to egress-proxy.
  - PRD 0013 (supervise plane) `cred-proxy-block` MCP tool stays;
    its proposed file format updates per the new route shape.
- Removal of the old cred-proxy code: `bot_bottle/cred_proxy.py`,
  `cred_proxy_server.py`, `backend/docker/cred_proxy.py`,
  `provision/cred_proxy.py`, the `Dockerfile.cred-proxy`. Tests
  updated.

### Out of scope

- Pipelock CA path: pipelock keeps generating its *own* CA for
  any internal TLS termination it still does (e.g., on the
  egress-proxy → upstream leg if pipelock is the MITM there).
  Whether pipelock needs that CA at all post-cutover is an open
  question (probably no — egress-proxy already terminated; pipelock
  is now downstream of a plain-HTTP forward from egress-proxy).
- Glob / regex matching in `path_allowlist`. v1 ships prefix
  matching; expressive forms are a follow-up.
- An MCP tool for the agent to propose `path_allowlist`
  additions. Today the operator manages this via the manifest +
  the existing `routes edit <bottle>` TUI verb (renamed to
  `egress-proxy edit <bottle>`).

## Proposed design

### Topology

```
[Agent] --HTTP_PROXY=egress-proxy-->
           [egress-proxy (mitmproxy)]
              MITM with per-bottle CA
              path_allowlist enforcement
              Authorization header injection
            --HTTPS_PROXY=pipelock-->
                  [pipelock]
                    hostname allowlist
                    DLP body scan
                  --egress-->  Internet
```

Universal coverage: every HTTP/HTTPS request the agent makes hits
egress-proxy first. cred-proxy's URL convention
(`http://cred-proxy:9099/...`) goes away — there's no need for the
agent to address the proxy by name because it's already on the
default proxy path.

### Manifest

```yaml
egress_proxy:
  routes:
    # Authenticated route — `auth` block carries the injection
    # config. path_allowlist optional.
    - host: "api.github.com"
      auth:
        scheme: "Bearer"
        token_ref: "GH_PAT"
      path_allowlist:
        - "/repos/didericis/"
        - "/users/didericis"
    # Unauthenticated path-filtered route — `auth` omitted
    # entirely (presence/absence of the key is the auth signal).
    - host: "github.com"
      path_allowlist:
        - "/didericis/"
    # Bare-pass route: no auth, no path constraint. Useful when
    # you want a host to skip path filtering but still be
    # DLP-scanned by pipelock on the outbound leg.
    - host: "api.anthropic.com"
```

Route matching is on `host` (was `path` prefix). The hostname
gates whether a route applies; `path_allowlist` (if present)
constrains the URL path under that host. The optional `auth`
block carries credential-injection config:

- Omit `auth` → no Authorization header injected (replaces the
  earlier draft's `auth_scheme: "none"`).
- `auth.scheme` → one of `Bearer`, `token` (the values
  cred-proxy supports today; sidesteps the gitea-token quirk).
- `auth.token_ref` → host env var holding the secret. Same
  semantics as cred-proxy's `TokenRef` field today.

Validation: `auth` (if present) must contain both `scheme` and
`token_ref`. An empty `auth: {}` is an error rather than a
synonym for "no auth" — that's what omission is for.

### mitmproxy addon shape

The egress-proxy ships a small Python addon that:

- Loads the per-bottle routes from `/etc/egress-proxy/routes.yaml`
  (rendered by the prepare step, docker-cp'd in like cred-proxy's
  current routes.json).
- On `request` hook: match `flow.request.host` → route. If no route
  matches → forward unchanged (pipelock will hostname-gate it). If
  route matches and has `path_allowlist`, check `flow.request.path`
  against the prefix list; 403 with a clear reason if no match.
- On approved requests: strip inbound Authorization. If the route
  carries an `auth` block, inject `Authorization: <auth.scheme>
  <token-from-env-named-by-auth.token_ref>`. If the route omits
  `auth`, leave Authorization unset.
- SIGHUP / file-mtime watch on `routes.yaml` for hot-reload (same
  cadence as today's cred-proxy SIGHUP path).

mitmproxy's standard CA generation handles per-host leaf certs at
SNI time. The per-bottle CA is generated at bottle launch (was
pipelock's tls-init step; now egress-proxy's). Agent's trust store
gets the egress-proxy CA installed in place of pipelock's.

### Trust-domain concentration

The egress-proxy now holds:

- Every credential the bottle declared in `egress_proxy.routes[]`
  (OAuth tokens, PATs, npm tokens).
- The per-bottle MITM CA private key.

This is a deliberate concentration. With the previous split:

- cred-proxy held tokens.
- pipelock held the CA.

A memory disclosure in cred-proxy exposed tokens; in pipelock,
the CA. Both were bad; neither exposed everything.

The new egress-proxy in one disclosure exposes both. Mitigations:

- mitmproxy runs as an unprivileged user inside the container.
- Tokens live in the container's environ (same as cred-proxy today).
  The CA private key is mounted from the host's stage_dir (mode 600).
- Pipelock stays as a separate sidecar, so a compromise of
  egress-proxy doesn't disable pipelock's hostname check + DLP on
  the outbound leg — the attacker can forge certs to the agent but
  can't easily exfil from inside the agent without pipelock
  noticing.

The user (per PR #25 discussion) accepted this concentration in
exchange for the one-sidecar consolidation. The PRD records it
explicitly.

### Migration — hard cutover

No backward-compat alias for `bottle.cred_proxy.routes[]`. At
manifest load:

- `cred_proxy:` block → `die()` with a clear pointer at this PRD
  and a migration recipe (rename to `egress_proxy:`, rename
  `path` → `host`, drop the agent-side URL prefix).
- `cred_proxy_routes` field on existing dataclasses removed.
- `Dockerfile.cred-proxy` deleted.
- `bot_bottle/cred_proxy*.py` deleted.
- `bot_bottle/backend/docker/cred_proxy*.py` consolidated into
  `egress_proxy*.py`.
- Provisioner files renamed.
- PRDs 0010 (cred-proxy), 0014 (cred-proxy-block remediation)
  retroactively annotated as "superseded by 0017" — old text
  preserved, header updated.

### Implementation chunks

Plausibly three implementation PRs after this PRD lands:

1. **egress-proxy sidecar core.** Dockerfile + mitmproxy addon +
   `routes.yaml` schema + lifecycle (prepare / start / stop / SIGHUP).
2. **Manifest + provisioner migration.** Rename cred-proxy
   throughout the codebase, hard-fail on legacy manifests, update
   agent CA trust to point at egress-proxy.
3. **PRD 0014 retargeting.** cred-proxy-block remediation's apply
   path repointed at egress-proxy (SIGHUP, audit log, etc.).
   Supervise tool description updated.

## Open questions

- **mitmproxy addon distribution.** Mount the addon Python file
  from stage_dir, or bake it into the image. Mount is more
  hot-reloadable; bake-in is more reproducible. Recommend bake-in,
  with routes.yaml as the only mounted state.
- **Path match semantics.** Prefix-only for v1 (matches PRD 0017
  v1 spirit). Globs / regex are a follow-up if operators ask.
- **Mode for the `Authorization` strip on inbound.** Pipelock has a
  similar strip in `sensitive_headers`. Confirm there's no
  double-strip causing a real header the agent set to disappear
  unexpectedly. Probably want egress-proxy to be the only stripper
  for routes that match.
- **Pipelock's TLS interception post-cutover.** Today pipelock
  MITMs the cred-proxy → upstream leg using its own CA. After the
  cutover, that leg starts as a CONNECT tunnel from egress-proxy
  (egress-proxy treats pipelock as a plain HTTPS forward proxy).
  Does pipelock still need to MITM? Probably no — egress-proxy
  already terminated, body content is already inspected upstream
  by egress-proxy's addons (or could be). But that means moving
  DLP from pipelock to egress-proxy, which expands egress-proxy's
  trust-domain *further*. Punted to the implementation PR to
  decide.
- **Performance.** Two MITM hops in the worst case (agent ↔
  egress-proxy and pipelock ↔ upstream if pipelock keeps its
  interception). Measure under realistic load; if it's a problem,
  the answer is probably to disable pipelock's TLS interception
  and let it operate at hostname-only.
- **Agent's existing dotfile rewrites.** Today cred-proxy
  provisions ~/.npmrc with `registry=http://cred-proxy:9099/npm/`,
  ~/.gitconfig with `insteadOf` rules, etc. After the cutover
  none of those rewrites are strictly necessary for routing
  (HTTPS_PROXY catches everything), but they may still be useful
  for canonicalisation (so the agent's `npm install` doesn't
  surprise itself by talking to a different registry). Decide per
  dotfile in the migration PR.

## References

- PRD 0010 — cred-proxy (superseded by this PRD).
- PRD 0014 — cred-proxy-block remediation (retargeted).
- PRD 0013 — supervise plane (tool descriptions updated).
- PR #25 — the supervise loop, whose `_apply_pipelock_url`
  docstring flagged the original "path filtering belongs
  somewhere" follow-up.
- mitmproxy — https://mitmproxy.org/ — chosen as the egress-proxy
  engine because it's the canonical scriptable MITM forward proxy.