didericis/bot-bottle

PRD 0005: mitmproxy TLS interception for pipelock content scanning #8

Closed

didericis wants to merge 6 commits from mitmproxy-tls-interception into main

Author	SHA1	Message	Date
didericis	22bc13dc3c	feat(mitmproxy): integration tests for the bumped HTTPS path test / unit (pull_request) Successful in 20s Details test / integration (pull_request) Successful in 15s Details Fourth and final step of PRD 0005. Two new end-to-end tests that exercise the full chain agent -> mitmproxy(bump) -> addon -> pipelock -> upstream and pin the two paths the addon implements. - test_mitmproxy_blocks_secret_https_post: HTTPS variant of the existing test_pipelock_blocks_secret_post. Posts a credential pattern in the body over HTTPS through the bottle. mitmproxy bumps the CONNECT (the agent trusts the per-bottle ephemeral CA installed by provision_ca), the addon forwards the decrypted request to pipelock, pipelock returns 403 with the known `blocked: ...` body shape, and the addon short-circuits the flow with status=403 + X-Pipelock-Bridge: block. The two-axis assertion (status + header) proves the addon-mediated path is what produced the block, not some other layer. - test_mitmproxy_allows_normal_https: hits raw.githubusercontent.com (a baked-in allowlist host) over HTTPS through the bottle. Verifies the addon's allow path: mitmproxy bumps, addon forwards to pipelock for the scan, pipelock allows, mitmproxy proceeds to the real upstream, response comes back through. The absence of X-Pipelock-Bridge on the response is the signal that the addon didn't short-circuit. Body length sanity-checks that the response is real upstream content, not a synthesized stub. Both probes are stdlib-only Node (http.request CONNECT + tls.connect on the tunneled socket) — pulling in undici as a dep would be the clean way to do HTTPS-through-proxy but is out of scope. The earlier integration tests still pass with mitmproxy in path: their assertions hold under the new topology, though their semantic coverage shifts (e.g. test_pipelock_allow_node now exercises mitmproxy's CONNECT-200 path rather than pipelock's host allowlist on CONNECT). Updating those tests is a follow-up. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>	2026-05-12 13:46:09 -04:00
didericis	c4de42ea3c	feat(mitmproxy): render mitmproxy in the dry-run preflight Third step of PRD 0005. The preflight now surfaces the TLS- intercept layer so the operator sees it before agreeing to launch. - Text output: one new line under the egress summary — "tls intercept : mitmproxy (per-bottle ephemeral CA, generated at launch)". - JSON output (--format=json contract): new egress.mitm: { enabled: true, ca_fingerprint: null } block. Fingerprint is always null at dry-run because the CA only exists after the sidecar starts; real launches print it as a stderr log line from provision_ca. - Pin the new shape in the dry-run integration test. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>	2026-05-12 13:40:31 -04:00
didericis	21054212d4	feat(mitmproxy): wire the sidecar into the bottle launch lifecycle Second step of PRD 0005. The mitmproxy sidecar from the previous commit now actually runs alongside pipelock when a bottle launches. - BottleBackend gains a non-abstract provision_ca with a default no-op so non-Docker backends aren't forced to implement TLS interception. provision() orchestrates ca → prompt → skills → ssh → git; CA goes first so trust is set up before anything else runs inside the agent. - DockerBottlePlan gains `mitmproxy_plan: MitmproxyProxyPlan`. The prepare step builds it alongside the existing pipelock plan; no new manifest schema or host-side scratch files. - DockerBottleBackend grows self._mitm, threads it through prepare and launch. Mirror of the existing self._proxy pattern. - launch.py brings the mitmproxy sidecar up between pipelock and the agent container, passing pipelock's service-name URL via env. ExitStack callback handles teardown in reverse order. - The agent's HTTPS_PROXY / HTTP_PROXY now point at mitmproxy (not pipelock directly). Three new -e flags inject the CA trust trio (NODE_EXTRA_CA_CERTS / SSL_CERT_FILE / REQUESTS_CA_BUNDLE) at docker run time; Docker propagates those into docker exec so the claude process sees them without per-exec threading. - New provisioner backend/docker/provision/ca.py extracts the CA cert from the running mitmproxy sidecar, copies it into the agent at /usr/local/share/ca-certificates/claude-bottle-mitm.crt, runs update-ca-certificates, and emits a stderr line with the SHA-256 fingerprint (stdlib ssl + hashlib; no subprocess). Cleanup needs no change — `docker ps --filter name=^claude-bottle-` already catches the new claude-bottle-mitm-<slug> containers. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>	2026-05-12 13:38:51 -04:00
didericis	e579c3d4fd	feat(mitmproxy): vendor the addon and Docker sidecar lifecycle First step of PRD 0005. Three new files for the mitmproxy-in-front-of-pipelock topology — wiring into the bottle launch comes in the next commit. - claude_bottle/mitmproxy/__init__.py: abstract MitmproxyProxy base + MitmproxyProxyPlan. Mirrors the PipelockProxy shape (prepare / start / stop) and adds extract_ca_cert for the CA cert hand-off into the agent. - claude_bottle/mitmproxy/addon.py: the vendored Python addon mitmproxy loads inside the sidecar. Forwards each decrypted request to pipelock as a plain HTTP forward-proxy call, inspects the response, and short-circuits the flow with 403 on a pipelock block (status=403 + body starts with `blocked: `, pinned empirically against pipelock 2.3.0 in the impl spike). Self-contained — no claude_bottle imports — so it loads in a sidecar that doesn't have claude_bottle on its path. - claude_bottle/backend/docker/mitmproxy.py: DockerMitmproxyProxy with create / cp / network connect / start lifecycle. Pinned to mitmproxy/mitmproxy@sha256:00b77b5d… (multi-arch manifest for v12.2.3). - tests/unit/test_mitmproxy_verdict.py: pins the verdict fingerprint so a pipelock-side body shape change breaks loudly. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>	2026-05-12 13:32:36 -04:00
didericis	c2eacac49f	docs(prd): update 0005 after open-question walkthrough test / unit (pull_request) Successful in 17s Details test / integration (pull_request) Successful in 15s Details Re-grounds the design after walking the eight original open questions interactively. Two structural changes: - Topology A → A'. A spike confirmed mitmproxy's `upstream` mode re-wraps decrypted flows in a new CONNECT to the upstream proxy, which would have left pipelock seeing only ciphertext (the very gap this PRD set out to close). The fix is to run mitmproxy in `regular` mode and ship a vendored Python addon that forwards each decrypted request to pipelock as a plain HTTP forward-proxy call. Pipelock is unchanged. - mitmproxy owns CA generation. The research note's preference for a host-side openssl / cryptography CA turned out to be unnecessary — mitmproxy generates a fresh CA on startup; the public cert is `docker cp`'d into the agent. No new host-side crypto deps. Dry-run can't render a fingerprint (CA doesn't exist yet); launches print it once to stderr. Other Q3–Q8 resolutions folded in: Debian-base `update-ca-certificates` confirmed, mitmproxy 12 verified to speak h2 on both halves, selective-bump deferred to v2, response-body and MCP scanning deferred to v2, domain-fronting deferred to v2. Open questions rewritten — what remains is addon-implementation specifics (pipelock 403-body fingerprint, env-var inheritance through docker exec, addon test fixtures). Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>	2026-05-12 12:54:27 -04:00
didericis	fe71249005	docs(prd): add 0005 mitmproxy TLS interception test / unit (pull_request) Successful in 10s Details test / integration (pull_request) Successful in 13s Details Captures the design for putting a mitmproxy sidecar in front of pipelock on the egress path so pipelock's body / header / MCP scanners see plaintext for the HTTPS hosts in the default allowlist. Implements Topology A from docs/research/tls-mitm-for-pipelock.md with a per-bottle ephemeral CA, no manifest schema change in v1, and selective-bumping deferred until a pinning host appears. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>	2026-05-12 12:20:24 -04:00