Audit: control-plane exposure on macOS consolidated backend + WebSocket DLP gap (security/architecture/quality) #400

New Issue

2026-07-17T04:43:57-04:00

didericis-claude commented

2026-07-17 04:43:57 -04:00

Full-repo audit covering security, architecture, and general quality. Read at the tip of main (dfc693e) with macos-consolidated-gateway (#399, 4a607ad) assumed merged, per the audit request — the shared consolidated gateway is treated as live on both the Firecracker and macOS backends.

Findings are ranked by severity. The headline is a control-plane exposure that is specific to the macOS consolidated backend and defeats the product's core guarantees; everything else is lower-stakes.

🔴 Critical

1. The orchestrator control plane is unauthenticated and, on the macOS consolidated backend, directly reachable by every agent

Where: bot_bottle/orchestrator/control_plane.py (no auth on any route), bot_bottle/backend/macos_container/infra.py (--host 0.0.0.0 --port 8099 in the shared infra container), bot_bottle/backend/macos_container/gateway.py / launch.py (agent and infra container share GATEWAY_NETWORK).

The invariant that's broken. registry.py::by_source_ip documents the precondition the whole design leans on: source-IP attribution is safe "only where the source IP is unspoofable and the control plane is reachable only by the trusted gateway." The control plane has no authentication of its own — dispatch() routes every request straight through; the only guard anywhere is the (source_ip, identity_token) check inside /resolve and /attribute. The mutating endpoints have no token check at all.

Why it's exposed on macOS. On Firecracker the boundary holds: the nft table in scripts/firecracker-netpool.sh DNATs agent traffic to the infra VM only on the gateway ports {9099, 9100, 9420} and drops everything else, so port 8099 (control plane) is unreachable from a bottle VM. On the macOS backend the infra container runs both the control plane (8099) and the gateway data plane (9099/9100/9420) in one container (infra.py::_init_script), and the agent sits on the same --internal Apple network (gateway.py::ensure_networks). container network create --internal gives no per-port filtering — containers on the same network reach each other on all ports. So every agent can reach http://<gateway_ip>:8099.

What an agent can do once it reaches 8099. The agent already knows its own identity token — it's handed to the agent process in HTTPS_PROXY=http://bottle:<token>@<gw>:9099 (launch.py::_identity_proxy_env) — and its own source IP from its NIC. From inside the bottle:

GET /bottles → enumerate every sibling bottle on the host, including each one's policy and metadata (BottleRecord.redacted() omits only the token). Cross-bottle disclosure.
PUT /bottles/<id>/policy → rewrite its own egress allowlist to allow-all (no token required, just a bottle_id from the list above). The gateway serves the new routes on the next /resolve. This defeats the per-bottle egress allowlist — the product's primary control.
POST /resolve with its own (source_ip, identity_token) → the response includes tokens, the real upstream credentials the gateway injects on the bottle's behalf (control_plane.py:188, orch.tokens_for). This directly defeats the README's "Tokens the agent never sees" guarantee — the agent reads them straight out.
POST /supervise/respond → self-approve its own pending supervise proposals (egress-allow, token-allow). The human approval gate becomes a formality the agent can click through for itself.
POST /bottles / DELETE /bottles/<id> → launch or tear down arbitrary bottles on the host, affecting other agents.

Net: on the macOS consolidated backend a prompt-injected or misbehaving agent can lift the egress allowlist, read the credentials it was never supposed to see, and rubber-stamp its own approvals — i.e. every guarantee in the README's feature list collapses to the agent's own discretion.

Fix direction (any one closes it; defense-in-depth wants more than one):

Authenticate the control plane. It's the real fix and it's backend-agnostic. A per-host shared secret the host CLI + gateway hold (env-injected, not on the agent network) gates all of /bottles*, /supervise/respond, and the tokens field of /resolve. /resolve and /attribute would still additionally require the caller's (source_ip, identity_token).
Separate the admin surface from the agent-facing surface so agents can't route to 8099 at all — bind the control plane to an interface the agent network doesn't share, or front the gateway data-plane ports on a proxy container distinct from the control-plane container (mirrors the docker two-container split the consolidation collapsed).
Scope the macOS network the way Firecracker's nft does — only expose 9099/9100/9420 to agents. Apple's container networking may not support this cleanly, which is exactly why the auth fix is the durable one.

Also verify: the docker consolidated backend. If its control-plane container sits on the agent's --internal network the same exposure applies there; only Firecracker's port-scoped nft demonstrably prevents it today.

🟠 High

2. WebSocket DLP is silently inert in the consolidated (multi-tenant) gateway

Where: bot_bottle/egress_addon.py::websocket_message (~L623–658).

Unlike request()/response(), which resolve the calling bottle's config per request via _resolve_flow, websocket_message scans against self.config and self._supervise_slug. In consolidated mode no per-bottle routes.yaml is written (the egress addon runs purely off BOT_BOTTLE_ORCHESTRATOR_URL), so self.config stays Config(routes=()). match_route therefore returns None and the handler returns without scanning or killing the frame. The code comment concedes this ("inert until websocket routing is made source-IP-aware").

Impact: for any host that made it past the CONNECT allowlist, WebSocket frames flow with no outbound credential DLP and no inbound prompt-injection DLP on the shared gateway — a live exfil channel for streaming agents. It fails open, and silently, which is the wrong direction for a DLP control. At minimum it should fail closed (kill WS frames when the addon can't resolve a per-flow config) until per-flow WS resolution lands.

🟡 Medium — architecture / maintainability

3. Two live tenancy code paths (single-tenant vs consolidated) across every gateway-facing module

egress_addon.py, git_http_backend.py, policy_resolver.py, and the supervise plumbing each carry a "legacy single-tenant" branch alongside the consolidated one, described as transitional. Every branch is security-relevant and has to be kept correct in parallel — finding #2 is exactly a case where one path (WS) never got the consolidated treatment. Recommend tracking the single-tenant removal as a real deprecation with a target, not an open-ended "until every backend runs consolidated."

4. Flat-file import-shim duplication for gateway-bundled modules

egress_addon_core, dlp_detectors, policy_resolver, git_http_backend, yaml_subset, egress_dlp_config are COPYed flat into the gateway image and every one uses a try: import X / except ImportError: from .X shim, plus hand-maintained duplicate constants (e.g. IDENTITY_HEADER, GIT_GATE_TIMEOUT_SECS copied rather than imported). A rename or a missed file in Dockerfile.gateway fails only at runtime inside the container, not in host tests or pyright. Worth a packaging step (or a single bundled subpackage) that makes the gateway bundle a build artifact of the real package rather than a parallel flat copy.

5. Custom YAML subset parser is a security-relevant parse surface

yaml_subset.py (685 lines) is a hand-rolled parser sitting on the manifest + egress-policy trust boundary. It's well unit-tested (597-line test file), but a bespoke parser in this position warrants fuzzing (malformed/adversarial policy blobs, deeply nested input, resource-exhaustion cases) rather than example-based tests alone — a parser bug here is a policy-bypass or DoS.

Positives worth recording

Attribution crypto is done right: 256-bit secrets.token_urlsafe tokens, hmac.compare_digest comparison, both-signals-required, fail-closed on ambiguity (registry.py::attribute).
The egress DLP detector set is genuinely thorough — encoded-variant expansion (base64/32/hex/gzip/url), NFKD unicode normalization against confusables, alnum-projection + sliding-window fragmentation resistance, and the _closest_pair O(n log n) rewrite explicitly to kill a quadratic DoS on attacker-controlled bodies.
Fail-closed is the consistent default across resolve_client_config, resolve_sandbox_root, and the gateway resolvers — unattributed/errored/unparseable all deny.
The Firecracker nft boundary is well-constructed: per-slot anti-spoof source-IP pinning, port-scoped DNAT, its own table at priority -10. It's precisely the model the macOS backend needs and lacks (finding #1).

Filed from an automated audit. The Critical is the actionable one and is contained to the macOS consolidated backend as written — Firecracker's port-scoped nft blocks the same attack today.

Full-repo audit covering **security**, **architecture**, and **general quality**. Read at the tip of `main` (`dfc693e`) *with `macos-consolidated-gateway` (#399, `4a607ad`) assumed merged*, per the audit request — the shared consolidated gateway is treated as live on both the Firecracker and macOS backends. Findings are ranked by severity. The headline is a control-plane exposure that is specific to the macOS consolidated backend and defeats the product's core guarantees; everything else is lower-stakes. --- ## 🔴 Critical ### 1. The orchestrator control plane is unauthenticated and, on the macOS consolidated backend, directly reachable by every agent **Where:** `bot_bottle/orchestrator/control_plane.py` (no auth on any route), `bot_bottle/backend/macos_container/infra.py` (`--host 0.0.0.0 --port 8099` in the shared infra container), `bot_bottle/backend/macos_container/gateway.py` / `launch.py` (agent and infra container share `GATEWAY_NETWORK`). **The invariant that's broken.** `registry.py::by_source_ip` documents the precondition the whole design leans on: source-IP attribution is safe *"only where the source IP is unspoofable **and the control plane is reachable only by the trusted gateway**."* The control plane has **no authentication of its own** — `dispatch()` routes every request straight through; the only guard anywhere is the `(source_ip, identity_token)` check inside `/resolve` and `/attribute`. The mutating endpoints have no token check at all. **Why it's exposed on macOS.** On Firecracker the boundary holds: the nft table in `scripts/firecracker-netpool.sh` DNATs agent traffic to the infra VM only on the gateway ports `{9099, 9100, 9420}` and drops everything else, so port **8099** (control plane) is unreachable from a bottle VM. On the macOS backend the infra container runs **both** the control plane (8099) and the gateway data plane (9099/9100/9420) in one container (`infra.py::_init_script`), and the agent sits on the same `--internal` Apple network (`gateway.py::ensure_networks`). `container network create --internal` gives no per-port filtering — containers on the same network reach each other on **all** ports. So every agent can reach `http://<gateway_ip>:8099`. **What an agent can do once it reaches 8099.** The agent already knows its own identity token — it's handed to the agent process in `HTTPS_PROXY=http://bottle:<token>@<gw>:9099` (`launch.py::_identity_proxy_env`) — and its own source IP from its NIC. From inside the bottle: - `GET /bottles` → enumerate every sibling bottle on the host, including each one's `policy` and `metadata` (`BottleRecord.redacted()` omits only the token). Cross-bottle disclosure. - `PUT /bottles/<id>/policy` → **rewrite its own egress allowlist to allow-all** (no token required, just a bottle_id from the list above). The gateway serves the new routes on the next `/resolve`. This defeats the per-bottle egress allowlist — the product's primary control. - `POST /resolve` with its own `(source_ip, identity_token)` → the response includes `tokens`, the **real upstream credentials** the gateway injects on the bottle's behalf (`control_plane.py:188`, `orch.tokens_for`). This directly defeats the README's "**Tokens the agent never sees**" guarantee — the agent reads them straight out. - `POST /supervise/respond` → **self-approve its own pending supervise proposals** (egress-allow, token-allow). The human approval gate becomes a formality the agent can click through for itself. - `POST /bottles` / `DELETE /bottles/<id>` → launch or tear down arbitrary bottles on the host, affecting other agents. Net: on the macOS consolidated backend a prompt-injected or misbehaving agent can lift the egress allowlist, read the credentials it was never supposed to see, and rubber-stamp its own approvals — i.e. every guarantee in the README's feature list collapses to the agent's own discretion. **Fix direction** (any one closes it; defense-in-depth wants more than one): - **Authenticate the control plane.** It's the real fix and it's backend-agnostic. A per-host shared secret the host CLI + gateway hold (env-injected, not on the agent network) gates all of `/bottles*`, `/supervise/respond`, and the `tokens` field of `/resolve`. `/resolve` and `/attribute` would still additionally require the caller's `(source_ip, identity_token)`. - **Separate the admin surface from the agent-facing surface** so agents can't route to 8099 at all — bind the control plane to an interface the agent network doesn't share, or front the gateway data-plane ports on a proxy container distinct from the control-plane container (mirrors the docker two-container split the consolidation collapsed). - **Scope the macOS network** the way Firecracker's nft does — only expose 9099/9100/9420 to agents. Apple's `container` networking may not support this cleanly, which is exactly why the auth fix is the durable one. **Also verify:** the docker consolidated backend. If its control-plane container sits on the agent's `--internal` network the same exposure applies there; only Firecracker's port-scoped nft demonstrably prevents it today. --- ## 🟠 High ### 2. WebSocket DLP is silently inert in the consolidated (multi-tenant) gateway **Where:** `bot_bottle/egress_addon.py::websocket_message` (~L623–658). Unlike `request()`/`response()`, which resolve the calling bottle's config per request via `_resolve_flow`, `websocket_message` scans against `self.config` and `self._supervise_slug`. In consolidated mode no per-bottle `routes.yaml` is written (the egress addon runs purely off `BOT_BOTTLE_ORCHESTRATOR_URL`), so `self.config` stays `Config(routes=())`. `match_route` therefore returns `None` and the handler returns **without scanning or killing the frame**. The code comment concedes this ("inert until websocket routing is made source-IP-aware"). **Impact:** for any host that made it past the CONNECT allowlist, WebSocket frames flow with **no outbound credential DLP and no inbound prompt-injection DLP** on the shared gateway — a live exfil channel for streaming agents. It fails *open*, and silently, which is the wrong direction for a DLP control. At minimum it should fail closed (kill WS frames when the addon can't resolve a per-flow config) until per-flow WS resolution lands. --- ## 🟡 Medium — architecture / maintainability ### 3. Two live tenancy code paths (single-tenant vs consolidated) across every gateway-facing module `egress_addon.py`, `git_http_backend.py`, `policy_resolver.py`, and the supervise plumbing each carry a "legacy single-tenant" branch alongside the consolidated one, described as transitional. Every branch is security-relevant and has to be kept correct in parallel — finding #2 is exactly a case where one path (WS) never got the consolidated treatment. Recommend tracking the single-tenant removal as a real deprecation with a target, not an open-ended "until every backend runs consolidated." ### 4. Flat-file import-shim duplication for gateway-bundled modules `egress_addon_core`, `dlp_detectors`, `policy_resolver`, `git_http_backend`, `yaml_subset`, `egress_dlp_config` are COPYed flat into the gateway image and every one uses a `try: import X / except ImportError: from .X` shim, plus hand-maintained duplicate constants (e.g. `IDENTITY_HEADER`, `GIT_GATE_TIMEOUT_SECS` copied rather than imported). A rename or a missed file in `Dockerfile.gateway` fails only at runtime inside the container, not in host tests or pyright. Worth a packaging step (or a single bundled subpackage) that makes the gateway bundle a build artifact of the real package rather than a parallel flat copy. ### 5. Custom YAML subset parser is a security-relevant parse surface `yaml_subset.py` (685 lines) is a hand-rolled parser sitting on the manifest + egress-policy trust boundary. It's well unit-tested (597-line test file), but a bespoke parser in this position warrants fuzzing (malformed/adversarial policy blobs, deeply nested input, resource-exhaustion cases) rather than example-based tests alone — a parser bug here is a policy-bypass or DoS. --- ## Positives worth recording - Attribution crypto is done right: 256-bit `secrets.token_urlsafe` tokens, `hmac.compare_digest` comparison, both-signals-required, fail-closed on ambiguity (`registry.py::attribute`). - The egress DLP detector set is genuinely thorough — encoded-variant expansion (base64/32/hex/gzip/url), NFKD unicode normalization against confusables, alnum-projection + sliding-window fragmentation resistance, and the `_closest_pair` O(n log n) rewrite explicitly to kill a quadratic DoS on attacker-controlled bodies. - Fail-closed is the consistent default across `resolve_client_config`, `resolve_sandbox_root`, and the gateway resolvers — unattributed/errored/unparseable all deny. - The Firecracker nft boundary is well-constructed: per-slot anti-spoof source-IP pinning, port-scoped DNAT, its own table at priority -10. It's precisely the model the macOS backend needs and lacks (finding #1). --- *Filed from an automated audit. The Critical is the actionable one and is contained to the macOS consolidated backend as written — Firecracker's port-scoped nft blocks the same attack today.*

didericis-claude added the

Priority

Critical

1

Kind/Security labels 2026-07-17 04:43:57 -04:00

didericis-claude commented

2026-07-17 04:51:52 -04:00

Fixing the 🔴 Critical (control-plane exposure) in branch macos-consolidated-gateway (#399).

Approach: fix direction #1 — authenticate the control plane — since it's the backend-agnostic, durable fix and closes the exposure on both macOS and docker (docker's orchestrator container is on the shared GATEWAY_NETWORK too, so the same attack applies there, as the audit flagged).

A per-host control-plane secret (~/.bot-bottle/control-plane-token, 0600, minted once) is required on every route except GET /health, checked with hmac.compare_digest. It's held only by the trusted callers, never on the agent network:

control plane verifies it (injected as env into the orchestrator/infra container);
gateway (PolicyResolver) sends it on /resolve + /attribute (same env, read in-container);
host CLI + launcher (OrchestratorClient) send it, read from the host file.

The agent never receives the env var or the host file, so from a bottle every /bottles*, /resolve, /attribute, and /supervise/* call gets 401 — closing the allowlist-rewrite, the tokens (upstream-credential) read, and the supervise self-approval. The existing (source_ip, identity_token) check on /resolve+/attribute stays as defense-in-depth.

Enforced when configured: macOS + docker inject the token (→ enforced, vuln closed). Firecracker's port-scoped nft already blocks 8099 from agent VMs, so it stays protected today; wiring the same secret into its infra-VM init is a clean fast-follow for defense-in-depth (left out here to avoid churning the prebuilt-artifact hash).

The other findings (WS DLP fail-open #2, tenancy-path/duplication/parser items #3–5) are out of scope for this fix and remain open.

Fixing the 🔴 Critical (control-plane exposure) in branch `macos-consolidated-gateway` (#399). **Approach:** fix direction #1 — authenticate the control plane — since it's the backend-agnostic, durable fix and closes the exposure on both macOS *and* docker (docker's orchestrator container is on the shared `GATEWAY_NETWORK` too, so the same attack applies there, as the audit flagged). A per-host **control-plane secret** (`~/.bot-bottle/control-plane-token`, 0600, minted once) is required on every route except `GET /health`, checked with `hmac.compare_digest`. It's held only by the trusted callers, never on the agent network: - **control plane** verifies it (injected as env into the orchestrator/infra container); - **gateway** (`PolicyResolver`) sends it on `/resolve` + `/attribute` (same env, read in-container); - **host CLI + launcher** (`OrchestratorClient`) send it, read from the host file. The agent never receives the env var or the host file, so from a bottle every `/bottles*`, `/resolve`, `/attribute`, and `/supervise/*` call gets `401` — closing the allowlist-rewrite, the `tokens` (upstream-credential) read, and the supervise self-approval. The existing `(source_ip, identity_token)` check on `/resolve`+`/attribute` stays as defense-in-depth. Enforced **when configured**: macOS + docker inject the token (→ enforced, vuln closed). Firecracker's port-scoped nft already blocks 8099 from agent VMs, so it stays protected today; wiring the same secret into its infra-VM init is a clean fast-follow for defense-in-depth (left out here to avoid churning the prebuilt-artifact hash). The other findings (WS DLP fail-open #2, tenancy-path/duplication/parser items #3–5) are out of scope for this fix and remain open.

didericis referenced this issue from a commit

2026-07-17 05:15:23 -04:00

fix(security): authenticate the orchestrator control plane (issue #400 Critical)

didericis-claude commented

2026-07-17 05:20:18 -04:00

Fixed in ca91fc4 on macos-consolidated-gateway (#399).

The control plane now requires the per-host secret on every route but GET /health (hmac-compared). Verified end-to-end on real Apple Container, running the exact attacks from the report inside a live agent (which holds its own identity token but not the admin secret):

From the agent (port 8099)	before	after
`GET /bottles` (enumerate siblings)	200	401
`POST /resolve` (lift upstream creds)	200 + `tokens`	401
`PUT /bottles/<id>/policy` (allow-all)	200	401
`POST /supervise/respond` (self-approve)	200	401
`DELETE /bottles/<id>` (nuke a bottle)	200	401

And the legitimate paths still work: the host CLI (holds the secret) lists bottles fine while an unauthenticated request gets 401, and egress policy still enforces (200 allowed / 403 denied) — which proves the gateway authenticates to /resolve with the injected secret rather than being locked out itself.

Scope notes:

docker is injected + enforced too (its orchestrator container is on the shared gateway network, same exposure you flagged).
Firecracker stays protected by its port-scoped nft; wiring the same secret into its infra-VM init is a clean fast-follow (left out to avoid churning the prebuilt-artifact hash). The mechanism is in place — it just needs the env set there.
Findings #2 (WS DLP fail-open) and #3–5 remain open; out of scope for this fix.

1829 unit tests pass (incl. new control-plane auth tests), pyright clean, pylint 9.91.

Fixed in `ca91fc4` on `macos-consolidated-gateway` (#399). The control plane now requires the per-host secret on every route but `GET /health` (hmac-compared). Verified **end-to-end on real Apple Container**, running the exact attacks from the report inside a live agent (which holds its own identity token but not the admin secret): | From the agent (port 8099) | before | after | | --- | --- | --- | | `GET /bottles` (enumerate siblings) | 200 | **401** | | `POST /resolve` (lift upstream creds) | 200 + `tokens` | **401** | | `PUT /bottles/<id>/policy` (allow-all) | 200 | **401** | | `POST /supervise/respond` (self-approve) | 200 | **401** | | `DELETE /bottles/<id>` (nuke a bottle) | 200 | **401** | And the legitimate paths still work: the host CLI (holds the secret) lists bottles fine while an unauthenticated request gets 401, and **egress policy still enforces (200 allowed / 403 denied)** — which proves the gateway authenticates to `/resolve` with the injected secret rather than being locked out itself. Scope notes: - **docker** is injected + enforced too (its orchestrator container is on the shared gateway network, same exposure you flagged). - **Firecracker** stays protected by its port-scoped nft; wiring the same secret into its infra-VM init is a clean fast-follow (left out to avoid churning the prebuilt-artifact hash). The mechanism is in place — it just needs the env set there. - Findings **#2 (WS DLP fail-open)** and **#3–5** remain open; out of scope for this fix. 1829 unit tests pass (incl. new control-plane auth tests), pyright clean, pylint 9.91.

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: didericis/bot-bottle#400