Compare commits

..

18 Commits

Author SHA1 Message Date
didericis-codex 3dbf1780b4 docs(prd): require shared storage permissions
prd-number-check / require-numbered-prds (pull_request) Successful in 11s
test / image-input-builds (pull_request) Successful in 45s
test / integration-docker (pull_request) Successful in 1m8s
test / unit (pull_request) Successful in 2m30s
test / coverage (pull_request) Successful in 25s
tracker-policy-pr / check-pr (pull_request) Failing after 12m49s
2026-07-27 04:45:45 +00:00
didericis-codex ff4da6f41e docs(prd): bound heavy gateway operations
prd-number-check / require-numbered-prds (pull_request) Successful in 10s
test / unit (pull_request) Successful in 54s
test / coverage (pull_request) Has been skipped
test / integration-docker (pull_request) Has been cancelled
test / image-input-builds (pull_request) Successful in 53s
tracker-policy-pr / check-pr (pull_request) Failing after 14m19s
2026-07-27 04:16:56 +00:00
didericis-codex 3bb90da11c docs(prd): require cleanup execution integrity
test / image-input-builds (pull_request) Failing after 14m11s
test / unit (pull_request) Failing after 14m19s
prd-number-check / require-numbered-prds (pull_request) Failing after 14m25s
test / integration-docker (pull_request) Has been cancelled
test / coverage (pull_request) Has been skipped
tracker-policy-pr / check-pr (pull_request) Failing after 12m36s
2026-07-27 03:55:18 +00:00
didericis-codex 0146450951 docs(prd): extend authoritative cleanup boundaries
prd-number-check / require-numbered-prds (pull_request) Successful in 13s
test / image-input-builds (pull_request) Successful in 3m18s
test / unit (pull_request) Successful in 57s
test / integration-docker (pull_request) Successful in 1m9s
test / coverage (pull_request) Successful in 19s
tracker-policy-pr / check-pr (pull_request) Failing after 10m30s
2026-07-27 03:31:46 +00:00
didericis-codex ecaf23cdb5 docs(prd): define authoritative failure boundaries
prd-number-check / require-numbered-prds (pull_request) Successful in 9s
test / unit (pull_request) Successful in 52s
test / integration-docker (pull_request) Waiting to run
test / coverage (pull_request) Blocked by required conditions
test / image-input-builds (pull_request) Successful in 2m42s
tracker-policy-pr / check-pr (pull_request) Failing after 12s
2026-07-27 02:58:07 +00:00
didericis-codex 6e46a9b191 fix(orchestrator): satisfy adapter type contracts
prd-number-check / require-numbered-prds (pull_request) Successful in 12s
lint / lint (push) Successful in 53s
test / image-input-builds (pull_request) Successful in 1m2s
test / integration-docker (pull_request) Successful in 58s
test / unit (pull_request) Successful in 46s
test / coverage (pull_request) Successful in 21s
tracker-policy-pr / check-pr (pull_request) Failing after 7s
2026-07-27 02:19:11 +00:00
didericis-codex a59e495faa refactor(egress): split request policy pipeline stages 2026-07-27 02:19:11 +00:00
didericis-codex b8818948a0 feat(firecracker): enumerate running bottles 2026-07-27 02:19:11 +00:00
didericis-codex b09952045a fix(orchestrator): bound unauthenticated HTTP requests 2026-07-27 02:19:11 +00:00
didericis-codex de192359ee fix(security): authenticate persisted egress secrets 2026-07-27 02:19:11 +00:00
didericis-codex 7d9933edc0 fix(docker): fail closed on network address scan errors 2026-07-27 02:19:11 +00:00
didericis-codex 2bc9ef8ec0 fix(firecracker): abort cleanup on scan errors 2026-07-27 02:19:11 +00:00
didericis-codex 47b6bead69 refactor(backend): type enumeration failures 2026-07-27 02:19:11 +00:00
didericis-codex 15ecada022 fix(docker): surface enumeration failures 2026-07-27 02:19:11 +00:00
didericis-codex e2222bd96b fix(security): prohibit unauthenticated orchestrator 2026-07-27 02:19:11 +00:00
didericis-claude 74ec9843f0 chore(coverage): relax thresholds to cut low-value churn
prd-number-check / require-numbered-prds (pull_request) Successful in 9s
tracker-policy-pr / check-pr (pull_request) Successful in 10s
test / unit (pull_request) Successful in 49s
test / image-input-builds (pull_request) Successful in 50s
test / integration-docker (pull_request) Successful in 56s
test / coverage (pull_request) Successful in 15s
test / image-input-builds (push) Successful in 46s
Update Quality Badges / update-badges (push) Successful in 56s
test / coverage (push) Successful in 22s
test / integration-docker (push) Successful in 1m3s
test / unit (push) Successful in 48s
lint / lint (push) Successful in 2m59s
Lower the diff-coverage gate from 90% to 80% and the critical-module
target from 90% to 85%. The 90% diff gate forced back-fill tests on
nearly every changed line; 80% keeps new code honest without the churn.
Global coverage stays informational per ADR 0004 (no new gate added).

Updates scripts/diff_coverage.py, scripts/coverage.sh,
scripts/critical-modules.txt, .gitea/workflows/test.yml, and records the
change as a dated revision in ADR 0004.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-26 21:45:08 -04:00
didericis-claude ed9fc76f97 fix(docker): only heal on the ParseAddr poison signature, not any inspect error
prd-number-check / require-numbered-prds (pull_request) Successful in 5s
test / unit (pull_request) Successful in 50s
tracker-policy-pr / check-pr (pull_request) Successful in 6s
test / image-input-builds (pull_request) Successful in 59s
test / integration-docker (pull_request) Successful in 1m6s
test / coverage (pull_request) Successful in 42s
Update Quality Badges / update-badges (push) Successful in 49s
lint / lint (push) Successful in 1m2s
test / coverage (push) Successful in 24s
test / integration-docker (push) Successful in 1m3s
test / unit (push) Successful in 48s
test / image-input-builds (push) Successful in 2m44s
Address review: the self-heal classified every non-absence `network inspect`
failure as a poisoned network and force-removed the shared gateway. But
inspect also fails on transient daemon/API errors, permission failures,
timeouts, or a bad context — destroying a healthy gateway on that guess would
tear the network out from under every live bottle.

Now the destructive path runs only for the known poison signature (docker's
`ParseAddr` error from the malformed `::1/64` IPv6 gateway). The absent case
(`No such network`) still just creates; any other inspect failure raises a
clear GatewayError without mutating shared state.

Adds a regression test asserting a generic inspect error issues neither
`docker rm --force` nor `docker network rm` and surfaces the error.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-27 01:41:09 +00:00
didericis-claude bd8a146a46 fix(docker): heal a poisoned IPv6 gateway network, not just avoid creating one
test / integration-docker (pull_request) Successful in 1m15s
test / image-input-builds (pull_request) Successful in 1m14s
lint / lint (push) Successful in 3m28s
tracker-policy-pr / check-pr (pull_request) Failing after 10m28s
test / unit (pull_request) Failing after 10m39s
prd-number-check / require-numbered-prds (pull_request) Failing after 10m45s
test / coverage (pull_request) Has been skipped
PR #515 stopped bot-bottle from *creating* a gateway network with a
malformed IPv6 subnet (--ipv6=false), but it can't recover a network that
is *already* poisoned. On a daemon that default-enables IPv6, the fixed-name
`bot-bottle-gateway` network gets an `fdd0::/64` subnet whose `::1/64`
gateway trips docker's own netip.ParseAddr, so `docker network inspect`/`ls`
exit non-zero — poisoning every command that reads networks.

Such a network can survive on a shared runner from a pre-fix or concurrent
launch. `_ensure_network` never healed it: its migrate/recreate branch only
ran when `network inspect` *succeeded*, but a poisoned network makes inspect
*fail*, so the code fell through to `network create`, which no-ops on
"already exists" — leaving the poison in place. The next subnet read then
failed with ConsolidatedLaunchError (test_multitenant_isolation), and a bare
`docker network ls` failed too (test_orphan_cleanup).

Fix, two parts:
- gateway.py: when `network inspect` fails for a reason other than "no such
  network", treat the network as poisoned and force-remove + recreate it
  IPv4-only. Absent-vs-poisoned is distinguished by the inspect stderr.
- test.yml: add an integration-docker preflight that drops the leftover
  gateway network (and its container) before the suite, so direct `network
  ls`/`inspect` calls in tests are clean even on a daemon where the in-code
  heal can't run because inspect itself is what's broken.

Adds unit coverage for the poisoned-heal and the absent-create split.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-27 01:23:25 +00:00
33 changed files with 922 additions and 454 deletions
+16 -2
View File
@@ -102,6 +102,20 @@ jobs:
python3 --version python3 --version
python3 cli.py backend status --backend=docker python3 cli.py backend status --backend=docker
- name: Preflight — clear any leftover poisoned gateway network
run: |
# The gateway network has a fixed name and persists across jobs on
# this shared runner. A pre-fix or concurrent launch can leave it with
# a malformed IPv6 subnet that trips docker's own ParseAddr in
# `network inspect` (see PR #515); the code now self-heals it, but the
# heal can't run if `network inspect` is what's broken on some daemon
# versions. Drop the network here so this run recreates it IPv4-only.
# Remove the attached gateway container first (else `network rm` fails
# on active endpoints); both are recreated by ensure_running. Harmless
# when absent.
docker rm --force bot-bottle-orch-gateway 2>/dev/null || true
docker network rm bot-bottle-gateway 2>/dev/null || true
- name: Run integration tests (docker) with coverage - name: Run integration tests (docker) with coverage
env: env:
BOT_BOTTLE_BACKEND: docker BOT_BOTTLE_BACKEND: docker
@@ -284,7 +298,7 @@ jobs:
- name: Combined coverage (unit + docker integration) - name: Combined coverage (unit + docker integration)
run: PYTHON=python3 bash scripts/coverage.sh aggregate critical run: PYTHON=python3 bash scripts/coverage.sh aggregate critical
- name: Diff-coverage gate (changed lines >= 90%) - name: Diff-coverage gate (changed lines >= 80%)
run: | run: |
git fetch --no-tags origin main:refs/remotes/origin/main git fetch --no-tags origin main:refs/remotes/origin/main
python3 scripts/diff_coverage.py --base origin/main --min 90 python3 scripts/diff_coverage.py --base origin/main --min 80
+3
View File
@@ -30,6 +30,7 @@ if TYPE_CHECKING:
BottleImages, BottleImages,
BottlePlan, BottlePlan,
BottleSpec, BottleSpec,
EnumerationError,
ExecResult, ExecResult,
) )
from .selection import ( from .selection import (
@@ -59,6 +60,7 @@ _LAZY_MODULES: dict[str, str] = {
"BottleImages": "base", "BottleImages": "base",
"BottleBackend": "base", "BottleBackend": "base",
"BackendStatus": "base", "BackendStatus": "base",
"EnumerationError": "base",
"get_bottle_backend": "selection", "get_bottle_backend": "selection",
"known_backend_names": "selection", "known_backend_names": "selection",
"has_backend": "selection", "has_backend": "selection",
@@ -100,6 +102,7 @@ __all__ = [
"BottlePlan", "BottlePlan",
"BottleSpec", "BottleSpec",
"ExecResult", "ExecResult",
"EnumerationError",
"CommitCancelled", "CommitCancelled",
"Freezer", "Freezer",
"get_freezer", "get_freezer",
+4
View File
@@ -42,6 +42,10 @@ class BackendStatus(enum.IntEnum):
READY = 0 READY = 0
class EnumerationError(RuntimeError):
"""A backend could not produce an authoritative live-resource snapshot."""
@dataclass(frozen=True) @dataclass(frozen=True)
class BottleSpec: class BottleSpec:
"""CLI-supplied intent. Backend-agnostic — each backend's prepare """CLI-supplied intent. Backend-agnostic — each backend's prepare
+27 -13
View File
@@ -16,6 +16,7 @@ from pathlib import Path
from typing import Any from typing import Any
from ...log import die, warn from ...log import die, warn
from ..base import EnumerationError
# --- Lifecycle helpers (PRD 0018 chunk 3) ---------------------------------- # --- Lifecycle helpers (PRD 0018 chunk 3) ----------------------------------
@@ -52,19 +53,20 @@ def slug_from_compose_project(project: str) -> str:
def list_compose_projects( def list_compose_projects(
*, include_stopped: bool = True, warn_on_error: bool = True, *,
include_stopped: bool = True,
warn_on_error: bool = True,
raise_on_error: bool = False,
) -> list[str]: ) -> list[str]:
"""All compose project names starting with `bot-bottle-`. """All compose project names starting with `bot-bottle-`.
`include_stopped=True` (default) runs `docker compose ls --all` `include_stopped=True` (default) runs `docker compose ls --all`
so exited projects appear too; pass False to get only projects so exited projects appear too; pass False to get only projects
with at least one running container. with at least one running container.
Returns [] on docker daemon errors or malformed output rather Best-effort callers get ``[]`` on Docker errors or malformed output.
than raising — callers should treat the empty list as "no Enumeration callers pass ``raise_on_error=True`` so a failed query is not
projects discoverable", not "no projects exist". `warn_on_error` reported as an authoritative empty result.
stays true for explicit operator commands like cleanup, but active """
discovery paths set it false so dashboard refreshes don't spam
stderr while Docker Desktop is stopped."""
argv = ["docker", "compose", "ls", "--format", "json"] argv = ["docker", "compose", "ls", "--format", "json"]
if include_stopped: if include_stopped:
argv.insert(3, "--all") argv.insert(3, "--all")
@@ -72,19 +74,27 @@ def list_compose_projects(
result = subprocess.run( result = subprocess.run(
argv, capture_output=True, text=True, check=False, argv, capture_output=True, text=True, check=False,
) )
except FileNotFoundError: except FileNotFoundError as exc:
# docker binary not on PATH — same shape as a daemon-down if raise_on_error:
# error from the caller's POV: no projects discoverable. raise EnumerationError(
"docker compose ls failed: docker not found"
) from exc
return [] return []
if result.returncode != 0: if result.returncode != 0:
message = f"docker compose ls failed: {result.stderr.strip()}"
if raise_on_error:
raise EnumerationError(message)
if warn_on_error: if warn_on_error:
warn(f"docker compose ls failed: {result.stderr.strip()}") warn(message)
return [] return []
try: try:
projects = json.loads(result.stdout or "[]") projects = json.loads(result.stdout or "[]")
except json.JSONDecodeError as e: except json.JSONDecodeError as e:
message = f"docker compose ls returned malformed JSON: {e}"
if raise_on_error:
raise EnumerationError(message) from e
if warn_on_error: if warn_on_error:
warn(f"docker compose ls returned malformed JSON: {e}") warn(message)
return [] return []
names: list[str] = [] names: list[str] = []
for p in projects: for p in projects:
@@ -97,7 +107,10 @@ def list_compose_projects(
def list_active_slugs( def list_active_slugs(
*, include_stopped: bool = False, warn_on_error: bool = True, *,
include_stopped: bool = False,
warn_on_error: bool = True,
raise_on_error: bool = False,
) -> list[str]: ) -> list[str]:
"""Slugs (project name minus prefix) of currently-running """Slugs (project name minus prefix) of currently-running
bottles. Used by the dashboard's operator-edit verbs to choose bottles. Used by the dashboard's operator-edit verbs to choose
@@ -108,6 +121,7 @@ def list_active_slugs(
for p in list_compose_projects( for p in list_compose_projects(
include_stopped=include_stopped, include_stopped=include_stopped,
warn_on_error=warn_on_error, warn_on_error=warn_on_error,
raise_on_error=raise_on_error,
) )
) if slug ) if slug
) )
@@ -68,6 +68,11 @@ def _network_container_ips(network: str) -> list[str]:
"docker", "network", "inspect", "--format", "docker", "network", "inspect", "--format",
"{{range .Containers}}{{.IPv4Address}} {{end}}", network, "{{range .Containers}}{{.IPv4Address}} {{end}}", network,
]) ])
if proc.returncode != 0:
detail = proc.stderr.strip() or f"exit {proc.returncode}"
raise ConsolidatedLaunchError(
f"could not inspect addresses on gateway network {network}: {detail}"
)
ips: list[str] = [] ips: list[str] = []
for entry in proc.stdout.split(): for entry in proc.stdout.split():
ips.append(entry.split("/", 1)[0]) ips.append(entry.split("/", 1)[0])
+12 -12
View File
@@ -1,9 +1,8 @@
"""Active-agent enumeration for the docker backend. """Active-agent enumeration for the docker backend.
Returns `ActiveAgent` records the CLI `active` command and the Returns `ActiveAgent` records the CLI `active` command and the
dashboard agents pane consume. Empty when docker isn't reachable dashboard agents pane consume. Docker query failures raise rather
— gated by `has_backend('docker')` at the cross-backend caller than masquerading as an authoritative empty result.
so this module trusts that docker is available when called.
The parser (`_parse_services_by_project`) is exposed for direct The parser (`_parse_services_by_project`) is exposed for direct
unit testing; the docker `docker ps` invocation is in unit testing; the docker `docker ps` invocation is in
@@ -13,17 +12,18 @@ from __future__ import annotations
import subprocess import subprocess
from .. import ActiveAgent from .. import ActiveAgent, EnumerationError
from ...bottle_state import read_metadata from ...bottle_state import read_metadata
from .compose import compose_project_name, list_active_slugs from .compose import compose_project_name, list_active_slugs
def enumerate_active() -> list[ActiveAgent]: def enumerate_active() -> list[ActiveAgent]:
"""All currently-running docker-backed agents. Caller is """All currently-running docker-backed agents."""
responsible for gating on `has_backend('docker')` if it slugs = list_active_slugs(
matters; if docker is missing the `docker ps` call below include_stopped=False,
returns an empty list silently.""" warn_on_error=False,
slugs = list_active_slugs(include_stopped=False, warn_on_error=False) raise_on_error=True,
)
if not slugs: if not slugs:
return [] return []
services_by_project = _query_services_by_project() services_by_project = _query_services_by_project()
@@ -74,8 +74,8 @@ def _query_services_by_project() -> dict[str, set[str]]:
], ],
capture_output=True, text=True, check=False, capture_output=True, text=True, check=False,
) )
except FileNotFoundError: except FileNotFoundError as exc:
return {} raise EnumerationError("docker ps failed: docker not found") from exc
if r.returncode != 0: if r.returncode != 0:
return {} raise EnumerationError(f"docker ps failed: {r.stderr.strip()}")
return _parse_services_by_project(r.stdout or "") return _parse_services_by_project(r.stdout or "")
+35 -4
View File
@@ -140,12 +140,43 @@ class DockerGateway(Gateway):
marker = inspected.stdout.strip() marker = inspected.stdout.strip()
if marker in {"", self._subnet}: if marker in {"", self._subnet}:
return return
if inspected.returncode == 0: # Inspectable but mislabelled: the stale auto-IPAM network created
# Migrate the stale auto-IPAM network created by older releases. # by older releases. Replace it below.
# Removing the fixed gateway is safe here: this launch recreates it. stale = True
else:
# inspect failed. Classify by stderr — do NOT assume "not absent"
# implies "poisoned": a transient daemon/API error, permission
# failure, timeout, or bad context also fails here, and destroying
# the shared gateway on that guess would tear the network out from
# under every live bottle.
err = inspected.stderr.lower()
if "no such network" in err or "not found" in err:
# Absent: nothing to replace — create it below.
stale = False
elif "parseaddr" in err:
# Present but poisoned. A daemon that default-enables IPv6
# attaches an fdd0::/64 subnet whose `::1/64` gateway trips
# docker's own netip.ParseAddr in `network inspect`/`ls`, so the
# command exits non-zero with that signature. A fixed release
# never *creates* such a network, but one can survive on a
# shared host from an older or concurrent launch — and
# `--ipv6=false` alone can't heal it, since the create below only
# no-ops on "already exists". Force-replace it so later reads
# (e.g. `_network_cidr` pinning a source IP) stop failing.
stale = True
else:
# Unrecognized failure: no evidence the network is malformed.
# Surface it rather than mutate shared state on a guess.
raise GatewayError(
f"gateway network {self.network} could not be inspected: "
f"{inspected.stderr.strip()}"
)
if stale:
# Migrate the stale/poisoned network. Removing the fixed gateway is
# safe here: this launch recreates it.
run_docker(["docker", "rm", "--force", self.name]) run_docker(["docker", "rm", "--force", self.name])
removed = run_docker(["docker", "network", "rm", self.network]) removed = run_docker(["docker", "network", "rm", self.network])
if removed.returncode != 0: if removed.returncode != 0 and "no such network" not in removed.stderr.lower():
raise GatewayError( raise GatewayError(
f"gateway network {self.network} needs explicit subnet " f"gateway network {self.network} needs explicit subnet "
f"{self._subnet} but could not be replaced: " f"{self._subnet} but could not be replaced: "
+17 -5
View File
@@ -29,6 +29,7 @@ import subprocess
from pathlib import Path from pathlib import Path
from ...log import info from ...log import info
from .. import EnumerationError
from . import util from . import util
from .bottle_cleanup_plan import FirecrackerBottleCleanupPlan from .bottle_cleanup_plan import FirecrackerBottleCleanupPlan
@@ -62,12 +63,23 @@ def _scan_processes(run_root: Path) -> tuple[set[str], list[int]]:
* ``orphan_pids`` — firecracker pids whose run dir no longer exists * ``orphan_pids`` — firecracker pids whose run dir no longer exists
(a lingering VMM to kill). (a lingering VMM to kill).
""" """
result = subprocess.run( try:
["pgrep", "-a", "firecracker"], result = subprocess.run(
capture_output=True, text=True, check=False, ["pgrep", "-a", "firecracker"],
) capture_output=True, text=True, check=False,
if result.returncode != 0: )
except OSError as exc:
raise EnumerationError(
f"could not enumerate Firecracker processes: {exc}"
) from exc
if result.returncode == 1:
# pgrep's documented "no processes matched" result.
return set(), [] return set(), []
if result.returncode != 0:
detail = (result.stderr or "").strip() or f"exit {result.returncode}"
raise EnumerationError(
f"could not enumerate Firecracker processes: {detail}"
)
live: set[str] = set() live: set[str] = set()
orphan_pids: list[int] = [] orphan_pids: list[int] = []
for line in result.stdout.splitlines(): for line in result.stdout.splitlines():
+22 -4
View File
@@ -1,14 +1,32 @@
"""Active-agent enumeration for the Firecracker backend. """Active-agent enumeration for the Firecracker backend.
The backend is disabled during the companion-container removal (#385) — it can't Running bottles are the Firecracker processes whose ``--config-file`` points
launch bottles, so there are none to enumerate. Real enumeration returns at an existing per-bottle run directory. The same authoritative process scan
with the backend's consolidated relaunch (#354). protects cleanup from deleting live VMs; operational scan failures propagate
as ``EnumerationError`` instead of masquerading as an empty host.
""" """
from __future__ import annotations from __future__ import annotations
from ...bottle_state import read_metadata
from .. import ActiveAgent from .. import ActiveAgent
from .cleanup import live_run_dirs
def enumerate_active() -> list[ActiveAgent]: def enumerate_active() -> list[ActiveAgent]:
return [] out: list[ActiveAgent] = []
for run_dir in live_run_dirs():
slug = run_dir.name
metadata = read_metadata(slug)
out.append(ActiveAgent(
backend_name="firecracker",
slug=slug,
agent_name=metadata.agent_name if metadata else "?",
started_at=metadata.started_at if metadata else "",
# Firecracker uses the shared gateway, so there are no
# per-bottle gateway service containers to report.
services=(),
label=metadata.label if metadata else "",
color=metadata.color if metadata else "",
))
return out
+12 -11
View File
@@ -5,7 +5,7 @@ from __future__ import annotations
import subprocess import subprocess
from ...bottle_state import read_metadata from ...bottle_state import read_metadata
from .. import ActiveAgent from .. import ActiveAgent, EnumerationError
from .infra import INFRA_NAME, ORCHESTRATOR_NAME from .infra import INFRA_NAME, ORCHESTRATOR_NAME
# The name every agent container carries: `bot-bottle-<slug>`. Exported # The name every agent container carries: `bot-bottle-<slug>`. Exported
@@ -20,17 +20,18 @@ CONTAINER_NAME_PREFIX = "bot-bottle-"
_INFRA_NAMES = frozenset({INFRA_NAME, ORCHESTRATOR_NAME}) _INFRA_NAMES = frozenset({INFRA_NAME, ORCHESTRATOR_NAME})
class EnumerationError(RuntimeError):
"""container list failed; the resulting live set is not authoritative."""
def enumerate_active() -> list[ActiveAgent]: def enumerate_active() -> list[ActiveAgent]:
result = subprocess.run( try:
["container", "list", "--quiet"], result = subprocess.run(
capture_output=True, ["container", "list", "--quiet"],
text=True, capture_output=True,
check=False, text=True,
) check=False,
)
except FileNotFoundError as exc:
raise EnumerationError(
"container list failed: container CLI not found"
) from exc
if result.returncode != 0: if result.returncode != 0:
raise EnumerationError( raise EnumerationError(
f"container list failed: " f"container list failed: "
+37 -24
View File
@@ -389,19 +389,9 @@ class EgressAddon:
self._passthrough_conns.discard(conn_id) self._passthrough_conns.discard(conn_id)
async def request(self, flow: http.HTTPFlow) -> None: async def request(self, flow: http.HTTPFlow) -> None:
config, slug, env = self._request_context(flow)
request_path, _, query = flow.request.path.partition("?") request_path, _, query = flow.request.path.partition("?")
# Reuse the context stashed by http_connect for HTTPS flows (one
# orchestrator round-trip per connection). Plain-HTTP flows have no
# prior CONNECT stash, so resolve now and stash for response/websocket.
meta = getattr(flow, "metadata", None)
if isinstance(meta, dict) and _FLOW_CTX_KEY in meta:
config, slug, env = meta[_FLOW_CTX_KEY]
self._request_token(flow) # strip identity headers; token already resolved
else:
config, slug, env = self._resolve_flow(flow)
self._stash_flow_ctx(flow, config, slug, env)
# Introspection ("_egress.local/allowlist") reports the calling bottle's # Introspection ("_egress.local/allowlist") reports the calling bottle's
# own resolved routes — served after resolution so it reflects this # own resolved routes — served after resolution so it reflects this
# bottle's policy, not a stale global. # bottle's policy, not a stale global.
@@ -422,6 +412,29 @@ class EgressAddon:
# the path/query the git checks below rely on. # the path/query the git checks below rely on.
request_path, _, query = flow.request.path.partition("?") request_path, _, query = flow.request.path.partition("?")
if not self._allow_git_request(flow, config, request_path, query):
return
self._apply_route_policy(flow, config, route, request_path, env)
def _request_context(
self, flow: http.HTTPFlow,
) -> tuple[Config, str, "typing.Mapping[str, str]"]:
"""Resolve one bottle context, reusing the HTTPS CONNECT snapshot."""
meta = getattr(flow, "metadata", None)
if isinstance(meta, dict) and _FLOW_CTX_KEY in meta:
config, slug, env = meta[_FLOW_CTX_KEY]
self._request_token(flow)
return config, slug, env
config, slug, env = self._resolve_flow(flow)
self._stash_flow_ctx(flow, config, slug, env)
return config, slug, env
def _allow_git_request(
self, flow: http.HTTPFlow, config: Config,
request_path: str, query: str,
) -> bool:
"""Apply the HTTPS Git push/fetch boundary before general routing."""
if is_git_push_request(request_path, query): if is_git_push_request(request_path, query):
self._block( self._block(
flow, flow,
@@ -430,20 +443,20 @@ class EgressAddon:
"git-gate's pre-receive hook).", "git-gate's pre-receive hook).",
ctx=self._req_ctx(flow), ctx=self._req_ctx(flow),
) )
return return False
if not is_git_fetch_request(request_path, query):
if is_git_fetch_request(request_path, query): return True
git_decision = decide_git_fetch( git_decision = decide_git_fetch(config.routes, flow.request.pretty_host)
config.routes, flow.request.pretty_host, if git_decision.action != "block":
) return True
if git_decision.action == "block": self._block(flow, git_decision.reason, ctx=self._req_ctx(flow))
self._block( return False
flow,
git_decision.reason,
ctx=self._req_ctx(flow),
)
return
def _apply_route_policy(
self, flow: http.HTTPFlow, config: Config, route: Route | None,
request_path: str, env: "typing.Mapping[str, str]",
) -> None:
"""Strip agent auth, evaluate the route, then inject gateway auth."""
# Strip agent-set Authorization after DLP scan so smuggled tokens # Strip agent-set Authorization after DLP scan so smuggled tokens
# are caught above; the route may inject gateway-owned auth below. # are caught above; the route may inject gateway-owned auth below.
# Routes with preserve_auth=True pass the header through as-is so the # Routes with preserve_auth=True pass the header through as-is so the
+11 -4
View File
@@ -1,6 +1,7 @@
"""Run the orchestrator control plane as a plain process (PRD 0070 dev-harness). """Run the orchestrator control plane as a plain process (PRD 0070 dev-harness).
python -m bot_bottle.orchestrator [--host H] [--port P] [--db PATH] BOT_BOTTLE_ORCHESTRATOR_TOKEN=<signing-key> \
python -m bot_bottle.orchestrator [--host H] [--port P] [--db PATH]
The PRD sequences the orchestrator as a plain-process dev-harness first, so The PRD sequences the orchestrator as a plain-process dev-harness first, so
the consolidation core (registry + attribution + HTTP control plane + live the consolidation core (registry + attribution + HTTP control plane + live
@@ -16,12 +17,13 @@ import secrets
from pathlib import Path from pathlib import Path
from .. import log from .. import log
from .store.store_manager import StoreManager from ..trust_domain import CONTROL_PLANE
from .broker import LaunchBroker, StubBroker from .broker import LaunchBroker, StubBroker
from .server import make_server
from .docker_broker import DockerBroker from .docker_broker import DockerBroker
from .store.registry_store import RegistryStore, default_db_path from .server import make_server
from .service import OrchestratorCore from .service import OrchestratorCore
from .store.store_manager import StoreManager
from .store.registry_store import RegistryStore, default_db_path
def main(argv: list[str] | None = None) -> int: def main(argv: list[str] | None = None) -> int:
@@ -38,6 +40,11 @@ def main(argv: list[str] | None = None) -> int:
help="launch broker: 'stub' records requests; 'docker' runs containers", help="launch broker: 'stub' records requests; 'docker' runs containers",
) )
args = parser.parse_args(argv) args = parser.parse_args(argv)
if not CONTROL_PLANE.key_from_env():
log.die(
f"{CONTROL_PLANE.key_env} is required; refusing to start the "
"orchestrator without caller authentication"
)
registry = RegistryStore(args.db) registry = RegistryStore(args.db)
registry.migrate() registry.migrate()
+92 -33
View File
@@ -59,8 +59,10 @@ import http.server
import json import json
import math import math
import os import os
import socket
import socketserver import socketserver
import sys import sys
import threading
import typing import typing
from urllib.parse import urlsplit from urllib.parse import urlsplit
@@ -80,6 +82,9 @@ Json = dict[str, object]
# token at all, and a compromised gateway holds only `gateway` — neither can # token at all, and a compromised gateway holds only `gateway` — neither can
# drive the operator routes (approve proposals, rewrite policy, read tokens). # drive the operator routes (approve proposals, rewrite policy, read tokens).
ORCHESTRATOR_AUTH_HEADER = "x-bot-bottle-orchestrator-auth" ORCHESTRATOR_AUTH_HEADER = "x-bot-bottle-orchestrator-auth"
MAX_BODY_BYTES = 1 * 1024 * 1024
REQUEST_TIMEOUT_SECONDS = 10.0
MAX_REQUEST_THREADS = 32
# The routes the data plane (role `gateway`) is allowed to reach — exactly the # The routes the data plane (role `gateway`) is allowed to reach — exactly the
# per-request lookups PolicyResolver makes. Every other authenticated route is # per-request lookups PolicyResolver makes. Every other authenticated route is
@@ -116,9 +121,8 @@ def dispatch( # pylint: disable=too-many-return-statements,too-many-branches
no I/O beyond the orchestrator so it is fully testable without a socket. no I/O beyond the orchestrator so it is fully testable without a socket.
`role` is the caller's verified control-plane role (`gateway` or `cli`), or `role` is the caller's verified control-plane role (`gateway` or `cli`), or
None for an unauthenticated request; an open-mode server (no signing key None for an unauthenticated request. Every route except `GET /health`
configured see `OrchestratorServer`) passes `cli`. Every route except requires a role: a missing role is 401, and a role that
`GET /health` requires a role: a missing role is 401, and a role that
doesn't cover the route is 403 — so a `gateway` data-plane token can reach doesn't cover the route is 403 — so a `gateway` data-plane token can reach
`/resolve` + `/supervise/{propose,poll}` but not the operator routes `/resolve` + `/supervise/{propose,poll}` but not the operator routes
(rewrite policy, read injected tokens, approve its own supervise proposals). (rewrite policy, read injected tokens, approve its own supervise proposals).
@@ -372,10 +376,33 @@ class Handler(http.server.BaseHTTPRequestHandler):
plane down for the caller.""" plane down for the caller."""
server = self.server server = self.server
assert isinstance(server, OrchestratorServer) assert isinstance(server, OrchestratorServer)
length = int(self.headers.get("Content-Length") or 0)
body = self.rfile.read(length) if length > 0 else b""
role = server.role_for(self.headers.get(ORCHESTRATOR_AUTH_HEADER, "")) role = server.role_for(self.headers.get(ORCHESTRATOR_AUTH_HEADER, ""))
route = urlsplit(self.path).path.rstrip("/") or "/"
if not (method == "GET" and route == "/health") and role is None:
self._write_json(
401, {"error": "control-plane authentication required"},
)
return
length_header = self.headers.get("Content-Length")
try: try:
length = int(length_header) if length_header is not None else 0
except ValueError:
self._write_json(400, {"error": "invalid Content-Length"})
return
if length < 0:
self._write_json(400, {"error": "invalid Content-Length"})
return
if length > MAX_BODY_BYTES:
self._write_json(413, {"error": "request body too large"})
return
try:
body = self.rfile.read(length) if length else b""
except (TimeoutError, socket.timeout):
self._write_json(408, {"error": "request body read timed out"})
return
try:
status: int
payload: Json
status, payload = dispatch( status, payload = dispatch(
server.orchestrator, method, self.path, body, role=role) server.orchestrator, method, self.path, body, role=role)
except Exception as e: # noqa: BLE001 — the control plane must stay up except Exception as e: # noqa: BLE001 — the control plane must stay up
@@ -388,6 +415,9 @@ class Handler(http.server.BaseHTTPRequestHandler):
) )
sys.stderr.flush() sys.stderr.flush()
status, payload = 500, {"error": "internal error"} status, payload = 500, {"error": "internal error"}
self._write_json(status, payload)
def _write_json(self, status: int, payload: Json) -> None:
data = json.dumps(payload).encode() data = json.dumps(payload).encode()
self.send_response(status) self.send_response(status)
self.send_header("Content-Type", "application/json") self.send_header("Content-Type", "application/json")
@@ -414,51 +444,80 @@ class OrchestratorServer(socketserver.ThreadingMixIn, http.server.HTTPServer):
Holds the per-host control-plane *signing key* (from Holds the per-host control-plane *signing key* (from
`$BOT_BOTTLE_ORCHESTRATOR_TOKEN`, injected by the launcher into the `$BOT_BOTTLE_ORCHESTRATOR_TOKEN`, injected by the launcher into the
orchestrator process only) and verifies each request's role-scoped token orchestrator process only) and verifies each request's role-scoped token
against it. When a key is set, every route but `/health` requires a valid against it. Every route but `/health` requires a valid token whose role
token whose role covers the route; when it is unset the server runs **open** covers the route. Construction fails when the key is absent so a new or
(full `cli` access) and says so loudly at startup a fail-visible fallback misconfigured launcher cannot accidentally expose an open control plane."""
for tests and any backend that hasn't wired the key yet (e.g. Firecracker,
whose nft boundary already blocks agents from the control-plane port)."""
daemon_threads = True daemon_threads = True
allow_reuse_address = True allow_reuse_address = True
def __init__(self, address: tuple[str, int], orchestrator: OrchestratorCore) -> None: def __init__(
self,
address: tuple[str, int],
orchestrator: OrchestratorCore,
*,
signing_key: str,
) -> None:
self.orchestrator = orchestrator self.orchestrator = orchestrator
# The control-plane trust domain's signing key, as injected into THIS self._signing_key = signing_key.strip()
# (the owning) process by the launcher (#476). Unset → open mode below.
self._signing_key = CONTROL_PLANE.key_from_env()
if not self._signing_key: if not self._signing_key:
sys.stderr.write( raise ValueError(
"orchestrator: WARNING — no control-plane signing key " "orchestrator control-plane signing key is required; "
f"(${CONTROL_PLANE.key_env}); running WITHOUT caller " "refusing to start without caller authentication"
"authentication. Any client that can reach this port can drive "
"it. Backends that put the control plane on an agent-reachable "
"network MUST set this.\n"
) )
sys.stderr.flush() self._request_slots = threading.BoundedSemaphore(MAX_REQUEST_THREADS)
super().__init__(address, Handler) super().__init__(address, Handler)
def get_request(self) -> tuple[socket.socket, typing.Any]:
request, client_address = super().get_request()
request.settimeout(REQUEST_TIMEOUT_SECONDS)
return request, client_address
def process_request(
self, request: typing.Any, client_address: typing.Any,
) -> None:
# Bound concurrency before ThreadingMixIn creates a worker. Backpressure
# stays in the accept loop instead of allocating an unbounded thread per
# slow or malicious connection.
self._request_slots.acquire()
try:
super().process_request(request, client_address)
except BaseException:
self._request_slots.release()
raise
def process_request_thread(
self, request: typing.Any, client_address: typing.Any,
) -> None:
try:
super().process_request_thread(request, client_address)
finally:
self._request_slots.release()
def role_for(self, presented: str) -> str | None: def role_for(self, presented: str) -> str | None:
"""The role the request is authorized as, or None if unauthenticated. """The verified caller role, or None for a missing/invalid token."""
Open mode (no signing key) grants full `cli` access the fail-visible
fallback. Otherwise verify the presented signed token; a missing/invalid
token yields None ( 401), a valid one yields its `gateway`/`cli`
role ( per-route 401/403 in `dispatch`)."""
if not self._signing_key:
return ROLE_CLI
return CONTROL_PLANE.verify(presented, self._signing_key) return CONTROL_PLANE.verify(presented, self._signing_key)
def make_server( def make_server(
orchestrator: OrchestratorCore, host: str = "127.0.0.1", port: int = 0 orchestrator: OrchestratorCore,
host: str = "127.0.0.1",
port: int = 0,
*,
signing_key: str | None = None,
) -> OrchestratorServer: ) -> OrchestratorServer:
"""Build (but do not start) a control-plane server. `port=0` binds an """Build an authenticated control-plane server.
ephemeral port read `server.server_address` for the actual one."""
return OrchestratorServer((host, port), orchestrator) ``signing_key=None`` reads the owning process's injected environment.
Empty or missing keys are rejected by :class:`OrchestratorServer`.
"""
key = CONTROL_PLANE.key_from_env() if signing_key is None else signing_key
return OrchestratorServer(
(host, port), orchestrator, signing_key=key,
)
__all__ = [ __all__ = [
"dispatch", "Handler", "OrchestratorServer", "make_server", "Json", "dispatch", "Handler", "OrchestratorServer", "make_server", "Json",
"ORCHESTRATOR_AUTH_HEADER", "ORCHESTRATOR_AUTH_HEADER", "MAX_BODY_BYTES",
] ]
+11 -3
View File
@@ -366,15 +366,23 @@ class OrchestratorCore:
value with *env_var_secret*, and restores ``_tokens[bottle_id]``. value with *env_var_secret*, and restores ``_tokens[bottle_id]``.
Returns True on success, False when no stored secrets exist for this Returns True on success, False when no stored secrets exist for this
bottle or decryption fails (wrong key / corrupt data).""" bottle or decryption fails (wrong key / corrupt data)."""
from .store.secret_store import decrypt_value from .store.secret_store import decrypt_value, encrypt_value, is_legacy_blob
encrypted = self.registry.get_agent_secrets(bottle_id) encrypted = self.registry.get_agent_secrets(bottle_id)
if not encrypted: if not encrypted:
return False return False
try: try:
self._tokens[bottle_id] = {k: decrypt_value(env_var_secret, v) decrypted = {
for k, v in encrypted.items()} k: decrypt_value(env_var_secret, v) for k, v in encrypted.items()
}
except ValueError: except ValueError:
return False return False
self._tokens[bottle_id] = decrypted
if any(is_legacy_blob(value) for value in encrypted.values()):
migrated = {
key: encrypt_value(env_var_secret, value)
for key, value in decrypted.items()
}
self.registry.store_agent_secrets(bottle_id, migrated)
return True return True
# --- consolidated gateway ---------------------------------------------- # --- consolidated gateway ----------------------------------------------
+80 -18
View File
@@ -12,9 +12,15 @@ reattachment path reads ENV_VAR_SECRET from the running agent container via
``POST /bottles/<id>/reprovision_gateway``; the orchestrator decrypts the ``POST /bottles/<id>/reprovision_gateway``; the orchestrator decrypts the
stored rows and re-populates ``_tokens``. stored rows and re-populates ``_tokens``.
Encryption scheme: HMAC-SHA256 used as a PRF in CTR mode (stdlib-only, Encryption scheme: encrypt-then-MAC using independent HMAC-SHA256-derived
no external deps). Each value is encrypted independently. The output blob is encryption and authentication subkeys (stdlib-only, no external deps). Each
``nonce (16 bytes) || ciphertext`` encoded as URL-safe base64 (no padding). value is encrypted independently. New output blobs are:
``version || nonce (16 bytes) || ciphertext || tag (32 bytes)``
encoded as URL-safe base64 (no padding). The version marker lets the reader
accept legacy ``nonce || ciphertext`` rows long enough to rewrite them in the
authenticated format after a successful reprovision.
keystream_block_i = HMAC-SHA256(key, nonce || i.to_bytes(4, "big")) keystream_block_i = HMAC-SHA256(key, nonce || i.to_bytes(4, "big"))
ciphertext_i = plaintext_i XOR keystream_block_i[:len(plaintext_i)] ciphertext_i = plaintext_i XOR keystream_block_i[:len(plaintext_i)]
@@ -30,6 +36,8 @@ import secrets
_KEY_BYTES = 32 # 256-bit key from ENV_VAR_SECRET _KEY_BYTES = 32 # 256-bit key from ENV_VAR_SECRET
_NONCE_BYTES = 16 # 128-bit random nonce per encrypt call _NONCE_BYTES = 16 # 128-bit random nonce per encrypt call
_BLOCK = 32 # HMAC-SHA256 output width == one keystream block _BLOCK = 32 # HMAC-SHA256 output width == one keystream block
_TAG_BYTES = 32
_VERSION = b"BBSE1"
# Env-var name the agent container receives at startup. # Env-var name the agent container receives at startup.
ENV_VAR_SECRET_NAME = "ENV_VAR_SECRET" ENV_VAR_SECRET_NAME = "ENV_VAR_SECRET"
@@ -41,7 +49,13 @@ def new_env_var_secret() -> str:
def _b64dec(s: str) -> bytes: def _b64dec(s: str) -> bytes:
return base64.urlsafe_b64decode(s + "=" * (-len(s) % 4)) return base64.b64decode(
s + "=" * (-len(s) % 4), altchars=b"-_", validate=True,
)
def _subkey(key: bytes, purpose: bytes) -> bytes:
return hmac.new(key, b"bot-bottle-secret-store:" + purpose, hashlib.sha256).digest()
def _keystream(key: bytes, nonce: bytes, block_index: int) -> bytes: def _keystream(key: bytes, nonce: bytes, block_index: int) -> bytes:
@@ -53,42 +67,90 @@ def _keystream(key: bytes, nonce: bytes, block_index: int) -> bytes:
def encrypt_value(secret_b64: str, plaintext: str) -> str: def encrypt_value(secret_b64: str, plaintext: str) -> str:
"""Encrypt a single string value with *secret_b64* (the ENV_VAR_SECRET). """Encrypt a single string value with *secret_b64* (the ENV_VAR_SECRET).
Returns a URL-safe base64 blob ``nonce || ciphertext`` suitable for Returns a URL-safe base64 authenticated blob suitable for
the ``bottled_agent_secrets.value`` column.""" the ``bottled_agent_secrets.value`` column."""
key = _b64dec(secret_b64) key = _b64dec(secret_b64)
encryption_key = _subkey(key, b"encryption")
authentication_key = _subkey(key, b"authentication")
pt = plaintext.encode() pt = plaintext.encode()
nonce = secrets.token_bytes(_NONCE_BYTES) nonce = secrets.token_bytes(_NONCE_BYTES)
ct = bytearray() ct = bytearray()
for i in range(0, len(pt), _BLOCK): for i in range(0, len(pt), _BLOCK):
chunk = pt[i : i + _BLOCK] chunk = pt[i : i + _BLOCK]
ks = _keystream(key, nonce, i)[: len(chunk)] ks = _keystream(encryption_key, nonce, i // _BLOCK)[: len(chunk)]
ct.extend(p ^ k for p, k in zip(chunk, ks)) ct.extend(p ^ k for p, k in zip(chunk, ks))
return base64.urlsafe_b64encode(nonce + bytes(ct)).rstrip(b"=").decode() authenticated = _VERSION + nonce + bytes(ct)
tag = hmac.new(authentication_key, authenticated, hashlib.sha256).digest()
return base64.urlsafe_b64encode(authenticated + tag).rstrip(b"=").decode()
def decrypt_value(secret_b64: str, blob_b64: str) -> str: def is_legacy_blob(blob_b64: str) -> bool:
"""Decrypt a blob produced by :func:`encrypt_value`. """Whether *blob_b64* uses the pre-authentication storage format."""
Returns the original plaintext string. Raises ``ValueError`` for malformed
input or a key mismatch (wrong key produces garbage, not an error, unless
the plaintext is non-UTF-8 treat all such failures as wrong key)."""
key = _b64dec(secret_b64)
try: try:
blob = _b64dec(blob_b64) return not _b64dec(blob_b64).startswith(_VERSION)
except Exception as exc: except (ValueError, TypeError):
raise ValueError(f"invalid ciphertext blob: {exc}") from exc return False
def _decrypt_legacy(key: bytes, blob: bytes) -> str:
"""Read the original ``nonce || ciphertext`` format for migration only."""
if len(blob) < _NONCE_BYTES: if len(blob) < _NONCE_BYTES:
raise ValueError("ciphertext blob too short") raise ValueError("ciphertext blob too short")
nonce, ciphertext = blob[:_NONCE_BYTES], blob[_NONCE_BYTES:] nonce, ciphertext = blob[:_NONCE_BYTES], blob[_NONCE_BYTES:]
pt = bytearray() pt = bytearray()
# The legacy format used the byte offset as the PRF counter.
for i in range(0, len(ciphertext), _BLOCK): for i in range(0, len(ciphertext), _BLOCK):
chunk = ciphertext[i : i + _BLOCK] chunk = ciphertext[i : i + _BLOCK]
ks = _keystream(key, nonce, i)[: len(chunk)] ks = _keystream(key, nonce, i)[: len(chunk)]
pt.extend(c ^ k for c, k in zip(chunk, ks)) pt.extend(c ^ k for c, k in zip(chunk, ks))
try: try:
return bytes(pt).decode() return bytes(pt).decode()
except UnicodeDecodeError as exc:
raise ValueError(f"decryption produced non-UTF-8 output: {exc}") from exc
def decrypt_value(secret_b64: str, blob_b64: str) -> str:
"""Decrypt a blob produced by :func:`encrypt_value`.
Returns the original plaintext string. Raises ``ValueError`` for malformed
input, authentication failure, or a key mismatch. Legacy unauthenticated
rows remain readable so callers can migrate them immediately."""
key = _b64dec(secret_b64)
try:
blob = _b64dec(blob_b64)
except (ValueError, TypeError) as exc:
raise ValueError(f"invalid ciphertext blob: {exc}") from exc
if not blob.startswith(_VERSION):
return _decrypt_legacy(key, blob)
minimum = len(_VERSION) + _NONCE_BYTES + _TAG_BYTES
if len(blob) < minimum:
raise ValueError("ciphertext blob too short")
authenticated, supplied_tag = blob[:-_TAG_BYTES], blob[-_TAG_BYTES:]
authentication_key = _subkey(key, b"authentication")
expected_tag = hmac.new(
authentication_key, authenticated, hashlib.sha256,
).digest()
if not hmac.compare_digest(supplied_tag, expected_tag):
raise ValueError("ciphertext authentication failed")
nonce_start = len(_VERSION)
nonce = blob[nonce_start : nonce_start + _NONCE_BYTES]
ciphertext = blob[nonce_start + _NONCE_BYTES : -_TAG_BYTES]
encryption_key = _subkey(key, b"encryption")
pt = bytearray()
for i in range(0, len(ciphertext), _BLOCK):
chunk = ciphertext[i : i + _BLOCK]
ks = _keystream(encryption_key, nonce, i // _BLOCK)[: len(chunk)]
pt.extend(c ^ k for c, k in zip(chunk, ks))
try:
return bytes(pt).decode()
except UnicodeDecodeError as exc: except UnicodeDecodeError as exc:
raise ValueError(f"decryption produced non-UTF-8 output (wrong key?): {exc}") from exc raise ValueError(f"decryption produced non-UTF-8 output (wrong key?): {exc}") from exc
__all__ = ["ENV_VAR_SECRET_NAME", "new_env_var_secret", "encrypt_value", "decrypt_value"] __all__ = [
"ENV_VAR_SECRET_NAME",
"new_env_var_secret",
"encrypt_value",
"decrypt_value",
"is_legacy_blob",
]
+3 -4
View File
@@ -40,7 +40,7 @@ from .paths import (
class ProvisioningError(RuntimeError): class ProvisioningError(RuntimeError):
"""A control-plane auth invariant would be violated (e.g. starting the """A control-plane auth invariant would be violated (e.g. starting the
orchestrator without its signing key which would run OPEN).""" orchestrator without its signing key)."""
@dataclass(frozen=True) @dataclass(frozen=True)
@@ -67,9 +67,8 @@ class TrustDomain:
def key_from_env(self, environ: Mapping[str, str] | None = None) -> str: def key_from_env(self, environ: Mapping[str, str] | None = None) -> str:
"""The signing key as the owning process sees it — read from `key_env` """The signing key as the owning process sees it — read from `key_env`
(default `os.environ`). "" when unset; the caller decides whether that is (default `os.environ`). ``""`` when unset; owning services reject that
fatal (`ControlPlaneProvisioning`) or the open-mode fallback value rather than start without authentication."""
(`OrchestratorServer`)."""
env = os.environ if environ is None else environ env = os.environ if environ is None else environ
return env.get(self.key_env, "").strip() return env.get(self.key_env, "").strip()
+6 -2
View File
@@ -3,6 +3,10 @@
- **Status:** Accepted - **Status:** Accepted
- **Date:** 2026-06-25 - **Date:** 2026-06-25
- **Deciders:** didericis - **Deciders:** didericis
- **Revised:** 2026-07-27 — thresholds relaxed (critical minimum 90→85%,
diff-coverage gate 90→80%) to cut low-value test churn on changed lines.
The risk-weighting structure and the "global is informational" rule are
unchanged.
## Context ## Context
@@ -34,7 +38,7 @@ a regression (Goodhart's law).
Coverage is **risk-weighted**, measured over the **combined unit + Coverage is **risk-weighted**, measured over the **combined unit +
integration** suites, with three rules: integration** suites, with three rules:
1. **Critical modules must remain ≥ 90%.** The curated security/logic core 1. **Critical modules must remain ≥ 85%.** The curated security/logic core
covers the host and gateway egress policy, manifest trust boundary, covers the host and gateway egress policy, manifest trust boundary,
git-gate enforcement, supervise protocol/server, YAML parser, and bottle git-gate enforcement, supervise protocol/server, YAML parser, and bottle
state. The concrete module list lives in `scripts/critical-modules.txt`; state. The concrete module list lives in `scripts/critical-modules.txt`;
@@ -55,7 +59,7 @@ integration** suites, with three rules:
The forward-looking guard is a **diff-coverage gate** The forward-looking guard is a **diff-coverage gate**
(`scripts/diff_coverage.py`): new/changed executable lines on a branch (`scripts/diff_coverage.py`): new/changed executable lines on a branch
must be ≥ 90% covered. This catches regressions where they are must be ≥ 80% covered. This catches regressions where they are
introduced without forcing a back-fill crusade through legacy glue. The introduced without forcing a back-fill crusade through legacy glue. The
gate skips lines in omitted files (there is no coverage data for them), gate skips lines in omitted files (there is no coverage data for them),
so the omit list cannot launder *new* logic into the dark: anything that so the omit list cannot launder *new* logic into the dark: anything that
@@ -56,8 +56,7 @@ key.
- Rewriting the HMAC primitive: `orchestrator_auth.mint/verify` gain an optional - Rewriting the HMAC primitive: `orchestrator_auth.mint/verify` gain an optional
`roles=` arg (default unchanged) so a key can carry a different role set; `roles=` arg (default unchanged) so a key can carry a different role set;
nothing else changes. nothing else changes.
- Network topology, the plane split (#469), or the server's open-mode fallback - Network topology or the plane split (#469).
for tests.
## Design ## Design
@@ -0,0 +1,197 @@
# PRD 0082: Authoritative failure boundaries
- **Status:** Draft
- **Author:** codex
- **Created:** 2026-07-27
- **Issue:** #444
## Summary
Make every security- or lifecycle-sensitive snapshot distinguish authoritative
empty state from unavailable state, and make every destructive or
resource-consuming boundary revalidate the assumptions it acts on. This
finishes the focused quality work begun under #444 without broad rewrites:
cleanup cannot act on stale identities, policy introspection cannot publish a
fabricated empty policy, gateway servers bound untrusted work, and daemon
shutdown does not emit uncaught background-thread failures. Shared
control-plane storage and gateway credential provisioning also enforce their
filesystem security contract before sensitive data is written.
## Problem
Several paths are individually fail-closed but compose into unsafe or
misleading behavior:
1. `cleanup` prepares a plan, waits indefinitely for operator confirmation,
then kills stored PIDs and removes stored paths without checking that those
identities still describe the same orphan. A PID may be reused or a run
directory may become active during the prompt.
2. The supervisor reuses egress's deny-all fallback for
`list-egress-routes`. Deny-all is correct for enforcement, but presenting it
as a successful empty route table can cause a later replace-all proposal to
discard live routes.
3. The supervisor and Git HTTP services accept bounded declared body sizes but
use blocking reads and unbounded request threads. An untrusted bottle can
exhaust the shared gateway with slow or parallel requests.
4. macOS cleanup enumerates containers and networks independently and treats a
failed query as an empty class, so a partial snapshot can still become a
destructive plan.
5. Gateway log-pump threads race stream closure during shutdown and emit
uncaught exceptions even when shutdown otherwise succeeds.
6. Firecracker discovers VMs through whitespace-split `pgrep -a` output.
A configured cache path containing spaces can hide a live VM from the
snapshot and make its run directory appear orphaned.
7. Docker cleanup asks compose for its project snapshot in best-effort mode.
A transient query failure can therefore become an empty stopped-project
set and authorize deletion of associated state directories.
8. Firecracker artifact downloads and registry publication have no network
deadline, so an unresponsive registry can hold setup or release work
indefinitely.
9. Authenticated secret blobs select the unauthenticated legacy decoder when
their in-band version prefix is changed, allowing storage tampering to
bypass tag verification.
10. Cleanup executes the entire post-confirmation snapshot rather than the
intersection with what the operator saw, and mutation failures are not
reflected in the command result.
11. Git smart-HTTP can retain sixteen 100 MiB request bodies concurrently,
cleanup mutations have no subprocess deadline, and Firecracker signalling
failures bypass shared mutation accounting.
12. SQLite creates the shared control-plane database before its mode is
restricted, then suppresses permission-repair failures. Gateway transports
also differ in whether copied deploy-key modes are preserved.
These are one design problem: state used to authorize deletion, replacement,
or resource allocation must be authoritative at the point of use.
## Goals / Success Criteria
- Cleanup never signals a PID or recursively deletes a path solely because it
appeared in a pre-confirmation snapshot.
- Firecracker cleanup proves immediately before action that a PID is still the
same Firecracker process and that a run directory is still orphaned.
- Firecracker process discovery reads NUL-delimited argv from `/proc`; paths
are never reconstructed from whitespace-delimited process listings.
- All backend cleanup discovery primitives raise a typed enumeration error on
operational failure. No backend may independently continue from a partial
snapshot.
- Shared cleanup control flow lives in the backend layer; concrete backends
override resource-specific discovery and validation primitives rather than
each implementing a bespoke failure policy.
- `list-egress-routes` returns an MCP error when attribution or policy
resolution is unavailable. A genuine, authoritatively resolved empty policy
remains a successful empty list.
- Supervisor and Git HTTP request bodies have total read deadlines, and each
service bounds concurrent request work. Limits apply to authenticated
callers because bottles themselves are untrusted.
- Gateway child-output pumping treats expected stream closure during shutdown
as completion while preserving diagnostics for unexpected failures.
- Artifact pull, existence-check, and publication requests use explicit
network deadlines.
- Persisted secrets accept only the authenticated format. The schema migration
intentionally clears legacy rows; local agents are reprovisioned rather
than retaining a ciphertext-controlled downgrade path.
- Cleanup executes only resources present in both the displayed and current
authoritative plans, attempts every approved mutation, and returns failure
when any mutation does not complete.
- Git request bodies spool to disk behind a separate heavy-work semaphore;
cleanup commands have configurable deadlines; Firecracker signalling
failures aggregate while identity-verification uncertainty still aborts.
- The shared database directory and file are private before SQLite writes any
control-plane state; an inability to enforce those modes aborts startup.
- Gateway credential directories and files receive explicit private modes
inside the gateway, independent of Docker, Apple Container, or SSH copy
semantics.
- Unit tests cover PID/path reuse, partial backend enumeration, transient
policy resolution failure, slow bodies, concurrency saturation, and stream
closure races.
## Non-goals
- Further decomposition solely to reduce module line counts.
- Replacing gateway stdlib HTTP services with a web framework.
- Changing egress matching, DLP decisions, proposal semantics, or backend
launch behavior beyond the synchronization required for safe cleanup.
- Making cleanup silently skip uncertain resources. Uncertainty is an
operator-visible failure.
## Design
### Shared backend control flow
Follow the backend architecture rule used by gateway attachment: shared
behavior lives above concrete backends; subclasses provide primitives, not
control flow.
Cleanup remains previewable, but confirmation authorizes a *new authoritative
evaluation*, not blind execution of the displayed object. The shared flow:
1. asks each available backend for a preview;
2. displays the union and asks for confirmation;
3. refreshes each non-empty backend plan;
4. validates destructive identities immediately before action;
5. aborts loudly if the refreshed plan or any identity cannot be proven safe.
Backend-specific primitives define how to identify a resource. Firecracker
uses process start identity plus canonical config/run paths; container
backends use authoritative CLI queries and stable resource names/labels.
Container engines expose destructive name-based commands without a portable
compare-and-delete operation. Cleanup therefore refreshes after confirmation
and requires every discovery query to succeed, minimizing but not claiming to
eliminate the final name-reuse race. A future engine-specific stable-ID
primitive may close that residual window without moving control flow back
into each backend.
### Enforcement state versus introspection state
Egress enforcement retains its deny-all fallback because uncertainty must not
grant network access. Supervisor introspection uses a strict resolver path:
unattributed callers and resolver failures become typed MCP errors, while a
successfully resolved policy containing zero routes returns `routes: []`.
### Gateway resource boundaries
Both stdlib servers set a per-connection body deadline before reading and use a
bounded request executor or semaphore. Saturated capacity fails quickly with a
service-unavailable response. Existing size caps remain independent:
supervisor proposals retain the 1 MiB cap and Git pack requests retain their
larger protocol-appropriate cap.
### Shutdown diagnostics
The gateway output pump catches only stream-closure exceptions expected after
the supervisor closes child pipes. Other I/O failures remain visible and are
reported through the supervisor's normal diagnostic channel.
### Shared filesystem security
The common SQLite store owns database creation for every backend. It creates
the parent directory and an empty database with private modes before opening
SQLite, repairs existing modes, verifies the resulting state, and propagates
every enforcement failure. Backend launchers do not duplicate this policy.
The backend-neutral gateway provisioner likewise applies directory and file
modes after transport copies complete. This avoids relying on copy behavior
that differs among Docker, Apple Container, and Firecracker's SSH transport.
## Implementation chunks
1. Existing fail-closed security and backend enumeration fixes.
2. FastAPI orchestrator transport and bounded control-plane bodies.
3. Egress request-policy and outbound-DLP pipeline extraction.
4. Supervisor MCP dispatch extraction.
5. Shared cleanup refresh/revalidation plus authoritative macOS discovery.
6. Strict supervisor introspection and bounded supervisor/Git HTTP work.
7. Gateway shutdown log-pump closure handling.
8. Lossless Firecracker process identities, authoritative Docker cleanup
queries, and bounded Firecracker artifact transfers.
9. Mandatory authenticated secret storage, shared cleanup-plan intersection
and mutation accounting, and contained Git backend process failures.
10. Disk-spooled and separately bounded Git bodies, cleanup command deadlines,
and classified Firecracker signalling failures.
11. Fail-closed shared database creation and backend-neutral gateway credential
permissions.
## Open questions
None.
-273
View File
@@ -1,273 +0,0 @@
# PRD prd-new: Host control server
- **Status:** Draft
- **Author:** Claude
- **Created:** 2026-07-26
- **Issue:** #468
## Summary
Promote the in-process launch broker into a standalone **host control
server**: the single privileged component on the host. Both the CLI and the
orchestrator drive it over HTTP; it brokers agent launches, owns the
orchestrator's own lifecycle, and is the sole writer of host-durable state (the
tamper-evident audit record). This closes the three gaps between today's
well-formed broker *contract* ([`orchestrator/broker.py`](../../bot_bottle/orchestrator/broker.py))
and a real out-of-process service — transport, durable provisioned secret,
and a disciplined op vocabulary — and splits host state by
owner and lifetime. The prize: **the CLI no longer needs the Docker socket**,
which is what finally lets a dedicated Gitea runner user drop the
root-equivalent `docker` group (PRD 0070, "Relationship to other work").
## Problem
Container launches run directly from a short-lived CLI process against the
Docker socket. That socket is root-equivalent, so every host that launches
bottles hands root to whoever invokes the CLI — including a CI runner user we
want to keep unprivileged. PRD 0070 already argues for replacing the fat socket
with a **thin, structured, auditable** launch broker, and the contract for that
broker exists and is tested in-process. But it is *only* in-process:
`LaunchBroker.submit(token)` is a method call from
`OrchestratorCore.launch_bottle` ([`service.py:116`](../../bot_bottle/orchestrator/service.py)),
and `DockerBroker` is on no production path — every backend starts the
orchestrator with `--broker stub` ([`__main__.py:54`](../../bot_bottle/orchestrator/__main__.py)).
Three gaps stand between that scaffold and a host service:
1. **No transport.** `submit` is an in-process call. A real service needs a
`BrokerClient` that POSTs the signed token and a host-side HTTP server that
verifies and acts.
2. **The signing secret is ephemeral and self-generated.**
[`__main__.py:53`](../../bot_bottle/orchestrator/__main__.py) does
`secrets.token_bytes(32)` and hands the *same value* to signer and verifier —
viable only because they share a process. A separate daemon needs the secret
provisioned out of band and durable across orchestrator restarts.
3. **The op vocabulary is `launch` / `teardown` only.** Everything else
host-privileged still lives in the CLI, so the schema has to grow — carefully,
since PRD 0070's security argument rests on "structured requests only, static
flags + ids."
Separately, host state has no clear owner. `OrchestratorCore.reconcile` takes
`live_source_ips` as a parameter *only because the orchestrator cannot see the
backend* ([`service.py:137`](../../bot_bottle/orchestrator/service.py)); the
egress traffic log is written to the container's stderr; and there is no durable,
tamper-evident home for the audit record that survives orchestrator destruction.
## Goals / Success Criteria
- A standalone host control server that the CLI and orchestrator reach over
**HTTP**, with three entry paths working end to end:
- `web console -(iroh)-> orchestrator -(http)-> host controller -> launch`
- `cli -(http)-> orchestrator -(http)-> host controller -> launch`
- `cli -(http)-> host controller` — start / restart / status of the
orchestrator **itself** (the bootstrap/recovery path #391 targets).
- The launch op is expressed as a **signed JWT of static flags + ids only**,
verified against a closed schema.
- The signing secret is **provisioned out of band and durable** across
orchestrator restarts (a `TrustDomain` per #476, with a key the orchestrator
never holds for the host controller's *own* endpoints).
- Host-privileged operations move off the CLI to the control server; **the CLI
no longer opens the Docker socket** for bottle operations.
- `Orchestrator.reconcile` no longer takes `live_source_ips` — live-bottle
enumeration becomes an internal control-server call.
- Host-durable state lands as an **append-only, hash-chained JSONL** audit log
owned solely by the host controller; operational state stays SQLite owned
solely by the orchestrator.
## Non-goals
- **Removing standing privilege.** This converts on-demand privilege (a CLI the
user invokes) into standing privilege (a daemon under launchd/systemd). The
win is that the privilege is *narrower* (structured requests vs. a raw socket),
not that it disappears. "Always running" is an accepted new property.
- **Asymmetric signing.** We stay HS256 — see Design / "Signing stays
symmetric."
- **Integrity against a live compromised orchestrator.** Host-location of the
audit log does not buy this: the orchestrator makes the decisions being audited
and can forge or omit entries wherever the file lives. An off-box copy is the
answer, tracked separately.
- **A single unified DB for all state.** Impossible over a guest-kernel share
(SQLite locking is not coherent); state is split by owner and lifetime instead.
- **The generic `SecretProvider` (#355)** and **remote terminal design (#478)**
both ride the same door but are their own work.
## Design
### Topology
The host controller is the sole privileged component. The orchestrator becomes a
client of it for launches, and the CLI becomes a client of it for *both* bottle
operations (indirectly, through the orchestrator) and orchestrator lifecycle
(directly, for bootstrap/recovery — startup can't route through the thing being
started).
```
web console ─(iroh)─▶ orchestrator ─┐
├─(http, signed JWT)─▶ host controller ─▶ launch
cli ────────(http)──▶ orchestrator ─┘
cli ────────(http, bearer)──────────────────────────────▶ host controller (orchestrator lifecycle)
```
### Transport: `BrokerClient` + host server
`LaunchBroker.submit(token)` keeps its exact signature and semantics; only the
*wire* changes. A new `BrokerClient` implements the same submit contract by
POSTing the signed token to the host controller (stdlib `urllib`, like the
existing [`orchestrator/client.py`](../../bot_bottle/orchestrator/client.py)),
and the host controller's launch handler is the existing `verify_request` +
`_launch`/`_teardown` path, now reached over HTTP instead of a method call. The
in-process `StubBroker` stays for the dev-harness and tests; `DockerBroker`'s
`_launch`/`_teardown` bodies move behind the server unchanged. Because the client
satisfies the same interface `OrchestratorCore` already depends on, the core does
not change to gain a real backend.
### Signing stays symmetric (HS256)
PRD 0070 nominally specifies asymmetric; the code is HS256 and we keep it.
Asymmetric matters when the verifier is *less* privileged than the signer — here
it is the reverse: the host controller (verifier) is strictly more privileged
than the orchestrator (signer), and a controller that could forge orchestrator
requests gains nothing, since it is already the component that launches. Staying
symmetric also honors the no-runtime-deps policy (stdlib has no Ed25519). This
matches the reasoning already inlined in `broker.py`'s module docstring.
### Replay protection is out of scope (tracked in #494)
Once the launch token travels over a wire, a captured token could be replayed —
`sign_request` already emits `jti`/`iat` but `verify_request` reads neither, so
there is no expiry window or `jti` cache today. Enforcing that (an `iat` window +
a self-trimming `jti` cache) is a pure in-process change that lands independently
of this work, and it is deferred to **#494** rather than gating the MVP of the
host control server. Nothing here depends on it; it can merge before or after.
### Op vocabulary and the "ids + static flags" rule (gap 3)
Each op moved off the CLI widens the privileged surface, so growth is governed by
one explicit rule, enforced in `verify_request`'s schema check:
> A broker op carries **only ids and enumerated static flags** — a bottle id, a
> pool slot, a **content-addressed** image ref chosen from a fixed set, an op
> name from a closed vocabulary. Never a free-form path, argv, command, or
> caller-supplied filesystem location. If an operation cannot be expressed that
> way, it does not become a broker op.
Operations that fit and move off the CLI (all today in
`backend/*/consolidated_launch.py`, driven by a short-lived CLI process):
| Op | What it does | Fits the rule because |
|---|---|---|
| `launch` / `teardown` | existing | ids + slot + image ref |
| `orchestrator.ensure_running` | start the infra container | no arguments |
| `orchestrator.{start,restart,status}` | lifecycle (the #391 path) | no arguments |
| `list_live` | enumerate running bottles for reconcile | no arguments; returns ids/IPs |
| `allocate_ip` | `next_free_ip` over `_network_container_ips` | no arguments; returns an IP |
| `provision_git_gate` | `cp`/`exec` a per-bottle deploy key into the gateway | bottle id + key handle, no path |
| `reprovision` | `docker exec printenv <ENV_VAR_SECRET>` on a live agent | bottle id + secret *name* |
Image **builds** stay with the orchestrator for v1 (PRD 0070 §Memory: builds run
control-plane-side; a dedicated slim build unit is later, #468-adjacent), so no
`build` broker op is added here.
With `list_live` as an internal control-server call, `Orchestrator.reconcile`'s
`live_source_ips` parameter goes away — the tell PRD 0070 called out that the
orchestrator couldn't see the backend disappears with it.
### Secret provisioning (gap 2)
The shared HS256 secret becomes a durable, out-of-band artifact via the
**`TrustDomain`** seam (#476,
[`trust_domain.py`](../../bot_bottle/trust_domain.py)):
- The **launch-broker secret** is a `TrustDomain` whose key
(`host_signing_key(<file>)`, minted 0600 on first use, durable under
`bot_bottle_root()`) is provisioned to the orchestrator (signer) and the host
controller (verifier). Durability across orchestrator restarts is what makes
re-adoption work — a restart re-verifies against the same key.
- The **host controller's own lifecycle endpoints** (the direct `cli -> host
controller` path) get a **separate** `TrustDomain` key the orchestrator never
holds — exactly the second domain #476's PRD reserves. The orchestrator must
not be able to mint the credentials used to start and stop it.
This reuses the seam #476 landed rather than re-deriving provisioning per
backend (the PR #471 bug class).
### One daemon, structurally separate handlers (open decision 1)
The audit writer and the broker live in **one daemon** for install simplicity,
but with **no shared parsing** and **different credentials per handler**:
- the **launch** handler requires the signed launch **JWT** (provenance +
un-coercible schema);
- the **audit-append** handler takes a plain **bearer token** and writes to the
JSONL log.
This does not defend against orchestrator compromise (it holds both creds) — it
stops a bug in the boring audit path from reaching the privileged launch path.
The launcher stays small enough to audit line-by-line, per PRD 0070.
### State ownership: split by owner and lifetime
A single mounted DB is impossible — SQLite locking is not coherent across guest
kernels over a share, which is why the macOS backend already uses a container-only
volume (`INFRA_DB_VOLUME`). So state splits three ways (depends on #469, which
gets `bot-bottle.db` off the data plane first):
| Owner | State | Home | Shape |
|---|---|---|---|
| **Orchestrator** | `orchestrator_bottles` registry; `bottled_agent_secrets` (encrypted egress tokens); `supervise_proposals` / `supervise_responses` | volume nothing else mounts (generalizing the macOS design) | **SQLite** — mutable, transactional, queried |
| **Host controller** | supervise audit entries; egress traffic log (today → container stderr); host-side config | host filesystem, survives orchestrator/volume destruction | **JSONL** — append-only |
| **Gateway** | none | — | after #469 the data plane holds no DB state |
The historical record is **JSONL, not SQLite**, because it is append-only, never
updated, never transactionally queried: `O_APPEND` writes are atomic, there is no
locking protocol to get wrong, hash-chaining for tamper-evidence is cheap, and it
survives container-runtime volume pruning (the #450 lesson) and stays readable
without the orchestrator running. Both halves of "the audit record" — supervise
decisions and the egress traffic log — land in the one place.
The orchestrator is **sole mounter and sole writer** of its SQLite volume; the
host controller is **sole writer** of the JSONL log, over the authenticated
audit-append channel.
## Implementation chunks
Ordered, each independently mergeable:
1. **`BrokerClient` + host launch server** over HTTP, reusing `verify_request`
and the existing `DockerBroker` bodies. Wire `OrchestratorCore` to a
`BrokerClient` behind a flag; keep `StubBroker` for the dev-harness. Closes
gap 1.
2. **Durable secret via `TrustDomain`** — provision the launch-broker key to
signer + verifier; add the host controller's own lifecycle `TrustDomain`.
Closes gap 2.
3. **Grow the op vocabulary** one op at a time (`list_live` first — it also
removes `reconcile`'s `live_source_ips`), each behind the ids + static-flags
rule. Closes gap 3.
4. **JSONL audit log** — the host-controller-owned, hash-chained historical
record with the plain-bearer audit-append handler; redirect the egress traffic
log into it.
5. **Drop the Docker socket from the CLI** once every host-privileged op it used
is a broker op — the payoff that unblocks the unprivileged Gitea runner user.
## Open questions
1. **Schema-width rule enforcement.** The "ids + static flags" rule is stated;
should `verify_request` reject unknown claim keys outright (strict schema) to
keep the surface from drifting? Leaning yes.
2. **Audit-append back-pressure.** What the audit handler does if the JSONL sink
is unavailable (fail-closed vs. buffer) — resolve before shipping chunk 5.
## References
- **PRD 0070** — the contract, the launch broker, and the state tiers this
implements.
- **#469** — get `bot-bottle.db` off the data plane (lands underneath this).
- **#476** ([`prd-new-control-plane-auth-provisioning`](prd-new-control-plane-auth-provisioning.md))
— the `TrustDomain` seam this plugs the host controller's key into.
- **#391** — backend-agnostic orchestrator restart (the bootstrap path).
- **#494** — enforce broker replay protection (`iat` window + `jti` cache); split
out of this PRD as an independent in-process change.
- **#386** — prebuilt images from the Gitea OCI registry (the fixed image set the
broker validates against).
- **#355** — generic `SecretProvider`.
- **#478** — remote terminal design.
+5 -5
View File
@@ -13,7 +13,7 @@
# are re-executed; no KVM or Docker dependency. # are re-executed; no KVM or Docker dependency.
# #
# Pass "critical" as the last argument in either mode to also report just the # Pass "critical" as the last argument in either mode to also report just the
# critical modules (ADR 0004 target: 90%). # critical modules (ADR 0004 target: 85%).
set -euo pipefail set -euo pipefail
cd "$(dirname "$0")/.." cd "$(dirname "$0")/.."
@@ -34,8 +34,8 @@ if [ "${1:-}" = "aggregate" ]; then
"$PY" -m coverage report -m "$PY" -m coverage report -m
if [ "${2:-}" = "critical" ]; then if [ "${2:-}" = "critical" ]; then
echo "== critical modules (ADR 0004 minimum: 90%) ==" >&2 echo "== critical modules (ADR 0004 minimum: 85%) ==" >&2
"$PY" -m coverage report --include="$CRITICAL" --fail-under=90 "$PY" -m coverage report --include="$CRITICAL" --fail-under=85
fi fi
exit 0 exit 0
fi fi
@@ -55,6 +55,6 @@ echo "== combined report ==" >&2
"$PY" -m coverage report -m "$PY" -m coverage report -m
if [ "${1:-}" = "critical" ]; then if [ "${1:-}" = "critical" ]; then
echo "== critical modules (ADR 0004 minimum: 90%) ==" >&2 echo "== critical modules (ADR 0004 minimum: 85%) ==" >&2
"$PY" -m coverage report --include="$CRITICAL" --fail-under=90 "$PY" -m coverage report --include="$CRITICAL" --fail-under=85
fi fi
+1 -1
View File
@@ -1,4 +1,4 @@
# Critical security/logic core held to the >=90% coverage bar by # Critical security/logic core held to the >=85% coverage bar by
# docs/decisions/0004-coverage-policy.md. # docs/decisions/0004-coverage-policy.md.
# #
# SINGLE SOURCE OF TRUTH: scripts/coverage.sh (the `critical` report) and # SINGLE SOURCE OF TRUTH: scripts/coverage.sh (the `critical` report) and
+3 -3
View File
@@ -13,8 +13,8 @@ policy.
Usage: Usage:
scripts/coverage.sh # produce .coverage first scripts/coverage.sh # produce .coverage first
python3 scripts/diff_coverage.py # gate against origin/main, min 90% python3 scripts/diff_coverage.py # gate against origin/main, min 80%
python3 scripts/diff_coverage.py --base main --min 85 python3 scripts/diff_coverage.py --base main --min 75
""" """
from __future__ import annotations from __future__ import annotations
@@ -74,7 +74,7 @@ def main() -> int:
ap = argparse.ArgumentParser() ap = argparse.ArgumentParser()
ap.add_argument("--base", default="origin/main", ap.add_argument("--base", default="origin/main",
help="git ref to diff against (default: origin/main)") help="git ref to diff against (default: origin/main)")
ap.add_argument("--min", type=float, default=90.0, ap.add_argument("--min", type=float, default=80.0,
help="minimum %% of changed executable lines covered") help="minimum %% of changed executable lines covered")
args = ap.parse_args() args = ap.parse_args()
+14
View File
@@ -18,6 +18,7 @@ from bot_bottle.backend.docker.compose import (
list_compose_projects, list_compose_projects,
slug_from_compose_project, slug_from_compose_project,
) )
from bot_bottle.backend import EnumerationError
class TestProjectNaming(unittest.TestCase): class TestProjectNaming(unittest.TestCase):
@@ -69,6 +70,19 @@ class TestComposeProjectListing(unittest.TestCase):
self.assertEqual([], list_active_slugs(warn_on_error=False)) self.assertEqual([], list_active_slugs(warn_on_error=False))
warn.assert_not_called() warn.assert_not_called()
def test_compose_ls_error_can_be_raised_for_enumeration(self):
with mock.patch(
"bot_bottle.backend.docker.compose.subprocess.run",
return_value=subprocess.CompletedProcess(
args=["docker"], returncode=1, stdout="", stderr="no daemon",
),
):
with self.assertRaisesRegex(EnumerationError, "no daemon"):
list_active_slugs(
warn_on_error=False,
raise_on_error=True,
)
if __name__ == "__main__": if __name__ == "__main__":
unittest.main() unittest.main()
+12
View File
@@ -7,6 +7,8 @@ from pathlib import Path
from unittest.mock import MagicMock, Mock, patch from unittest.mock import MagicMock, Mock, patch
from bot_bottle.backend.docker.consolidated_launch import ( from bot_bottle.backend.docker.consolidated_launch import (
ConsolidatedLaunchError,
_network_container_ips,
launch_consolidated, launch_consolidated,
deprovision_consolidated, deprovision_consolidated,
) )
@@ -86,6 +88,16 @@ class TestLaunchConsolidated(unittest.TestCase):
client.teardown_bottle.assert_called_once_with("b1") # no orphan left client.teardown_bottle.assert_called_once_with("b1") # no orphan left
class TestNetworkContainerIps(unittest.TestCase):
def test_fails_closed_when_network_inspection_fails(self) -> None:
result = Mock(returncode=1, stdout="", stderr="daemon unavailable")
with (
patch(f"{_MOD}.run_docker", return_value=result),
self.assertRaisesRegex(ConsolidatedLaunchError, "daemon unavailable"),
):
_network_container_ips("bot-bottle-gateway")
class TestTeardownConsolidated(unittest.TestCase): class TestTeardownConsolidated(unittest.TestCase):
def test_deregisters_and_deprovisions(self) -> None: def test_deregisters_and_deprovisions(self) -> None:
client = Mock() client = Mock()
@@ -19,13 +19,16 @@ of issue #77 — the dashboard now delegates to this layer.
from __future__ import annotations from __future__ import annotations
import subprocess
import tempfile import tempfile
import unittest import unittest
from pathlib import Path from pathlib import Path
from unittest.mock import patch
from tests.unit import use_bottle_root from tests.unit import use_bottle_root
from bot_bottle import bottle_state from bot_bottle import bottle_state
from bot_bottle.backend.docker import enumerate as _enumerate from bot_bottle.backend.docker import enumerate as _enumerate
from bot_bottle.backend import EnumerationError
class TestParseServicesByProject(unittest.TestCase): class TestParseServicesByProject(unittest.TestCase):
@@ -72,6 +75,18 @@ class TestParseServicesByProject(unittest.TestCase):
self.assertEqual({"bot-bottle-dev-abc": {"egress"}}, out) self.assertEqual({"bot-bottle-dev-abc": {"egress"}}, out)
class TestQueryServicesByProject(unittest.TestCase):
def test_docker_ps_failure_is_not_an_empty_result(self):
with patch(
"bot_bottle.backend.docker.enumerate.subprocess.run",
return_value=subprocess.CompletedProcess(
args=["docker"], returncode=1, stdout="", stderr="daemon down",
),
):
with self.assertRaisesRegex(EnumerationError, "daemon down"):
_enumerate._query_services_by_project()
class _FakeHomeMixin: class _FakeHomeMixin:
def _setup_fake_home(self) -> None: def _setup_fake_home(self) -> None:
self._tmp = tempfile.TemporaryDirectory(prefix="enum-active.") self._tmp = tempfile.TemporaryDirectory(prefix="enum-active.")
+29 -1
View File
@@ -15,6 +15,7 @@ import unittest
from pathlib import Path from pathlib import Path
from unittest.mock import patch from unittest.mock import patch
from bot_bottle.backend import EnumerationError
from bot_bottle.backend.firecracker import cleanup as fc_cleanup from bot_bottle.backend.firecracker import cleanup as fc_cleanup
from bot_bottle.backend.firecracker.bottle_cleanup_plan import ( from bot_bottle.backend.firecracker.bottle_cleanup_plan import (
FirecrackerBottleCleanupPlan, FirecrackerBottleCleanupPlan,
@@ -58,10 +59,37 @@ class TestProcessScan(unittest.TestCase):
self.assertEqual({str(run_root / "live-a")}, live) self.assertEqual({str(run_root / "live-a")}, live)
self.assertEqual([222], orphan_pids) self.assertEqual([222], orphan_pids)
def test_scan_empty_when_pgrep_fails(self): def test_scan_empty_when_pgrep_finds_no_processes(self):
with patch.object(fc_cleanup.subprocess, "run", return_value=_proc(returncode=1)): with patch.object(fc_cleanup.subprocess, "run", return_value=_proc(returncode=1)):
self.assertEqual((set(), []), fc_cleanup._scan_processes(Path("/x"))) self.assertEqual((set(), []), fc_cleanup._scan_processes(Path("/x")))
def test_scan_raises_when_pgrep_errors(self):
proc = subprocess.CompletedProcess(
[], 2, stdout="", stderr="invalid process expression",
)
with (
patch.object(fc_cleanup.subprocess, "run", return_value=proc),
self.assertRaisesRegex(EnumerationError, "invalid process expression"),
):
fc_cleanup._scan_processes(Path("/x"))
def test_prepare_cleanup_does_not_plan_deletions_when_scan_errors(self):
with tempfile.TemporaryDirectory() as tmp:
run_root = Path(tmp)
live = run_root / "live-a"
live.mkdir()
with (
patch.object(fc_cleanup, "_run_root", return_value=run_root),
patch.object(
fc_cleanup.subprocess,
"run",
return_value=_proc(returncode=2),
),
self.assertRaises(EnumerationError),
):
fc_cleanup.prepare_cleanup()
self.assertTrue(live.is_dir())
def test_live_run_dirs_returns_paths_in_stable_order(self): def test_live_run_dirs_returns_paths_in_stable_order(self):
with patch.object(fc_cleanup, "_run_root", return_value=Path("/run")), \ with patch.object(fc_cleanup, "_run_root", return_value=Path("/run")), \
patch.object(fc_cleanup, "_scan_processes", patch.object(fc_cleanup, "_scan_processes",
+64
View File
@@ -0,0 +1,64 @@
"""Unit tests for Firecracker active-agent enumeration."""
from __future__ import annotations
import unittest
from pathlib import Path
from unittest.mock import patch
from bot_bottle.backend import EnumerationError
from bot_bottle.backend.firecracker import enumerate as fc_enumerate
from bot_bottle.bottle_state import BottleMetadata
class TestEnumerateActive(unittest.TestCase):
def test_maps_live_run_dirs_to_active_agents(self) -> None:
metadata = BottleMetadata(
identity="dev-a",
agent_name="claude",
cwd="",
copy_cwd=False,
started_at="2026-07-26T12:00:00Z",
label="review",
color="blue",
)
with (
patch.object(
fc_enumerate, "live_run_dirs",
return_value=(Path("/cache/run/dev-a"),),
),
patch.object(fc_enumerate, "read_metadata", return_value=metadata),
):
agents = fc_enumerate.enumerate_active()
self.assertEqual(1, len(agents))
self.assertEqual("firecracker", agents[0].backend_name)
self.assertEqual("dev-a", agents[0].slug)
self.assertEqual("claude", agents[0].agent_name)
self.assertEqual("review", agents[0].label)
self.assertEqual((), agents[0].services)
def test_missing_metadata_uses_safe_defaults(self) -> None:
with (
patch.object(
fc_enumerate, "live_run_dirs",
return_value=(Path("/cache/run/dev-a"),),
),
patch.object(fc_enumerate, "read_metadata", return_value=None),
):
agent = fc_enumerate.enumerate_active()[0]
self.assertEqual("?", agent.agent_name)
self.assertEqual("", agent.started_at)
def test_process_scan_failure_propagates(self) -> None:
with (
patch.object(
fc_enumerate, "live_run_dirs",
side_effect=EnumerationError("pgrep failed"),
),
self.assertRaisesRegex(EnumerationError, "pgrep failed"),
):
fc_enumerate.enumerate_active()
if __name__ == "__main__":
unittest.main()
+10 -1
View File
@@ -5,6 +5,7 @@ from __future__ import annotations
import unittest import unittest
from unittest.mock import patch from unittest.mock import patch
from bot_bottle.backend import EnumerationError
from bot_bottle.backend.macos_container import cleanup, enumerate as enum_mod from bot_bottle.backend.macos_container import cleanup, enumerate as enum_mod
from bot_bottle.backend.macos_container.bottle_cleanup_plan import ( from bot_bottle.backend.macos_container.bottle_cleanup_plan import (
MacosContainerBottleCleanupPlan, MacosContainerBottleCleanupPlan,
@@ -69,10 +70,18 @@ class TestMacosContainerEnumerate(unittest.TestCase):
self.assertEqual(["dev-abc"], [a.slug for a in agents]) self.assertEqual(["dev-abc"], [a.slug for a in agents])
def test_raises_when_the_cli_fails(self): def test_raises_when_the_cli_fails(self):
from bot_bottle.backend.macos_container.enumerate import EnumerationError
with self.assertRaises(EnumerationError): with self.assertRaises(EnumerationError):
self._enumerate("", returncode=1) self._enumerate("", returncode=1)
def test_raises_typed_error_when_cli_is_missing(self):
with (
patch.object(
enum_mod.subprocess, "run", side_effect=FileNotFoundError,
),
self.assertRaisesRegex(EnumerationError, "CLI not found"),
):
enum_mod.enumerate_active()
if __name__ == "__main__": if __name__ == "__main__":
unittest.main() unittest.main()
+82
View File
@@ -248,6 +248,88 @@ class TestDockerGateway(unittest.TestCase):
calls, calls,
) )
def test_ensure_running_replaces_poisoned_ipv6_network(self) -> None:
# A daemon that default-enables IPv6 leaves the gateway network with a
# malformed fdd0::/64 gateway, so `docker network inspect` exits
# non-zero with a ParseAddr error (not "No such network"). `--ipv6=false`
# can't heal an already-poisoned network — the create just no-ops on
# "already exists" — so _ensure_network must force-remove and recreate
# it, else every later subnet read keeps failing.
calls: list[list[str]] = []
def fake(argv: list[str], **_kw: object) -> Mock:
calls.append(argv)
if argv[:2] == ["docker", "ps"]:
return _proc(stdout="")
if argv[:3] == ["docker", "network", "inspect"]:
return _proc(
returncode=1,
stderr='ParseAddr("fdd0:0:0:4::1/64"): unexpected character, '
'want colon (at "/64")',
)
return _proc()
with patch(_RUN_DOCKER, side_effect=fake):
self.sc.connect_to_orchestrator(_ORCH_URL, _TOKEN)
self.assertIn(["docker", "rm", "--force", self.sc.name], calls)
self.assertIn(["docker", "network", "rm", self.sc.network], calls)
creates = [c for c in calls if c[:3] == ["docker", "network", "create"]]
self.assertEqual(
[[
"docker", "network", "create",
"--ipv6=false",
"--subnet", DEFAULT_GATEWAY_SUBNET,
"--label",
f"bot-bottle.gateway-subnet={DEFAULT_GATEWAY_SUBNET}",
self.sc.network,
]],
creates,
)
def test_ensure_running_creates_network_when_inspect_reports_absent(self) -> None:
# The absent case (inspect fails with "No such network") must NOT try to
# remove anything — it just creates. Guards the poisoned-vs-absent split.
calls: list[list[str]] = []
def fake(argv: list[str], **_kw: object) -> Mock:
calls.append(argv)
if argv[:3] == ["docker", "network", "inspect"]:
return _proc(returncode=1, stderr="Error: No such network: x")
return _proc(stdout="") if argv[:2] == ["docker", "ps"] else _proc()
with patch(_RUN_DOCKER, side_effect=fake):
self.sc.connect_to_orchestrator(_ORCH_URL, _TOKEN)
self.assertNotIn(["docker", "network", "rm", self.sc.network], calls)
creates = [c for c in calls if c[:3] == ["docker", "network", "create"]]
self.assertEqual(1, len(creates))
def test_ensure_running_does_not_destroy_on_generic_inspect_error(self) -> None:
# A generic inspect failure (daemon hiccup, permission, timeout) is NOT
# evidence of a poisoned network. Only the ParseAddr poison signature may
# take the destructive heal path; anything else must surface as an error
# without tearing down a possibly-healthy shared gateway.
calls: list[list[str]] = []
def fake(argv: list[str], **_kw: object) -> Mock:
calls.append(argv)
if argv[:2] == ["docker", "ps"]:
return _proc(stdout="")
if argv[:3] == ["docker", "network", "inspect"]:
return _proc(
returncode=1,
stderr="Cannot connect to the Docker daemon at unix:///var/run/docker.sock",
)
return _proc()
with patch(_RUN_DOCKER, side_effect=fake):
with self.assertRaises(GatewayError):
self.sc.connect_to_orchestrator(_ORCH_URL, _TOKEN)
# No mutation of the shared gateway: neither the container nor the
# network is removed, and nothing is recreated.
self.assertNotIn(["docker", "network", "rm", self.sc.network], calls)
self.assertFalse(any(c[:3] == ["docker", "rm", "--force"] for c in calls))
self.assertEqual([], [c for c in calls if c[:3] == ["docker", "network", "create"]])
def test_ca_cert_pem_reads_from_container(self) -> None: def test_ca_cert_pem_reads_from_container(self) -> None:
with patch(_RUN_DOCKER, return_value=_proc(stdout=_CA_PEM)) as m: with patch(_RUN_DOCKER, return_value=_proc(stdout=_CA_PEM)) as m:
self.assertEqual(_CA_PEM, self.sc.ca_cert_pem()) self.assertEqual(_CA_PEM, self.sc.ca_cert_pem())
+25 -9
View File
@@ -2,6 +2,9 @@
from __future__ import annotations from __future__ import annotations
import base64
import hashlib
import hmac
import unittest import unittest
from bot_bottle.orchestrator.store.secret_store import ( from bot_bottle.orchestrator.store.secret_store import (
@@ -68,15 +71,28 @@ class TestDecryptErrors(unittest.TestCase):
def test_wrong_key_raises_value_error(self) -> None: def test_wrong_key_raises_value_error(self) -> None:
ct = encrypt_value(self.secret, "secret-token") ct = encrypt_value(self.secret, "secret-token")
other_key = new_env_var_secret() other_key = new_env_var_secret()
# Wrong key produces garbage bytes; decrypt_value raises ValueError with self.assertRaisesRegex(ValueError, "authentication failed"):
# when the result is non-UTF-8 (which is very likely for 12-char data). decrypt_value(other_key, ct)
# We allow it to succeed only if garbage happens to be valid UTF-8, but
# the plaintext must not match. def test_tampered_ciphertext_raises_value_error(self) -> None:
try: raw = bytearray(base64.urlsafe_b64decode(
result = decrypt_value(other_key, ct) encrypt_value(self.secret, "secret-token") + "=="
self.assertNotEqual("secret-token", result) ))
except ValueError: raw[22] ^= 1
pass tampered = base64.urlsafe_b64encode(raw).rstrip(b"=").decode()
with self.assertRaisesRegex(ValueError, "authentication failed"):
decrypt_value(self.secret, tampered)
def test_reads_legacy_ciphertext_for_migration(self) -> None:
key = base64.urlsafe_b64decode(self.secret + "==")
nonce = b"0123456789abcdef"
plaintext = b"legacy-token"
stream = hmac.new(
key, nonce + (0).to_bytes(4, "big"), hashlib.sha256,
).digest()
ciphertext = bytes(p ^ k for p, k in zip(plaintext, stream))
legacy = base64.urlsafe_b64encode(nonce + ciphertext).rstrip(b"=").decode()
self.assertEqual("legacy-token", decrypt_value(self.secret, legacy))
def test_truncated_blob_raises_value_error(self) -> None: def test_truncated_blob_raises_value_error(self) -> None:
with self.assertRaises(ValueError): with self.assertRaises(ValueError):
+70 -18
View File
@@ -7,6 +7,7 @@ server tests), plus one real-socket round-trip to prove the handler wiring.
from __future__ import annotations from __future__ import annotations
import base64 import base64
import http.client
import io import io
import json import json
import secrets import secrets
@@ -22,7 +23,7 @@ from unittest.mock import MagicMock, patch
from bot_bottle.orchestrator_auth import ROLE_CLI, ROLE_GATEWAY, mint from bot_bottle.orchestrator_auth import ROLE_CLI, ROLE_GATEWAY, mint
from bot_bottle.orchestrator.broker import StubBroker from bot_bottle.orchestrator.broker import StubBroker
from bot_bottle.orchestrator.server import dispatch, make_server from bot_bottle.orchestrator.server import MAX_BODY_BYTES, dispatch, make_server
from bot_bottle.orchestrator.store.registry_store import BottleRecord, RegistryStore from bot_bottle.orchestrator.store.registry_store import BottleRecord, RegistryStore
from bot_bottle.orchestrator.service import OrchestratorCore from bot_bottle.orchestrator.service import OrchestratorCore
from bot_bottle.orchestrator.store.store_manager import StoreManager from bot_bottle.orchestrator.store.store_manager import StoreManager
@@ -251,11 +252,51 @@ class TestDispatch(unittest.TestCase):
class TestServerRoundTrip(unittest.TestCase): class TestServerRoundTrip(unittest.TestCase):
def _raw_status(self, content_length: str, *, authenticated: bool = True) -> int:
tmp = tempfile.TemporaryDirectory()
self.addCleanup(tmp.cleanup)
key = "request-limits-key"
server = make_server(
_orchestrator(Path(tmp.name) / "r.db"),
"127.0.0.1", 0, signing_key=key,
)
self.addCleanup(server.server_close)
threading.Thread(target=server.serve_forever, daemon=True).start()
self.addCleanup(server.shutdown)
conn = http.client.HTTPConnection(
str(server.server_address[0]), server.server_address[1], timeout=5,
)
self.addCleanup(conn.close)
conn.putrequest("POST", "/bottles")
conn.putheader("Content-Length", content_length)
if authenticated:
conn.putheader(
"x-bot-bottle-orchestrator-auth", mint(ROLE_CLI, key),
)
conn.endheaders()
return conn.getresponse().status
def test_rejects_malformed_content_length(self) -> None:
self.assertEqual(400, self._raw_status("not-a-number"))
def test_rejects_oversized_body_without_reading_it(self) -> None:
self.assertEqual(413, self._raw_status(str(MAX_BODY_BYTES + 1)))
def test_rejects_unauthenticated_request_before_reading_body(self) -> None:
self.assertEqual(
401,
self._raw_status(str(MAX_BODY_BYTES), authenticated=False),
)
def test_http_register_health_attribute(self) -> None: def test_http_register_health_attribute(self) -> None:
tmp = tempfile.TemporaryDirectory() tmp = tempfile.TemporaryDirectory()
self.addCleanup(tmp.cleanup) self.addCleanup(tmp.cleanup)
orch = _orchestrator(Path(tmp.name) / "r.db") orch = _orchestrator(Path(tmp.name) / "r.db")
server = make_server(orch, "127.0.0.1", 0) signing_key = "round-trip-key"
auth = mint(ROLE_CLI, signing_key)
server = make_server(
orch, "127.0.0.1", 0, signing_key=signing_key,
)
self.addCleanup(server.server_close) self.addCleanup(server.server_close)
thread = threading.Thread(target=server.serve_forever, daemon=True) thread = threading.Thread(target=server.serve_forever, daemon=True)
thread.start() thread.start()
@@ -267,7 +308,10 @@ class TestServerRoundTrip(unittest.TestCase):
reg = json.load(urllib.request.urlopen( reg = json.load(urllib.request.urlopen(
urllib.request.Request( urllib.request.Request(
f"{base}/bottles", data=_body({"source_ip": "10.243.0.7"}), f"{base}/bottles", data=_body({"source_ip": "10.243.0.7"}),
method="POST", headers={"Content-Type": "application/json"}, method="POST", headers={
"Content-Type": "application/json",
"x-bot-bottle-orchestrator-auth": auth,
},
), timeout=5, ), timeout=5,
)) ))
self.assertTrue(reg["bottle_id"]) self.assertTrue(reg["bottle_id"])
@@ -279,7 +323,10 @@ class TestServerRoundTrip(unittest.TestCase):
urllib.request.Request( urllib.request.Request(
f"{base}/attribute", f"{base}/attribute",
data=_body({"source_ip": "10.243.0.7", "identity_token": reg["identity_token"]}), data=_body({"source_ip": "10.243.0.7", "identity_token": reg["identity_token"]}),
method="POST", headers={"Content-Type": "application/json"}, method="POST", headers={
"Content-Type": "application/json",
"x-bot-bottle-orchestrator-auth": auth,
},
), timeout=5, ), timeout=5,
)) ))
self.assertEqual(reg["bottle_id"], attr["bottle_id"]) self.assertEqual(reg["bottle_id"], attr["bottle_id"])
@@ -287,15 +334,25 @@ class TestServerRoundTrip(unittest.TestCase):
def test_internal_failure_is_contextual_but_redacted(self) -> None: def test_internal_failure_is_contextual_but_redacted(self) -> None:
orch = MagicMock() orch = MagicMock()
orch.registry.all.side_effect = RuntimeError("SENSITIVE request value") orch.registry.all.side_effect = RuntimeError("SENSITIVE request value")
signing_key = "failure-path-key"
auth = mint(ROLE_CLI, signing_key)
with patch("sys.stderr", io.StringIO()) as stderr: with patch("sys.stderr", io.StringIO()) as stderr:
server = make_server(orch, "127.0.0.1", 0) server = make_server(
orch, "127.0.0.1", 0, signing_key=signing_key,
)
self.addCleanup(server.server_close) self.addCleanup(server.server_close)
thread = threading.Thread(target=server.serve_forever, daemon=True) thread = threading.Thread(target=server.serve_forever, daemon=True)
thread.start() thread.start()
self.addCleanup(server.shutdown) self.addCleanup(server.shutdown)
host, port = server.server_address[0], server.server_address[1] host, port = server.server_address[0], server.server_address[1]
with self.assertRaises(urllib.error.HTTPError) as raised: with self.assertRaises(urllib.error.HTTPError) as raised:
urllib.request.urlopen(f"http://{host}:{port}/bottles", timeout=5) urllib.request.urlopen(
urllib.request.Request(
f"http://{host}:{port}/bottles",
headers={"x-bot-bottle-orchestrator-auth": auth},
),
timeout=5,
)
payload = json.loads(raised.exception.read()) payload = json.loads(raised.exception.read())
output = stderr.getvalue() output = stderr.getvalue()
self.assertEqual({"error": "internal error"}, payload) self.assertEqual({"error": "internal error"}, payload)
@@ -369,8 +426,9 @@ class TestOrchestratorAuth(unittest.TestCase):
self.assertIsNotNone(self.orch.registry.get(rec.bottle_id)) self.assertIsNotNone(self.orch.registry.get(rec.bottle_id))
def _server_with_key(self, signing_key: str): def _server_with_key(self, signing_key: str):
with patch.dict("os.environ", {"BOT_BOTTLE_ORCHESTRATOR_TOKEN": signing_key}): server = make_server(
server = make_server(self.orch, "127.0.0.1", 0) self.orch, "127.0.0.1", 0, signing_key=signing_key,
)
self.addCleanup(server.server_close) self.addCleanup(server.server_close)
threading.Thread(target=server.serve_forever, daemon=True).start() threading.Thread(target=server.serve_forever, daemon=True).start()
self.addCleanup(server.shutdown) self.addCleanup(server.shutdown)
@@ -399,16 +457,10 @@ class TestOrchestratorAuth(unittest.TestCase):
self.assertEqual(403, self._status(f"{base}/bottles", header=gateway_tok)) self.assertEqual(403, self._status(f"{base}/bottles", header=gateway_tok))
self.assertEqual(200, self._status(f"{base}/bottles", header=cli_tok)) self.assertEqual(200, self._status(f"{base}/bottles", header=cli_tok))
def test_unconfigured_server_runs_open(self) -> None: def test_unconfigured_server_refuses_to_start(self) -> None:
"""No signing key set (tests / nft-protected Firecracker): open mode with patch.dict("os.environ", {}, clear=True):
grants full cli access, so existing round-trip behavior is unchanged.""" with self.assertRaisesRegex(ValueError, "signing key is required"):
with patch.dict("os.environ", {}, clear=False): make_server(self.orch, "127.0.0.1", 0)
import os
os.environ.pop("BOT_BOTTLE_ORCHESTRATOR_TOKEN", None)
server = make_server(self.orch, "127.0.0.1", 0)
self.addCleanup(server.server_close)
self.assertEqual(ROLE_CLI, server.role_for(""))
self.assertEqual(ROLE_CLI, server.role_for("anything"))
class TestDispatchSupervise(unittest.TestCase): class TestDispatchSupervise(unittest.TestCase):
+1 -2
View File
@@ -78,8 +78,7 @@ class TestControlPlaneProvisioning(unittest.TestCase):
self.assertEqual("key", prov.orchestrator_key()) self.assertEqual("key", prov.orchestrator_key())
def test_orchestrator_key_fail_closes_when_empty(self) -> None: def test_orchestrator_key_fail_closes_when_empty(self) -> None:
# Invariant 4: the orchestrator must never start without a key — it would # Invariant 4: the orchestrator must never start without a key. There is no
# run OPEN and grant every caller that reaches it full `cli`. There is no
# topology opt-out: a separate host does not stop the gateway (or any # topology opt-out: a separate host does not stop the gateway (or any
# other caller) from reaching the control-plane listener. # other caller) from reaching the control-plane listener.
prov = ControlPlaneProvisioning() prov = ControlPlaneProvisioning()