feat(orchestrator): durable launch-broker secret via TrustDomain (#468 )

Chunk 2 of the host-control-server stack: close the PRD's **durable secret** gap and replace chunk 1's BOT_BOTTLE_BROKER_SECRET stopgap. - trust_domain.py: two new domains. LAUNCH_BROKER holds the durable HS256 key both the orchestrator (signer) and the host control server (verifier) share for the broker's launch JWT — a host-canonical key file minted 0600 on first use, so a restarted orchestrator re-verifies against the same key. HOST_CONTROLLER is the separate domain for the controller's own lifecycle endpoints, keyed by a key the orchestrator never holds (its role is `host`, deliberately outside control-plane ROLES). LaunchBrokerProvisioning is the fail-closed seam. - orchestrator_auth.py: ROLE_HOST, outside ROLES. - paths.py: key-file + env-var constants for both domains. Key resolution is split by owner (addresses codex review on #497): broker_secret(allow_host_file=...) — the host controller / dev-harness (True) may mint/read the durable host key file it owns; the GUEST orchestrator (--broker http, default False) must be *injected* the key and fails closed if it isn't. A guest that fell back to the host file would mint a process-local key unrelated to the host controller's, so startup would succeed but every launch would 401 — this prevents that silent divergence. Tested: domain boundary + separation, provisioning fail-closed, and broker_secret env-only (guest) vs host-file (host) resolution. pyright clean; pylint 9.86. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
fix(secret): authenticate bottled-secret encryption (#468 )
2026-07-26 22:35:26 +00:00 · 2026-07-26 22:35:21 +00:00 · 2026-07-26 22:35:21 +00:00 · 2026-07-26 22:35:21 +00:00 · 2026-07-26 22:35:13 +00:00 · 2026-07-26 22:35:13 +00:00
25 changed files with 1563 additions and 1037 deletions
@@ -102,20 +102,6 @@ jobs:
          python3 --version
          python3 cli.py backend status --backend=docker
      - name: Preflight — clear any leftover poisoned gateway network
        run: |
          # The gateway network has a fixed name and persists across jobs on
          # this shared runner. A pre-fix or concurrent launch can leave it with
          # a malformed IPv6 subnet that trips docker's own ParseAddr in
          # `network inspect` (see PR #515); the code now self-heals it, but the
          # heal can't run if `network inspect` is what's broken on some daemon
          # versions. Drop the network here so this run recreates it IPv4-only.
          # Remove the attached gateway container first (else `network rm` fails
          # on active endpoints); both are recreated by ensure_running. Harmless
          # when absent.
          docker rm --force bot-bottle-orch-gateway 2>/dev/null || true
          docker network rm bot-bottle-gateway 2>/dev/null || true
      - name: Run integration tests (docker) with coverage
        env:
          BOT_BOTTLE_BACKEND: docker
@@ -298,7 +284,7 @@ jobs:
      - name: Combined coverage (unit + docker integration)
        run: PYTHON=python3 bash scripts/coverage.sh aggregate critical
-      - name: Diff-coverage gate (changed lines >= 80%)
+      - name: Diff-coverage gate (changed lines >= 90%)
        run: |
          git fetch --no-tags origin main:refs/remotes/origin/main
-          python3 scripts/diff_coverage.py --base origin/main --min 80
+          python3 scripts/diff_coverage.py --base origin/main --min 90
@@ -140,43 +140,12 @@ class DockerGateway(Gateway):
            marker = inspected.stdout.strip()
            if marker in {"", self._subnet}:
                return
-            # Inspectable but mislabelled: the stale auto-IPAM network created
+        if inspected.returncode == 0:
-            # by older releases. Replace it below.
+            # Migrate the stale auto-IPAM network created by older releases.
-            stale = True
+            # Removing the fixed gateway is safe here: this launch recreates it.
        else:
            # inspect failed. Classify by stderr — do NOT assume "not absent"
            # implies "poisoned": a transient daemon/API error, permission
            # failure, timeout, or bad context also fails here, and destroying
            # the shared gateway on that guess would tear the network out from
            # under every live bottle.
            err = inspected.stderr.lower()
            if "no such network" in err or "not found" in err:
                # Absent: nothing to replace — create it below.
                stale = False
            elif "parseaddr" in err:
                # Present but poisoned. A daemon that default-enables IPv6
                # attaches an fdd0::/64 subnet whose `::1/64` gateway trips
                # docker's own netip.ParseAddr in `network inspect`/`ls`, so the
                # command exits non-zero with that signature. A fixed release
                # never *creates* such a network, but one can survive on a
                # shared host from an older or concurrent launch — and
                # `--ipv6=false` alone can't heal it, since the create below only
                # no-ops on "already exists". Force-replace it so later reads
                # (e.g. `_network_cidr` pinning a source IP) stop failing.
                stale = True
            else:
                # Unrecognized failure: no evidence the network is malformed.
                # Surface it rather than mutate shared state on a guess.
                raise GatewayError(
                    f"gateway network {self.network} could not be inspected: "
                    f"{inspected.stderr.strip()}"
                )
        if stale:
            # Migrate the stale/poisoned network. Removing the fixed gateway is
            # safe here: this launch recreates it.
            run_docker(["docker", "rm", "--force", self.name])
            removed = run_docker(["docker", "network", "rm", self.network])
-            if removed.returncode != 0 and "no such network" not in removed.stderr.lower():
+            if removed.returncode != 0:
                raise GatewayError(
                    f"gateway network {self.network} needs explicit subnet "
                    f"{self._subnet} but could not be replaced: "
@@ -17,7 +17,10 @@ from pathlib import Path
 from .. import log
 from .store.store_manager import StoreManager
-from .broker import LaunchBroker, StubBroker
+from ..paths import LAUNCH_BROKER_KEY_ENV
 from .broker import StubBroker, SubmitBroker
 from .broker_client import BrokerClient
 from .host_server import DEFAULT_PORT, broker_secret
 from .server import make_server
 from .docker_broker import DockerBroker
 from .store.registry_store import RegistryStore, default_db_path
@@ -34,8 +37,13 @@ def main(argv: list[str] | None = None) -> int:
        help=f"registry DB path (default: {default_db_path()})",
    )
    parser.add_argument(
-        "--broker", choices=("stub", "docker"), default="stub",
+        "--broker", choices=("stub", "docker", "http"), default="stub",
-        help="launch broker: 'stub' records requests; 'docker' runs containers",
+        help="launch broker: 'stub' records requests; 'docker' runs containers "
             "in-process; 'http' relays signed requests to a host control server",
    )
    parser.add_argument(
        "--host-controller-url", default=f"http://127.0.0.1:{DEFAULT_PORT}",
        help="host control server URL (used only with --broker http)",
    )
    args = parser.parse_args(argv)
@@ -47,11 +55,27 @@ def main(argv: list[str] | None = None) -> int:
    # operator reaches it over HTTP (never a second, disconnected DB).
    StoreManager(registry.db_path).migrate()
-    # An ephemeral signing secret ties the orchestrator (signer) to its
+    # A signing secret ties the orchestrator (signer) to its broker (verifier).
-    # broker (verifier). 'stub' records launches instead of starting
+    # 'stub' records launches instead of starting anything; 'docker' runs real
-    # anything; 'docker' runs real containers (firecracker drops in later).
+    # containers in-process; 'http' relays signed requests to a separate host
-    secret = secrets.token_bytes(32)
+    # control server, which verifies and launches. For 'stub'/'docker' the secret
-    broker: LaunchBroker = DockerBroker(secret) if args.broker == "docker" else StubBroker(secret)
+    # is ephemeral (signer and verifier share this process). For 'http' it must be
    # the SAME key the host controller holds — and this process is the *guest*
    # (signer), so it must be given that key by injection, NOT mint its own
    # process-local one (which would diverge from the host's and 401 every launch).
    broker: SubmitBroker
    if args.broker == "http":
        secret = broker_secret()  # env-injected only; no host-file fallback here
        if secret is None:
            parser.error(
                f"--broker http requires the launch-broker key injected as "
                f"${LAUNCH_BROKER_KEY_ENV} (the host controller owns/mints it); the "
                "orchestrator must not mint its own or it would diverge from the host's"
            )
        broker = BrokerClient(args.host_controller_url)
    else:
        secret = secrets.token_bytes(32)
        broker = DockerBroker(secret) if args.broker == "docker" else StubBroker(secret)
    orchestrator = OrchestratorCore(registry, broker, secret)
    server = make_server(orchestrator, host=args.host, port=args.port)
@@ -29,6 +29,7 @@ import json
 import secrets
 import time
 from dataclasses import dataclass
 from typing import Protocol
 _JWT_HEADER = {"alg": "HS256", "typ": "JWT"}
 _ALLOWED_OPS = ("launch", "teardown")
@@ -37,7 +38,21 @@ _ALLOWED_OPS = ("launch", "teardown")
 class BrokerAuthError(Exception):
    """A broker request failed provenance or schema verification —
    bad/absent signature, malformed token, or a payload that doesn't match
-    the fixed launch-request shape. Fail-closed: the broker must not act."""
+    the fixed launch-request shape. Fail-closed: the broker must not act.
    A **definite** negative: nothing was launched, so a caller may safely roll
    back as if the op never happened."""
 class BrokerUnavailableError(Exception):
    """A brokered request could not be carried to a verdict: the broker (or the
    wire to it) was unreachable, timed out, or dropped the response.
    Crucially **ambiguous** — unlike `BrokerAuthError`, the op MAY already have
    taken effect on the backend before the response was lost, so a caller must
    NOT assume it did nothing (e.g. must not roll a registry row back as if no
    launch happened, which would orphan a running container). Only the in-process
    brokers never raise this; the out-of-process `BrokerClient` does."""
@dataclass(frozen=True)
@@ -123,6 +138,16 @@ def verify_request(token: str, secret: bytes) -> LaunchRequest:
 # --- the broker itself ------------------------------------------------------
 class SubmitBroker(Protocol):
    """The single method `OrchestratorCore` depends on: verify a signed token and
    perform its op, returning the verified request. Both the in-process
    `LaunchBroker` and the out-of-process `BrokerClient` (which relays the token
    to the host control server) satisfy it structurally, so the core is unchanged
    whether the backend is local or a real host service."""
    def submit(self, token: str) -> LaunchRequest: ...
 class LaunchBroker(abc.ABC):
    """Verifies a signed request came from the orchestrator, then performs
    the backend-native launch/teardown. Subclasses implement `_launch` /
@@ -168,7 +193,9 @@ class StubBroker(LaunchBroker):
 __all__ = [
    "BrokerAuthError",
    "BrokerUnavailableError",
    "LaunchRequest",
    "SubmitBroker",
    "LaunchBroker",
    "StubBroker",
    "sign_request",
@@ -0,0 +1,126 @@
 """Orchestrator-side broker transport (issue #468, chunk 1).
 The signer's half of the launch-broker transport gap. `BrokerClient` satisfies
 the exact `submit(token)` contract `OrchestratorCore` already depends on (see
 `broker.SubmitBroker`), but instead of verifying and launching in-process it POSTs
 the signed token to the host control server over HTTP (stdlib `urllib`, like
 `orchestrator/client.py`). Because it is drop-in for that interface, wiring a real
 out-of-process backend does not change the core: it still signs a request and
 calls `submit()`; only the wire is new.
 A provenance/schema rejection from the host controller (HTTP 401) is re-raised as
 the same `BrokerAuthError` the in-process broker raises, so the launch path's
 rollback-on-failure (`OrchestratorCore.launch_bottle`) behaves identically whether
 the broker is local or remote.
 """
 from __future__ import annotations
 import json
 import urllib.error
 import urllib.request
 from .broker import BrokerAuthError, BrokerUnavailableError, LaunchRequest
 DEFAULT_TIMEOUT_SECONDS = 5.0
 class BrokerClientError(RuntimeError):
    """The host control server *responded*, but with an unexpected status other
    than the fail-closed 401 (which surfaces as `BrokerAuthError`) — e.g. a 502
    backend failure or a malformed body. A definite negative: the host processed
    the request and it did not launch. (A *no-response* failure — unreachable /
    timeout / dropped — is the ambiguous `BrokerUnavailableError` instead.)"""
 class BrokerClient:
    """Drop-in `submit(token)` that relays a signed request to the host control
    server. Holds no secret — provenance rides entirely in the signed token, so a
    caller that can reach this client still cannot forge a launch."""
    def __init__(self, base_url: str, *, timeout: float = DEFAULT_TIMEOUT_SECONDS) -> None:
        self._base = base_url.rstrip("/")
        self._timeout = timeout
    def submit(self, token: str) -> LaunchRequest:
        """POST the signed token to the host controller and return the request it
        verified and acted on.
        Raises `BrokerAuthError` on a fail-closed 401 (bad provenance/schema —
        the same exception the in-process broker raises); `BrokerClientError` if
        the host *responds* with any other non-success status or a malformed
        body (a definite negative); or `BrokerUnavailableError` if no response is
        obtained (unreachable / timeout / dropped) — the **ambiguous** case, where
        the host may already have acted, so the caller must not roll back."""
        data = json.dumps({"token": token}).encode()
        req = urllib.request.Request(
            f"{self._base}/broker", data=data, method="POST",
            headers={"Content-Type": "application/json"},
        )
        try:
            with urllib.request.urlopen(req, timeout=self._timeout) as resp:
                return _request_from(_json_object(resp.read()))
        except urllib.error.HTTPError as e:
            detail = _error_detail(e)
            if e.code == 401:
                raise BrokerAuthError(
                    detail or "host controller rejected the request"
                ) from e
            raise BrokerClientError(
                f"POST /broker: HTTP {e.code} {detail}".rstrip()
            ) from e
        except (urllib.error.URLError, TimeoutError, OSError) as e:
            # No usable response — unreachable, timed out, or the connection
            # dropped mid-exchange. Ambiguous: the request may already have
            # launched the bottle, so this is NOT a definite failure.
            raise BrokerUnavailableError(f"POST /broker: {e}") from e
 def _json_object(raw: bytes) -> dict[str, object]:
    """Parse a JSON object, tolerating an empty or malformed body (→ {}), like
    the orchestrator client — a bad body becomes a clean 'missing field' error
    downstream rather than an opaque JSON crash."""
    if not raw:
        return {}
    try:
        obj = json.loads(raw)
    except ValueError:
        return {}
    return obj if isinstance(obj, dict) else {}
 def _error_detail(e: urllib.error.HTTPError) -> str:
    """The `error` string from a structured error response, best-effort — an
    error body may be absent or unreadable, in which case there is no detail."""
    try:
        detail = _json_object(e.read()).get("error", "")
    except Exception:  # noqa: BLE001 — the error body is advisory only
        return ""
    return detail if isinstance(detail, str) else ""
 def _request_from(payload: dict[str, object]) -> LaunchRequest:
    """Reconstruct the verified `LaunchRequest` the controller echoed, so the
    returned value matches the in-process broker's (which returns the request it
    acted on). A missing op/bottle_id means a malformed response."""
    op = payload.get("op")
    bottle_id = payload.get("bottle_id")
    if not isinstance(op, str) or not isinstance(bottle_id, str) or not bottle_id:
        raise BrokerClientError("host controller response missing op/bottle_id")
    source_ip = payload.get("source_ip")
    image_ref = payload.get("image_ref")
    slot = payload.get("slot")
    return LaunchRequest(
        op=op,
        bottle_id=bottle_id,
        source_ip=source_ip if isinstance(source_ip, str) else "",
        image_ref=image_ref if isinstance(image_ref, str) else "",
        slot=slot if isinstance(slot, int) and not isinstance(slot, bool) else None,
    )
 __all__ = [
    "BrokerClient",
    "BrokerClientError",
    "DEFAULT_TIMEOUT_SECONDS",
 ]
@@ -0,0 +1,287 @@
 """Host control server (issue #468) — the launch broker as a real host service.
 Chunk 1 of the host-control-server stack closes the **transport** gap the PRD
 opens with: today `LaunchBroker.submit(token)` is an in-process method call from
 `OrchestratorCore`, and a real host service needs it reachable over the wire.
 This module is that service — the single privileged host component — reached over
 **HTTP** (the universal transport 0070 chose), mirroring the orchestrator control
 plane's shape (`orchestrator/server.py`): a pure `dispatch()` for socket-free
 testing, wrapped by a thin stdlib `http.server` adapter.
    GET  /health  -> 200 {"status": "ok"}
    POST /broker  -> 200 {"op", "bottle_id", "source_ip", "image_ref", "slot"}
                     400 (bad body) | 401 (bad provenance/schema) | 502 (backend)
                     body: {"token": "<signed launch/teardown JWT>"}
 Only the **signed token** crosses the wire; the server holds the shared HS256
 secret and a real `LaunchBroker` (e.g. `DockerBroker`) and runs the existing
 `verify_request` + `_launch`/`_teardown` path behind the endpoint, so nothing
 free-form ever reaches it. Provenance/schema failures are fail-closed 401s that
 never touch the backend (`LaunchBroker.submit` verifies before acting), and a
 backend launch failure is a 502 the caller must surface — neither takes the
 controller down.
 The signed launch token *is* the endpoint's authentication (its provenance is the
 whole point of the JWS), so `/broker` needs no separate caller credential; the
 host controller's own lifecycle endpoints, which do, arrive with the `host`-role
 tokens of the separate `HOST_CONTROLLER` trust domain in a later chunk.
 The shared signing secret is the durable **launch-broker `TrustDomain` key**
 (#468/#476): a host-canonical key file minted 0600 on first use, provisioned to
 the orchestrator (signer) and this server (verifier). A backend launcher injects
 it via `$BOT_BOTTLE_LAUNCH_BROKER_KEY`; a host-side dev-harness process reads the
 key file directly. Durability is the point — a restarted orchestrator re-verifies
 against the same key, so re-adoption works.
 """
 from __future__ import annotations
 import argparse
 import http.server
 import json
 import os
 import socketserver
 import sys
 import typing
 from urllib.parse import urlsplit
 from .. import log
 from ..paths import LAUNCH_BROKER_KEY_ENV
 from ..trust_domain import LAUNCH_BROKER
 from .broker import BrokerAuthError, LaunchBroker
 from .docker_broker import DockerBroker
 # JSON body payload type (parsed request / rendered response).
 Json = dict[str, object]
 # Default host-controller port. Distinct from the orchestrator control plane
 # (8099) — a separate privileged component listening on its own socket.
 DEFAULT_PORT = 8091
 # Cap on the request body. A signed broker request is tiny, so rejecting anything
 # larger *before reading it* keeps a caller that can merely reach the socket (no
 # signed token needed) from exhausting memory or a handler thread with a huge
 # Content-Length — the signed token, not mere reachability, is the authority.
 MAX_BODY_BYTES = 64 * 1024
 # Per-request socket timeout, bounding how long a stalled / slow-loris caller can
 # hold a handler thread on this privileged listener.
 REQUEST_TIMEOUT_SECONDS = 15
 def _parse_json_object(body: bytes) -> Json:
    """Parse a JSON object body. Raises ValueError for non-objects / bad JSON."""
    if not body:
        return {}
    obj = json.loads(body)  # raises json.JSONDecodeError (a ValueError)
    if not isinstance(obj, dict):
        raise ValueError("request body must be a JSON object")
    return obj
 def broker_secret(
    environ: typing.Mapping[str, str] | None = None, *, allow_host_file: bool = False,
 ) -> bytes | None:
    """The shared launch-broker HS256 secret, as this process should use it.
    Always prefers the key injected into this process's env
    (`$BOT_BOTTLE_LAUNCH_BROKER_KEY`). `allow_host_file` decides the fallback when
    it is absent, and the distinction is a security boundary:
    - **Host-side** processes — the host controller and the host dev-harness — pass
      ``allow_host_file=True`` to read (minting on first use) the durable host key
      file (``bot_bottle_root()/launch-broker-key``) they legitimately own.
    - The **guest orchestrator** (``--broker http``) keeps the default ``False``.
      It runs inside a container/VM whose ``bot_bottle_root()`` is process-local,
      so minting a file there would silently create a key UNRELATED to the host
      controller's — startup would succeed but every launch would be rejected 401.
      It must instead be *given* the key by its launcher, and fail closed (None)
      if it wasn't, rather than diverge.
    None when no key is available (a guest with no injection, or an unwritable
    host root)."""
    key = LAUNCH_BROKER.key_from_env(environ)
    if not key and allow_host_file:
        try:
            key = LAUNCH_BROKER.signing_key()  # host-canonical, minted on first use
        except OSError:
            return None
    return key.encode("utf-8") if key else None
 def dispatch(  # pylint: disable=too-many-return-statements
    broker: LaunchBroker, method: str, path: str, body: bytes,
 ) -> tuple[int, Json]:
    """Route one host-control request to a (status, payload) pair. Pure — the
    only side effect is the broker's own backend launch — so routing is testable
    without a socket.
    Total by design: a provenance/schema failure becomes 401 and a backend launch
    failure becomes 502 rather than raising, so one bad request can neither act
    on the backend nor take the controller down for the next caller."""
    route = urlsplit(path).path.rstrip("/") or "/"
    if method == "GET" and route == "/health":
        return 200, {"status": "ok"}
    if method == "POST" and route == "/broker":
        try:
            data = _parse_json_object(body)
        except ValueError as e:
            return 400, {"error": f"invalid JSON: {e}"}
        token = data.get("token")
        if not isinstance(token, str) or not token:
            return 400, {"error": "token (string) is required"}
        try:
            req = broker.submit(token)
        except BrokerAuthError as e:
            # Fail-closed: bad signature, malformed token, or off-schema payload.
            # `submit` verifies before acting, so nothing was launched.
            return 401, {"error": f"broker auth failed: {e}"}
        except Exception as e:  # noqa: BLE001 — a backend launch failure (docker
            # down, image gone) is operational, not a control-plane bug; the
            # caller must see it as a distinct 502, and the server must stay up.
            return 502, {"error": f"backend launch failed: {e}"}
        return 200, {
            "op": req.op,
            "bottle_id": req.bottle_id,
            "source_ip": req.source_ip,
            "image_ref": req.image_ref,
            "slot": req.slot,
        }
    return 404, {"error": "not found"}
 class Handler(http.server.BaseHTTPRequestHandler):
    """Thin stdlib adapter: read the body, call `dispatch`, write JSON."""
    # Socket timeout per request (applied by StreamRequestHandler.setup) so a
    # stalled caller can't pin a handler thread on this privileged listener.
    timeout = REQUEST_TIMEOUT_SECONDS
    # Quiet by default; opt back into stdlib access logging with
    # BOT_BOTTLE_HOST_CONTROLLER_DEBUG (the controller has its own logging).
    def log_message(self, format: str, *args: typing.Any) -> None:  # noqa: A002
        if os.environ.get("BOT_BOTTLE_HOST_CONTROLLER_DEBUG"):
            super().log_message(format, *args)
    def _serve(self, method: str) -> None:
        """Read the request body (bounded), dispatch it, and write the JSON
        reply. A dispatch that raises (it shouldn't — dispatch is total) still
        returns a 500 rather than dropping the connection."""
        server = self.server
        assert isinstance(server, HostControlServer)
        try:
            length = int(self.headers.get("Content-Length") or 0)
        except ValueError:
            self._reply(400, {"error": "invalid Content-Length"})
            return
        if length < 0 or length > MAX_BODY_BYTES:
            # Reject before reading: nothing legitimate is this big, so an
            # oversized declared length is a bug or a resource-exhaustion attempt.
            self._reply(413, {"error": "request body too large"})
            return
        body = self.rfile.read(length) if length > 0 else b""
        try:
            status, payload = dispatch(server.broker, method, self.path, body)
        except Exception as e:  # noqa: BLE001 — the controller must stay up
            sys.stderr.write(f"host controller: {method} {self.path} failed: {e!r}\n")
            sys.stderr.flush()
            status, payload = 500, {"error": f"internal error: {e}"}
        self._reply(status, payload)
    def _reply(self, status: int, payload: typing.Mapping[str, object]) -> None:
        """Write one JSON response with an explicit Content-Length."""
        data = json.dumps(payload).encode()
        self.send_response(status)
        self.send_header("Content-Type", "application/json")
        self.send_header("Content-Length", str(len(data)))
        self.end_headers()
        self.wfile.write(data)
    def do_GET(self) -> None:
        self._serve("GET")
    def do_POST(self) -> None:
        self._serve("POST")
 class HostControlServer(socketserver.ThreadingMixIn, http.server.HTTPServer):
    """Threading HTTP server that carries the launch broker for its handlers.
    The broker holds the shared signing secret and performs the backend-native
    launch/teardown; the server itself keeps no secret of its own — provenance
    rides entirely in each request's signed token."""
    daemon_threads = True
    allow_reuse_address = True
    def __init__(self, address: tuple[str, int], broker: LaunchBroker) -> None:
        self.broker = broker
        super().__init__(address, Handler)
 def make_host_server(
    broker: LaunchBroker, host: str = "127.0.0.1", port: int = DEFAULT_PORT
 ) -> HostControlServer:
    """Build (but do not start) a host control server. `port=0` binds an
    ephemeral port — read `server.server_address` for the actual one."""
    return HostControlServer((host, port), broker)
 def main(argv: list[str] | None = None) -> int:
    """Run the host control server as a plain process (dev-harness).
        python -m bot_bottle.orchestrator.host_server [--host H] [--port P]
    Fail-closed: without the launch-broker key the server can verify no request's
    provenance, so it refuses to start rather than run a launcher that accepts
    unsigned input. As the host-side owner of the key, it may mint/read the host
    key file (`allow_host_file=True`)."""
    parser = argparse.ArgumentParser(prog="bot_bottle.orchestrator.host_server")
    parser.add_argument("--host", default="127.0.0.1", help="bind address")
    parser.add_argument("--port", type=int, default=DEFAULT_PORT, help="bind port (0 = ephemeral)")
    args = parser.parse_args(argv)
    secret = broker_secret(allow_host_file=True)
    if secret is None:
        sys.stderr.write(
            f"host controller: refusing to start without the launch-broker key "
            f"(${LAUNCH_BROKER_KEY_ENV}, or a writable host root to mint it) — it "
            "could verify no request's provenance and would relay unsigned "
            "launches to the backend\n"
        )
        sys.stderr.flush()
        return 2
    broker = DockerBroker(secret)
    server = make_host_server(broker, host=args.host, port=args.port)
    bound_host, bound_port = server.server_address[0], server.server_address[1]
    log.info(
        "host control server listening",
        context={"host": bound_host, "port": bound_port},
    )
    try:
        server.serve_forever()
    except KeyboardInterrupt:
        log.info("host controller shutting down")
    finally:
        server.server_close()
    return 0
 __all__ = [
    "dispatch",
    "Handler",
    "HostControlServer",
    "make_host_server",
    "broker_secret",
    "main",
    "Json",
    "DEFAULT_PORT",
 ]
 if __name__ == "__main__":
    raise SystemExit(main())
@@ -25,7 +25,7 @@ import json
 from collections.abc import Iterable
 from datetime import datetime, timezone
-from .broker import LaunchBroker, LaunchRequest, sign_request
+from .broker import BrokerUnavailableError, LaunchRequest, SubmitBroker, sign_request
 from .store.registry_store import DEFAULT_REAP_GRACE_SECONDS, BottleRecord, RegistryStore
 from .supervisor import (
    AuditEntry,
@@ -62,7 +62,7 @@ class OrchestratorCore:
    def __init__(
        self,
        registry: RegistryStore,
-        broker: LaunchBroker,
+        broker: SubmitBroker,
        sign_secret: bytes,
        supervisor: Supervisor | None = None,
    ) -> None:
@@ -111,14 +111,23 @@ class OrchestratorCore:
            image_ref=image_ref,
            slot=slot,
        )
        launched = False
        try:
            self._broker.submit(sign_request(req, self._secret))
-            launched = True
+        except BrokerUnavailableError:
-        finally:
+            # Ambiguous delivery failure (timeout / dropped response): the broker
-            if not launched:
+            # may already have launched the bottle before the response was lost.
-                self.registry.deregister(rec.bottle_id)
+            # Do NOT deregister — that would orphan a running container with no
-                self._tokens.pop(rec.bottle_id, None)
+            # registry row (reconcile reaps rows, never containers). Keep the row
            # so reconcile reaps it iff the bottle is not actually live; surface
            # the error so the caller knows the launch is unconfirmed.
            raise
        except Exception:
            # A definite failure — a fail-closed rejection, a backend launch
            # error, or the host reporting it did not launch: nothing is running,
            # so roll the registry entry back to leave no orphan.
            self.registry.deregister(rec.bottle_id)
            self._tokens.pop(rec.bottle_id, None)
            raise
        return rec
    def teardown_bottle(self, bottle_id: str) -> bool:
@@ -12,12 +12,22 @@ reattachment path reads ENV_VAR_SECRET from the running agent container via
 ``POST /bottles/<id>/reprovision_gateway``; the orchestrator decrypts the
 stored rows and re-populates ``_tokens``.
-Encryption scheme: HMAC-SHA256 used as a PRF in CTR mode (stdlib-only,
+Encryption scheme: HMAC-SHA256 used as a PRF in CTR mode, **authenticated**
-no external deps). Each value is encrypted independently. The output blob is
+encrypt-then-MAC (stdlib-only, no external deps). Each value is encrypted
-``nonce (16 bytes) || ciphertext`` encoded as URL-safe base64 (no padding).
+independently. The output blob is ``nonce (16 bytes) || ciphertext || tag
 (32 bytes)`` encoded as URL-safe base64 (no padding).
    keystream_block_i = HMAC-SHA256(key, nonce || i.to_bytes(4, "big"))
    ciphertext_i      = plaintext_i XOR keystream_block_i[:len(plaintext_i)]
    mac_key           = HMAC-SHA256(key, "bottled-secret-mac-v1")
    tag               = HMAC-SHA256(mac_key, nonce || ciphertext)
 The tag is what makes a **wrong key deterministically detectable**: without it,
 CTR decryption with the wrong key yields garbage that only fails when it isn't
 valid UTF-8 (so ``reprovision`` would sometimes "succeed" with a wrong
 ENV_VAR_SECRET and inject garbage egress tokens). The MAC key is derived from
 the ENV_VAR_SECRET by a domain-separated HMAC so the same key never both
 generates the keystream and signs the tag with the same message shape.
 """
 from __future__ import annotations
@@ -29,6 +39,7 @@ import secrets
 _KEY_BYTES = 32    # 256-bit key from ENV_VAR_SECRET
 _NONCE_BYTES = 16  # 128-bit random nonce per encrypt call
 _TAG_BYTES = 32    # HMAC-SHA256 authentication tag
 _BLOCK = 32        # HMAC-SHA256 output width == one keystream block
 # Env-var name the agent container receives at startup.
@@ -50,45 +61,58 @@ def _keystream(key: bytes, nonce: bytes, block_index: int) -> bytes:
    ).digest()
 def _tag(key: bytes, nonce: bytes, ciphertext: bytes) -> bytes:
    """The authentication tag over ``nonce || ciphertext``, keyed by a MAC
    subkey domain-separated from the keystream key."""
    mac_key = hmac.new(key, b"bottled-secret-mac-v1", hashlib.sha256).digest()
    return hmac.new(mac_key, nonce + ciphertext, hashlib.sha256).digest()
 def _ctr(key: bytes, nonce: bytes, data: bytes) -> bytes:
    """CTR keystream XOR — its own inverse, so it both encrypts and decrypts."""
    out = bytearray()
    for i in range(0, len(data), _BLOCK):
        chunk = data[i : i + _BLOCK]
        ks = _keystream(key, nonce, i)[: len(chunk)]
        out.extend(b ^ k for b, k in zip(chunk, ks))
    return bytes(out)
 def encrypt_value(secret_b64: str, plaintext: str) -> str:
    """Encrypt a single string value with *secret_b64* (the ENV_VAR_SECRET).
-    Returns a URL-safe base64 blob ``nonce || ciphertext`` suitable for
+    Returns a URL-safe base64 blob ``nonce || ciphertext || tag`` suitable for
    the ``bottled_agent_secrets.value`` column."""
    key = _b64dec(secret_b64)
    pt = plaintext.encode()
    nonce = secrets.token_bytes(_NONCE_BYTES)
-    ct = bytearray()
+    ct = _ctr(key, nonce, plaintext.encode())
-    for i in range(0, len(pt), _BLOCK):
+    tag = _tag(key, nonce, ct)
-        chunk = pt[i : i + _BLOCK]
+    return base64.urlsafe_b64encode(nonce + ct + tag).rstrip(b"=").decode()
        ks = _keystream(key, nonce, i)[: len(chunk)]
        ct.extend(p ^ k for p, k in zip(chunk, ks))
    return base64.urlsafe_b64encode(nonce + bytes(ct)).rstrip(b"=").decode()
 def decrypt_value(secret_b64: str, blob_b64: str) -> str:
    """Decrypt a blob produced by :func:`encrypt_value`.
    Returns the original plaintext string. Raises ``ValueError`` for malformed
-    input or a key mismatch (wrong key produces garbage, not an error, unless
+    input, a **wrong key**, or a tampered ciphertext — all caught by the
-    the plaintext is non-UTF-8 — treat all such failures as wrong key)."""
+    authentication tag before any plaintext is returned, so a wrong
    ENV_VAR_SECRET is rejected deterministically (never a garbage token)."""
    key = _b64dec(secret_b64)
    try:
        blob = _b64dec(blob_b64)
    except Exception as exc:
        raise ValueError(f"invalid ciphertext blob: {exc}") from exc
-    if len(blob) < _NONCE_BYTES:
+    if len(blob) < _NONCE_BYTES + _TAG_BYTES:
        raise ValueError("ciphertext blob too short")
-    nonce, ciphertext = blob[:_NONCE_BYTES], blob[_NONCE_BYTES:]
+    nonce = blob[:_NONCE_BYTES]
-    pt = bytearray()
+    tag = blob[-_TAG_BYTES:]
-    for i in range(0, len(ciphertext), _BLOCK):
+    ciphertext = blob[_NONCE_BYTES:-_TAG_BYTES]
-        chunk = ciphertext[i : i + _BLOCK]
+    if not hmac.compare_digest(tag, _tag(key, nonce, ciphertext)):
-        ks = _keystream(key, nonce, i)[: len(chunk)]
+        raise ValueError("ciphertext failed authentication (wrong key or tampered)")
        pt.extend(c ^ k for c, k in zip(chunk, ks))
    try:
-        return bytes(pt).decode()
+        return _ctr(key, nonce, ciphertext).decode()
-    except UnicodeDecodeError as exc:
+    except UnicodeDecodeError as exc:  # pragma: no cover - authenticated, so unreachable
-        raise ValueError(f"decryption produced non-UTF-8 output (wrong key?): {exc}") from exc
+        raise ValueError(f"decryption produced non-UTF-8 output: {exc}") from exc
 __all__ = ["ENV_VAR_SECRET_NAME", "new_env_var_secret", "encrypt_value", "decrypt_value"]
@@ -36,6 +36,13 @@ ROLE_GATEWAY = "gateway"
 ROLE_CLI = "cli"
 ROLES: frozenset[str] = frozenset({ROLE_GATEWAY, ROLE_CLI})
 # The host controller's own lifecycle role (#468). Deliberately OUTSIDE `ROLES`:
 # it belongs to a separate trust domain (`HOST_CONTROLLER`) signed by a key the
 # orchestrator never holds, so the orchestrator's control-plane key can neither
 # mint nor accept it — the orchestrator must not be able to forge the credential
 # used to start and stop it.
 ROLE_HOST = "host"
 _ALG = "HS256"
@@ -103,4 +110,4 @@ def verify(token: str, secret: str, *, roles: frozenset[str] = ROLES) -> str | N
    return role if isinstance(role, str) and role in roles else None
-__all__ = ["ROLE_GATEWAY", "ROLE_CLI", "ROLES", "mint", "verify"]
+__all__ = ["ROLE_GATEWAY", "ROLE_CLI", "ROLE_HOST", "ROLES", "mint", "verify"]
@@ -47,6 +47,22 @@ ORCHESTRATOR_TOKEN_ENV = "BOT_BOTTLE_ORCHESTRATOR_TOKEN"
 # cannot forge a higher-privilege `cli` token (issue #469 review).
 ORCHESTRATOR_AUTH_JWT_ENV = "BOT_BOTTLE_ORCHESTRATOR_AUTH_JWT"
 # The durable launch-broker signing key: the HS256 secret the orchestrator
 # (signer) and the host control server (verifier) share to sign/verify launch
 # requests (#468). A host-canonical key file (minted 0600 on first use) so it
 # survives orchestrator restarts — re-adoption re-verifies against the same key —
 # instead of the ephemeral per-process secret of the in-process broker.
 LAUNCH_BROKER_KEY_FILENAME = "launch-broker-key"
 LAUNCH_BROKER_KEY_ENV = "BOT_BOTTLE_LAUNCH_BROKER_KEY"
 # The host controller's OWN key, for its lifecycle endpoints (the direct
 # cli -> host controller path that starts/stops the orchestrator). Separate from
 # the launch-broker key and never held by the orchestrator: the controller starts
 # and stops the orchestrator, so the orchestrator must not be able to mint the
 # credentials used to drive it (#468/#476).
 HOST_CONTROLLER_KEY_FILENAME = "host-controller-key"
 HOST_CONTROLLER_KEY_ENV = "BOT_BOTTLE_HOST_CONTROLLER_KEY"
 HOST_CONTROLLER_AUTH_JWT_ENV = "BOT_BOTTLE_HOST_CONTROLLER_AUTH_JWT"
 # The host directory holding the gateway's persistent mitmproxy CA. Bind-mounted
 # into the infra/gateway container at mitmproxy's confdir so the self-generated
 # CA survives container recreation — every agent installs this one CA to trust
@@ -142,6 +158,11 @@ __all__ = [
    "ORCHESTRATOR_TOKEN_FILENAME",
    "ORCHESTRATOR_TOKEN_ENV",
    "ORCHESTRATOR_AUTH_JWT_ENV",
    "LAUNCH_BROKER_KEY_FILENAME",
    "LAUNCH_BROKER_KEY_ENV",
    "HOST_CONTROLLER_KEY_FILENAME",
    "HOST_CONTROLLER_KEY_ENV",
    "HOST_CONTROLLER_AUTH_JWT_ENV",
    "GATEWAY_CA_DIRNAME",
    "bot_bottle_root",
    "host_db_path",
@@ -29,8 +29,13 @@ from collections.abc import Mapping
 from dataclasses import dataclass
 from . import orchestrator_auth
-from .orchestrator_auth import ROLE_GATEWAY
+from .orchestrator_auth import ROLE_GATEWAY, ROLE_HOST
 from .paths import (
    HOST_CONTROLLER_AUTH_JWT_ENV,
    HOST_CONTROLLER_KEY_ENV,
    HOST_CONTROLLER_KEY_FILENAME,
    LAUNCH_BROKER_KEY_ENV,
    LAUNCH_BROKER_KEY_FILENAME,
    ORCHESTRATOR_AUTH_JWT_ENV,
    ORCHESTRATOR_TOKEN_ENV,
    ORCHESTRATOR_TOKEN_FILENAME,
@@ -99,6 +104,40 @@ CONTROL_PLANE = TrustDomain(
 )
 # The launch-broker domain (#468): durable key material for the broker's own
 # signed launch requests (`broker.py`'s HS256 launch JWT), shared by the
 # orchestrator (signer) and the host control server (verifier). Unlike
 # `CONTROL_PLANE` it mints no role tokens — the broker's provenance is the launch
 # JWT, not a role token — so its `roles` set is empty and it is used only as a
 # provider of durable, host-canonical key material (`signing_key` / `key_from_env`).
 # The durability is the point: the key survives orchestrator restarts, so a
 # restarted orchestrator re-verifies against the same key instead of the
 # ephemeral per-process secret the in-process broker used.
 LAUNCH_BROKER = TrustDomain(
    name="launch-broker",
    key_filename=LAUNCH_BROKER_KEY_FILENAME,
    roles=frozenset(),
    key_env=LAUNCH_BROKER_KEY_ENV,
    token_env="",
 )
 # The host controller's own domain (#468) — the SECOND domain #476 reserves. Its
 # key, which the orchestrator never holds, signs the `host`-role tokens the CLI
 # presents on the host controller's lifecycle endpoints (start / restart / status
 # of the orchestrator itself). Keeping it separate from `CONTROL_PLANE` is the
 # whole point: the host controller starts and stops the orchestrator, so the
 # orchestrator must not be able to mint the credentials used to drive it. (The
 # lifecycle endpoints themselves arrive in a later chunk; the domain is
 # established here alongside the durable launch-broker key.)
 HOST_CONTROLLER = TrustDomain(
    name="host-controller",
    key_filename=HOST_CONTROLLER_KEY_FILENAME,
    roles=frozenset({ROLE_HOST}),
    key_env=HOST_CONTROLLER_KEY_ENV,
    token_env=HOST_CONTROLLER_AUTH_JWT_ENV,
 )
@dataclass(frozen=True)
 class ControlPlaneProvisioning:
    """The one seam every backend launcher uses to provision control-plane auth,
@@ -132,9 +171,56 @@ class ControlPlaneProvisioning:
        return self.domain.mint(ROLE_GATEWAY)
@dataclass(frozen=True)
 class LaunchBrokerProvisioning:
    """The seam that provisions the host-side launch broker's durable keys (#468),
    the counterpart to `ControlPlaneProvisioning`. Both the orchestrator (signer)
    and the host control server (verifier) receive the SAME launch-broker key
    (carry it in `broker_domain.key_env`); the host controller ALSO receives its
    own lifecycle key (`controller_domain.key_env`) the orchestrator never holds.
    Fail-closed like the control-plane seam: minting returns "" only if the host
    root is unwritable, and an empty launch-broker key would leave the verifier
    unable to authenticate any launch — so we raise rather than hand back a key
    that would make the host controller reject (or, if a caller defaulted it,
    accept) unsigned input."""
    broker_domain: TrustDomain = LAUNCH_BROKER
    controller_domain: TrustDomain = HOST_CONTROLLER
    def broker_key(self) -> str:
        """The durable launch-broker key both the orchestrator and the host
        control server must receive (in `broker_domain.key_env`). Raises rather
        than return ""."""
        key = self.broker_domain.signing_key()
        if not key:
            raise ProvisioningError(
                f"refusing to provision the {self.broker_domain.name} broker "
                "without a signing key: the host controller could then verify no "
                "launch request's provenance"
            )
        return key
    def controller_key(self) -> str:
        """The host controller's own lifecycle key — provisioned ONLY to the host
        controller (in `controller_domain.key_env`), never to the orchestrator, so
        the orchestrator cannot mint the `host`-role tokens that start and stop
        it. Raises rather than return ""."""
        key = self.controller_domain.signing_key()
        if not key:
            raise ProvisioningError(
                f"refusing to provision the {self.controller_domain.name} without "
                "a signing key: its lifecycle endpoints would authenticate no one"
            )
        return key
 __all__ = [
    "ProvisioningError",
    "TrustDomain",
    "CONTROL_PLANE",
    "LAUNCH_BROKER",
    "HOST_CONTROLLER",
    "ControlPlaneProvisioning",
    "LaunchBrokerProvisioning",
 ]
@@ -3,10 +3,6 @@
 - **Status:** Accepted
 - **Date:** 2026-06-25
 - **Deciders:** didericis
 - **Revised:** 2026-07-27 — thresholds relaxed (critical minimum 90→85%,
  diff-coverage gate 90→80%) to cut low-value test churn on changed lines.
  The risk-weighting structure and the "global is informational" rule are
  unchanged.
 ## Context
@@ -38,7 +34,7 @@ a regression (Goodhart's law).
 Coverage is **risk-weighted**, measured over the **combined unit +
 integration** suites, with three rules:
-1. **Critical modules must remain ≥ 85%.** The curated security/logic core
+1. **Critical modules must remain ≥ 90%.** The curated security/logic core
   covers the host and gateway egress policy, manifest trust boundary,
   git-gate enforcement, supervise protocol/server, YAML parser, and bottle
   state. The concrete module list lives in `scripts/critical-modules.txt`;
@@ -59,7 +55,7 @@ integration** suites, with three rules:
 The forward-looking guard is a **diff-coverage gate**
 (`scripts/diff_coverage.py`): new/changed executable lines on a branch
-must be ≥ 80% covered. This catches regressions where they are
+must be ≥ 90% covered. This catches regressions where they are
 introduced without forcing a back-fill crusade through legacy glue. The
 gate skips lines in omitted files (there is no coverage data for them),
 so the omit list cannot launder *new* logic into the dark: anything that
@@ -0,0 +1,273 @@
 # PRD prd-new: Host control server
 - **Status:** Draft
 - **Author:** Claude
 - **Created:** 2026-07-26
 - **Issue:** #468
 ## Summary
 Promote the in-process launch broker into a standalone **host control
 server**: the single privileged component on the host. Both the CLI and the
 orchestrator drive it over HTTP; it brokers agent launches, owns the
 orchestrator's own lifecycle, and is the sole writer of host-durable state (the
 tamper-evident audit record). This closes the three gaps between today's
 well-formed broker *contract* ([`orchestrator/broker.py`](../../bot_bottle/orchestrator/broker.py))
 and a real out-of-process service — transport, durable provisioned secret,
 and a disciplined op vocabulary — and splits host state by
 owner and lifetime. The prize: **the CLI no longer needs the Docker socket**,
 which is what finally lets a dedicated Gitea runner user drop the
 root-equivalent `docker` group (PRD 0070, "Relationship to other work").
 ## Problem
 Container launches run directly from a short-lived CLI process against the
 Docker socket. That socket is root-equivalent, so every host that launches
 bottles hands root to whoever invokes the CLI — including a CI runner user we
 want to keep unprivileged. PRD 0070 already argues for replacing the fat socket
 with a **thin, structured, auditable** launch broker, and the contract for that
 broker exists and is tested in-process. But it is *only* in-process:
 `LaunchBroker.submit(token)` is a method call from
 `OrchestratorCore.launch_bottle` ([`service.py:116`](../../bot_bottle/orchestrator/service.py)),
 and `DockerBroker` is on no production path — every backend starts the
 orchestrator with `--broker stub` ([`__main__.py:54`](../../bot_bottle/orchestrator/__main__.py)).
 Three gaps stand between that scaffold and a host service:
 1. **No transport.** `submit` is an in-process call. A real service needs a
   `BrokerClient` that POSTs the signed token and a host-side HTTP server that
   verifies and acts.
 2. **The signing secret is ephemeral and self-generated.**
   [`__main__.py:53`](../../bot_bottle/orchestrator/__main__.py) does
   `secrets.token_bytes(32)` and hands the *same value* to signer and verifier —
   viable only because they share a process. A separate daemon needs the secret
   provisioned out of band and durable across orchestrator restarts.
 3. **The op vocabulary is `launch` / `teardown` only.** Everything else
   host-privileged still lives in the CLI, so the schema has to grow — carefully,
   since PRD 0070's security argument rests on "structured requests only, static
   flags + ids."
 Separately, host state has no clear owner. `OrchestratorCore.reconcile` takes
 `live_source_ips` as a parameter *only because the orchestrator cannot see the
 backend* ([`service.py:137`](../../bot_bottle/orchestrator/service.py)); the
 egress traffic log is written to the container's stderr; and there is no durable,
 tamper-evident home for the audit record that survives orchestrator destruction.
 ## Goals / Success Criteria
 - A standalone host control server that the CLI and orchestrator reach over
  **HTTP**, with three entry paths working end to end:
  - `web console -(iroh)-> orchestrator -(http)-> host controller -> launch`
  - `cli -(http)-> orchestrator -(http)-> host controller -> launch`
  - `cli -(http)-> host controller` — start / restart / status of the
    orchestrator **itself** (the bootstrap/recovery path #391 targets).
 - The launch op is expressed as a **signed JWT of static flags + ids only**,
  verified against a closed schema.
 - The signing secret is **provisioned out of band and durable** across
  orchestrator restarts (a `TrustDomain` per #476, with a key the orchestrator
  never holds for the host controller's *own* endpoints).
 - Host-privileged operations move off the CLI to the control server; **the CLI
  no longer opens the Docker socket** for bottle operations.
 - `Orchestrator.reconcile` no longer takes `live_source_ips` — live-bottle
  enumeration becomes an internal control-server call.
 - Host-durable state lands as an **append-only, hash-chained JSONL** audit log
  owned solely by the host controller; operational state stays SQLite owned
  solely by the orchestrator.
 ## Non-goals
 - **Removing standing privilege.** This converts on-demand privilege (a CLI the
  user invokes) into standing privilege (a daemon under launchd/systemd). The
  win is that the privilege is *narrower* (structured requests vs. a raw socket),
  not that it disappears. "Always running" is an accepted new property.
 - **Asymmetric signing.** We stay HS256 — see Design / "Signing stays
  symmetric."
 - **Integrity against a live compromised orchestrator.** Host-location of the
  audit log does not buy this: the orchestrator makes the decisions being audited
  and can forge or omit entries wherever the file lives. An off-box copy is the
  answer, tracked separately.
 - **A single unified DB for all state.** Impossible over a guest-kernel share
  (SQLite locking is not coherent); state is split by owner and lifetime instead.
 - **The generic `SecretProvider` (#355)** and **remote terminal design (#478)** —
  both ride the same door but are their own work.
 ## Design
 ### Topology
 The host controller is the sole privileged component. The orchestrator becomes a
 client of it for launches, and the CLI becomes a client of it for *both* bottle
 operations (indirectly, through the orchestrator) and orchestrator lifecycle
 (directly, for bootstrap/recovery — startup can't route through the thing being
 started).
 ```
 web console ─(iroh)─▶ orchestrator ─┐
                                    ├─(http, signed JWT)─▶ host controller ─▶ launch
 cli ────────(http)──▶ orchestrator ─┘
 cli ────────(http, bearer)──────────────────────────────▶ host controller   (orchestrator lifecycle)
 ```
 ### Transport: `BrokerClient` + host server
 `LaunchBroker.submit(token)` keeps its exact signature and semantics; only the
 *wire* changes. A new `BrokerClient` implements the same submit contract by
 POSTing the signed token to the host controller (stdlib `urllib`, like the
 existing [`orchestrator/client.py`](../../bot_bottle/orchestrator/client.py)),
 and the host controller's launch handler is the existing `verify_request` +
 `_launch`/`_teardown` path, now reached over HTTP instead of a method call. The
 in-process `StubBroker` stays for the dev-harness and tests; `DockerBroker`'s
 `_launch`/`_teardown` bodies move behind the server unchanged. Because the client
 satisfies the same interface `OrchestratorCore` already depends on, the core does
 not change to gain a real backend.
 ### Signing stays symmetric (HS256)
 PRD 0070 nominally specifies asymmetric; the code is HS256 and we keep it.
 Asymmetric matters when the verifier is *less* privileged than the signer — here
 it is the reverse: the host controller (verifier) is strictly more privileged
 than the orchestrator (signer), and a controller that could forge orchestrator
 requests gains nothing, since it is already the component that launches. Staying
 symmetric also honors the no-runtime-deps policy (stdlib has no Ed25519). This
 matches the reasoning already inlined in `broker.py`'s module docstring.
 ### Replay protection is out of scope (tracked in #494)
 Once the launch token travels over a wire, a captured token could be replayed —
 `sign_request` already emits `jti`/`iat` but `verify_request` reads neither, so
 there is no expiry window or `jti` cache today. Enforcing that (an `iat` window +
 a self-trimming `jti` cache) is a pure in-process change that lands independently
 of this work, and it is deferred to **#494** rather than gating the MVP of the
 host control server. Nothing here depends on it; it can merge before or after.
 ### Op vocabulary and the "ids + static flags" rule (gap 3)
 Each op moved off the CLI widens the privileged surface, so growth is governed by
 one explicit rule, enforced in `verify_request`'s schema check:
 > A broker op carries **only ids and enumerated static flags** — a bottle id, a
 > pool slot, a **content-addressed** image ref chosen from a fixed set, an op
 > name from a closed vocabulary. Never a free-form path, argv, command, or
 > caller-supplied filesystem location. If an operation cannot be expressed that
 > way, it does not become a broker op.
 Operations that fit and move off the CLI (all today in
 `backend/*/consolidated_launch.py`, driven by a short-lived CLI process):
 | Op | What it does | Fits the rule because |
 |---|---|---|
 | `launch` / `teardown` | existing | ids + slot + image ref |
 | `orchestrator.ensure_running` | start the infra container | no arguments |
 | `orchestrator.{start,restart,status}` | lifecycle (the #391 path) | no arguments |
 | `list_live` | enumerate running bottles for reconcile | no arguments; returns ids/IPs |
 | `allocate_ip` | `next_free_ip` over `_network_container_ips` | no arguments; returns an IP |
 | `provision_git_gate` | `cp`/`exec` a per-bottle deploy key into the gateway | bottle id + key handle, no path |
 | `reprovision` | `docker exec printenv <ENV_VAR_SECRET>` on a live agent | bottle id + secret *name* |
 Image **builds** stay with the orchestrator for v1 (PRD 0070 §Memory: builds run
 control-plane-side; a dedicated slim build unit is later, #468-adjacent), so no
 `build` broker op is added here.
 With `list_live` as an internal control-server call, `Orchestrator.reconcile`'s
 `live_source_ips` parameter goes away — the tell PRD 0070 called out that the
 orchestrator couldn't see the backend disappears with it.
 ### Secret provisioning (gap 2)
 The shared HS256 secret becomes a durable, out-of-band artifact via the
 **`TrustDomain`** seam (#476,
 [`trust_domain.py`](../../bot_bottle/trust_domain.py)):
 - The **launch-broker secret** is a `TrustDomain` whose key
  (`host_signing_key(<file>)`, minted 0600 on first use, durable under
  `bot_bottle_root()`) is provisioned to the orchestrator (signer) and the host
  controller (verifier). Durability across orchestrator restarts is what makes
  re-adoption work — a restart re-verifies against the same key.
 - The **host controller's own lifecycle endpoints** (the direct `cli -> host
  controller` path) get a **separate** `TrustDomain` key the orchestrator never
  holds — exactly the second domain #476's PRD reserves. The orchestrator must
  not be able to mint the credentials used to start and stop it.
 This reuses the seam #476 landed rather than re-deriving provisioning per
 backend (the PR #471 bug class).
 ### One daemon, structurally separate handlers (open decision 1)
 The audit writer and the broker live in **one daemon** for install simplicity,
 but with **no shared parsing** and **different credentials per handler**:
 - the **launch** handler requires the signed launch **JWT** (provenance +
  un-coercible schema);
 - the **audit-append** handler takes a plain **bearer token** and writes to the
  JSONL log.
 This does not defend against orchestrator compromise (it holds both creds) — it
 stops a bug in the boring audit path from reaching the privileged launch path.
 The launcher stays small enough to audit line-by-line, per PRD 0070.
 ### State ownership: split by owner and lifetime
 A single mounted DB is impossible — SQLite locking is not coherent across guest
 kernels over a share, which is why the macOS backend already uses a container-only
 volume (`INFRA_DB_VOLUME`). So state splits three ways (depends on #469, which
 gets `bot-bottle.db` off the data plane first):
 | Owner | State | Home | Shape |
 |---|---|---|---|
 | **Orchestrator** | `orchestrator_bottles` registry; `bottled_agent_secrets` (encrypted egress tokens); `supervise_proposals` / `supervise_responses` | volume nothing else mounts (generalizing the macOS design) | **SQLite** — mutable, transactional, queried |
 | **Host controller** | supervise audit entries; egress traffic log (today → container stderr); host-side config | host filesystem, survives orchestrator/volume destruction | **JSONL** — append-only |
 | **Gateway** | none | — | after #469 the data plane holds no DB state |
 The historical record is **JSONL, not SQLite**, because it is append-only, never
 updated, never transactionally queried: `O_APPEND` writes are atomic, there is no
 locking protocol to get wrong, hash-chaining for tamper-evidence is cheap, and it
 survives container-runtime volume pruning (the #450 lesson) and stays readable
 without the orchestrator running. Both halves of "the audit record" — supervise
 decisions and the egress traffic log — land in the one place.
 The orchestrator is **sole mounter and sole writer** of its SQLite volume; the
 host controller is **sole writer** of the JSONL log, over the authenticated
 audit-append channel.
 ## Implementation chunks
 Ordered, each independently mergeable:
 1. **`BrokerClient` + host launch server** over HTTP, reusing `verify_request`
   and the existing `DockerBroker` bodies. Wire `OrchestratorCore` to a
   `BrokerClient` behind a flag; keep `StubBroker` for the dev-harness. Closes
   gap 1.
 2. **Durable secret via `TrustDomain`** — provision the launch-broker key to
   signer + verifier; add the host controller's own lifecycle `TrustDomain`.
   Closes gap 2.
 3. **Grow the op vocabulary** one op at a time (`list_live` first — it also
   removes `reconcile`'s `live_source_ips`), each behind the ids + static-flags
   rule. Closes gap 3.
 4. **JSONL audit log** — the host-controller-owned, hash-chained historical
   record with the plain-bearer audit-append handler; redirect the egress traffic
   log into it.
 5. **Drop the Docker socket from the CLI** once every host-privileged op it used
   is a broker op — the payoff that unblocks the unprivileged Gitea runner user.
 ## Open questions
 1. **Schema-width rule enforcement.** The "ids + static flags" rule is stated;
   should `verify_request` reject unknown claim keys outright (strict schema) to
   keep the surface from drifting? Leaning yes.
 2. **Audit-append back-pressure.** What the audit handler does if the JSONL sink
   is unavailable (fail-closed vs. buffer) — resolve before shipping chunk 5.
 ## References
 - **PRD 0070** — the contract, the launch broker, and the state tiers this
  implements.
 - **#469** — get `bot-bottle.db` off the data plane (lands underneath this).
 - **#476** ([`prd-new-control-plane-auth-provisioning`](prd-new-control-plane-auth-provisioning.md))
  — the `TrustDomain` seam this plugs the host controller's key into.
 - **#391** — backend-agnostic orchestrator restart (the bootstrap path).
 - **#494** — enforce broker replay protection (`iat` window + `jti` cache); split
  out of this PRD as an independent in-process change.
 - **#386** — prebuilt images from the Gitea OCI registry (the fixed image set the
  broker validates against).
 - **#355** — generic `SecretProvider`.
 - **#478** — remote terminal design.
@@ -32,17 +32,9 @@ not a principled scope exclusion: both are major hosted sandbox platforms and
 belong in this landscape even though they target platform builders rather than
 bot-bottle's local single-operator workflow.
 Updated 2026-07-27 after a scan of recent Show HN launches: **Black LLAB,
 Eve, CloudRouter, Nucleus, yolo-cage, and Sandbox Agent SDK** added as a
 dated entrant cohort. They sharpen the comparison on three axes the original
 table underweighted: the browser/preview loop, parallel-agent operator UX, and
 a provider-neutral automation/session API.
 ## Summary
-The main table compares bot-bottle against fifteen canonical
+The main table compares bot-bottle against fifteen isolation/sandbox tools.
 isolation/sandbox tools; a later section evaluates six recent HN entrants
 without widening an already unwieldy table.
 Governance/pre-action authorization and credential-only layers are covered
 separately because they don't provide VM or container isolation. None
 duplicate bot-bottle's combination of local
@@ -550,199 +542,6 @@ them.
  framework runtime is not compromised.
 - **Maturity**: Specification + reference implementation, 2026.
 ## Recent HN entrants (added 2026-07-27)
 These are grouped by launch date rather than promoted into the main table.
 Several are young or sparsely documented, and putting them beside mature
 runtime platforms with false precision would obscure the useful comparison.
 The HN launch posts are the evidence snapshot; feature claims should be
 rechecked against their repositories before relying on them for a security
 decision.
 ### Black LLAB
 - **Source**: https://github.com/isaacdear/black-llab ;
  HN launch https://news.ycombinator.com/item?id=47402394
 - **Isolation/locality**: Local Docker environment, with an isolated container
  created for each agent task. Shared host kernel; no stronger boundary is
  claimed.
 - **Agent integration**: General local/cloud model workspace. Its headline is
  dynamic routing of simple prompts to local models and complex prompts to
  hosted models, with code execution and web scraping inside the task
  container.
 - **Network/credentials**: No default-deny egress, payload inspection, or
  host-side credential injection documented in the launch.
 - **Competitive read**: Superficial overlap ("a container per agent task"),
  but not a direct security-policy competitor. Its useful challenge is the
  integrated model-selection UX, which bot-bottle intentionally leaves to the
  selected agent provider.
 - **Maturity**: Early solo project; HN launch received 1 point.
 ### Eve
 - **Source**: https://eve.new/ ;
  HN launch https://news.ycombinator.com/item?id=47721255
 - **Isolation/locality**: Managed, hosted Linux sandbox per user/session
  (claimed 2 vCPU, 4 GB RAM, 10 GB disk), with filesystem, code execution,
  headless Chromium, and service connectors.
 - **Agent integration**: End-user OpenClaw-style agent product. An orchestrator
  routes subtasks to specialist models and can run parallel subagents that
  coordinate through a shared filesystem. Web UI and iMessage are primary
  interaction surfaces.
 - **Network/credentials**: Broad connectors are a product feature; the launch
  does not document bot-bottle-style default-deny route policy, content DLP,
  or credentials held outside the sandbox.
 - **Competitive read**: Adjacent, not direct. Eve sells a managed colleague;
  bot-bottle lets an operator run existing coding-agent CLIs under local
  containment. Eve nevertheless demonstrates the appeal of background work,
  live progress, browser capability, and mobile notification.
 - **Maturity**: Commercial hosted product; HN launch received 71 points and
  39 comments.
 ### CloudRouter
 - **Source**: https://github.com/manaflow-ai/manaflow/tree/main/packages/cloudrouter ;
  HN launch https://news.ycombinator.com/item?id=47006393
 - **Isolation/locality**: Claude Code or Codex runs locally and provisions
  remote cloud VMs/GPUs for execution. Project files are uploaded to the VM;
  each machine exposes auth-protected VNC, VS Code, and Jupyter surfaces.
 - **Agent integration**: A skill plus CLI lets the coding agent itself start,
  command, inspect, and tear down machines. Browser automation is integrated,
  including snapshots and screenshots. Parallel disposable compute is the
  central workflow.
 - **Network/credentials**: The launch emphasizes remote resource isolation and
  authenticated UI endpoints, not default-deny guest egress, payload DLP, or
  proxy-held application credentials.
 - **Competitive read**: The closest recent workflow competitor. It directly
  addresses parallel coding agents, environmental conflict, and closing the
  browser/test loop, but trades local custody for elastic cloud compute.
  Cloud VMs and GPUs could be a future bot-bottle backend; they do not replace
  its manifest/policy layer.
 - **Maturity**: Active open-source monorepo project; HN launch received
  138 points and 36 comments.
 ### Nucleus
 - **Source**: https://github.com/coproduct-opensource/nucleus ;
  HN launch https://news.ycombinator.com/item?id=46855770
 - **Isolation/locality**: Firecracker microVM with an enforcing MCP tool proxy.
 - **Agent integration/config**: Compositional permission envelope for
  read/write/run actions. The envelope is non-escalating and can tighten or
  terminate, with scoped approval tokens for gated operations.
 - **Network/credentials**: Default-deny egress, DNS allowlist, iptables drift
  detection, time/budget caps, and hash-chained audit logging are claimed.
  Remote append-only audit storage and attestation were roadmap items at
  launch.
 - **Competitive read**: Direct on security architecture, especially
  non-escalating policy and tamper-evident audit. It is an early execution/tool
  proxy rather than a provider-neutral, one-command coding-agent product. Its
  tool-level action envelope is semantically finer than bot-bottle's network
  boundary; bot-bottle is stronger on turnkey agent/provider integration,
  credential custody, Git mediation, and long-running operator workflow.
 - **Maturity**: Early OSS experiment; HN launch received 3 points.
 ### yolo-cage
 - **Source**: https://github.com/borenstein/yolo-cage ;
  HN launch https://news.ycombinator.com/item?id=46706796
 - **Isolation/locality**: Local sandbox for running multiple coding agents in
  YOLO mode. The launch discussion describes a VM boundary.
 - **Agent integration**: Built around the native Claude Code experience and
  motivated by running many agents in parallel without permission-prompt
  fatigue.
 - **Network/Git/credentials**: Strict egress filtering, configurable HTTP
  middleware, and mediated `git`/`gh` dispatch are the main value. The launch
  discussion explicitly identifies provider credential handling as unfinished
  and difficult because Claude state spans multiple host paths.
 - **Competitive read**: The closest new threat-model competitor. It shares
  bot-bottle's premise that filesystem isolation alone is insufficient and
  that Git plus authorized HTTP channels need mediation. bot-bottle currently
  leads on cross-provider support, proxy-held Claude/Codex/forge credentials,
  typed per-role manifests, content DLP, and supervision. yolo-cage's simpler
  pitch and narrower Claude-first setup may be easier to explain.
 - **Maturity**: Early local tool; HN launch received 60 points and 76 comments.
 ### Sandbox Agent SDK
 - **Source**: https://github.com/rivet-dev/sandbox-agent ;
  HN launch https://news.ycombinator.com/item?id=46795584
 - **Isolation/locality**: Does not provide the isolation primitive. It runs
  inside E2B, Daytona, Modal, Cloudflare Containers, Agent Computer, BoxLite,
  Docker, or another sandbox provider. Embedded mode can also run locally
  without a sandbox.
 - **Agent integration**: Provider-neutral Rust server/SDK exposing a common
  HTTP/SSE/OpenAPI interface across Claude Code, Codex, OpenCode, Cursor, Amp,
  and Pi, plus a universal event/session schema for external storage and
  replay. It also exposes filesystem, managed-process, terminal, MCP, skills,
  custom-tool, and computer-use APIs. TypeScript is the primary SDK surface.
 - **Network/credentials**: Delegated to the chosen sandbox provider.
 - **Credential posture**: Its documented convenience command extracts real
  OpenAI/Anthropic credentials from local agent configuration and passes them
  as environment variables into the sandbox. That is materially weaker than
  bot-bottle's host-side credential custody, but it is an integration choice,
  not a structural limitation: a sandbox provider could put a credential
  proxy underneath the same SDK.
 - **Competitive read**: A serious architectural threat despite not supplying
  isolation. Sandbox Agent is trying to standardize the boundary *above* the
  sandbox: one client protocol, session model, and UI/control surface across
  every coding agent and runtime. If that boundary becomes the ecosystem
  standard, users and application builders may choose a sandbox provider plus
  Sandbox Agent rather than a vertically integrated launcher. bot-bottle's
  manifests would then be valuable chiefly as a local policy/backend
  implementation unless they expose an equally usable control contract.
 - **Maturity**: Apache 2.0, ~1.5k stars and 426 commits at the 2026-07-27
  check; HN launch received 41 points.
 #### Why the Sandbox Agent architecture is strategically different
 The manifest and the universal control protocol solve different layers:
 - A bot-bottle manifest is a **trusted launch-time policy composition**. It
  selects the agent role, isolation backend, image, skills, egress routes,
  credentials, Git mediation, and supervision policy. Crucially, identity and
  secret references live on the host side of the trust boundary.
 - Sandbox Agent is a **runtime control and observation protocol**. A remote
  client creates sessions, sends messages, handles permissions, configures
  skills/MCP, manipulates files/processes/desktops, and streams normalized
  events. It deliberately delegates sandbox lifecycle, Git management,
  storage, network policy, and credential security to other products.
 That makes it complementary in a component diagram but competitive in product
 architecture. The layer that becomes the stable integration point tends to own
 the ecosystem. Three plausible threat paths matter:
 1. **Standard control plane, interchangeable runtimes.** Applications integrate
   once with Sandbox Agent and treat E2B, Daytona, BoxLite, Docker, or a future
   local microVM as replaceable compute. A provider that bundles adequate
   egress and credential custody makes bot-bottle's end-to-end launcher less
   necessary.
 2. **Policy grows upward.** Sandbox Agent already configures permissions,
   skills, MCP, custom tools, filesystem/process access, and computer use. If
   it adds a declarative, host-verifiable policy document, the overlap with
   agent/bottle manifests becomes substantial even if enforcement remains
   delegated.
 3. **UI and session ownership.** Its universal transcript schema, Inspector,
   React components, event replay, and remote terminal/computer APIs can become
   the natural basis for desktop, web, and mobile agent managers. bot-bottle's
   security layer could remain stronger while losing the operator surface and
   distribution channel.
 The counter-position is not to claim that manifests and an API are mutually
 exclusive. The defensible split is:
 - bot-bottle owns the trusted policy and enforcement plane outside the agent;
 - a provider-neutral protocol owns agent process control and normalized
  events; and
 - the operator UI consumes both.
 This suggests an explicit compatibility decision rather than parallel,
 accidental protocol design: evaluate running Sandbox Agent inside a bottle and
 exposing it only through the authenticated bot-bottle control plane. If its
 schema is suitable, adopting it could turn a threat into an integration while
 keeping manifests as the higher-trust policy source. If it is unsuitable,
 bot-bottle should still publish a stable provider-neutral session/event API so
 frontends do not depend on Claude/Codex/Pi-specific process behavior.
 ## Comparison table
 *Isolation/sandbox tools only. AGT and OAP are governance layers — see their per-project notes above.*
@@ -817,70 +616,6 @@ would be a *backend* bot-bottle could call, not a competitor to its
 manifest layer. endo-familiar is in a different paradigm entirely:
 capability passing rather than kernel boundaries.
 **Recent entrants change two parts of this read.** yolo-cage is closer to the
 actual threat model than agent-safehouse or litterbox: it combines a VM-style
 boundary with mediated Git and filtered HTTP specifically for parallel coding
 agents. Sandbox Agent SDK is the more important strategic entrant even though
 it supplies no isolation. It can become the standard agent-control layer above
 all of these runtimes, including a future bot-bottle backend. CloudRouter is
 the clearest workflow challenge because its browser/desktop/GPU loop makes
 parallel agents visibly more capable, not merely safer.
 ## Gap evaluation after the 2026-07-27 entrant scan
 ### Material gaps
 1. **A stable provider-neutral control and event protocol.** This is the
   largest newly visible gap. bot-bottle normalizes launch/provisioning across
   providers, but an external UI or orchestrator still lacks one documented
   contract for creating a Claude/Codex/Pi session, sending input, handling
   permission/supervision events, streaming normalized output, reconnecting,
   and replaying history. Sandbox Agent SDK addresses exactly this layer and
   is already portable across many sandbox providers.
 2. **Browser/preview closure.** CloudRouter and Eve make a browser or desktop
   part of the standard agent environment and expose screenshots/live viewing
   to the operator. bot-bottle can run dev servers and supports nested
   containers, but it does not present a first-class browser/computer-use
   primitive or an auth-protected preview surface. For coding agents expected
   to verify UI work, this is a real product gap.
 3. **Unified parallel-session operator UX.** Named persistent bottles and
   supervision provide the substrate, but the recent products make task
   switching, live progress, notifications, terminal attach, diffs, and
   session history the product. Security depth will not compensate for a
   visibly rougher daily loop.
 4. **Normalized transcript persistence and replay.** bot-bottle preserves
   provider-specific state for resume; it does not expose a provider-neutral
   event record suitable for audit, replay, analytics, or a web/mobile client.
   This is both a UX gap and an audit gap.
 ### Important, but not necessarily bot-bottle features
 - **Cloud VM/GPU provisioning.** Valuable for elastic workloads and could be a
  backend, but it conflicts with the local-custody default and should not
  displace core policy work.
 - **Automatic model routing.** Black LLAB and Eve sell task-to-model routing.
  bot-bottle's provider-template boundary can host that choice without making
  it part of the trusted sandbox policy.
 - **A thousand SaaS connectors.** This broadens capability and blast radius.
  The bot-bottle-native answer should remain explicit, scoped forge/egress
  associations rather than connector count as a goal.
 - **SDK-driven sandbox lifecycle as the primary configuration model.** Useful
  for platform builders, but not a replacement for reviewable, host-owned
  manifests. A control API and a declarative policy source are compatible;
  neither should silently become the other.
 ### Areas where bot-bottle remains ahead
 - real provider and forge credentials remain outside the agent process rather
  than being extracted into its environment;
 - authorized HTTP payloads are scanned, not merely destination-filtered;
 - Git writes traverse a distinct gate with secret scanning and host-held
  upstream credentials;
 - role policy is host-owned, composable, and separate from untrusted repo
  content; and
 - local Firecracker/Apple Container execution preserves operator custody
  without requiring a hosted sandbox platform.
 ## Borrowable ideas
 ### Already shipped or otherwise addressed
@@ -907,19 +642,6 @@ parallel agents visibly more capable, not merely safer.
 ### Still worth considering
 - **Sandbox Agent compatibility or an equivalent stable protocol (highest
  priority):** spike running its server inside a bottle behind bot-bottle's
  authenticated control plane. Compare its session/event schema, permission
  model, restore semantics, and provider coverage with current provider
  adapters. Adopt compatibility if it preserves the host-owned trust boundary;
  otherwise specify bot-bottle's own stable API before building another UI.
 - **First-class browser/preview loop** (from CloudRouter and Eve): give a
  bottle an optional browser/computer-use capability plus an operator-visible,
  authenticated preview/screenshot surface. Treat its network access as part
  of the bottle policy, not an implicit bypass.
 - **Provider-neutral transcript/event persistence** (from Sandbox Agent SDK):
  retain enough normalized structure for replay and audit while preserving the
  provider-native state needed for exact resume.
 - **Live network activity in the supervisor TUI** (from Docker sbx): show
  allowed and blocked connections and let the operator propose policy changes
  from the existing supervision surface.
@@ -930,11 +652,10 @@ parallel agents visibly more capable, not merely safer.
  closer review. This needs a carefully specified trust model before it can be
  more than a heuristic.
-Not worth borrowing: SDK-first *policy configuration* as used by boxlite /
+Not worth borrowing: the SDK-first programmatic API style of boxlite /
-microsandbox (cuts against the reviewable declarative-manifest stance), and
+microsandbox (cuts against the declarative-manifest stance), and the
-the hosted-SaaS custody model of tilde.run (cuts against the "infrastructure I
+hosted-SaaS dashboard model of tilde.run (cuts against the
-control" goal). A provider-neutral runtime-control API is a separate concern
+"infrastructure I control" goal).
 and is worth borrowing.
 ## Publishing and positioning verdict
@@ -958,15 +679,9 @@ bot-bottle remains unusual in combining:
 The practical wedge is “as easy as native yolo, with declarative role policy
 and self-hosted custody,” including scoped access to private LAN/Tailnet
 services that cloud-first runtimes cannot provide without additional network
-plumbing. The main competitive risks are now:
+plumbing. The main competitive risks are a local wrapper such as claudebox or
-
+Docker sbx growing a role-manifest layer, and GUI products such as SuperHQ
- a local wrapper such as yolo-cage, claudebox, or Docker sbx growing a
+adding equivalent policy and audit depth.
  role-manifest and credential-custody layer;
 - Sandbox Agent SDK becoming the standard control/session boundary and making
  the runtime beneath it interchangeable; and
 - GUI products such as SuperHQ or CloudRouter adding equivalent policy and
  audit depth before bot-bottle closes the browser/preview and
  parallel-session UX gaps.
 ## Caveats
@@ -1,536 +0,0 @@
 # Sandbox Agent SDK and bot-bottle: protocol versus product
 This note asks whether [Sandbox Agent SDK](https://github.com/rivet-dev/sandbox-agent)
 and bot-bottle compete for the same architectural layer, whether bot-bottle
 can productize the turnkey ecosystem/DX layer above it, and how far the
 Docker/OCI analogy actually holds.
 Research conducted 2026-07-27. Sandbox Agent SDK was at the `0.4.x` line,
 Apache 2.0, and documented support for Claude Code, Codex, OpenCode, Cursor,
 Amp, and Pi at the time of review.
 ## Summary
 **The projects are complementary at the component boundary and competitive at
 the product boundary.** Sandbox Agent SDK normalizes how software controls a
 coding-agent process inside an arbitrary sandbox. bot-bottle decides what
 sandbox to create, what trusted role and policy it receives, how credentials
 and Git access cross the boundary, how traffic is constrained, and how an
 operator launches and supervises the result.
 The Docker analogy is useful with one correction:
 - Sandbox Agent SDK is not equivalent to Linux container APIs or OCI itself.
  It is closer to a **containerd shim plus a portable exec/session API for
  coding agents**. It adapts incompatible agent processes to one HTTP/SSE
  contract.
 - A future independent agent-session specification would be the closer OCI
  analogue.
 - bot-bottle can credibly occupy the **Docker Engine / Compose / Desktop**
  layer: packaging, policy composition, lifecycle, networking, credentials,
  storage, operator UX, and a one-command experience above interchangeable
  agent adapters and isolation runtimes.
 That is a viable position, but “turnkey wrapper” undersells it. A thin wrapper
 is replaceable. The valuable product is a **turnkey, policy-first coding-agent
 runtime** whose manifest compiles trusted operator intent into multiple
 enforcement planes. Sandbox Agent SDK may be one internal process-control
 component of that product.
 The recommended direction is:
 1. Keep the bot-bottle manifest as the host-owned source of trusted policy.
 2. Spike Sandbox Agent SDK as the in-bottle provider/session adapter.
 3. Expose a stable, provider-neutral bot-bottle control API, compatible with
   Sandbox Agent where practical.
 4. Keep security decisions and authoritative audit outside the sandbox.
 5. Build the ecosystem around policy packs, agent images, skills, backends,
   operator UI, and trusted integrations—not around a proprietary transcript
   protocol.
 ## What each project is today
 ### Sandbox Agent SDK
 Sandbox Agent is a Rust server that runs alongside the coding agent. A client
 connects over HTTP, streams events over SSE, and uses one API across agent
 implementations. Its documented surface includes:
 - creating and restoring agent sessions;
 - sending messages and streaming normalized events;
 - handling permissions;
 - configuring MCP servers, skills, and custom tools;
 - filesystem and managed-process APIs;
 - interactive terminal access;
 - computer-use/desktop operations;
 - a universal session/transcript schema;
 - an Inspector UI, React components, CLI, TypeScript SDK, and OpenAPI spec.
 It can run in embedded mode or inside E2B, Daytona, Modal, Cloudflare
 Containers, Agent Computer, BoxLite, Docker, and other environments. It
 explicitly leaves these concerns to the caller or sandbox provider:
 - sandbox creation and lifecycle;
 - Git repository management;
 - durable session storage;
 - network policy;
 - isolation strength; and
 - secure credential delivery.
 Its documented credential convenience path extracts real provider credentials
 from local agent configuration and passes them into the sandbox environment.
 That is convenient but is not an acceptable security boundary for bot-bottle.
 Sources:
 - [Sandbox Agent repository and architecture](https://github.com/rivet-dev/sandbox-agent)
 - [Sandbox Agent documentation](https://sandboxagent.dev/docs)
 - [HTTP API](https://sandboxagent.dev/docs/api-reference)
 - [Universal session/transcript schema](https://sandboxagent.dev/docs/session-transcript-schema)
 ### bot-bottle
 bot-bottle is a host-side launch, policy, and enforcement system for existing
 coding-agent CLIs. Its current architecture includes:
 - agent and bottle manifests with composition via `extends:`;
 - a host-only trust boundary for roles, identity, and secret references;
 - provider templates and plugins for Claude Code, Codex, Pi, and custom
  providers;
 - Firecracker on KVM Linux and Apple Container on macOS, with Docker fallback;
 - image construction and provider-specific provisioning;
 - default-deny inspected egress with path/method/header policy;
 - payload DLP on authorized channels;
 - real credentials held outside the agent and injected by the gateway;
 - Git mediation, upstream credential custody, and gitleaks scanning;
 - a per-host authenticated orchestrator and shared gateway;
 - named bottle lifecycle, resume, supervision, and audit state; and
 - a CLI/TUI intended to make full-permission agents operationally tolerable.
 The provider layer currently normalizes launch-time concerns—command, image,
 prompt delivery, files, skills, environment, verification, and provider-owned
 egress routes. It does **not** yet expose a stable provider-neutral runtime
 contract for sessions, messages, transcripts, terminals, or normalized events.
 That is the gap Sandbox Agent directly illuminates.
 Sources in this repository:
 - [`README.md`](../../README.md)
 - [`0070-per-host-orchestrator.md`](../prds/0070-per-host-orchestrator.md)
 - [`0026-agent-provider-templates.md`](../prds/0026-agent-provider-templates.md)
 - [`0053-user-provider-plugins.md`](../prds/0053-user-provider-plugins.md)
 - [`agent_provider.py`](../../bot_bottle/agent_provider.py)
 ## The layer model
 The cleanest architecture has four layers:
 | Layer | Responsibility | Likely owner |
 |---|---|---|
 | Operator product | Install, select a role, launch, observe, intervene, resume, review changes | bot-bottle |
 | Trusted policy and lifecycle | Compose manifest, choose backend/image, hold credentials, enforce egress/Git, persist authoritative audit | bot-bottle |
 | Agent control protocol | Start provider process, create session, send input, stream normalized events, terminal/computer operations | Sandbox Agent or a compatible protocol |
 | Isolation primitive | VM/container/process boundary, filesystem, CPU/memory, networking substrate | Firecracker, Apple Container, Docker, E2B, Daytona, BoxLite, etc. |
 The important boundary is between trusted policy/lifecycle and agent control.
 The agent-control daemon runs in the environment being treated as untrusted.
 It can report what the agent says happened, but it cannot authoritatively prove
 that policy was enforced. Egress decisions, credential custody, Git scanning,
 bottle identity, and security audit must remain outside it.
 ### Proposed composition
 ```text
 operator UI / CLI / API
          |
          v
 bot-bottle orchestrator  (trusted)
  - resolves manifest
  - owns bottle identity and lifecycle
  - stores authoritative audit
  - authenticates clients
          |
          +--------------------------+
          |                          |
          v                          v
 isolation backend              shared gateway  (trusted)
 Firecracker / Apple / Docker   - egress policy + DLP
          |                    - credential injection
          |                    - Git mediation
          v
 bottle / guest  (untrusted)
  - Sandbox Agent server
  - Claude Code / Codex / Pi subprocess
  - workspace, skills, MCP configuration
 ```
 The bot-bottle manifest would compile into both sides:
 - **outside the bottle:** backend, network, egress, credentials, Git,
  supervision, identity, and authoritative lifecycle;
 - **inside the bottle:** selected provider, prompt, skills, MCP configuration,
  startup arguments, and non-secret session metadata.
 Sandbox Agent should never receive real secrets merely because its API offers
 a credential extraction helper. Provider and forge requests should continue
 to use bot-bottle's placeholder/proxy pattern.
 ## How accurate is the Docker/OCI analogy?
 ### The useful part
 The container ecosystem separates low-level execution from a product that
 ordinary developers operate. OCI defines interoperable image, runtime, and
 distribution specifications. Docker Engine adds a daemon, API, CLI, object
 model, images, networks, volumes, and lifecycle; Docker Desktop and related
 products add installation, updates, UI, integrations, policy, and team
 workflows.
 The same separation can exist for coding agents:
 | Container ecosystem | Agent-sandbox ecosystem |
 |---|---|
 | OCI/runtime contract | A future open agent session/event contract |
 | `runc` / runtime adapter | Claude/Codex/Pi adapter |
 | containerd shim and task/exec API | Sandbox Agent server and HTTP/SSE session API |
 | containerd / CRI-style lifecycle | Sandbox-provider lifecycle APIs |
 | Docker Engine / Compose | bot-bottle orchestrator + manifests + backends + gateway |
 | Docker Desktop / Hub ecosystem | bot-bottle desktop/mobile UX, policy packs, agent images, skills, trusted integrations |
 Sandbox Agent makes coding-agent processes portable in roughly the way a shim
 makes runtimes consumable through a common lifecycle interface. bot-bottle can
 make the entire safe-agent system usable without asking the operator to
 assemble that plumbing.
 Official container references:
 - [Open Container Initiative](https://opencontainers.org/)
 - [OCI Runtime Specification](https://github.com/opencontainers/runtime-spec)
 - [Docker Engine architecture](https://docs.docker.com/engine/)
 - [Docker alternative runtimes and containerd shims](https://docs.docker.com/engine/daemon/alternative-runtimes/)
 ### Where the analogy breaks
 1. **Sandbox Agent is an implementation, not an independent standard.**
   Its OpenAPI document is public, but the project currently owns the server,
   adapters, schema, and evolution. OCI is an independently governed set of
   specifications with multiple implementations.
 2. **It sits above, not below, the isolation boundary.** Linux namespaces,
   cgroups, VMs, and OCI runtimes create the boundary. Sandbox Agent controls a
   process after some other system has created that boundary.
 3. **It reaches into product territory.** Inspector, React components,
   computer-use APIs, skills/MCP configuration, transcripts, and restoration
   are not merely low-level primitives. Sandbox Agent can continue growing
   upward into the same UI and orchestration space bot-bottle might occupy.
 4. **Coding agents are semantically uneven.** Normalizing a container
   lifecycle is easier than claiming full behavioral parity across Claude
   Code, Codex, Cursor, Amp, OpenCode, and Pi. A universal schema can become a
   lowest common denominator or accumulate provider-specific escape hatches.
 5. **The security contract is not standardized.** An agent-session API says
   little about whether credentials are visible, egress is controlled, Git is
   mediated, or audit is trustworthy. Those are core bot-bottle concerns.
 The positioning should therefore say “Docker-like product layer above an open
 agent-control protocol,” not “Sandbox Agent is OCI” or “bot-bottle implements
 OCI for agents.”
 ## Can bot-bottle be the turnkey product layer?
 Yes, if it owns substantially more than launch syntax.
 The turnkey promise is:
 > Choose a trusted role, point it at a project, and run any supported coding
 > agent with full permissions. bot-bottle builds the environment, isolates it,
 > supplies only the capabilities it needs, keeps credentials outside, mediates
 > external writes, and gives the operator one place to watch and intervene.
 That product has several defensible jobs:
 ### 1. Packaging and reproducibility
 - provider and toolchain images;
 - pinned, verified build inputs;
 - skills and MCP configuration;
 - role/bottle composition;
 - cached startup and portable environment definitions; and
 - compatibility testing across agents and backends.
 ### 2. Trusted policy compilation
 The manifest is valuable because one reviewable document compiles into:
 - an isolation plan;
 - gateway routes and DLP policy;
 - credential slots;
 - Git-gate repositories and identities;
 - provider configuration;
 - supervision behavior; and
 - operator-facing preflight.
 Sandbox Agent's runtime configuration does not replace this. The policy must
 be resolved before an untrusted guest or agent-control daemon exists.
 ### 3. Security enforcement
 - dedicated-kernel isolation where available;
 - no direct guest route to the internet;
 - credentials injected outside the agent;
 - content inspection on allowed destinations;
 - Git secrets scanning and upstream-key custody;
 - fail-closed policy resolution; and
 - authoritative host-side audit.
 This is the strongest current differentiation from a generic
 “Sandbox Agent + Docker/E2B” assembly.
 ### 4. Lifecycle and operations
 - install and host preflight;
 - image build/update;
 - start, stop, resume, cleanup, and migration;
 - concurrent named agents;
 - state recovery after crashes;
 - live supervision and policy remediation; and
 - backend selection without changing the role definition.
 ### 5. Ecosystem and DX
 A product layer can support:
 - curated provider images;
 - signed policy/bottle packs;
 - reusable role templates;
 - skills and MCP bundles;
 - backend plugins;
 - an authenticated desktop/web/mobile operator client;
 - browser/preview integration;
 - normalized transcripts and change review; and
 - team policy distribution and compliance exports.
 The analogy to Docker is strongest here: users adopt the coherent workflow and
 ecosystem, not because the low-level process API is proprietary.
 ## Business and product positioning
 “Turnkey wrapper” is understandable internally but weak externally. It implies
 that the hard work lives underneath and that another wrapper can replace it.
 Prefer one of:
 - **The policy-first runtime for coding agents**
 - **Run any coding agent with full permissions, without giving it your host or
  credentials**
 - **A turnkey local control plane for isolated coding agents**
 - **Docker-like packaging and operations for coding agents, with the security
  boundary outside the agent**
 The open/product split could resemble the container ecosystem:
 ### Open foundation
 - manifest schema and composition;
 - local CLI and core orchestrator;
 - provider adapters;
 - Firecracker/Apple Container/Docker backends;
 - gateway policy format and enforcement;
 - Sandbox Agent compatibility;
 - local audit and supervision; and
 - conformance tests for providers/backends/policy.
 ### Productizable ecosystem/DX
 - polished desktop and mobile clients;
 - fleet/remote-host management;
 - signed and curated role/image/policy registry;
 - team policy distribution and administrative controls;
 - durable searchable transcripts and audit exports;
 - SSO, RBAC, retention, and tamper-evident audit;
 - managed update/compatibility channels;
 - remote browser/preview relay;
 - enterprise support; and
 - optional managed build/cache infrastructure.
 OCI itself is not the thing Docker sells. Interoperability expands the market;
 the product captures value through reliable packaging, workflow, distribution,
 management, and trust. bot-bottle should follow that logic rather than trying
 to make its session protocol the moat.
 ## Strategic threat from Sandbox Agent
 Sandbox Agent is a real threat for three reasons:
 1. **It can become the integration default.** A frontend or agent platform can
   integrate one API and choose among many agents and sandbox vendors.
 2. **It can own session data and UI.** The universal event schema, Inspector,
   React components, restoration, terminal, and computer-use APIs give it a
   natural path toward the operator surface.
 3. **Sandbox providers can move upward.** If E2B, Daytona, BoxLite, or another
   runtime combines Sandbox Agent with adequate network policy and credential
   custody, it can offer much of the turnkey stack.
 The threat is not that its manifest syntax is better. It currently has no
 equivalent trusted policy composition. The threat is that **the ecosystem may
 standardize around its API before bot-bottle has a stable external control
 surface**. In that world bot-bottle is evaluated as one sandbox provider,
 while the SDK and its consumers own the user relationship.
 ## Why bot-bottle can still win its layer
 Sandbox Agent's scope exclusions align with bot-bottle's deepest work:
 - it does not choose or operate the sandbox provider;
 - it does not mediate Git;
 - it does not own network policy;
 - it does not securely deliver credentials;
 - it does not durably store sessions; and
 - it cannot make guest-generated telemetry authoritative.
 Those are not incidental features. Together they define the trusted system
 around an untrusted coding agent. bot-bottle also has a narrower and coherent
 initial customer: a developer or small operator who wants existing agent CLIs
 to run locally with broad permissions and bounded consequences.
 The durable advantage is therefore:
 > Sandbox Agent makes agents controllable. bot-bottle makes them safe and
 > operable.
 That sentence remains true only if bot-bottle closes its operator-DX gaps.
 Security without a browser/preview loop, stable API, normalized session view,
 and good parallel-task UX risks becoming an invisible backend feature.
 ## Integration options
 ### Option A — Embed Sandbox Agent inside each bottle
 bot-bottle launches Sandbox Agent as the provider process supervisor and
 connects it to the host orchestrator through a bottle-scoped authenticated
 channel.
 **Benefits**
 - immediate provider-neutral session API;
 - more supported agents;
 - normalized streaming and transcripts;
 - terminal, filesystem, process, and computer-use primitives;
 - Inspector/React ecosystem; and
 - less provider-specific reverse engineering in bot-bottle.
 **Risks**
 - `0.x` API/schema churn;
 - extra binary and release-supply-chain dependency;
 - lowest-common-denominator normalization;
 - conflict with provider-native resume state;
 - an in-guest daemon is attacker-controlled after guest compromise;
 - duplicate orchestration responsibilities; and
 - upstream can move into policy/lifecycle and compete more directly.
 **Security rule**
 Treat every event and state claim from Sandbox Agent as untrusted telemetry.
 Never delegate egress authorization, credential release, bottle identity,
 authoritative audit, or Git policy to it.
 ### Option B — Implement a Sandbox Agent-compatible endpoint
 bot-bottle maps the external protocol onto its existing provider adapters and
 process model without running the upstream server.
 **Benefits**
 - ecosystem compatibility with tighter component control;
 - no in-guest daemon dependency; and
 - room to preserve bot-bottle-native lifecycle semantics.
 **Risks**
 - large and continuing compatibility burden;
 - “full feature coverage” is expensive across all providers;
 - accidental protocol fork; and
 - effort diverted from policy and UX differentiation.
 ### Option C — Define an independent bot-bottle session API
 Build only the control surface bot-bottle needs.
 **Benefits**
 - clean fit with the trust model and persistent named bottles;
 - no upstream dependency; and
 - deliberate support for supervision and security events.
 **Risks**
 - recreates a fast-growing open-source project;
 - no existing client ecosystem;
 - slower browser/desktop/mobile work; and
 - increases the chance that Sandbox Agent becomes the de facto standard first.
 ### Recommendation
 Start with **Option A as a bounded compatibility spike**, not a product
 commitment. Do not begin with a clean-room competing protocol.
 The spike should answer:
 1. Can Claude Code, Codex, and Pi retain exact native resume behavior?
 2. Can Sandbox Agent run without receiving real provider credentials?
 3. Can its server be reached through a bottle-scoped authenticated channel
   without exposing the orchestrator or broadening guest egress?
 4. Which permission events overlap or conflict with bot-bottle supervision?
 5. Can normalized events be stored while clearly separating untrusted
   transcript telemetry from authoritative gateway/Git audit?
 6. Can manifest skills, MCP servers, prompt, and startup arguments compile
   deterministically into its configuration?
 7. Does its versioning policy permit a compatibility contract bot-bottle can
   support?
 8. What image-size, startup-time, and update burden does the binary add?
 If the answers are favorable, adopt it behind a bot-bottle-owned interface and
 pin/test the supported version. If not, implement the smallest compatible
 subset needed by external clients before inventing a wholly separate API.
 ## Product roadmap implications
 The competitor scan and this architecture comparison reorder the likely work:
 1. **Provider-neutral control/session compatibility spike**
 2. **Stable authenticated external bot-bottle API**
 3. **Normalized transcript/event persistence**
 4. **Parallel-session operator UI**
 5. **Browser/preview/computer-use capability**
 6. **Policy/image/skill distribution and signing**
 7. **Remote host/fleet management**
 This does not mean pausing security work. It means exposing the shipped
 security work through a product surface that can compete with the SDK-plus-
 sandbox ecosystem.
 ## Decision
 Treat Sandbox Agent SDK as a potentially standard **agent process-control
 layer**, not as a sandbox replacement and not as a minor complementary
 library. Position bot-bottle one layer above it:
 - manifests express trusted role and environment policy;
 - bot-bottle compiles and enforces that policy across host, gateway, Git, and
  isolation backends;
 - Sandbox Agent or a compatible protocol controls the selected agent process;
  and
 - bot-bottle owns the turnkey operator experience.
 The Docker analogy is strategically sound when stated as:
 > Sandbox Agent can be the portable task/exec protocol; bot-bottle can be the
 > opinionated engine, Compose-like policy layer, and Desktop-like operator
 > product.
 It is not sound when stated as:
 > Sandbox Agent is OCI and bot-bottle is Docker.
 There is no independent OCI-equivalent agent specification yet, and Sandbox
 Agent already reaches into UI/session territory. Compatibility should be
 pursued quickly, while the trusted manifest/enforcement plane and operator
 experience remain the parts bot-bottle deliberately owns.
@@ -13,7 +13,7 @@
 #     are re-executed; no KVM or Docker dependency.
 #
 # Pass "critical" as the last argument in either mode to also report just the
-# critical modules (ADR 0004 target: 85%).
+# critical modules (ADR 0004 target: 90%).
 set -euo pipefail
 cd "$(dirname "$0")/.."
@@ -34,8 +34,8 @@ if [ "${1:-}" = "aggregate" ]; then
    "$PY" -m coverage report -m
    if [ "${2:-}" = "critical" ]; then
-        echo "== critical modules (ADR 0004 minimum: 85%) ==" >&2
+        echo "== critical modules (ADR 0004 minimum: 90%) ==" >&2
-        "$PY" -m coverage report --include="$CRITICAL" --fail-under=85
+        "$PY" -m coverage report --include="$CRITICAL" --fail-under=90
    fi
    exit 0
 fi
@@ -55,6 +55,6 @@ echo "== combined report ==" >&2
 "$PY" -m coverage report -m
 if [ "${1:-}" = "critical" ]; then
-    echo "== critical modules (ADR 0004 minimum: 85%) ==" >&2
+    echo "== critical modules (ADR 0004 minimum: 90%) ==" >&2
-    "$PY" -m coverage report --include="$CRITICAL" --fail-under=85
+    "$PY" -m coverage report --include="$CRITICAL" --fail-under=90
 fi
@@ -1,4 +1,4 @@
-# Critical security/logic core held to the >=85% coverage bar by
+# Critical security/logic core held to the >=90% coverage bar by
 # docs/decisions/0004-coverage-policy.md.
 #
 # SINGLE SOURCE OF TRUTH: scripts/coverage.sh (the `critical` report) and
@@ -13,8 +13,8 @@ policy.
 Usage:
    scripts/coverage.sh                 # produce .coverage first
-    python3 scripts/diff_coverage.py    # gate against origin/main, min 80%
+    python3 scripts/diff_coverage.py    # gate against origin/main, min 90%
-    python3 scripts/diff_coverage.py --base main --min 75
+    python3 scripts/diff_coverage.py --base main --min 85
 """
 from __future__ import annotations
@@ -74,7 +74,7 @@ def main() -> int:
    ap = argparse.ArgumentParser()
    ap.add_argument("--base", default="origin/main",
                    help="git ref to diff against (default: origin/main)")
-    ap.add_argument("--min", type=float, default=80.0,
+    ap.add_argument("--min", type=float, default=90.0,
                    help="minimum %% of changed executable lines covered")
    args = ap.parse_args()
@@ -0,0 +1,117 @@
 """Unit: orchestrator-side broker client (issue #468, chunk 1). HTTP mocked."""
 from __future__ import annotations
 import io
 import json
 import unittest
 import urllib.error
 from unittest.mock import MagicMock, patch
 from bot_bottle.orchestrator.broker import (
    BrokerAuthError,
    BrokerUnavailableError,
    LaunchRequest,
 )
 from bot_bottle.orchestrator.broker_client import BrokerClient, BrokerClientError
 _URLOPEN = "bot_bottle.orchestrator.broker_client.urllib.request.urlopen"
 def _resp(payload: object) -> MagicMock:
    m = MagicMock()
    m.__enter__.return_value.read.return_value = json.dumps(payload).encode()
    return m
 def _http_error(code: int, payload: object = None) -> urllib.error.HTTPError:
    body = json.dumps(payload).encode() if payload is not None else b""
    return urllib.error.HTTPError(
        "http://host/broker", code, "err", {}, io.BytesIO(body))  # type: ignore[arg-type]
 class TestSubmit(unittest.TestCase):
    def setUp(self) -> None:
        self.c = BrokerClient("http://host:8091")
    def test_returns_the_verified_request(self) -> None:
        echo = {
            "op": "launch", "bottle_id": "b1", "source_ip": "10.0.0.1",
            "image_ref": "img", "slot": 3,
        }
        with patch(_URLOPEN, return_value=_resp(echo)):
            got = self.c.submit("tok")
        self.assertEqual(
            LaunchRequest(op="launch", bottle_id="b1", source_ip="10.0.0.1",
                          image_ref="img", slot=3),
            got,
        )
    def test_posts_token_to_broker_endpoint(self) -> None:
        with patch(_URLOPEN, return_value=_resp({"op": "teardown", "bottle_id": "b1"})) as m:
            self.c.submit("signed-token")
        request = m.call_args.args[0]
        self.assertEqual("POST", request.get_method())
        self.assertTrue(request.full_url.endswith("/broker"))
        self.assertEqual({"token": "signed-token"}, json.loads(request.data))
    def test_401_raises_broker_auth_error(self) -> None:
        # A fail-closed provenance/schema rejection surfaces as the SAME exception
        # the in-process broker raises, so the launch path's rollback is identical.
        with patch(_URLOPEN, side_effect=_http_error(401, {"error": "bad signature"})):
            with self.assertRaises(BrokerAuthError):
                self.c.submit("forged")
    def test_502_is_a_definite_client_error(self) -> None:
        # The host responded — it processed the request and did not launch, so a
        # definite BrokerClientError (the caller may safely roll back).
        with patch(_URLOPEN, side_effect=_http_error(502, {"error": "docker down"})):
            with self.assertRaises(BrokerClientError):
                self.c.submit("tok")
    def test_unreachable_is_ambiguous_unavailable(self) -> None:
        # No response at all — the request may already have launched, so the
        # AMBIGUOUS BrokerUnavailableError (the caller must NOT roll back).
        with patch(_URLOPEN, side_effect=urllib.error.URLError("refused")):
            with self.assertRaises(BrokerUnavailableError):
                self.c.submit("tok")
    def test_timeout_is_ambiguous_unavailable(self) -> None:
        # A dropped/late response after the request was sent is the exact orphan
        # risk: the host may have launched. Must be ambiguous, not a definite fail.
        with patch(_URLOPEN, side_effect=TimeoutError("read timed out")):
            with self.assertRaises(BrokerUnavailableError):
                self.c.submit("tok")
    def test_malformed_success_body_raises(self) -> None:
        with patch(_URLOPEN, return_value=_resp({"op": "launch"})):  # missing bottle_id
            with self.assertRaises(BrokerClientError):
                self.c.submit("tok")
    def test_empty_error_body_is_tolerated(self) -> None:
        # An error with no readable JSON body still classifies by status code.
        with patch(_URLOPEN, side_effect=_http_error(401)):
            with self.assertRaises(BrokerAuthError):
                self.c.submit("forged")
    def test_non_json_success_body_raises(self) -> None:
        # A 200 whose body isn't JSON is tolerated into {} then fails the
        # missing-field check — a definite client error, not a crash.
        m = MagicMock()
        m.__enter__.return_value.read.return_value = b"not json at all"
        with patch(_URLOPEN, return_value=m):
            with self.assertRaises(BrokerClientError):
                self.c.submit("tok")
    def test_unreadable_error_body_is_tolerated(self) -> None:
        # An HTTPError whose body can't be read (fp=None) still classifies by
        # status — the error detail is best-effort.
        err = urllib.error.HTTPError(
            "http://host/broker", 502, "err", {}, None)  # type: ignore[arg-type]
        with patch(_URLOPEN, side_effect=err):
            with self.assertRaises(BrokerClientError):
                self.c.submit("tok")
 if __name__ == "__main__":
    unittest.main()
@@ -248,88 +248,6 @@ class TestDockerGateway(unittest.TestCase):
            calls,
        )
    def test_ensure_running_replaces_poisoned_ipv6_network(self) -> None:
        # A daemon that default-enables IPv6 leaves the gateway network with a
        # malformed fdd0::/64 gateway, so `docker network inspect` exits
        # non-zero with a ParseAddr error (not "No such network"). `--ipv6=false`
        # can't heal an already-poisoned network — the create just no-ops on
        # "already exists" — so _ensure_network must force-remove and recreate
        # it, else every later subnet read keeps failing.
        calls: list[list[str]] = []
        def fake(argv: list[str], **_kw: object) -> Mock:
            calls.append(argv)
            if argv[:2] == ["docker", "ps"]:
                return _proc(stdout="")
            if argv[:3] == ["docker", "network", "inspect"]:
                return _proc(
                    returncode=1,
                    stderr='ParseAddr("fdd0:0:0:4::1/64"): unexpected character, '
                    'want colon (at "/64")',
                )
            return _proc()
        with patch(_RUN_DOCKER, side_effect=fake):
            self.sc.connect_to_orchestrator(_ORCH_URL, _TOKEN)
        self.assertIn(["docker", "rm", "--force", self.sc.name], calls)
        self.assertIn(["docker", "network", "rm", self.sc.network], calls)
        creates = [c for c in calls if c[:3] == ["docker", "network", "create"]]
        self.assertEqual(
            [[
                "docker", "network", "create",
                "--ipv6=false",
                "--subnet", DEFAULT_GATEWAY_SUBNET,
                "--label",
                f"bot-bottle.gateway-subnet={DEFAULT_GATEWAY_SUBNET}",
                self.sc.network,
            ]],
            creates,
        )
    def test_ensure_running_creates_network_when_inspect_reports_absent(self) -> None:
        # The absent case (inspect fails with "No such network") must NOT try to
        # remove anything — it just creates. Guards the poisoned-vs-absent split.
        calls: list[list[str]] = []
        def fake(argv: list[str], **_kw: object) -> Mock:
            calls.append(argv)
            if argv[:3] == ["docker", "network", "inspect"]:
                return _proc(returncode=1, stderr="Error: No such network: x")
            return _proc(stdout="") if argv[:2] == ["docker", "ps"] else _proc()
        with patch(_RUN_DOCKER, side_effect=fake):
            self.sc.connect_to_orchestrator(_ORCH_URL, _TOKEN)
        self.assertNotIn(["docker", "network", "rm", self.sc.network], calls)
        creates = [c for c in calls if c[:3] == ["docker", "network", "create"]]
        self.assertEqual(1, len(creates))
    def test_ensure_running_does_not_destroy_on_generic_inspect_error(self) -> None:
        # A generic inspect failure (daemon hiccup, permission, timeout) is NOT
        # evidence of a poisoned network. Only the ParseAddr poison signature may
        # take the destructive heal path; anything else must surface as an error
        # without tearing down a possibly-healthy shared gateway.
        calls: list[list[str]] = []
        def fake(argv: list[str], **_kw: object) -> Mock:
            calls.append(argv)
            if argv[:2] == ["docker", "ps"]:
                return _proc(stdout="")
            if argv[:3] == ["docker", "network", "inspect"]:
                return _proc(
                    returncode=1,
                    stderr="Cannot connect to the Docker daemon at unix:///var/run/docker.sock",
                )
            return _proc()
        with patch(_RUN_DOCKER, side_effect=fake):
            with self.assertRaises(GatewayError):
                self.sc.connect_to_orchestrator(_ORCH_URL, _TOKEN)
        # No mutation of the shared gateway: neither the container nor the
        # network is removed, and nothing is recreated.
        self.assertNotIn(["docker", "network", "rm", self.sc.network], calls)
        self.assertFalse(any(c[:3] == ["docker", "rm", "--force"] for c in calls))
        self.assertEqual([], [c for c in calls if c[:3] == ["docker", "network", "create"]])
    def test_ca_cert_pem_reads_from_container(self) -> None:
        with patch(_RUN_DOCKER, return_value=_proc(stdout=_CA_PEM)) as m:
            self.assertEqual(_CA_PEM, self.sc.ca_cert_pem())
@@ -0,0 +1,306 @@
 """Unit tests for the host control server (issue #468, chunk 1).
 Mostly exercises the pure `dispatch()` (socket-free, like the orchestrator
 server tests), plus a real-socket round-trip through `BrokerClient` that proves
 the full sign -> POST -> verify -> act seam over HTTP.
 """
 from __future__ import annotations
 import http.client
 import io
 import json
 import os
 import secrets
 import tempfile
 import threading
 import typing
 import unittest
 from unittest.mock import MagicMock, patch
 from bot_bottle.orchestrator.broker import (
    BrokerAuthError,
    LaunchBroker,
    LaunchRequest,
    StubBroker,
    sign_request,
 )
 from bot_bottle.orchestrator.broker_client import BrokerClient
 from bot_bottle.orchestrator.host_server import (
    MAX_BODY_BYTES,
    Handler,
    HostControlServer,
    broker_secret,
    dispatch,
    main,
    make_host_server,
 )
 from bot_bottle.paths import LAUNCH_BROKER_KEY_ENV
 def _body(obj: object) -> bytes:
    return json.dumps(obj).encode()
 class _RaisingBroker(LaunchBroker):
    """A broker whose backend launch always fails — exercises the 502 path (an
    operational backend failure, distinct from a fail-closed provenance 401)."""
    def _launch(self, req: LaunchRequest) -> None:
        raise RuntimeError("docker down")
    def _teardown(self, req: LaunchRequest) -> None:
        raise RuntimeError("docker down")
 class TestDispatch(unittest.TestCase):
    def setUp(self) -> None:
        self.secret = secrets.token_bytes(16)
        self.broker = StubBroker(self.secret)
    def _token(self, **kwargs: object) -> str:
        return sign_request(LaunchRequest(**kwargs), self.secret)  # type: ignore[arg-type]
    def test_health(self) -> None:
        status, payload = dispatch(self.broker, "GET", "/health", b"")
        self.assertEqual(200, status)
        self.assertEqual("ok", payload["status"])
    def test_broker_launch_verifies_and_acts(self) -> None:
        token = self._token(
            op="launch", bottle_id="b1", source_ip="10.243.0.1",
            image_ref="img", slot=2,
        )
        status, payload = dispatch(self.broker, "POST", "/broker", _body({"token": token}))
        self.assertEqual(200, status)
        self.assertEqual("launch", payload["op"])
        self.assertEqual("b1", payload["bottle_id"])
        self.assertEqual("img", payload["image_ref"])
        self.assertEqual(2, payload["slot"])
        self.assertEqual(["b1"], [r.bottle_id for r in self.broker.launched])
    def test_broker_teardown_acts(self) -> None:
        token = self._token(op="teardown", bottle_id="b1")
        status, _ = dispatch(self.broker, "POST", "/broker", _body({"token": token}))
        self.assertEqual(200, status)
        self.assertEqual(["b1"], [r.bottle_id for r in self.broker.torn_down])
    def test_forged_token_is_401_and_nothing_acted(self) -> None:
        forged = sign_request(
            LaunchRequest(op="launch", bottle_id="b1"), secrets.token_bytes(16))
        status, payload = dispatch(self.broker, "POST", "/broker", _body({"token": forged}))
        self.assertEqual(401, status)
        self.assertIn("broker auth failed", str(payload["error"]))
        self.assertEqual([], self.broker.launched)  # fail-closed: never launched
    def test_backend_failure_is_502(self) -> None:
        broker = _RaisingBroker(self.secret)
        token = self._token(op="launch", bottle_id="b1", image_ref="img")
        status, payload = dispatch(broker, "POST", "/broker", _body({"token": token}))
        self.assertEqual(502, status)
        self.assertIn("backend launch failed", str(payload["error"]))
    def test_missing_token_is_400(self) -> None:
        status, _ = dispatch(self.broker, "POST", "/broker", _body({}))
        self.assertEqual(400, status)
    def test_bad_json_is_400(self) -> None:
        status, _ = dispatch(self.broker, "POST", "/broker", b"{not json")
        self.assertEqual(400, status)
    def test_empty_body_is_missing_token_400(self) -> None:
        # Empty body parses to {} (no token) → 400, never reaching the broker.
        status, _ = dispatch(self.broker, "POST", "/broker", b"")
        self.assertEqual(400, status)
        self.assertEqual([], self.broker.launched)
    def test_non_object_body_is_400(self) -> None:
        status, _ = dispatch(self.broker, "POST", "/broker", b"[1, 2]")
        self.assertEqual(400, status)
    def test_unknown_route_404(self) -> None:
        status, _ = dispatch(self.broker, "GET", "/nope", b"")
        self.assertEqual(404, status)
    def test_trailing_slash_normalized(self) -> None:
        status, _ = dispatch(self.broker, "GET", "/health/", b"")
        self.assertEqual(200, status)
 class TestBrokerSecret(unittest.TestCase):
    """The durable launch-broker key (#468/#476): prefer the env-injected key,
    else the durable host key file, so signer and verifier resolve the same one."""
    def test_reads_injected_key_from_env(self) -> None:
        # The injected key is honoured regardless of allow_host_file — both the
        # host controller and the guest orchestrator take an injected key.
        self.assertEqual(
            b"injected-key", broker_secret({LAUNCH_BROKER_KEY_ENV: "injected-key"}))
        self.assertEqual(
            b"injected-key",
            broker_secret({LAUNCH_BROKER_KEY_ENV: "injected-key"}, allow_host_file=True))
    def test_guest_without_injection_fails_closed(self) -> None:
        # The default (guest orchestrator): no env key and NO host-file fallback,
        # so it returns None rather than mint a divergent process-local key.
        self.assertIsNone(broker_secret({}))
    def test_host_side_falls_back_to_the_durable_key_file(self) -> None:
        # allow_host_file=True (host controller / dev-harness): mint/read the
        # durable host key file, the same key on every call (restart re-adoption).
        with tempfile.TemporaryDirectory() as root:
            with patch.dict("os.environ", {"BOT_BOTTLE_ROOT": root}, clear=False):
                os.environ.pop(LAUNCH_BROKER_KEY_ENV, None)
                first = broker_secret(allow_host_file=True)
                second = broker_secret(allow_host_file=True)
        self.assertTrue(first)
        self.assertEqual(first, second)
 class TestSeamRoundTrip(unittest.TestCase):
    """The whole point of chunk 1: a request signed by the orchestrator side is
    POSTed to a real host control server, verified there, and acted on — over
    HTTP, not an in-process call."""
    def _serve(self, broker: LaunchBroker) -> BrokerClient:
        server = make_host_server(broker, "127.0.0.1", 0)
        self.addCleanup(server.server_close)
        threading.Thread(target=server.serve_forever, daemon=True).start()
        self.addCleanup(server.shutdown)
        host, port = server.server_address[0], server.server_address[1]
        return BrokerClient(f"http://{host}:{port}")
    def test_sign_post_verify_act_over_http(self) -> None:
        secret = secrets.token_bytes(16)
        broker = StubBroker(secret)
        client = self._serve(broker)
        req = LaunchRequest(
            op="launch", bottle_id="b1", source_ip="10.0.0.1", image_ref="img", slot=1)
        got = client.submit(sign_request(req, secret))
        self.assertEqual(req, got)  # the controller echoes the verified request
        self.assertEqual(["b1"], [r.bottle_id for r in broker.launched])
    def test_forged_token_raises_broker_auth_error_over_http(self) -> None:
        secret = secrets.token_bytes(16)
        broker = StubBroker(secret)
        client = self._serve(broker)
        forged = sign_request(
            LaunchRequest(op="launch", bottle_id="b1"), secrets.token_bytes(16))
        with self.assertRaises(BrokerAuthError):
            client.submit(forged)
        self.assertEqual([], broker.launched)  # fail-closed across the wire
 class TestRequestLimits(unittest.TestCase):
    """The privileged listener must not let a caller that can merely reach the
    socket (no signed token) exhaust it via an oversized declared body — and it
    rejects on the Content-Length *header*, before reading the body."""
    def _addr(self) -> tuple[str, int]:
        self.broker = StubBroker(secrets.token_bytes(16))
        server = make_host_server(self.broker, "127.0.0.1", 0)
        self.addCleanup(server.server_close)
        threading.Thread(target=server.serve_forever, daemon=True).start()
        self.addCleanup(server.shutdown)
        host, port = server.server_address[:2]
        return typing.cast(str, host), port
    def test_oversized_content_length_is_rejected_before_reading(self) -> None:
        host, port = self._addr()
        conn = http.client.HTTPConnection(host, port, timeout=5)
        self.addCleanup(conn.close)
        # Declare an oversized body but send only a sliver: the server must reject
        # on the header before reading, so the caller gets a clean, deterministic
        # 413 (no large unread body to race a connection reset).
        conn.putrequest("POST", "/broker", skip_accept_encoding=True)
        conn.putheader("Content-Type", "application/json")
        conn.putheader("Content-Length", str(MAX_BODY_BYTES + 1))
        conn.endheaders()
        conn.send(b"{}")  # far short of the declared length; never read
        resp = conn.getresponse()
        self.assertEqual(413, resp.status)
        self.assertEqual([], self.broker.launched)  # never reached the broker
 class TestServeUnit(unittest.TestCase):
    """Drive `Handler._serve` directly (no socket). The real per-request handler
    runs in a daemon thread whose coverage/trace data is lost, so the
    bounded-body and error paths are exercised here in the main thread instead."""
    def _handler(self, broker: LaunchBroker, headers: dict[str, str],
                 body: bytes = b"") -> tuple[Handler, MagicMock]:
        server = HostControlServer.__new__(HostControlServer)
        server.broker = broker
        h = Handler.__new__(Handler)
        h.server = server
        h.headers = headers  # type: ignore[assignment] — dict is a valid .get() stand-in
        h.path = "/broker"
        h.rfile = io.BytesIO(body)
        h.wfile = io.BytesIO()
        send_response = MagicMock()
        h.send_response = send_response  # type: ignore[method-assign]
        h.send_header = MagicMock()  # type: ignore[method-assign]
        h.end_headers = MagicMock()  # type: ignore[method-assign]
        return h, send_response
    def test_oversized_content_length_is_413(self) -> None:
        broker = StubBroker(secrets.token_bytes(16))
        h, send_response = self._handler(broker, {"Content-Length": str(MAX_BODY_BYTES + 1)})
        h.do_POST()  # exercises do_POST -> _serve
        send_response.assert_called_once_with(413)
        self.assertEqual([], broker.launched)  # rejected before the broker
    def test_invalid_content_length_is_400(self) -> None:
        h, send_response = self._handler(StubBroker(secrets.token_bytes(16)),
                                         {"Content-Length": "not-a-number"})
        h._serve("POST")
        send_response.assert_called_once_with(400)
    def test_valid_request_dispatches_200(self) -> None:
        secret = secrets.token_bytes(16)
        broker = StubBroker(secret)
        body = _body({"token": sign_request(
            LaunchRequest(op="teardown", bottle_id="b1"), secret)})
        h, send_response = self._handler(broker, {"Content-Length": str(len(body))}, body)
        h._serve("POST")
        send_response.assert_called_once_with(200)
        self.assertEqual(["b1"], [r.bottle_id for r in broker.torn_down])
    def test_dispatch_exception_becomes_500(self) -> None:
        # dispatch is total, but the handler still guards it: a raised dispatch
        # returns 500 rather than dropping the connection.
        h, send_response = self._handler(
            StubBroker(secrets.token_bytes(16)), {"Content-Length": "0"})
        with patch("bot_bottle.orchestrator.host_server.dispatch",
                   side_effect=RuntimeError("boom")):
            h._serve("POST")
        send_response.assert_called_once_with(500)
    def test_health_over_do_get(self) -> None:
        h, send_response = self._handler(StubBroker(secrets.token_bytes(16)), {})
        h.path = "/health"
        h.do_GET()
        send_response.assert_called_once_with(200)
 class TestMain(unittest.TestCase):
    def test_fail_closed_without_secret(self) -> None:
        with patch("bot_bottle.orchestrator.host_server.broker_secret",
                   return_value=None):
            self.assertEqual(2, main(["--port", "0"]))
    def test_serves_then_shuts_down_cleanly(self) -> None:
        fake = MagicMock()
        fake.server_address = ("127.0.0.1", 0)
        fake.serve_forever.side_effect = KeyboardInterrupt
        with patch("bot_bottle.orchestrator.host_server.broker_secret",
                   return_value=b"k"), \
             patch("bot_bottle.orchestrator.host_server.make_host_server",
                   return_value=fake):
            self.assertEqual(0, main(["--port", "0"]))
        fake.serve_forever.assert_called_once()
        fake.server_close.assert_called_once()
 if __name__ == "__main__":
    unittest.main()
@@ -0,0 +1,67 @@
 """Unit: the orchestrator dev-harness entrypoint (`python -m bot_bottle.orchestrator`).
 Exercises broker selection (stub / docker / http) and the fail-closed http path,
 patching `make_server` so the serve loop returns instead of blocking.
 """
 from __future__ import annotations
 import os
 import secrets
 import tempfile
 import unittest
 from pathlib import Path
 from unittest.mock import MagicMock, patch
 from bot_bottle.orchestrator.__main__ import main
 from bot_bottle.paths import LAUNCH_BROKER_KEY_ENV
 def _fake_server() -> MagicMock:
    fake = MagicMock()
    fake.server_address = ("127.0.0.1", 0)
    # Break out of serve_forever immediately, exercising the try/finally.
    fake.serve_forever.side_effect = KeyboardInterrupt
    return fake
 class TestMain(unittest.TestCase):
    def _run(self, broker: str, env: dict[str, str] | None = None) -> tuple[int, MagicMock]:
        fake = _fake_server()
        with tempfile.TemporaryDirectory() as d:
            argv = ["--db", str(Path(d) / "r.db"), "--port", "0", "--broker", broker]
            with patch("bot_bottle.orchestrator.__main__.make_server", return_value=fake), \
                    patch.dict("os.environ", env or {}, clear=False):
                if env is None:
                    os.environ.pop(LAUNCH_BROKER_KEY_ENV, None)
                rc = main(argv)
        return rc, fake
    def test_stub_broker_serves_and_closes(self) -> None:
        rc, fake = self._run("stub")
        self.assertEqual(0, rc)
        fake.serve_forever.assert_called_once()
        fake.server_close.assert_called_once()
    def test_docker_broker_serves(self) -> None:
        rc, _ = self._run("docker")
        self.assertEqual(0, rc)
    def test_http_broker_with_injected_key_serves(self) -> None:
        # The guest orchestrator takes the launch-broker key by injection.
        rc, _ = self._run(
            "http", env={LAUNCH_BROKER_KEY_ENV: secrets.token_urlsafe(16)})
        self.assertEqual(0, rc)
    def test_http_broker_without_injected_key_exits(self) -> None:
        # Fail-closed: no host-file fallback for the guest, so a missing injected
        # key is a usage error rather than a silently-minted divergent key.
        with tempfile.TemporaryDirectory() as d:
            with patch.dict("os.environ", {}, clear=False):
                os.environ.pop(LAUNCH_BROKER_KEY_ENV, None)
                with self.assertRaises(SystemExit):
                    main(["--db", str(Path(d) / "r.db"), "--broker", "http"])
 if __name__ == "__main__":
    unittest.main()
@@ -2,10 +2,12 @@
 from __future__ import annotations
 import base64
 import unittest
 from bot_bottle.orchestrator.store.secret_store import (
    ENV_VAR_SECRET_NAME,
    _NONCE_BYTES,
    decrypt_value,
    encrypt_value,
    new_env_var_secret,
@@ -65,22 +67,27 @@ class TestDecryptErrors(unittest.TestCase):
    def setUp(self) -> None:
        self.secret = new_env_var_secret()
-    def test_wrong_key_raises_value_error(self) -> None:
+    def test_wrong_key_always_raises_value_error(self) -> None:
        # Deterministic: the authentication tag rejects a wrong key every time,
        # so reprovision can never inject a garbage token. Repeat across many
        # random keys (the old unauthenticated scheme let ~5% through when the
        # garbage happened to decode as valid UTF-8).
        for _ in range(200):
            ct = encrypt_value(self.secret, "secret-token")
            with self.assertRaises(ValueError):
                decrypt_value(new_env_var_secret(), ct)
    def test_tampered_ciphertext_raises_value_error(self) -> None:
        ct = encrypt_value(self.secret, "secret-token")
-        other_key = new_env_var_secret()
+        raw = bytearray(base64.urlsafe_b64decode(ct + "=" * (-len(ct) % 4)))
-        # Wrong key produces garbage bytes; decrypt_value raises ValueError
+        raw[_NONCE_BYTES] ^= 0x01  # flip a bit in the ciphertext body → tag mismatch
-        # when the result is non-UTF-8 (which is very likely for 12-char data).
+        tampered = base64.urlsafe_b64encode(bytes(raw)).rstrip(b"=").decode()
-        # We allow it to succeed only if garbage happens to be valid UTF-8, but
+        with self.assertRaises(ValueError):
-        # the plaintext must not match.
+            decrypt_value(self.secret, tampered)
        try:
            result = decrypt_value(other_key, ct)
            self.assertNotEqual("secret-token", result)
        except ValueError:
            pass
    def test_truncated_blob_raises_value_error(self) -> None:
        with self.assertRaises(ValueError):
-            decrypt_value(self.secret, "dG9vc2hvcnQ")  # "tooshort" — under 16 nonce bytes
+            decrypt_value(self.secret, "dG9vc2hvcnQ")  # "tooshort" — under nonce+tag
    def test_invalid_base64_raises_value_error(self) -> None:
        with self.assertRaises(ValueError):
@@ -11,7 +11,12 @@ from contextlib import closing
 from pathlib import Path
 from unittest.mock import patch
-from bot_bottle.orchestrator.broker import LaunchBroker, LaunchRequest, StubBroker
+from bot_bottle.orchestrator.broker import (
    BrokerUnavailableError,
    LaunchBroker,
    LaunchRequest,
    StubBroker,
 )
 from bot_bottle.orchestrator.store.registry_store import RegistryStore
 from bot_bottle.orchestrator.service import OrchestratorCore
 from bot_bottle.orchestrator.store.secret_store import new_env_var_secret
@@ -25,8 +30,8 @@ from bot_bottle.orchestrator.supervisor import (
 class _FailingBroker(LaunchBroker):
-    """Verifies the token like any broker, then fails the launch — to
+    """Verifies the token like any broker, then fails the launch *definitely* —
-    exercise the orchestrator's registry rollback."""
+    to exercise the orchestrator's registry rollback."""
    def _launch(self, req: LaunchRequest) -> None:
        raise RuntimeError("launch failed")
@@ -35,6 +40,18 @@ class _FailingBroker(LaunchBroker):
        pass
 class _UnavailableBroker(LaunchBroker):
    """Verifies the token, then raises the *ambiguous* BrokerUnavailableError —
    the host may already have launched — so the orchestrator must KEEP the
    registry row rather than orphan a running container."""
    def _launch(self, req: LaunchRequest) -> None:
        raise BrokerUnavailableError("delivery dropped after send")
    def _teardown(self, req: LaunchRequest) -> None:
        pass
 class TestOrchestrator(unittest.TestCase):
    def setUp(self) -> None:
        self._tmp = tempfile.TemporaryDirectory()
@@ -144,11 +161,20 @@ class TestOrchestrator(unittest.TestCase):
        self.assertIsNotNone(self.orch.resolve("10.243.0.3", rec.identity_token))
        self.assertIsNone(self.orch.resolve("10.243.0.3", "wrong-token"))
-    def test_launch_rolls_back_registry_on_broker_failure(self) -> None:
+    def test_launch_rolls_back_registry_on_definite_broker_failure(self) -> None:
        orch = OrchestratorCore(self.store, _FailingBroker(self.secret), self.secret)
        with self.assertRaises(RuntimeError):
            orch.launch_bottle("10.243.0.9")
-        self.assertEqual([], self.store.all())  # no orphan
+        self.assertEqual([], self.store.all())  # no orphan row
    def test_launch_keeps_registry_on_ambiguous_broker_failure(self) -> None:
        # The host may already have launched the bottle before the response was
        # lost, so deregistering would orphan a running container with no row.
        # The row is kept for reconcile to reap iff the bottle is not live.
        orch = OrchestratorCore(self.store, _UnavailableBroker(self.secret), self.secret)
        with self.assertRaises(BrokerUnavailableError):
            orch.launch_bottle("10.243.0.9")
        self.assertEqual(1, len(self.store.all()))  # row survives — no orphan container
    def test_gateway_status_reports_unconfigured(self) -> None:
        # The orchestrator no longer owns a standalone gateway lifecycle; the
@@ -6,10 +6,13 @@ import unittest
 from unittest.mock import patch
 from bot_bottle import orchestrator_auth
-from bot_bottle.orchestrator_auth import ROLE_CLI, ROLE_GATEWAY
+from bot_bottle.orchestrator_auth import ROLE_CLI, ROLE_GATEWAY, ROLE_HOST
 from bot_bottle.trust_domain import (
    CONTROL_PLANE,
    HOST_CONTROLLER,
    LAUNCH_BROKER,
    ControlPlaneProvisioning,
    LaunchBrokerProvisioning,
    ProvisioningError,
    TrustDomain,
 )
@@ -101,5 +104,73 @@ class TestControlPlaneProvisioning(unittest.TestCase):
        self.assertNotEqual(ROLE_CLI, CONTROL_PLANE.verify(tok, "k"))
 class TestLaunchBrokerAndHostControllerDomains(unittest.TestCase):
    """The real #468 domains: the launch-broker key (shared by orchestrator +
    host controller) and the host controller's own lifecycle key."""
    def test_launch_broker_mints_no_role_tokens(self) -> None:
        # Empty role set — it provides durable key material for the broker's own
        # launch JWT, not orchestrator_auth role tokens.
        self.assertEqual(frozenset(), LAUNCH_BROKER.roles)
        with patch("bot_bottle.trust_domain.host_signing_key", return_value="k"):
            with self.assertRaises(ValueError):
                LAUNCH_BROKER.mint(ROLE_CLI)
    def test_host_controller_signs_host_role_only(self) -> None:
        with patch("bot_bottle.trust_domain.host_signing_key", return_value="k"):
            tok = HOST_CONTROLLER.mint(ROLE_HOST)
        self.assertEqual(ROLE_HOST, HOST_CONTROLLER.verify(tok, "k"))
        # A control-plane `cli` token (the orchestrator's key) never verifies as a
        # host-controller role — the orchestrator can't forge lifecycle creds.
        cli_tok = orchestrator_auth.mint(ROLE_CLI, "k")
        self.assertIsNone(HOST_CONTROLLER.verify(cli_tok, "k"))
    def test_control_plane_cannot_mint_the_host_role(self) -> None:
        # `host` is outside the control-plane role set on purpose.
        with patch("bot_bottle.trust_domain.host_signing_key", return_value="k"):
            with self.assertRaises(ValueError):
                CONTROL_PLANE.mint(ROLE_HOST)
    def test_the_three_domains_use_distinct_keys_and_env_vars(self) -> None:
        self.assertEqual(3, len({
            CONTROL_PLANE.key_filename,
            LAUNCH_BROKER.key_filename,
            HOST_CONTROLLER.key_filename,
        }))
        self.assertEqual(3, len({
            CONTROL_PLANE.key_env, LAUNCH_BROKER.key_env, HOST_CONTROLLER.key_env,
        }))
 class TestLaunchBrokerProvisioning(unittest.TestCase):
    def test_broker_key_returns_the_durable_key(self) -> None:
        prov = LaunchBrokerProvisioning()
        with patch("bot_bottle.trust_domain.host_signing_key", return_value="bk"):
            self.assertEqual("bk", prov.broker_key())
    def test_broker_key_fail_closes_when_empty(self) -> None:
        # An empty key would leave the host controller unable to verify any
        # launch — fail-closed rather than hand back a useless/dangerous key.
        prov = LaunchBrokerProvisioning()
        with patch("bot_bottle.trust_domain.host_signing_key", return_value=""):
            with self.assertRaises(ProvisioningError):
                prov.broker_key()
    def test_controller_key_is_distinct_from_the_broker_key(self) -> None:
        # The orchestrator holds the broker key but NEVER the controller key.
        prov = LaunchBrokerProvisioning()
        keys = {"launch-broker-key": "bk", "host-controller-key": "ck"}
        with patch("bot_bottle.trust_domain.host_signing_key",
                   side_effect=keys.__getitem__):
            self.assertEqual("bk", prov.broker_key())
            self.assertEqual("ck", prov.controller_key())
    def test_controller_key_fail_closes_when_empty(self) -> None:
        prov = LaunchBrokerProvisioning()
        with patch("bot_bottle.trust_domain.host_signing_key", return_value=""):
            with self.assertRaises(ProvisioningError):
                prov.controller_key()
 if __name__ == "__main__":
    unittest.main()
Author	SHA1	Message	Date
didericis-claude	4b17e6d683	feat(orchestrator): durable launch-broker secret via TrustDomain (#468 ) refresh-image-locks / refresh (push) Successful in 33s Details lint / lint (push) Successful in 1m7s Details test / image-input-builds (pull_request) Failing after 13m0s Details test / integration-docker (pull_request) Has been cancelled Details test / unit (pull_request) Failing after 10m48s Details test / coverage (pull_request) Has been skipped Details tracker-policy-pr / check-pr (pull_request) Successful in 13s Details Chunk 2 of the host-control-server stack: close the PRD's durable secret gap and replace chunk 1's BOT_BOTTLE_BROKER_SECRET stopgap. - trust_domain.py: two new domains. LAUNCH_BROKER holds the durable HS256 key both the orchestrator (signer) and the host control server (verifier) share for the broker's launch JWT — a host-canonical key file minted 0600 on first use, so a restarted orchestrator re-verifies against the same key. HOST_CONTROLLER is the separate domain for the controller's own lifecycle endpoints, keyed by a key the orchestrator never holds (its role is `host`, deliberately outside control-plane ROLES). LaunchBrokerProvisioning is the fail-closed seam. - orchestrator_auth.py: ROLE_HOST, outside ROLES. - paths.py: key-file + env-var constants for both domains. Key resolution is split by owner (addresses codex review on #497): broker_secret(allow_host_file=...) — the host controller / dev-harness (True) may mint/read the durable host key file it owns; the GUEST orchestrator (--broker http, default False) must be injected the key and fails closed if it isn't. A guest that fell back to the host file would mint a process-local key unrelated to the host controller's, so startup would succeed but every launch would 401 — this prevents that silent divergence. Tested: domain boundary + separation, provisioning fail-closed, and broker_secret env-only (guest) vs host-file (host) resolution. pyright clean; pylint 9.86. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-07-26 22:35:26 +00:00
didericis-claude	7a48ea2b0c	fix(secret): authenticate bottled-secret encryption (#468 ) refresh-image-locks / refresh (push) Successful in 42s Details lint / lint (push) Successful in 1m7s Details test / image-input-builds (pull_request) Successful in 1m12s Details test / unit (pull_request) Successful in 44s Details test / integration-docker (pull_request) Successful in 55s Details test / coverage (pull_request) Successful in 13s Details tracker-policy-pr / check-pr (pull_request) Successful in 7s Details The per-bottle egress-secret encryption (secret_store.py) was unauthenticated CTR/XOR: decrypting with the WRONG ENV_VAR_SECRET produced garbage that decrypt_value only rejected when it wasn't valid UTF-8. For short token values that garbage is coincidentally valid UTF-8 ~5% of the time, so `reprovision_from_secret` would occasionally "succeed" with a wrong key and inject a garbage egress credential — and test_reprovision_rejects_missing_rows_and_wrong_key failed ~5% of runs (flaky CI, surfaced by this stack's unit job). Switch to authenticated encrypt-then-MAC: append an HMAC-SHA256 tag over `nonce \|\| ciphertext`, keyed by a domain-separated MAC subkey derived from the ENV_VAR_SECRET. decrypt_value verifies the tag (constant-time) before returning any plaintext, so a wrong key or tampered ciphertext is rejected deterministically. Blob format is now `nonce \|\| ciphertext \|\| tag` (the stored rows are transient — re-written every launch — so no migration is needed). Pre-existing bug on main, unrelated to the transport work, but it blocks this stack's CI. Tests: wrong key rejected 200/200; tampered ciphertext rejected; round-trips unchanged. Deterministic now (was ~5% flaky). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-07-26 22:35:21 +00:00
didericis-claude	ec953ceda7	fix(orchestrator): address review on host control server transport (#468 ) Codex review on #496: - High — ambiguous delivery no longer orphans a launched bottle. A timeout / dropped response from the host controller is now the ambiguous BrokerUnavailableError (distinct from the definite BrokerAuthError / BrokerClientError). OrchestratorCore.launch_bottle keeps the registry row on the ambiguous case instead of deregistering — deregistering would orphan a running container with no record (reconcile reaps rows, never containers). The row is left for reconcile to reap iff the bottle is not actually live. Definite failures still roll back, so a real failure leaves no orphan row. - Medium — the privileged endpoint bounds request bodies. The host server rejects an oversized Content-Length with 413 before reading it, and sets a per-request socket timeout, so a caller that can merely reach the socket (no signed token) can't exhaust memory or a handler thread. Tests: ambiguous-keep vs definite-rollback in the launch path; the BrokerUnavailableError/BrokerClientError split in BrokerClient; the 413 body cap + handler error paths (driven in-thread, since daemon request threads lose coverage) plus a deterministic real-socket check that declares an oversized Content-Length but sends a sliver (rejection on the header, no unread-body reset race); and the __main__ entrypoint broker selection. Diff-coverage 98%; pyright clean; pylint 9.8. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-07-26 22:35:21 +00:00
didericis-claude	ed0f95f445	feat(orchestrator): host control server transport (#468 ) Chunk 1 of the host-control-server stack: close the PRD's transport gap. Today LaunchBroker.submit(token) is an in-process method call from OrchestratorCore; this makes it a real out-of-process service reached over HTTP. - host_server.py: the host control server. A pure dispatch() (POST /broker verifies a signed token via the existing verify_request + _launch/_teardown path, GET /health) wrapped by a thin http.server adapter, mirroring orchestrator/server.py. Only the signed token crosses the wire; provenance/schema failures are fail-closed 401s that never touch the backend, a backend launch failure is a 502. - broker_client.py: BrokerClient — a drop-in submit(token) that POSTs the signed token to the host controller. A 401 re-raises as BrokerAuthError so the launch path's rollback is identical local or remote. - broker.py: SubmitBroker Protocol — the one method OrchestratorCore depends on, satisfied by both LaunchBroker and BrokerClient, so the core is unchanged (service.py annotation only). - __main__.py: wire `--broker http` behind the shared-secret env var (BOT_BOTTLE_BROKER_SECRET, hex) — a chunk-1 stopgap the durable TrustDomain key (chunk 2, #476) replaces. Tested: pure-dispatch cases, BrokerClient with HTTP mocked, and a real-socket sign -> POST -> verify -> act round-trip (incl. fail-closed forged token). pyright clean; pylint 9.86. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-07-26 22:35:21 +00:00
didericis-claude	794e4e662d	docs: defer broker replay protection to its own issue (#494 ) prd-number-check / require-numbered-prds (pull_request) Failing after 7s Details tracker-policy-pr / check-pr (pull_request) Successful in 10s Details refresh-image-locks / refresh (push) Successful in 30s Details lint / lint (push) Successful in 1m6s Details Per PR review, replay protection is too heavy for the host control server MVP. Drop it from the four-gap framing (now three gaps), remove the enforcement design section and implementation chunk, and track the iat-window + jti-cache work in #494 as an independent in-process change. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-07-26 22:35:13 +00:00
didericis-claude	f2fe1f9b2d	docs: PRD for host-side control server (#468 ) Promote the in-process launch broker into a standalone host control server: the single privileged host component that brokers launches, owns orchestrator lifecycle, and is the sole writer of host-durable state. Closes the four broker gaps (transport, durable provisioned secret, replay protection, disciplined op vocabulary) and splits host state by owner and lifetime (orchestrator SQLite / host JSONL audit / gateway none). The payoff is dropping the Docker socket from the CLI. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-07-26 22:35:13 +00:00