feat(orchestrator): list_live broker op + reconcile via broker (#468)

Chunk 3 of the host-control-server stack: grow the broker op vocabulary
(PRD gap 3), starting with `list_live`, and invert `reconcile` onto it.

- broker: split the closed op vocabulary into mutation (`launch`/`teardown`,
  carry a bottle id + static flags) and query (`list_live`, carries nothing
  but its op name) kinds. `verify_request` now enforces a **strict schema**
  (open question 1, resolved yes): unknown claim keys are rejected, a mutation
  must name its bottle, and a query that smuggles any id/flag is refused.
- broker verb: `LaunchBroker.list_live` / `SubmitBroker.list_live` return the
  backend's live source IPs; a backend enumeration failure is converted to the
  single `BrokerUnavailableError` "live set unknown" signal. `DockerBroker`
  enumerates its labelled containers; `StubBroker` derives from launches (or a
  test override).
- host controller: `POST /broker/live` verifies a signed `list_live` token and
  returns `{source_ips}`; `BrokerClient.list_live` is its drop-in client.
- reconcile: `OrchestratorCore.reconcile()` drops the `live_source_ips`
  parameter and pulls the live set from the broker itself — the tell that the
  orchestrator couldn't see the backend goes away. **Fail-safe**: if the broker
  can't return an authoritative set the sweep is skipped, never run against an
  empty/partial set (which would reap healthy rows). The `/reconcile` HTTP
  contract + `OrchestratorClient.reconcile` become a bare trigger.

The macOS launcher's Apple-container enumeration stays for now; it becomes the
host controller's `list_live` when launch itself moves behind the broker (the
pulled-forward chunk 5, next in the stack).

Tests + pyright clean; pylint 10.0 on broker.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

bot_bottle/orchestrator/broker.py

@@ -32,7 +32,23 @@ from dataclasses import dataclass
 from typing import Protocol
 
 _JWT_HEADER = {"alg": "HS256", "typ": "JWT"}
-_ALLOWED_OPS = ("launch", "teardown")
+
+# The closed op vocabulary, split by kind (PRD "ids + static flags" rule):
+#   * mutation ops act on ONE bottle — they carry its id (+ static launch flags);
+#   * query ops enumerate host state — they carry NO ids or flags at all.
+# Each new host-privileged op is added here deliberately; `verify_request`
+# enforces the per-kind shape so the privileged surface can't drift.
+_MUTATION_OPS = ("launch", "teardown")
+_QUERY_OPS = ("list_live",)
+_ALLOWED_OPS = _MUTATION_OPS + _QUERY_OPS
+
+# Every claim key a signed request may carry — the request fields plus the two
+# per-signature envelope claims. A strict allow-list (open question 1, resolved
+# "yes"): an unknown key is rejected outright, so the schema can't silently widen
+# as ops are added.
+_ALLOWED_CLAIMS = frozenset(
+    {"op", "bottle_id", "source_ip", "image_ref", "slot", "jti", "iat"}
+)
 
 
 class BrokerAuthError(Exception):
@@ -56,17 +72,28 @@ class BrokerUnavailableError(Exception):
 
 
 @dataclass(frozen=True)
-class LaunchRequest:
-    """The structured, un-coercible launch/teardown request. Ids + flags
-    only — `image_ref` is a content-addressed id, never a path/argv."""
+class BrokerRequest:
+    """A structured, un-coercible broker request. Ids + static flags only —
+    `image_ref` is a content-addressed id, never a path/argv.
+
+    A **mutation** op (`launch`/`teardown`) names the bottle it acts on in
+    `bottle_id` (plus launch's static flags). A **query** op (`list_live`) acts
+    on no single bottle, so it carries nothing but its `op` name — `bottle_id`
+    and every flag stay empty, and `verify_request` rejects a query that smuggles
+    any in. `LaunchRequest` is kept as an alias for the launch/teardown callers."""
 
     op: str  # one of _ALLOWED_OPS
-    bottle_id: str
+    bottle_id: str = ""
     source_ip: str = ""
     image_ref: str = ""
     slot: int | None = None
 
 
+# Historical name — the request type predates the query ops. Kept so the launch
+# path (`OrchestratorCore`, `DockerBroker`) reads unchanged.
+LaunchRequest = BrokerRequest
+
+
 # --- minimal JWS/JWT (HS256), stdlib only ----------------------------------
 
 def _b64url(data: bytes) -> str:
@@ -118,20 +145,35 @@ def verify_request(token: str, secret: bytes) -> LaunchRequest:
         raise BrokerAuthError("unexpected header/alg")
     if not isinstance(claims, dict):
         raise BrokerAuthError("claims are not an object")
+    # Strict schema (open question 1): a signed request may carry ONLY the known
+    # claim keys, so the privileged surface can't widen by smuggling an extra
+    # claim past the fixed field checks below.
+    unknown = set(claims) - _ALLOWED_CLAIMS
+    if unknown:
+        raise BrokerAuthError(f"unknown claim(s): {', '.join(sorted(unknown))}")
 
     op = claims.get("op")
-    bottle_id = claims.get("bottle_id")
+    if not isinstance(op, str) or op not in _ALLOWED_OPS:
+        raise BrokerAuthError("request does not match the broker-request schema")
+    bottle_id = claims.get("bottle_id", "")
     source_ip = claims.get("source_ip", "")
     image_ref = claims.get("image_ref", "")
     slot = claims.get("slot")
     if (
-        not isinstance(op, str) or op not in _ALLOWED_OPS
-        or not isinstance(bottle_id, str) or not bottle_id
+        not isinstance(bottle_id, str)
         or not isinstance(source_ip, str) or not isinstance(image_ref, str)
         or not (slot is None or isinstance(slot, int))
     ):
-        raise BrokerAuthError("request does not match the launch-request schema")
-    return LaunchRequest(
+        raise BrokerAuthError("request does not match the broker-request schema")
+    # Per-kind shape: a mutation names exactly one bottle; a query names none and
+    # carries no flags (the "no arguments" rule for `list_live`). Enforcing both
+    # halves keeps `bottle_id`/flags from being a coercible field on a query op.
+    if op in _MUTATION_OPS:
+        if not bottle_id:
+            raise BrokerAuthError(f"{op} requires a bottle_id")
+    elif bottle_id or source_ip or image_ref or slot is not None:
+        raise BrokerAuthError(f"{op} is a query op and takes no ids or flags")
+    return BrokerRequest(
         op=op, bottle_id=bottle_id, source_ip=source_ip, image_ref=image_ref, slot=slot
     )
 
@@ -139,14 +181,18 @@ def verify_request(token: str, secret: bytes) -> LaunchRequest:
 # --- the broker itself ------------------------------------------------------
 
 class SubmitBroker(Protocol):
-    """The single method `OrchestratorCore` depends on: verify a signed token and
-    perform its op, returning the verified request. Both the in-process
-    `LaunchBroker` and the out-of-process `BrokerClient` (which relays the token
-    to the host control server) satisfy it structurally, so the core is unchanged
-    whether the backend is local or a real host service."""
+    """The broker surface `OrchestratorCore` depends on. `submit` verifies a
+    signed *mutation* token and performs its op, returning the verified request;
+    `list_live` verifies a signed `list_live` token and returns the source IPs of
+    the bottles the backend currently has running (reconcile's live set). Both the
+    in-process `LaunchBroker` and the out-of-process `BrokerClient` (which relays
+    the token to the host control server) satisfy it structurally, so the core is
+    unchanged whether the backend is local or a real host service."""
 
     def submit(self, token: str) -> LaunchRequest: ...
 
+    def list_live(self, token: str) -> list[str]: ...
+
 
 class LaunchBroker(abc.ABC):
     """Verifies a signed request came from the orchestrator, then performs
@@ -157,15 +203,35 @@ class LaunchBroker(abc.ABC):
         self._secret = secret
 
     def submit(self, token: str) -> LaunchRequest:
-        """Verify `token` and perform its op. Returns the verified request;
-        raises `BrokerAuthError` if provenance/shape fails."""
+        """Verify `token` and perform its mutation op. Returns the verified
+        request; raises `BrokerAuthError` if provenance/shape fails, or if the
+        token is a non-mutation op (a `list_live` token routed here by mistake)."""
         req = verify_request(token, self._secret)
         if req.op == "launch":
             self._launch(req)
-        else:
+        elif req.op == "teardown":
             self._teardown(req)
+        else:
+            raise BrokerAuthError(f"{req.op} is not a submit op")
         return req
 
+    def list_live(self, token: str) -> list[str]:
+        """Verify a `list_live` `token` and return the source IPs of the bottles
+        the backend currently has running. Raises `BrokerAuthError` on bad
+        provenance/shape or a non-`list_live` op; a backend enumeration failure
+        is converted to `BrokerUnavailableError` — the single "live set could not
+        be determined" signal reconcile catches to skip the sweep rather than
+        reaping healthy rows against a partial set (the out-of-process
+        `BrokerClient` raises the same on an enumeration 502 / no response)."""
+        req = verify_request(token, self._secret)
+        if req.op != "list_live":
+            raise BrokerAuthError(f"{req.op} is not a list_live op")
+        try:
+            return list(self._list_live())
+        except Exception as e:  # noqa: BLE001 — any enumeration failure means the
+            # live set is unknown; reconcile must skip, never reap against it.
+            raise BrokerUnavailableError(f"list_live enumeration failed: {e}") from e
+
     @abc.abstractmethod
     def _launch(self, req: LaunchRequest) -> None:
         ...
@@ -174,15 +240,31 @@ class LaunchBroker(abc.ABC):
     def _teardown(self, req: LaunchRequest) -> None:
         ...
 
+    @abc.abstractmethod
+    def _list_live(self) -> list[str]:
+        """Every running bottle's source IP, backend-native. Must raise (not
+        return a partial list) if the live set can't be determined
+        authoritatively, so reconcile skips rather than reaping healthy rows."""
+        ...
+
 
 class StubBroker(LaunchBroker):
     """Dev-harness broker: records verified requests without launching
-    anything. Exercises the full sign -> verify -> act contract in-process."""
+    anything. Exercises the full sign -> verify -> act contract in-process.
+
+    Its live set (`_list_live`) is, by default, every bottle it recorded a
+    launch for and no teardown since — the in-process analogue of enumerating
+    the backend. Tests that need to simulate a bottle dying out from under the
+    registry (the case reconcile exists for) set `live_source_ips` explicitly to
+    override that derived set."""
 
     def __init__(self, secret: bytes) -> None:
         super().__init__(secret)
         self.launched: list[LaunchRequest] = []
         self.torn_down: list[LaunchRequest] = []
+        # When not None, the exact live set `_list_live` reports — lets a test
+        # say "only these IPs are still up" regardless of what was launched.
+        self.live_source_ips: list[str] | None = None
 
     def _launch(self, req: LaunchRequest) -> None:
         self.launched.append(req)
@@ -190,10 +272,18 @@ class StubBroker(LaunchBroker):
     def _teardown(self, req: LaunchRequest) -> None:
         self.torn_down.append(req)
 
+    def _list_live(self) -> list[str]:
+        if self.live_source_ips is not None:
+            return list(self.live_source_ips)
+        torn = {r.bottle_id for r in self.torn_down}
+        return [r.source_ip for r in self.launched
+                if r.bottle_id not in torn and r.source_ip]
+
 
 __all__ = [
     "BrokerAuthError",
     "BrokerUnavailableError",
+    "BrokerRequest",
     "LaunchRequest",
     "SubmitBroker",
     "LaunchBroker",

bot_bottle/orchestrator/broker_client.py

@@ -75,6 +75,37 @@ class BrokerClient:
             # launched the bottle, so this is NOT a definite failure.
             raise BrokerUnavailableError(f"POST /broker: {e}") from e
 
+    def list_live(self, token: str) -> list[str]:
+        """POST the signed `list_live` token to the host controller and return
+        the source IPs of the bottles it enumerated as running.
+
+        Raises `BrokerAuthError` on a fail-closed 401 (bad provenance/schema);
+        `BrokerClientError` on any other non-success status or a malformed body
+        (a backend-enumeration 502 included); or `BrokerUnavailableError` if no
+        response is obtained. A query has no backend side effect, so — unlike
+        `submit` — every one of these is a *definite* "no live set"; reconcile
+        catches all three and skips the sweep rather than reaping against an
+        unknown or partial set."""
+        data = json.dumps({"token": token}).encode()
+        req = urllib.request.Request(
+            f"{self._base}/broker/live", data=data, method="POST",
+            headers={"Content-Type": "application/json"},
+        )
+        try:
+            with urllib.request.urlopen(req, timeout=self._timeout) as resp:
+                return _source_ips_from(_json_object(resp.read()))
+        except urllib.error.HTTPError as e:
+            detail = _error_detail(e)
+            if e.code == 401:
+                raise BrokerAuthError(
+                    detail or "host controller rejected the request"
+                ) from e
+            raise BrokerClientError(
+                f"POST /broker/live: HTTP {e.code} {detail}".rstrip()
+            ) from e
+        except (urllib.error.URLError, TimeoutError, OSError) as e:
+            raise BrokerUnavailableError(f"POST /broker/live: {e}") from e
+
 
 def _json_object(raw: bytes) -> dict[str, object]:
     """Parse a JSON object, tolerating an empty or malformed body (→ {}), like
@@ -99,6 +130,17 @@ def _error_detail(e: urllib.error.HTTPError) -> str:
     return detail if isinstance(detail, str) else ""
 
 
+def _source_ips_from(payload: dict[str, object]) -> list[str]:
+    """The `source_ips` list from a `/broker/live` response — every string
+    entry, ignoring any non-string the host controller should never send. A
+    missing/!list field is a malformed response (the query is meaningless
+    without it), so it fails rather than silently reconciling against []."""
+    raw = payload.get("source_ips")
+    if not isinstance(raw, list):
+        raise BrokerClientError("host controller response missing source_ips")
+    return [ip for ip in raw if isinstance(ip, str) and ip]
+
+
 def _request_from(payload: dict[str, object]) -> LaunchRequest:
     """Reconstruct the verified `LaunchRequest` the controller echoed, so the
     returned value matches the in-process broker's (which returns the request it

bot_bottle/orchestrator/client.py

@@ -15,7 +15,6 @@ from __future__ import annotations
 import json
 import urllib.error
 import urllib.request
-from collections.abc import Iterable
 from dataclasses import dataclass
 
 from ..log import debug
@@ -194,14 +193,13 @@ class OrchestratorClient:
             raise OrchestratorClientError(f"teardown {bottle_id}: HTTP {status}")
         return True
 
-    def reconcile(
-        self, live_source_ips: Iterable[str], *, grace_seconds: float | None = None,
-    ) -> list[str]:
-        """Drop registry rows for bottles that are no longer running
-        (`POST /reconcile`), returning the reaped bottle ids. `live_source_ips`
-        is the caller's enumeration of its live bottles — the orchestrator
-        can't see the backend from inside the orchestrator container/VM."""
-        body: dict[str, object] = {"live_source_ips": list(live_source_ips)}
+    def reconcile(self, *, grace_seconds: float | None = None) -> list[str]:
+        """Trigger the orchestrator's self-heal sweep (`POST /reconcile`),
+        returning the reaped bottle ids. The caller no longer enumerates the
+        live set: the orchestrator pulls it from the host controller
+        (`list_live`) itself, so this is a bare trigger with an optional
+        `grace_seconds`."""
+        body: dict[str, object] = {}
         if grace_seconds is not None:
             body["grace_seconds"] = grace_seconds
         payload = self._ok("POST", "/reconcile", body)

bot_bottle/orchestrator/docker_broker.py

@@ -71,6 +71,31 @@ class DockerBroker(LaunchBroker):
                 f"docker rm failed for {req.bottle_id}: {proc.stderr.strip()}"
             )
 
+    def _list_live(self) -> list[str]:
+        """Every running bottle container's network IP, from `docker ps` filtered
+        to this broker's label. Raises `DockerBrokerError` if the listing or any
+        inspect fails, so reconcile treats the live set as unknown and skips the
+        sweep rather than reaping healthy rows against a partial enumeration."""
+        listing = self._docker([
+            "docker", "ps", "--filter", f"label={BOTTLE_ID_LABEL}",
+            "--format", "{{.ID}}",
+        ])
+        if listing.returncode != 0:
+            raise DockerBrokerError(f"docker ps failed: {listing.stderr.strip()}")
+        ips: list[str] = []
+        for cid in (line for line in listing.stdout.splitlines() if line):
+            inspected = self._docker([
+                "docker", "inspect", "--format",
+                "{{range .NetworkSettings.Networks}}{{.IPAddress}} {{end}}", cid,
+            ])
+            if inspected.returncode != 0:
+                raise DockerBrokerError(
+                    f"docker inspect {cid} failed; live set is not authoritative: "
+                    f"{inspected.stderr.strip()}"
+                )
+            ips.extend(ip for ip in inspected.stdout.split() if ip)
+        return ips
+
 
 __all__ = [
     "DockerBroker",

bot_bottle/orchestrator/host_server.py

@@ -8,10 +8,19 @@ This module is that service — the single privileged host component — reached
 plane's shape (`orchestrator/server.py`): a pure `dispatch()` for socket-free
 testing, wrapped by a thin stdlib `http.server` adapter.
 
-    GET  /health  -> 200 {"status": "ok"}
-    POST /broker  -> 200 {"op", "bottle_id", "source_ip", "image_ref", "slot"}
-                     400 (bad body) | 401 (bad provenance/schema) | 502 (backend)
-                     body: {"token": "<signed launch/teardown JWT>"}
+    GET  /health       -> 200 {"status": "ok"}
+    POST /broker       -> 200 {"op", "bottle_id", "source_ip", "image_ref", "slot"}
+                          400 (bad body) | 401 (bad provenance/schema) | 502 (backend)
+                          body: {"token": "<signed launch/teardown JWT>"}
+    POST /broker/live  -> 200 {"source_ips": [...]}
+                          400 (bad body) | 401 (bad provenance/schema) | 502 (backend)
+                          body: {"token": "<signed list_live JWT>"}
+
+The op vocabulary grows one signed op at a time (PRD gap 3); each verb reaches
+the backend only through `verify_request`, so nothing free-form ever crosses the
+wire. `/broker/live` is the first query op — reconcile's live-bottle enumeration,
+which the orchestrator (blind to the backend from inside its container) now pulls
+from the host controller instead of being handed by the CLI.
 
 Only the **signed token** crosses the wire; the server holds the shared HS256
 secret and a real `LaunchBroker` (e.g. `DockerBroker`) and runs the existing
@@ -150,6 +159,26 @@ def dispatch(  # pylint: disable=too-many-return-statements
             "slot": req.slot,
         }
 
+    if method == "POST" and route == "/broker/live":
+        try:
+            data = _parse_json_object(body)
+        except ValueError as e:
+            return 400, {"error": f"invalid JSON: {e}"}
+        token = data.get("token")
+        if not isinstance(token, str) or not token:
+            return 400, {"error": "token (string) is required"}
+        try:
+            source_ips = broker.list_live(token)
+        except BrokerAuthError as e:
+            # Fail-closed: bad signature, malformed token, or a non-list_live op.
+            return 401, {"error": f"broker auth failed: {e}"}
+        except Exception as e:  # noqa: BLE001 — a backend enumeration failure
+            # (docker down, inspect failed) is operational, not a control-plane
+            # bug; surface it as a 502 so the caller skips reconcile rather than
+            # reaping against a partial set, and keep the controller up.
+            return 502, {"error": f"backend enumeration failed: {e}"}
+        return 200, {"source_ips": source_ips}
+
     return 404, {"error": "not found"}
 
 

bot_bottle/orchestrator/server.py

@@ -18,8 +18,8 @@ vsock / unix-socket portability caveats):
                                          body: {"env_var_secret"}
     DELETE /bottles/<bottle_id>        -> 200 {"torn_down": true} | 404  (teardown)
     POST   /reconcile                 -> 200 {"reaped": [bottle_id, ...]}
-                                         body: {"live_source_ips": [...],
-                                                ["grace_seconds"]}
+                                         body: {["grace_seconds"]}  (live set is
+                                         pulled from the host controller, not sent)
     POST   /attribute                 -> 200 {"bottle_id"} | 403
     POST   /resolve                   -> 200 {"bottle_id","policy"} | 403
                                          body: {"source_ip","identity_token"}
@@ -207,20 +207,15 @@ def dispatch(  # pylint: disable=too-many-return-statements,too-many-branches
         return 404, {"error": "no such bottle"}
 
     if method == "POST" and route == "/reconcile":
-        # Host-driven self-heal: the caller enumerates its live bottles (only
-        # the host can see the backend) and the orchestrator drops rows for
-        # every other active bottle. Trusted-caller only — an agent that could
-        # reach this would be able to unregister its neighbours.
+        # Host-driven self-heal trigger: the orchestrator pulls its own live set
+        # from the host controller (`list_live`) and drops rows for every other
+        # active bottle — the caller no longer supplies the live IPs. Trusted-
+        # caller only: an agent that could reach this would trigger a sweep that
+        # unregisters its neighbours. Body is optional (just `grace_seconds`).
         try:
             data = _parse_json_object(body)
         except ValueError as e:
             return 400, {"error": f"invalid JSON: {e}"}
-        raw_ips = data.get("live_source_ips")
-        if not isinstance(raw_ips, list):
-            return 400, {"error": "live_source_ips (list of strings) is required"}
-        if any(not isinstance(ip, str) or not ip for ip in raw_ips):
-            return 400, {"error": "live_source_ips must contain non-empty strings"}
-        live = raw_ips
         grace = data.get("grace_seconds")
         kwargs: dict[str, float] = {}
         if grace is not None:
@@ -230,7 +225,7 @@ def dispatch(  # pylint: disable=too-many-return-statements,too-many-branches
             if not math.isfinite(parsed_grace) or parsed_grace < 0:
                 return 400, {"error": "grace_seconds must be a non-negative finite number"}
             kwargs["grace_seconds"] = parsed_grace
-        return 200, {"reaped": orch.reconcile(live, **kwargs)}
+        return 200, {"reaped": orch.reconcile(**kwargs)}
 
     if method == "POST" and route == "/attribute":
         try:

bot_bottle/orchestrator/service.py

@@ -22,10 +22,17 @@ Launch lifecycle:
 from __future__ import annotations
 
 import json
-from collections.abc import Iterable
 from datetime import datetime, timezone
 
-from .broker import BrokerUnavailableError, LaunchRequest, SubmitBroker, sign_request
+from .. import log
+from .broker import (
+    BrokerAuthError,
+    BrokerUnavailableError,
+    LaunchRequest,
+    SubmitBroker,
+    sign_request,
+)
+from .broker_client import BrokerClientError
 from .store.registry_store import DEFAULT_REAP_GRACE_SECONDS, BottleRecord, RegistryStore
 from .supervisor import (
     AuditEntry,
@@ -145,22 +152,36 @@ class OrchestratorCore:
 
     def reconcile(
         self,
-        live_source_ips: Iterable[str],
         *,
         grace_seconds: float = DEFAULT_REAP_GRACE_SECONDS,
     ) -> list[str]:
         """Drop registry rows for bottles that are no longer running, and
         forget their in-memory egress tokens. Returns the reaped bottle ids.
 
-        The caller supplies the live set because only the host can enumerate
-        its own containers — the orchestrator runs *inside* the infra
-        container and has no view of the backend. Deliberately does not
-        broker a teardown: the container is already gone, so there is nothing
-        to stop, and a broker error must not stop the sweep from clearing
-        the row that would otherwise brick the next bottle at that address.
+        The live set is pulled from the **broker** (`list_live`), not passed in:
+        only the host can enumerate its own containers, and the host controller
+        is now that host component, so the orchestrator — blind to the backend
+        from inside its container — asks it over the same signed seam it launches
+        through. (Before the host control server this was a `live_source_ips`
+        argument the CLI handed in; the tell that the orchestrator couldn't see
+        the backend goes away with it — PRD gap 3.)
+
+        Fail-safe: if the broker can't return an authoritative live set
+        (unreachable, timed out, enumeration failed), the sweep is **skipped**,
+        never run against an empty/partial set — reaping a healthy bottle is far
+        worse than leaving an orphan one more cycle. Deliberately does not broker
+        a teardown: a reaped container is already gone, so there is nothing to
+        stop.
 
         See `RegistryStore.reap_absent` for why orphans accumulate and why
         they are harmful rather than merely untidy."""
+        token = sign_request(LaunchRequest(op="list_live"), self._secret)
+        try:
+            live_source_ips = self._broker.list_live(token)
+        except (BrokerUnavailableError, BrokerAuthError, BrokerClientError) as e:
+            log.info("reconcile skipped: live set unavailable",
+                     context={"error": str(e)})
+            return []
         reaped = self.registry.reap_absent(
             live_source_ips, grace_seconds=grace_seconds)
         for rec in reaped: