feat(orchestrator): list_live broker op + reconcile via broker (#468)
tracker-policy-pr / check-pr (pull_request) Successful in 10s
test / unit (pull_request) Successful in 52s
lint / lint (push) Successful in 58s
test / integration-docker (pull_request) Failing after 2m33s
test / coverage (pull_request) Has been skipped

Chunk 3 of the host-control-server stack: grow the broker op vocabulary
(PRD gap 3), starting with `list_live`, and invert `reconcile` onto it.

- broker: split the closed op vocabulary into mutation (`launch`/`teardown`,
  carry a bottle id + static flags) and query (`list_live`, carries nothing
  but its op name) kinds. `verify_request` now enforces a **strict schema**
  (open question 1, resolved yes): unknown claim keys are rejected, a mutation
  must name its bottle, and a query that smuggles any id/flag is refused.
- broker verb: `LaunchBroker.list_live` / `SubmitBroker.list_live` return the
  backend's live source IPs; a backend enumeration failure is converted to the
  single `BrokerUnavailableError` "live set unknown" signal. `DockerBroker`
  enumerates its labelled containers; `StubBroker` derives from launches (or a
  test override).
- host controller: `POST /broker/live` verifies a signed `list_live` token and
  returns `{source_ips}`; `BrokerClient.list_live` is its drop-in client.
- reconcile: `OrchestratorCore.reconcile()` drops the `live_source_ips`
  parameter and pulls the live set from the broker itself — the tell that the
  orchestrator couldn't see the backend goes away. **Fail-safe**: if the broker
  can't return an authoritative set the sweep is skipped, never run against an
  empty/partial set (which would reap healthy rows). The `/reconcile` HTTP
  contract + `OrchestratorClient.reconcile` become a bare trigger.

The macOS launcher's Apple-container enumeration stays for now; it becomes the
host controller's `list_live` when launch itself moves behind the broker (the
pulled-forward chunk 5, next in the stack).

Tests + pyright clean; pylint 10.0 on broker.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
2026-07-26 17:52:44 +00:00
parent 0e2af7de62
commit 7005b22bcf
16 changed files with 553 additions and 114 deletions
+42
View File
@@ -75,6 +75,37 @@ class BrokerClient:
# launched the bottle, so this is NOT a definite failure.
raise BrokerUnavailableError(f"POST /broker: {e}") from e
def list_live(self, token: str) -> list[str]:
"""POST the signed `list_live` token to the host controller and return
the source IPs of the bottles it enumerated as running.
Raises `BrokerAuthError` on a fail-closed 401 (bad provenance/schema);
`BrokerClientError` on any other non-success status or a malformed body
(a backend-enumeration 502 included); or `BrokerUnavailableError` if no
response is obtained. A query has no backend side effect, so — unlike
`submit` — every one of these is a *definite* "no live set"; reconcile
catches all three and skips the sweep rather than reaping against an
unknown or partial set."""
data = json.dumps({"token": token}).encode()
req = urllib.request.Request(
f"{self._base}/broker/live", data=data, method="POST",
headers={"Content-Type": "application/json"},
)
try:
with urllib.request.urlopen(req, timeout=self._timeout) as resp:
return _source_ips_from(_json_object(resp.read()))
except urllib.error.HTTPError as e:
detail = _error_detail(e)
if e.code == 401:
raise BrokerAuthError(
detail or "host controller rejected the request"
) from e
raise BrokerClientError(
f"POST /broker/live: HTTP {e.code} {detail}".rstrip()
) from e
except (urllib.error.URLError, TimeoutError, OSError) as e:
raise BrokerUnavailableError(f"POST /broker/live: {e}") from e
def _json_object(raw: bytes) -> dict[str, object]:
"""Parse a JSON object, tolerating an empty or malformed body (→ {}), like
@@ -99,6 +130,17 @@ def _error_detail(e: urllib.error.HTTPError) -> str:
return detail if isinstance(detail, str) else ""
def _source_ips_from(payload: dict[str, object]) -> list[str]:
"""The `source_ips` list from a `/broker/live` response — every string
entry, ignoring any non-string the host controller should never send. A
missing/!list field is a malformed response (the query is meaningless
without it), so it fails rather than silently reconciling against []."""
raw = payload.get("source_ips")
if not isinstance(raw, list):
raise BrokerClientError("host controller response missing source_ips")
return [ip for ip in raw if isinstance(ip, str) and ip]
def _request_from(payload: dict[str, object]) -> LaunchRequest:
"""Reconstruct the verified `LaunchRequest` the controller echoed, so the
returned value matches the in-process broker's (which returns the request it