feat(orchestrator): list_live broker op + reconcile via broker (#468)
tracker-policy-pr / check-pr (pull_request) Successful in 10s
test / unit (pull_request) Successful in 52s
lint / lint (push) Successful in 58s
test / integration-docker (pull_request) Failing after 2m33s
test / coverage (pull_request) Has been skipped

Chunk 3 of the host-control-server stack: grow the broker op vocabulary
(PRD gap 3), starting with `list_live`, and invert `reconcile` onto it.

- broker: split the closed op vocabulary into mutation (`launch`/`teardown`,
  carry a bottle id + static flags) and query (`list_live`, carries nothing
  but its op name) kinds. `verify_request` now enforces a **strict schema**
  (open question 1, resolved yes): unknown claim keys are rejected, a mutation
  must name its bottle, and a query that smuggles any id/flag is refused.
- broker verb: `LaunchBroker.list_live` / `SubmitBroker.list_live` return the
  backend's live source IPs; a backend enumeration failure is converted to the
  single `BrokerUnavailableError` "live set unknown" signal. `DockerBroker`
  enumerates its labelled containers; `StubBroker` derives from launches (or a
  test override).
- host controller: `POST /broker/live` verifies a signed `list_live` token and
  returns `{source_ips}`; `BrokerClient.list_live` is its drop-in client.
- reconcile: `OrchestratorCore.reconcile()` drops the `live_source_ips`
  parameter and pulls the live set from the broker itself — the tell that the
  orchestrator couldn't see the backend goes away. **Fail-safe**: if the broker
  can't return an authoritative set the sweep is skipped, never run against an
  empty/partial set (which would reap healthy rows). The `/reconcile` HTTP
  contract + `OrchestratorClient.reconcile` become a bare trigger.

The macOS launcher's Apple-container enumeration stays for now; it becomes the
host controller's `list_live` when launch itself moves behind the broker (the
pulled-forward chunk 5, next in the stack).

Tests + pyright clean; pylint 10.0 on broker.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
2026-07-26 17:52:44 +00:00
parent 0e2af7de62
commit 7005b22bcf
16 changed files with 553 additions and 114 deletions
@@ -166,9 +166,17 @@ def register_agent(
# collide with this bottle's — and `by_source_ip` fail-closes on ambiguity,
# which would resolve no policy at all and deny every host. Best-effort: a
# reconciliation failure must not block an otherwise-fine launch.
#
# The orchestrator now pulls the live set from the broker itself, so this is
# a bare trigger. Until macOS moves onto the host controller (the follow-up
# that routes launch through the broker), the orchestrator's broker is the
# stub, whose live set is the bottles it recorded launches for — so the sweep
# only under-reaps (leaves an orphan a cycle longer), never reaps a healthy
# bottle. The Apple-container enumeration below (`live_source_ips`) becomes
# that host controller's `list_live` there.
try:
client.reconcile(live_source_ips(endpoint.network))
except (OrchestratorClientError, EnumerationError) as e:
client.reconcile()
except OrchestratorClientError as e:
info(f"registry reconciliation skipped: {e}")
reg = provision_bottle(
client, source_ip, egress_plan, git_gate_plan, MacosGatewayTransport(),
+109 -19
View File
@@ -32,7 +32,23 @@ from dataclasses import dataclass
from typing import Protocol
_JWT_HEADER = {"alg": "HS256", "typ": "JWT"}
_ALLOWED_OPS = ("launch", "teardown")
# The closed op vocabulary, split by kind (PRD "ids + static flags" rule):
# * mutation ops act on ONE bottle — they carry its id (+ static launch flags);
# * query ops enumerate host state — they carry NO ids or flags at all.
# Each new host-privileged op is added here deliberately; `verify_request`
# enforces the per-kind shape so the privileged surface can't drift.
_MUTATION_OPS = ("launch", "teardown")
_QUERY_OPS = ("list_live",)
_ALLOWED_OPS = _MUTATION_OPS + _QUERY_OPS
# Every claim key a signed request may carry — the request fields plus the two
# per-signature envelope claims. A strict allow-list (open question 1, resolved
# "yes"): an unknown key is rejected outright, so the schema can't silently widen
# as ops are added.
_ALLOWED_CLAIMS = frozenset(
{"op", "bottle_id", "source_ip", "image_ref", "slot", "jti", "iat"}
)
class BrokerAuthError(Exception):
@@ -56,17 +72,28 @@ class BrokerUnavailableError(Exception):
@dataclass(frozen=True)
class LaunchRequest:
"""The structured, un-coercible launch/teardown request. Ids + flags
only — `image_ref` is a content-addressed id, never a path/argv."""
class BrokerRequest:
"""A structured, un-coercible broker request. Ids + static flags only —
`image_ref` is a content-addressed id, never a path/argv.
A **mutation** op (`launch`/`teardown`) names the bottle it acts on in
`bottle_id` (plus launch's static flags). A **query** op (`list_live`) acts
on no single bottle, so it carries nothing but its `op` name — `bottle_id`
and every flag stay empty, and `verify_request` rejects a query that smuggles
any in. `LaunchRequest` is kept as an alias for the launch/teardown callers."""
op: str # one of _ALLOWED_OPS
bottle_id: str
bottle_id: str = ""
source_ip: str = ""
image_ref: str = ""
slot: int | None = None
# Historical name — the request type predates the query ops. Kept so the launch
# path (`OrchestratorCore`, `DockerBroker`) reads unchanged.
LaunchRequest = BrokerRequest
# --- minimal JWS/JWT (HS256), stdlib only ----------------------------------
def _b64url(data: bytes) -> str:
@@ -118,20 +145,35 @@ def verify_request(token: str, secret: bytes) -> LaunchRequest:
raise BrokerAuthError("unexpected header/alg")
if not isinstance(claims, dict):
raise BrokerAuthError("claims are not an object")
# Strict schema (open question 1): a signed request may carry ONLY the known
# claim keys, so the privileged surface can't widen by smuggling an extra
# claim past the fixed field checks below.
unknown = set(claims) - _ALLOWED_CLAIMS
if unknown:
raise BrokerAuthError(f"unknown claim(s): {', '.join(sorted(unknown))}")
op = claims.get("op")
bottle_id = claims.get("bottle_id")
if not isinstance(op, str) or op not in _ALLOWED_OPS:
raise BrokerAuthError("request does not match the broker-request schema")
bottle_id = claims.get("bottle_id", "")
source_ip = claims.get("source_ip", "")
image_ref = claims.get("image_ref", "")
slot = claims.get("slot")
if (
not isinstance(op, str) or op not in _ALLOWED_OPS
or not isinstance(bottle_id, str) or not bottle_id
not isinstance(bottle_id, str)
or not isinstance(source_ip, str) or not isinstance(image_ref, str)
or not (slot is None or isinstance(slot, int))
):
raise BrokerAuthError("request does not match the launch-request schema")
return LaunchRequest(
raise BrokerAuthError("request does not match the broker-request schema")
# Per-kind shape: a mutation names exactly one bottle; a query names none and
# carries no flags (the "no arguments" rule for `list_live`). Enforcing both
# halves keeps `bottle_id`/flags from being a coercible field on a query op.
if op in _MUTATION_OPS:
if not bottle_id:
raise BrokerAuthError(f"{op} requires a bottle_id")
elif bottle_id or source_ip or image_ref or slot is not None:
raise BrokerAuthError(f"{op} is a query op and takes no ids or flags")
return BrokerRequest(
op=op, bottle_id=bottle_id, source_ip=source_ip, image_ref=image_ref, slot=slot
)
@@ -139,14 +181,18 @@ def verify_request(token: str, secret: bytes) -> LaunchRequest:
# --- the broker itself ------------------------------------------------------
class SubmitBroker(Protocol):
"""The single method `OrchestratorCore` depends on: verify a signed token and
perform its op, returning the verified request. Both the in-process
`LaunchBroker` and the out-of-process `BrokerClient` (which relays the token
to the host control server) satisfy it structurally, so the core is unchanged
whether the backend is local or a real host service."""
"""The broker surface `OrchestratorCore` depends on. `submit` verifies a
signed *mutation* token and performs its op, returning the verified request;
`list_live` verifies a signed `list_live` token and returns the source IPs of
the bottles the backend currently has running (reconcile's live set). Both the
in-process `LaunchBroker` and the out-of-process `BrokerClient` (which relays
the token to the host control server) satisfy it structurally, so the core is
unchanged whether the backend is local or a real host service."""
def submit(self, token: str) -> LaunchRequest: ...
def list_live(self, token: str) -> list[str]: ...
class LaunchBroker(abc.ABC):
"""Verifies a signed request came from the orchestrator, then performs
@@ -157,15 +203,35 @@ class LaunchBroker(abc.ABC):
self._secret = secret
def submit(self, token: str) -> LaunchRequest:
"""Verify `token` and perform its op. Returns the verified request;
raises `BrokerAuthError` if provenance/shape fails."""
"""Verify `token` and perform its mutation op. Returns the verified
request; raises `BrokerAuthError` if provenance/shape fails, or if the
token is a non-mutation op (a `list_live` token routed here by mistake)."""
req = verify_request(token, self._secret)
if req.op == "launch":
self._launch(req)
else:
elif req.op == "teardown":
self._teardown(req)
else:
raise BrokerAuthError(f"{req.op} is not a submit op")
return req
def list_live(self, token: str) -> list[str]:
"""Verify a `list_live` `token` and return the source IPs of the bottles
the backend currently has running. Raises `BrokerAuthError` on bad
provenance/shape or a non-`list_live` op; a backend enumeration failure
is converted to `BrokerUnavailableError` — the single "live set could not
be determined" signal reconcile catches to skip the sweep rather than
reaping healthy rows against a partial set (the out-of-process
`BrokerClient` raises the same on an enumeration 502 / no response)."""
req = verify_request(token, self._secret)
if req.op != "list_live":
raise BrokerAuthError(f"{req.op} is not a list_live op")
try:
return list(self._list_live())
except Exception as e: # noqa: BLE001 — any enumeration failure means the
# live set is unknown; reconcile must skip, never reap against it.
raise BrokerUnavailableError(f"list_live enumeration failed: {e}") from e
@abc.abstractmethod
def _launch(self, req: LaunchRequest) -> None:
...
@@ -174,15 +240,31 @@ class LaunchBroker(abc.ABC):
def _teardown(self, req: LaunchRequest) -> None:
...
@abc.abstractmethod
def _list_live(self) -> list[str]:
"""Every running bottle's source IP, backend-native. Must raise (not
return a partial list) if the live set can't be determined
authoritatively, so reconcile skips rather than reaping healthy rows."""
...
class StubBroker(LaunchBroker):
"""Dev-harness broker: records verified requests without launching
anything. Exercises the full sign -> verify -> act contract in-process."""
anything. Exercises the full sign -> verify -> act contract in-process.
Its live set (`_list_live`) is, by default, every bottle it recorded a
launch for and no teardown since — the in-process analogue of enumerating
the backend. Tests that need to simulate a bottle dying out from under the
registry (the case reconcile exists for) set `live_source_ips` explicitly to
override that derived set."""
def __init__(self, secret: bytes) -> None:
super().__init__(secret)
self.launched: list[LaunchRequest] = []
self.torn_down: list[LaunchRequest] = []
# When not None, the exact live set `_list_live` reports — lets a test
# say "only these IPs are still up" regardless of what was launched.
self.live_source_ips: list[str] | None = None
def _launch(self, req: LaunchRequest) -> None:
self.launched.append(req)
@@ -190,10 +272,18 @@ class StubBroker(LaunchBroker):
def _teardown(self, req: LaunchRequest) -> None:
self.torn_down.append(req)
def _list_live(self) -> list[str]:
if self.live_source_ips is not None:
return list(self.live_source_ips)
torn = {r.bottle_id for r in self.torn_down}
return [r.source_ip for r in self.launched
if r.bottle_id not in torn and r.source_ip]
__all__ = [
"BrokerAuthError",
"BrokerUnavailableError",
"BrokerRequest",
"LaunchRequest",
"SubmitBroker",
"LaunchBroker",
+42
View File
@@ -75,6 +75,37 @@ class BrokerClient:
# launched the bottle, so this is NOT a definite failure.
raise BrokerUnavailableError(f"POST /broker: {e}") from e
def list_live(self, token: str) -> list[str]:
"""POST the signed `list_live` token to the host controller and return
the source IPs of the bottles it enumerated as running.
Raises `BrokerAuthError` on a fail-closed 401 (bad provenance/schema);
`BrokerClientError` on any other non-success status or a malformed body
(a backend-enumeration 502 included); or `BrokerUnavailableError` if no
response is obtained. A query has no backend side effect, so — unlike
`submit` — every one of these is a *definite* "no live set"; reconcile
catches all three and skips the sweep rather than reaping against an
unknown or partial set."""
data = json.dumps({"token": token}).encode()
req = urllib.request.Request(
f"{self._base}/broker/live", data=data, method="POST",
headers={"Content-Type": "application/json"},
)
try:
with urllib.request.urlopen(req, timeout=self._timeout) as resp:
return _source_ips_from(_json_object(resp.read()))
except urllib.error.HTTPError as e:
detail = _error_detail(e)
if e.code == 401:
raise BrokerAuthError(
detail or "host controller rejected the request"
) from e
raise BrokerClientError(
f"POST /broker/live: HTTP {e.code} {detail}".rstrip()
) from e
except (urllib.error.URLError, TimeoutError, OSError) as e:
raise BrokerUnavailableError(f"POST /broker/live: {e}") from e
def _json_object(raw: bytes) -> dict[str, object]:
"""Parse a JSON object, tolerating an empty or malformed body (→ {}), like
@@ -99,6 +130,17 @@ def _error_detail(e: urllib.error.HTTPError) -> str:
return detail if isinstance(detail, str) else ""
def _source_ips_from(payload: dict[str, object]) -> list[str]:
"""The `source_ips` list from a `/broker/live` response — every string
entry, ignoring any non-string the host controller should never send. A
missing/!list field is a malformed response (the query is meaningless
without it), so it fails rather than silently reconciling against []."""
raw = payload.get("source_ips")
if not isinstance(raw, list):
raise BrokerClientError("host controller response missing source_ips")
return [ip for ip in raw if isinstance(ip, str) and ip]
def _request_from(payload: dict[str, object]) -> LaunchRequest:
"""Reconstruct the verified `LaunchRequest` the controller echoed, so the
returned value matches the in-process broker's (which returns the request it
+7 -9
View File
@@ -15,7 +15,6 @@ from __future__ import annotations
import json
import urllib.error
import urllib.request
from collections.abc import Iterable
from dataclasses import dataclass
from ..log import debug
@@ -194,14 +193,13 @@ class OrchestratorClient:
raise OrchestratorClientError(f"teardown {bottle_id}: HTTP {status}")
return True
def reconcile(
self, live_source_ips: Iterable[str], *, grace_seconds: float | None = None,
) -> list[str]:
"""Drop registry rows for bottles that are no longer running
(`POST /reconcile`), returning the reaped bottle ids. `live_source_ips`
is the caller's enumeration of its live bottles — the orchestrator
can't see the backend from inside the orchestrator container/VM."""
body: dict[str, object] = {"live_source_ips": list(live_source_ips)}
def reconcile(self, *, grace_seconds: float | None = None) -> list[str]:
"""Trigger the orchestrator's self-heal sweep (`POST /reconcile`),
returning the reaped bottle ids. The caller no longer enumerates the
live set: the orchestrator pulls it from the host controller
(`list_live`) itself, so this is a bare trigger with an optional
`grace_seconds`."""
body: dict[str, object] = {}
if grace_seconds is not None:
body["grace_seconds"] = grace_seconds
payload = self._ok("POST", "/reconcile", body)
+25
View File
@@ -71,6 +71,31 @@ class DockerBroker(LaunchBroker):
f"docker rm failed for {req.bottle_id}: {proc.stderr.strip()}"
)
def _list_live(self) -> list[str]:
"""Every running bottle container's network IP, from `docker ps` filtered
to this broker's label. Raises `DockerBrokerError` if the listing or any
inspect fails, so reconcile treats the live set as unknown and skips the
sweep rather than reaping healthy rows against a partial enumeration."""
listing = self._docker([
"docker", "ps", "--filter", f"label={BOTTLE_ID_LABEL}",
"--format", "{{.ID}}",
])
if listing.returncode != 0:
raise DockerBrokerError(f"docker ps failed: {listing.stderr.strip()}")
ips: list[str] = []
for cid in (line for line in listing.stdout.splitlines() if line):
inspected = self._docker([
"docker", "inspect", "--format",
"{{range .NetworkSettings.Networks}}{{.IPAddress}} {{end}}", cid,
])
if inspected.returncode != 0:
raise DockerBrokerError(
f"docker inspect {cid} failed; live set is not authoritative: "
f"{inspected.stderr.strip()}"
)
ips.extend(ip for ip in inspected.stdout.split() if ip)
return ips
__all__ = [
"DockerBroker",
+33 -4
View File
@@ -8,10 +8,19 @@ This module is that service — the single privileged host component — reached
plane's shape (`orchestrator/server.py`): a pure `dispatch()` for socket-free
testing, wrapped by a thin stdlib `http.server` adapter.
GET /health -> 200 {"status": "ok"}
POST /broker -> 200 {"op", "bottle_id", "source_ip", "image_ref", "slot"}
400 (bad body) | 401 (bad provenance/schema) | 502 (backend)
body: {"token": "<signed launch/teardown JWT>"}
GET /health -> 200 {"status": "ok"}
POST /broker -> 200 {"op", "bottle_id", "source_ip", "image_ref", "slot"}
400 (bad body) | 401 (bad provenance/schema) | 502 (backend)
body: {"token": "<signed launch/teardown JWT>"}
POST /broker/live -> 200 {"source_ips": [...]}
400 (bad body) | 401 (bad provenance/schema) | 502 (backend)
body: {"token": "<signed list_live JWT>"}
The op vocabulary grows one signed op at a time (PRD gap 3); each verb reaches
the backend only through `verify_request`, so nothing free-form ever crosses the
wire. `/broker/live` is the first query op — reconcile's live-bottle enumeration,
which the orchestrator (blind to the backend from inside its container) now pulls
from the host controller instead of being handed by the CLI.
Only the **signed token** crosses the wire; the server holds the shared HS256
secret and a real `LaunchBroker` (e.g. `DockerBroker`) and runs the existing
@@ -150,6 +159,26 @@ def dispatch( # pylint: disable=too-many-return-statements
"slot": req.slot,
}
if method == "POST" and route == "/broker/live":
try:
data = _parse_json_object(body)
except ValueError as e:
return 400, {"error": f"invalid JSON: {e}"}
token = data.get("token")
if not isinstance(token, str) or not token:
return 400, {"error": "token (string) is required"}
try:
source_ips = broker.list_live(token)
except BrokerAuthError as e:
# Fail-closed: bad signature, malformed token, or a non-list_live op.
return 401, {"error": f"broker auth failed: {e}"}
except Exception as e: # noqa: BLE001 — a backend enumeration failure
# (docker down, inspect failed) is operational, not a control-plane
# bug; surface it as a 502 so the caller skips reconcile rather than
# reaping against a partial set, and keep the controller up.
return 502, {"error": f"backend enumeration failed: {e}"}
return 200, {"source_ips": source_ips}
return 404, {"error": "not found"}
+8 -13
View File
@@ -18,8 +18,8 @@ vsock / unix-socket portability caveats):
body: {"env_var_secret"}
DELETE /bottles/<bottle_id> -> 200 {"torn_down": true} | 404 (teardown)
POST /reconcile -> 200 {"reaped": [bottle_id, ...]}
body: {"live_source_ips": [...],
["grace_seconds"]}
body: {["grace_seconds"]} (live set is
pulled from the host controller, not sent)
POST /attribute -> 200 {"bottle_id"} | 403
POST /resolve -> 200 {"bottle_id","policy"} | 403
body: {"source_ip","identity_token"}
@@ -207,20 +207,15 @@ def dispatch( # pylint: disable=too-many-return-statements,too-many-branches
return 404, {"error": "no such bottle"}
if method == "POST" and route == "/reconcile":
# Host-driven self-heal: the caller enumerates its live bottles (only
# the host can see the backend) and the orchestrator drops rows for
# every other active bottle. Trusted-caller only — an agent that could
# reach this would be able to unregister its neighbours.
# Host-driven self-heal trigger: the orchestrator pulls its own live set
# from the host controller (`list_live`) and drops rows for every other
# active bottle — the caller no longer supplies the live IPs. Trusted-
# caller only: an agent that could reach this would trigger a sweep that
# unregisters its neighbours. Body is optional (just `grace_seconds`).
try:
data = _parse_json_object(body)
except ValueError as e:
return 400, {"error": f"invalid JSON: {e}"}
raw_ips = data.get("live_source_ips")
if not isinstance(raw_ips, list):
return 400, {"error": "live_source_ips (list of strings) is required"}
if any(not isinstance(ip, str) or not ip for ip in raw_ips):
return 400, {"error": "live_source_ips must contain non-empty strings"}
live = raw_ips
grace = data.get("grace_seconds")
kwargs: dict[str, float] = {}
if grace is not None:
@@ -230,7 +225,7 @@ def dispatch( # pylint: disable=too-many-return-statements,too-many-branches
if not math.isfinite(parsed_grace) or parsed_grace < 0:
return 400, {"error": "grace_seconds must be a non-negative finite number"}
kwargs["grace_seconds"] = parsed_grace
return 200, {"reaped": orch.reconcile(live, **kwargs)}
return 200, {"reaped": orch.reconcile(**kwargs)}
if method == "POST" and route == "/attribute":
try:
+30 -9
View File
@@ -22,10 +22,17 @@ Launch lifecycle:
from __future__ import annotations
import json
from collections.abc import Iterable
from datetime import datetime, timezone
from .broker import BrokerUnavailableError, LaunchRequest, SubmitBroker, sign_request
from .. import log
from .broker import (
BrokerAuthError,
BrokerUnavailableError,
LaunchRequest,
SubmitBroker,
sign_request,
)
from .broker_client import BrokerClientError
from .store.registry_store import DEFAULT_REAP_GRACE_SECONDS, BottleRecord, RegistryStore
from .supervisor import (
AuditEntry,
@@ -145,22 +152,36 @@ class OrchestratorCore:
def reconcile(
self,
live_source_ips: Iterable[str],
*,
grace_seconds: float = DEFAULT_REAP_GRACE_SECONDS,
) -> list[str]:
"""Drop registry rows for bottles that are no longer running, and
forget their in-memory egress tokens. Returns the reaped bottle ids.
The caller supplies the live set because only the host can enumerate
its own containers — the orchestrator runs *inside* the infra
container and has no view of the backend. Deliberately does not
broker a teardown: the container is already gone, so there is nothing
to stop, and a broker error must not stop the sweep from clearing
the row that would otherwise brick the next bottle at that address.
The live set is pulled from the **broker** (`list_live`), not passed in:
only the host can enumerate its own containers, and the host controller
is now that host component, so the orchestrator — blind to the backend
from inside its container — asks it over the same signed seam it launches
through. (Before the host control server this was a `live_source_ips`
argument the CLI handed in; the tell that the orchestrator couldn't see
the backend goes away with it — PRD gap 3.)
Fail-safe: if the broker can't return an authoritative live set
(unreachable, timed out, enumeration failed), the sweep is **skipped**,
never run against an empty/partial set — reaping a healthy bottle is far
worse than leaving an orphan one more cycle. Deliberately does not broker
a teardown: a reaped container is already gone, so there is nothing to
stop.
See `RegistryStore.reap_absent` for why orphans accumulate and why
they are harmful rather than merely untidy."""
token = sign_request(LaunchRequest(op="list_live"), self._secret)
try:
live_source_ips = self._broker.list_live(token)
except (BrokerUnavailableError, BrokerAuthError, BrokerClientError) as e:
log.info("reconcile skipped: live set unavailable",
context={"error": str(e)})
return []
reaped = self.registry.reap_absent(
live_source_ips, grace_seconds=grace_seconds)
for rec in reaped: