feat(orchestrator): list_live broker op + reconcile via broker (#468)
tracker-policy-pr / check-pr (pull_request) Successful in 10s
test / unit (pull_request) Successful in 52s
lint / lint (push) Successful in 58s
test / integration-docker (pull_request) Failing after 2m33s
test / coverage (pull_request) Has been skipped

Chunk 3 of the host-control-server stack: grow the broker op vocabulary
(PRD gap 3), starting with `list_live`, and invert `reconcile` onto it.

- broker: split the closed op vocabulary into mutation (`launch`/`teardown`,
  carry a bottle id + static flags) and query (`list_live`, carries nothing
  but its op name) kinds. `verify_request` now enforces a **strict schema**
  (open question 1, resolved yes): unknown claim keys are rejected, a mutation
  must name its bottle, and a query that smuggles any id/flag is refused.
- broker verb: `LaunchBroker.list_live` / `SubmitBroker.list_live` return the
  backend's live source IPs; a backend enumeration failure is converted to the
  single `BrokerUnavailableError` "live set unknown" signal. `DockerBroker`
  enumerates its labelled containers; `StubBroker` derives from launches (or a
  test override).
- host controller: `POST /broker/live` verifies a signed `list_live` token and
  returns `{source_ips}`; `BrokerClient.list_live` is its drop-in client.
- reconcile: `OrchestratorCore.reconcile()` drops the `live_source_ips`
  parameter and pulls the live set from the broker itself — the tell that the
  orchestrator couldn't see the backend goes away. **Fail-safe**: if the broker
  can't return an authoritative set the sweep is skipped, never run against an
  empty/partial set (which would reap healthy rows). The `/reconcile` HTTP
  contract + `OrchestratorClient.reconcile` become a bare trigger.

The macOS launcher's Apple-container enumeration stays for now; it becomes the
host controller's `list_live` when launch itself moves behind the broker (the
pulled-forward chunk 5, next in the stack).

Tests + pyright clean; pylint 10.0 on broker.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
2026-07-26 17:52:44 +00:00
parent 0e2af7de62
commit 7005b22bcf
16 changed files with 553 additions and 114 deletions
@@ -8,6 +8,7 @@ from unittest.mock import Mock, patch
from bot_bottle.orchestrator.broker import (
BrokerAuthError,
BrokerUnavailableError,
LaunchRequest,
sign_request,
)
@@ -96,5 +97,50 @@ class TestDockerBroker(unittest.TestCase):
m.assert_not_called()
class TestDockerBrokerListLive(unittest.TestCase):
def setUp(self) -> None:
self.secret = secrets.token_bytes(16)
self.broker = DockerBroker(self.secret)
def test_enumerates_labeled_container_ips(self) -> None:
# `docker ps` lists two labeled containers; each inspect yields an IP.
ps = Mock(returncode=0, stdout="c1\nc2\n", stderr="")
i1 = Mock(returncode=0, stdout="10.0.0.1 \n", stderr="")
i2 = Mock(returncode=0, stdout="10.0.0.2 \n", stderr="")
with patch.object(self.broker, "_docker", side_effect=[ps, i1, i2]) as m:
got = self.broker._list_live()
self.assertEqual(["10.0.0.1", "10.0.0.2"], got)
# First call is the label-filtered `docker ps`.
ps_argv = m.call_args_list[0].args[0]
self.assertEqual(["docker", "ps", "--filter", f"label={BOTTLE_ID_LABEL}"], ps_argv[:4])
def test_no_containers_is_empty(self) -> None:
with patch.object(self.broker, "_docker",
return_value=Mock(returncode=0, stdout="", stderr="")):
self.assertEqual([], self.broker._list_live())
def test_ps_failure_raises(self) -> None:
with patch.object(self.broker, "_docker",
return_value=Mock(returncode=1, stdout="", stderr="daemon down")):
with self.assertRaises(DockerBrokerError):
self.broker._list_live()
def test_inspect_failure_raises(self) -> None:
ps = Mock(returncode=0, stdout="c1\n", stderr="")
bad = Mock(returncode=1, stdout="", stderr="no such container")
with patch.object(self.broker, "_docker", side_effect=[ps, bad]):
with self.assertRaises(DockerBrokerError):
self.broker._list_live()
def test_enumeration_failure_surfaces_as_unavailable_via_list_live(self) -> None:
# Through the verb, a backend failure is the single "live set unknown"
# signal (BrokerUnavailableError) reconcile catches to skip the sweep.
token = sign_request(LaunchRequest(op="list_live"), self.secret)
with patch.object(self.broker, "_docker",
return_value=Mock(returncode=1, stdout="", stderr="down")):
with self.assertRaises(BrokerUnavailableError):
self.broker.list_live(token)
if __name__ == "__main__":
unittest.main()