feat(orchestrator): list_live broker op + reconcile via broker (#468)
Chunk 3 of the host-control-server stack: grow the broker op vocabulary
(PRD gap 3), starting with `list_live`, and invert `reconcile` onto it.
- broker: split the closed op vocabulary into mutation (`launch`/`teardown`,
carry a bottle id + static flags) and query (`list_live`, carries nothing
but its op name) kinds. `verify_request` now enforces a **strict schema**
(open question 1, resolved yes): unknown claim keys are rejected, a mutation
must name its bottle, and a query that smuggles any id/flag is refused.
- broker verb: `LaunchBroker.list_live` / `SubmitBroker.list_live` return the
backend's live source IPs; a backend enumeration failure is converted to the
single `BrokerUnavailableError` "live set unknown" signal. `DockerBroker`
enumerates its labelled containers; `StubBroker` derives from launches (or a
test override).
- host controller: `POST /broker/live` verifies a signed `list_live` token and
returns `{source_ips}`; `BrokerClient.list_live` is its drop-in client.
- reconcile: `OrchestratorCore.reconcile()` drops the `live_source_ips`
parameter and pulls the live set from the broker itself — the tell that the
orchestrator couldn't see the backend goes away. **Fail-safe**: if the broker
can't return an authoritative set the sweep is skipped, never run against an
empty/partial set (which would reap healthy rows). The `/reconcile` HTTP
contract + `OrchestratorClient.reconcile` become a bare trigger.
The macOS launcher's Apple-container enumeration stays for now; it becomes the
host controller's `list_live` when launch itself moves behind the broker (the
pulled-forward chunk 5, next in the stack).
Tests + pyright clean; pylint 10.0 on broker.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
@@ -22,10 +22,17 @@ Launch lifecycle:
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
from collections.abc import Iterable
|
||||
from datetime import datetime, timezone
|
||||
|
||||
from .broker import BrokerUnavailableError, LaunchRequest, SubmitBroker, sign_request
|
||||
from .. import log
|
||||
from .broker import (
|
||||
BrokerAuthError,
|
||||
BrokerUnavailableError,
|
||||
LaunchRequest,
|
||||
SubmitBroker,
|
||||
sign_request,
|
||||
)
|
||||
from .broker_client import BrokerClientError
|
||||
from .store.registry_store import DEFAULT_REAP_GRACE_SECONDS, BottleRecord, RegistryStore
|
||||
from .supervisor import (
|
||||
AuditEntry,
|
||||
@@ -145,22 +152,36 @@ class OrchestratorCore:
|
||||
|
||||
def reconcile(
|
||||
self,
|
||||
live_source_ips: Iterable[str],
|
||||
*,
|
||||
grace_seconds: float = DEFAULT_REAP_GRACE_SECONDS,
|
||||
) -> list[str]:
|
||||
"""Drop registry rows for bottles that are no longer running, and
|
||||
forget their in-memory egress tokens. Returns the reaped bottle ids.
|
||||
|
||||
The caller supplies the live set because only the host can enumerate
|
||||
its own containers — the orchestrator runs *inside* the infra
|
||||
container and has no view of the backend. Deliberately does not
|
||||
broker a teardown: the container is already gone, so there is nothing
|
||||
to stop, and a broker error must not stop the sweep from clearing
|
||||
the row that would otherwise brick the next bottle at that address.
|
||||
The live set is pulled from the **broker** (`list_live`), not passed in:
|
||||
only the host can enumerate its own containers, and the host controller
|
||||
is now that host component, so the orchestrator — blind to the backend
|
||||
from inside its container — asks it over the same signed seam it launches
|
||||
through. (Before the host control server this was a `live_source_ips`
|
||||
argument the CLI handed in; the tell that the orchestrator couldn't see
|
||||
the backend goes away with it — PRD gap 3.)
|
||||
|
||||
Fail-safe: if the broker can't return an authoritative live set
|
||||
(unreachable, timed out, enumeration failed), the sweep is **skipped**,
|
||||
never run against an empty/partial set — reaping a healthy bottle is far
|
||||
worse than leaving an orphan one more cycle. Deliberately does not broker
|
||||
a teardown: a reaped container is already gone, so there is nothing to
|
||||
stop.
|
||||
|
||||
See `RegistryStore.reap_absent` for why orphans accumulate and why
|
||||
they are harmful rather than merely untidy."""
|
||||
token = sign_request(LaunchRequest(op="list_live"), self._secret)
|
||||
try:
|
||||
live_source_ips = self._broker.list_live(token)
|
||||
except (BrokerUnavailableError, BrokerAuthError, BrokerClientError) as e:
|
||||
log.info("reconcile skipped: live set unavailable",
|
||||
context={"error": str(e)})
|
||||
return []
|
||||
reaped = self.registry.reap_absent(
|
||||
live_source_ips, grace_seconds=grace_seconds)
|
||||
for rec in reaped:
|
||||
|
||||
Reference in New Issue
Block a user