fix(orchestrator): reap registry rows whose bottle is no longer running
A registry row only ever left the registry two ways: an explicit teardown_bottle (the launcher's cleanup callback) or the same-IP supersede sweep in register(). Neither runs when the launching CLI dies hard, so the row outlives its container. That orphan is not inert. Source IPs are recycled by the backend's DHCP and by_source_ip fail-closes on ambiguity, so a leftover row at a reused address resolves no policy at all for the next bottle that lands there — and a bottle with no policy denies every host, which surfaces to the agent as "host X is not in the allowlist" for hosts that were never the problem. Add reap_absent/reconcile and call it from the macOS launch path before registering, so each launch self-heals the registry. Restores the invariant the data plane needs: at most one active row per live address, and none for a dead one. The second half matters as much as the first — when several rows claim a *live* address the newest wins and the rest are swept, otherwise a recycled address stays ambiguous, which is exactly the bricked state. The host supplies the live set because the orchestrator runs inside the infra container and cannot see the backend. A grace window exempts rows younger than it, so reconciliation cannot race a bottle still coming up, and a reconcile failure is logged rather than blocking an otherwise-fine launch. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
@@ -15,6 +15,7 @@ from __future__ import annotations
|
||||
import json
|
||||
import urllib.error
|
||||
import urllib.request
|
||||
from collections.abc import Iterable
|
||||
from dataclasses import dataclass
|
||||
|
||||
from ..paths import host_control_plane_token
|
||||
@@ -147,6 +148,20 @@ class OrchestratorClient:
|
||||
raise OrchestratorClientError(f"teardown {bottle_id}: HTTP {status}")
|
||||
return True
|
||||
|
||||
def reconcile(
|
||||
self, live_source_ips: Iterable[str], *, grace_seconds: float | None = None,
|
||||
) -> list[str]:
|
||||
"""Drop registry rows for bottles that are no longer running
|
||||
(`POST /reconcile`), returning the reaped bottle ids. `live_source_ips`
|
||||
is the caller's enumeration of its live bottles — the orchestrator
|
||||
can't see the backend from inside the infra container."""
|
||||
body: dict[str, object] = {"live_source_ips": list(live_source_ips)}
|
||||
if grace_seconds is not None:
|
||||
body["grace_seconds"] = grace_seconds
|
||||
payload = self._ok("POST", "/reconcile", body)
|
||||
reaped = payload.get("reaped")
|
||||
return [r for r in reaped if isinstance(r, str)] if isinstance(reaped, list) else []
|
||||
|
||||
def set_policy(self, bottle_id: str, policy: str) -> bool:
|
||||
"""Live-reload a bottle's policy (`PUT /bottles/<id>/policy`). False on
|
||||
404 (unknown bottle)."""
|
||||
|
||||
Reference in New Issue
Block a user