refactor(macos): one infra container (control plane + gateway), fixes shared-DB races
Adopts the firecracker infra-VM pattern for macOS: the orchestrator control plane and the gateway data plane now run in a SINGLE Apple container instead of two. Apple Containers are lightweight VMs with separate kernels, so the prior two-container design had both guests writing one bot-bottle.db over virtiofs, where fcntl locks are not coherent across kernels — concurrent writes (the orchestrator's registry vs the gateway supervise daemon's queue) could corrupt it. One container = one kernel = coherent locking. The DB moves onto a container-only Apple volume (bot-bottle-mac-db), never bind-mounted from the host, so no host process opens the live file either. The host CLI already reaches registry + supervise state over the control-plane HTTP surface (cli/supervise.py uses OrchestratorClient), exactly as firecracker's VM-only DB requires. Two simplifications fall out of the single container: - No DNS dance: the control plane and gateway daemons reach each other over 127.0.0.1, so the orchestrator-before-gateway ordering (a workaround for Apple having no container DNS) is gone, along with the moved-IP recreate logic it needed. - Net -243 lines. Mechanics: the infra container runs from the gateway image with the control-plane source bind-mounted read-only (like the docker orchestrator, so a code change needs no rebuild) and a small sh -c init that starts both processes (mirrors firecracker's _infra_init). Also implements the macOS backend's ensure_orchestrator() and adds it to discover_orchestrator_url, so operator tools (supervise) can bring up / find the control plane on demand — previously the macOS backend died with "no orchestrator control plane". Verified end-to-end on real Apple Container 1.0.0: the single infra container comes up healthy (one address for control plane + gateway), both processes run, the DB is written on the container-only volume, host-side supervise works over HTTP, and a registered agent gets 200 for an allowed host / 403 for a denied one. 1824 unit tests pass with `container` absent (CI parity), pyright clean, pylint 9.89. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
@@ -15,6 +15,9 @@ caller has to start the agent in between. `ensure_gateway` runs first because
|
||||
the agent's proxy env needs the gateway's address at `container run` time; the
|
||||
agent's *own* address (the attribution key) only exists afterwards.
|
||||
|
||||
The control plane and the gateway are one **infra container** here (see
|
||||
`infra`), so `gateway_ip` and the control-plane host are the same address.
|
||||
|
||||
The consequence for the identity token: it is minted by registration, i.e.
|
||||
*after* the agent container exists, so it cannot be baked into the run-time
|
||||
env the way docker's compose spec does. It is delivered at `container exec`
|
||||
@@ -38,7 +41,7 @@ from ...orchestrator.registration import registration_inputs
|
||||
from ..docker.gateway_provision import deprovision_git_gate, provision_git_gate
|
||||
from .gateway import GATEWAY_NETWORK
|
||||
from .gateway_provision import AppleGatewayTransport
|
||||
from .orchestrator_service import MacosOrchestratorService, OrchestratorStartError
|
||||
from .infra import MacosInfraService, OrchestratorStartError
|
||||
|
||||
|
||||
class ConsolidatedLaunchError(RuntimeError):
|
||||
@@ -47,7 +50,9 @@ class ConsolidatedLaunchError(RuntimeError):
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class GatewayEndpoint:
|
||||
"""What the agent `container run` needs to reach the shared gateway."""
|
||||
"""What the agent `container run` needs to reach the shared gateway (the
|
||||
infra container). `gateway_ip` is that container's host-only address, the
|
||||
same host the control-plane URL points at."""
|
||||
|
||||
orchestrator_url: str
|
||||
gateway_ip: str # the gateway's address — the agent's proxy target
|
||||
@@ -68,19 +73,18 @@ class LaunchContext:
|
||||
|
||||
|
||||
def ensure_gateway(
|
||||
*, service: MacosOrchestratorService | None = None,
|
||||
*, service: MacosInfraService | None = None,
|
||||
) -> GatewayEndpoint:
|
||||
"""Ensure the orchestrator control plane + shared gateway are up, and
|
||||
report how to reach them. Idempotent — both are per-host singletons, so N
|
||||
bottle launches share the one pair. Call before starting the agent
|
||||
container: the agent's proxy env needs `gateway_ip` at run time."""
|
||||
service = service or MacosOrchestratorService()
|
||||
url = service.ensure_running()
|
||||
gateway = service.gateway(url)
|
||||
"""Ensure the per-host infra container (control plane + gateway) is up and
|
||||
report how to reach it. Idempotent — one singleton, so N bottle launches
|
||||
share it. Call before starting the agent container: the agent's proxy env
|
||||
needs `gateway_ip` at run time."""
|
||||
service = service or MacosInfraService()
|
||||
infra = service.ensure_running()
|
||||
return GatewayEndpoint(
|
||||
orchestrator_url=url,
|
||||
gateway_ip=gateway.ip_on_shared_network(),
|
||||
gateway_ca_pem=gateway.ca_cert_pem(),
|
||||
orchestrator_url=infra.control_plane_url,
|
||||
gateway_ip=infra.gateway_ip,
|
||||
gateway_ca_pem=service.ca_cert_pem(),
|
||||
network=service.network,
|
||||
)
|
||||
|
||||
|
||||
Reference in New Issue
Block a user