refactor(backend): base owns the gateway-attach flow, fail hard (0081 review)
lint / lint (push) Successful in 1m1s
test / unit (pull_request) Successful in 1m5s
tracker-policy-pr / check-pr (pull_request) Successful in 10s
test / integration-docker (pull_request) Failing after 1m30s
test / coverage (pull_request) Has been skipped

Addresses review on #519 (@didericis 6143 + the fail-hard direction over the
codex skip).

Base owns the flow (6143 / ADR 0006). `attach_bottled_agents_to_gateway()` is
now a concrete method on `BottleBackend` that delegates to
`gateway_attach.reconcile_running_bottles`; backends override only three
primitives — `_gateway_attach_resources()`, `_running_bottles()`,
`_attach_bottle_to_gateway()`. The shared control flow + error policy live in
one place so a backend can't drift onto a bespoke loop or a silent skip. Keeps
the reconcile flow + `GatewayAttachResources` out of `backend/base.py` (the
size guardrail) in a dedicated `backend/gateway_attach.py`. New ADR 0006 records
the "shared behaviour in the base backend, subclasses override primitives"
theme.

Fail hard, never skip (the maintainer's direction over the codex review's
skip). Any attach failure now aborts bring-up instead of being logged and
skipped: a bottle that silently can't reach the fresh gateway (its egress just
starts failing TLS) is worse than a loud failure. Every bottle is attempted and
the failures are raised together (aggregate `InfraLaunchError`) so one bring-up
surfaces the full blast radius. Resource gathering (the CA fetch) also fails
hard. PRD 0081 goal + design updated to match (reversed from the earlier
"tolerate per-bottle failures" draft).

Bumps the base.py guardrail cap 580->600 for the new contract surface (the
delegator + 3 abstract primitives); the flow itself lives in gateway_attach.py.

Refs #516, #519.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
2026-07-27 01:50:06 +00:00
parent 4f2ccaed29
commit 3c0d2fb66f
13 changed files with 420 additions and 251 deletions
+31 -19
View File
@@ -21,7 +21,7 @@ from abc import ABC, abstractmethod
from contextlib import AbstractContextManager, contextmanager
from dataclasses import dataclass
from pathlib import Path
from typing import Generator, Generic, Sequence, TypeVar
from typing import TYPE_CHECKING, Generator, Generic, Sequence, TypeVar
from ..agent_provider import AgentProvisionPlan, get_provider
from ..egress import EgressPlan
@@ -35,6 +35,9 @@ from ..workspace import WorkspacePlan, workspace_plan
from .print_util import print_multi, visible_agent_env_names
from .util import host_skill_dir
if TYPE_CHECKING:
from .gateway_attach import GatewayAttachResources
class BackendStatus(enum.IntEnum):
"""Return codes for BottleBackend.status(). READY == 0 so callsites
@@ -262,6 +265,11 @@ class Bottle(ABC):
PlanT = TypeVar("PlanT", bound=BottlePlan)
CleanupT = TypeVar("CleanupT", bound=BottleCleanupPlan)
# The backend-specific handle for one running bottle the reconcile attaches to
# the gateway (a run dir for firecracker, a container name for docker/macOS).
# Registry-style callers that don't care about the handle bind it to `Any`
# (`BottleBackend[Any, Any, Any]`).
AttachTargetT = TypeVar("AttachTargetT")
class InfraLaunchError(RuntimeError):
@@ -282,7 +290,7 @@ class BottleImages:
sidecar: str | Path = ""
class BottleBackend(ABC, Generic[PlanT, CleanupT]):
class BottleBackend(ABC, Generic[PlanT, CleanupT, AttachTargetT]):
"""Abstract base for selectable bottle backends. Concrete subclasses
(e.g. DockerBottleBackend) own their own prepare/launch impls.
Parameterized over the backend's concrete plan + cleanup-plan types
@@ -510,26 +518,30 @@ class BottleBackend(ABC, Generic[PlanT, CleanupT]):
compose ls`; firecracker cross-references its running gateway
containers against per-bottle metadata."""
@abstractmethod
def attach_bottled_agents_to_gateway(self) -> None:
"""Reconcile every already-running bottle against the freshly-(re)booted
gateway (PRD 0081). The host calls this on the gateway bring-up path —
the cold-boot branch only, never on an adopt of a healthy current
gateway — so a gateway rebuild/restart doesn't silently break running
bottles' gateway-dependent state.
"""Reconcile every running bottle against the freshly-(re)booted gateway
on the cold-boot bring-up path (PRD 0081). The shared flow + fail-hard
policy live in `gateway_attach`; backends override only the three
primitives below (ADR 0006)."""
from .gateway_attach import reconcile_running_bottles
reconcile_running_bottles(self)
Reconciling running bottles is a backend responsibility: only the
backend can enumerate its agents and reach them (firecracker over SSH,
docker/macOS over `exec`/`cp`), so every backend must implement it —
cross-backend by construction. Each per-bottle step is best-effort: one
unreachable or malformed bottle is logged and skipped so it never blocks
the others or the gateway coming up.
@abstractmethod
def _gateway_attach_resources(self) -> "GatewayAttachResources":
"""Gather what every bottle needs to (re)attach to the current gateway
(the CA now). Raises if the gateway isn't reachable (aborts reconcile)."""
Currently reconciles the **CA**: it replaces each agent's trusted CA
with the current gateway CA (unconditional — there is one gateway, so no
fingerprint match is needed), which rotates the CA for free on every
bring-up. Git-gate re-provision and egress-token restore fold into this
same reconcile on the same trigger (see #516)."""
@abstractmethod
def _running_bottles(self) -> Sequence[AttachTargetT]:
"""The live bottles as backend handles for `_attach_bottle_to_gateway`.
Raises if the live set can't be determined (fail hard, no partial set)."""
@abstractmethod
def _attach_bottle_to_gateway(
self, bottle: AttachTargetT, resources: "GatewayAttachResources",
) -> None:
"""(Re)attach one bottle to the current gateway (SSH / `exec`+`cp`).
Raises `InfraLaunchError` naming the bottle on failure."""
@classmethod
@abstractmethod
+16 -4
View File
@@ -33,6 +33,7 @@ from ...git_gate import GitGatePlan
from ...supervisor.plan import SupervisePlan
from ...manifest import Manifest
from .. import ActiveAgent, BottleBackend, BottleImages, BottleSpec
from ..gateway_attach import GatewayAttachResources
from . import cleanup as _cleanup
from . import enumerate as _enumerate
from . import launch as _launch
@@ -40,7 +41,7 @@ from . import resolve_plan as _resolve_plan
from .bottle import DockerBottle
from .bottle_cleanup_plan import DockerBottleCleanupPlan
from .bottle_plan import DockerBottlePlan
class DockerBottleBackend(BottleBackend["DockerBottlePlan", "DockerBottleCleanupPlan"]):
class DockerBottleBackend(BottleBackend["DockerBottlePlan", "DockerBottleCleanupPlan", str]):
"""Docker backend implementation. Selected by BOT_BOTTLE_BACKEND
when set to `docker`; retained as a legacy/example backend."""
@@ -141,6 +142,17 @@ class DockerBottleBackend(BottleBackend["DockerBottlePlan", "DockerBottleCleanup
def enumerate_active(self) -> Sequence[ActiveAgent]:
return _enumerate.enumerate_active()
def attach_bottled_agents_to_gateway(self) -> None:
from .infra_launch import attach_bottled_agents_to_gateway
attach_bottled_agents_to_gateway()
def _gateway_attach_resources(self) -> GatewayAttachResources:
from .infra import DockerInfraService
return GatewayAttachResources(
ca_pem=DockerInfraService().gateway().ca_cert_pem())
def _running_bottles(self) -> Sequence[str]:
from .infra_launch import running_agent_containers
return running_agent_containers()
def _attach_bottle_to_gateway(
self, bottle: str, resources: GatewayAttachResources,
) -> None:
from .infra_launch import push_ca_to_container
push_ca_to_container(bottle, resources.ca_pem)
+28 -45
View File
@@ -73,11 +73,32 @@ def _network_container_ips(network: str) -> list[str]:
return ips
def _push_ca_to_container(container: str, ca_pem: str) -> None:
"""Replace one running agent container's trusted gateway CA with `ca_pem`
and rebuild its trust store via `docker cp` + `docker exec`. Unconditional
install — there is one gateway, so no fingerprint match is needed. Raises
`InfraLaunchError` on failure."""
def running_agent_containers(
network: str = GATEWAY_NETWORK, gateway_name: str = INFRA_NAME,
) -> list[str]:
"""Every running agent container on the gateway `network` (the gateway
itself excluded) — the bottles the bring-up reconcile attaches to the fresh
gateway (PRD 0081). Raises `OSError` if `docker` can't be run (fail hard: a
reconcile that can't list its bottles must not look like "no bottles")."""
proc = run_docker([
"docker", "network", "inspect", "--format",
"{{range .Containers}}{{.Name}}\n{{end}}", network,
])
return [
name for name in (line.strip() for line in proc.stdout.splitlines())
if name and name != gateway_name
]
def push_ca_to_container(container: str, ca_pem: str) -> None:
"""(Re)attach one running agent container to the current gateway: replace
its trusted gateway CA with `ca_pem` and rebuild its trust store via
`docker cp` + `docker exec` (PRD 0081). Unconditional install — there is one
gateway, so no fingerprint match is needed.
Fail hard: raises `InfraLaunchError` (naming the container) on any failure —
the base reconcile aggregates and surfaces it rather than leaving the agent
silently unable to reach the gateway."""
mkdir = run_docker(
["docker", "exec", container, "mkdir", "-p",
"/usr/local/share/ca-certificates"]
@@ -105,45 +126,6 @@ def _push_ca_to_container(container: str, ca_pem: str) -> None:
)
def attach_bottled_agents_to_gateway(
ca_pem: str | None = None,
*,
network: str = GATEWAY_NETWORK,
gateway_name: str = INFRA_NAME,
) -> None:
"""Reconcile every running agent container against the freshly-(re)created
gateway (PRD 0081): replace each agent's trusted CA with the current gateway
CA, so a gateway rebuild doesn't silently break their TLS egress (#510).
Per-container steps are best-effort: one failure is logged and skipped so it
never blocks the others or the gateway coming up."""
if ca_pem is None:
ca_pem = DockerInfraService().gateway().ca_cert_pem()
try:
proc = run_docker([
"docker", "network", "inspect", "--format",
"{{range .Containers}}{{.Name}}\n{{end}}", network,
])
except OSError as exc:
log.info(f"CA reconcile skipped: {exc}")
return
reconciled = 0
for line in proc.stdout.splitlines():
name = line.strip()
if not name or name == gateway_name:
continue
try:
_push_ca_to_container(name, ca_pem)
reconciled += 1
except (OSError, InfraLaunchError) as exc:
log.info(f"CA reconcile skipped for {name}: {exc}")
if reconciled:
log.info(
"reconciled gateway CA into running docker bottle(s)",
context={"count": reconciled},
)
def _reprovision_running_bottles(
orchestrator_url: str,
network: str = GATEWAY_NETWORK,
@@ -250,7 +232,8 @@ def deprovision_consolidated(
__all__ = [
"LaunchContext",
"launch_consolidated",
"attach_bottled_agents_to_gateway",
"running_agent_containers",
"push_ca_to_container",
"deprovision_consolidated",
"InfraLaunchError",
]
+14 -4
View File
@@ -20,6 +20,7 @@ from ...git_gate import GitGatePlan
from ...manifest import Manifest
from ...supervisor.plan import SupervisePlan
from .. import ActiveAgent, BottleBackend, BottleImages, BottleSpec
from ..gateway_attach import GatewayAttachResources
from . import cleanup as _cleanup
from . import enumerate as _enumerate
from . import launch as _launch
@@ -31,7 +32,7 @@ from .bottle_plan import FirecrackerBottlePlan
class FirecrackerBottleBackend(
BottleBackend["FirecrackerBottlePlan", "FirecrackerBottleCleanupPlan"]
BottleBackend["FirecrackerBottlePlan", "FirecrackerBottleCleanupPlan", "Path"]
):
name = "firecracker"
@@ -119,9 +120,18 @@ class FirecrackerBottleBackend(
def enumerate_active(self) -> Sequence[ActiveAgent]:
return _enumerate.enumerate_active()
def attach_bottled_agents_to_gateway(self) -> None:
from .infra_launch import attach_bottled_agents_to_gateway
attach_bottled_agents_to_gateway()
def _gateway_attach_resources(self) -> GatewayAttachResources:
from .gateway import FirecrackerGateway
return GatewayAttachResources(ca_pem=FirecrackerGateway().ca_cert_pem())
def _running_bottles(self) -> Sequence[Path]:
return _cleanup.live_run_dirs()
def _attach_bottle_to_gateway(
self, bottle: Path, resources: GatewayAttachResources,
) -> None:
from .infra_launch import attach_ca_to_agent_vm
attach_ca_to_agent_vm(bottle, resources.ca_pem)
def supervise_mcp_url(self, plan: FirecrackerBottlePlan) -> str:
return plan.agent_supervise_url
+25 -40
View File
@@ -32,7 +32,6 @@ from dataclasses import dataclass
from pathlib import Path
from ...egress import EgressPlan
from ...gateway import GatewayError
from ...git_gate import GitGatePlan
from ...log import info
from ...orchestrator.client import OrchestratorClient, OrchestratorClientError
@@ -88,56 +87,42 @@ def persist_env_var_secret(private_key: Path, guest_ip: str, secret: str) -> Non
)
def _push_gateway_ca_to_agent(private_key: Path, guest_ip: str, ca_pem: str) -> None:
"""Replace one running agent VM's trusted gateway CA with `ca_pem` and
rebuild its trust store over SSH. Unconditional install — there is one
def attach_ca_to_agent_vm(run_dir: Path, ca_pem: str) -> None:
"""(Re)attach one running agent VM (identified by its `run_dir`) to the
current gateway: replace its trusted gateway CA with `ca_pem` and rebuild
its trust store over SSH (PRD 0081). Unconditional install — there is one
gateway, so no fingerprint match is needed. Bare-pipe input keeps the cert
off argv. Raises `InfraLaunchError` on failure."""
off argv.
Fail hard: raises `InfraLaunchError` (naming the bottle) if the run dir is
malformed or the push fails — the base reconcile aggregates and surfaces it
rather than leaving the agent silently unable to reach the gateway."""
guest_ip = _guest_ip_from_config(run_dir / "config.json")
private_key = run_dir / "bottle_id_ed25519"
if not guest_ip or not private_key.is_file():
raise InfraLaunchError(
f"{run_dir.name}: cannot resolve guest IP / SSH key to attach it "
f"to the gateway"
)
install = (
"umask 022; mkdir -p /usr/local/share/ca-certificates && "
f"cat > {AGENT_CA_PATH} && chmod 644 {AGENT_CA_PATH} && "
"update-ca-certificates"
)
proc = subprocess.run(
util.ssh_base_argv(private_key, guest_ip) + [install],
input=ca_pem, capture_output=True, text=True, check=False,
)
try:
proc = subprocess.run(
util.ssh_base_argv(private_key, guest_ip) + [install],
input=ca_pem, capture_output=True, text=True, check=False,
)
except OSError as exc:
raise InfraLaunchError(f"{run_dir.name} ({guest_ip}): CA push failed: {exc}") from exc
if proc.returncode != 0:
raise InfraLaunchError(
f"CA push to {guest_ip} failed: "
f"{run_dir.name} ({guest_ip}): CA push failed: "
f"{proc.stderr.strip() or '<no stderr>'}"
)
def attach_bottled_agents_to_gateway() -> None:
"""Reconcile every running agent VM against the freshly-booted gateway
(PRD 0081). Replaces each agent's trusted CA with the current gateway CA —
the gateway rootfs is ephemeral, so a cold boot mints a fresh CA and this is
what distributes it (a gateway rebuild doubles as a free CA rotation, #510).
Per-bottle steps are best-effort: one unreachable or malformed VM is logged
and skipped so it never blocks the others or the gateway coming up."""
gateway = FirecrackerGateway()
try:
ca_pem = gateway.ca_cert_pem()
except GatewayError as exc:
info(f"bring-up reconcile skipped: gateway CA unavailable: {exc}")
return
reconciled = 0
for run_dir in cleanup.live_run_dirs():
guest_ip = _guest_ip_from_config(run_dir / "config.json")
private_key = run_dir / "bottle_id_ed25519"
if not guest_ip or not private_key.is_file():
continue
try:
_push_gateway_ca_to_agent(private_key, guest_ip, ca_pem)
reconciled += 1
except (OSError, InfraLaunchError) as exc:
info(f"CA reconcile skipped for {guest_ip}: {exc}")
if reconciled:
info(f"reconciled gateway CA into {reconciled} running Firecracker bottle(s)")
def _reprovision_running_bottles(client: OrchestratorClient) -> None:
"""Read keys from live agent VMs and restore the restarted gateway."""
try:
@@ -210,7 +195,7 @@ def deprovision_consolidated(
__all__ = [
"LaunchContext",
"launch_consolidated",
"attach_bottled_agents_to_gateway",
"attach_ca_to_agent_vm",
"deprovision_consolidated",
"InfraLaunchError",
"OrchestratorStartError",
+54
View File
@@ -0,0 +1,54 @@
"""The shared gateway-attach reconcile flow (PRD 0081).
`BottleBackend.attach_bottled_agents_to_gateway()` delegates here so every
backend reconciles running bottles against a freshly-booted gateway *the same
way* the control flow + fail-hard policy live in one place and backends
override only the primitives (see ADR 0006). Kept out of `backend/base.py` so
the backend contract module stays lean (the base.py size guardrail).
"""
from __future__ import annotations
from dataclasses import dataclass
from typing import TYPE_CHECKING, Any
from .base import InfraLaunchError
if TYPE_CHECKING:
from .base import BottleBackend
@dataclass(frozen=True)
class GatewayAttachResources:
"""Everything a running bottle needs to (re)attach to the current gateway on
a cold boot (PRD 0081). Gathered once per reconcile and handed to every
bottle. The CA now; egress secrets and git-gate state join as the reconcile
grows (#516)."""
ca_pem: str
def reconcile_running_bottles(backend: "BottleBackend[Any, Any, Any]") -> None:
"""Reconcile every already-running bottle against the freshly-(re)booted
gateway: gather the attach resources once, then attach every live bottle.
Fail hard, never skip: a bottle that silently can't reach the fresh gateway
(its egress just starts failing TLS) is worse than a loud bring-up failure,
so any attach failure aborts bring-up. Every bottle is attempted first and
the failures are raised together, so one bring-up surfaces the full blast
radius rather than one bottle at a time."""
resources = backend._gateway_attach_resources()
failures: list[str] = []
for bottle in backend._running_bottles():
try:
backend._attach_bottle_to_gateway(bottle, resources)
except InfraLaunchError as exc:
failures.append(str(exc))
if failures:
raise InfraLaunchError(
f"could not attach {len(failures)} running bottle(s) to the "
f"freshly-booted gateway:\n " + "\n ".join(failures)
)
__all__ = ["GatewayAttachResources", "reconcile_running_bottles"]
+15 -4
View File
@@ -14,6 +14,7 @@ from ...git_gate import GitGatePlan
from ...supervisor.plan import SupervisePlan
from ...manifest import Manifest
from .. import ActiveAgent, BottleBackend, BottleImages, BottleSpec
from ..gateway_attach import GatewayAttachResources
from . import cleanup as _cleanup
from . import enumerate as _enumerate
from . import launch as _launch
@@ -25,7 +26,7 @@ from .bottle_plan import MacosContainerBottlePlan
class MacosContainerBottleBackend(
BottleBackend["MacosContainerBottlePlan", "MacosContainerBottleCleanupPlan"]
BottleBackend["MacosContainerBottlePlan", "MacosContainerBottleCleanupPlan", str]
):
"""Apple Container backend. Selected by
`BOT_BOTTLE_BACKEND=macos-container` or
@@ -117,9 +118,19 @@ class MacosContainerBottleBackend(
def enumerate_active(self) -> Sequence[ActiveAgent]:
return _enumerate.enumerate_active()
def attach_bottled_agents_to_gateway(self) -> None:
from .infra_launch import attach_bottled_agents_to_gateway
attach_bottled_agents_to_gateway()
def _gateway_attach_resources(self) -> GatewayAttachResources:
from .infra import MacosInfraService
return GatewayAttachResources(ca_pem=MacosInfraService().ca_cert_pem())
def _running_bottles(self) -> Sequence[str]:
from .infra_launch import running_agent_containers
return running_agent_containers()
def _attach_bottle_to_gateway(
self, bottle: str, resources: GatewayAttachResources,
) -> None:
from .infra_launch import push_ca_to_container
push_ca_to_container(bottle, resources.ca_pem)
def supervise_mcp_url(self, plan: MacosContainerBottlePlan) -> str:
return plan.agent_supervise_url
@@ -97,11 +97,23 @@ def ensure_gateway(
return endpoint
def _push_ca_to_container(name: str, ca_pem: str) -> None:
"""Replace one running agent container's trusted gateway CA with `ca_pem`
and rebuild its trust store via `container cp` + `container exec`.
Unconditional install there is one gateway, so no fingerprint match is
needed. Raises `InfraLaunchError` on failure."""
def running_agent_containers() -> list[str]:
"""Every running agent container's name — the bottles the bring-up reconcile
attaches to the fresh gateway (PRD 0081). Raises `EnumerationError` if the
live set can't be determined (fail hard rather than reconcile a partial
set)."""
return [f"{CONTAINER_NAME_PREFIX}{agent.slug}" for agent in enumerate_active()]
def push_ca_to_container(name: str, ca_pem: str) -> None:
"""(Re)attach one running agent container to the current gateway: replace
its trusted gateway CA with `ca_pem` and rebuild its trust store via
`container cp` + `container exec` (PRD 0081). Unconditional install there
is one gateway, so no fingerprint match is needed.
Fail hard: raises `InfraLaunchError` (naming the container) on any failure
the base reconcile aggregates and surfaces it rather than leaving the agent
silently unable to reach the gateway."""
mkdir = container_mod.run_container_argv(
["container", "exec", name, "mkdir", "-p",
"/usr/local/share/ca-certificates"]
@@ -131,32 +143,6 @@ def _push_ca_to_container(name: str, ca_pem: str) -> None:
)
def attach_bottled_agents_to_gateway(ca_pem: str | None = None) -> None:
"""Reconcile every running agent container against the freshly-(re)created
gateway (PRD 0081): replace each agent's trusted CA with the current gateway
CA, so a gateway rebuild doesn't silently break their TLS egress (#510).
Per-container steps are best-effort: one failure is logged and skipped so it
never blocks the others or the gateway coming up."""
if ca_pem is None:
ca_pem = MacosInfraService().ca_cert_pem()
try:
agents = list(enumerate_active())
except (EnumerationError, OSError) as exc:
info(f"CA reconcile skipped: {exc}")
return
reconciled = 0
for agent in agents:
name = f"{CONTAINER_NAME_PREFIX}{agent.slug}"
try:
_push_ca_to_container(name, ca_pem)
reconciled += 1
except (OSError, InfraLaunchError) as exc:
info(f"CA reconcile skipped for {name}: {exc}")
if reconciled:
info(f"reconciled gateway CA into {reconciled} running macOS bottle(s)")
def _reprovision_running_bottles(endpoint: GatewayEndpoint) -> None:
"""Recover keys from live Apple containers and restore gateway tokens."""
try:
@@ -259,7 +245,8 @@ __all__ = [
"GatewayEndpoint",
"LaunchContext",
"ensure_gateway",
"attach_bottled_agents_to_gateway",
"running_agent_containers",
"push_ca_to_container",
"live_source_ips",
"register_agent",
"deprovision_consolidated",
+3 -3
View File
@@ -26,10 +26,10 @@ from .base import ActiveAgent, BackendStatus, BottleBackend
# Tests may replace _backends with a {name: fake} dict via patch.object;
# _get_backends() returns the current module-level value as-is when it
# is not None, so test fakes take effect without triggering real imports.
_backends: dict[str, BottleBackend[Any, Any]] | None = None
_backends: dict[str, BottleBackend[Any, Any, Any]] | None = None
def _get_backends() -> dict[str, BottleBackend[Any, Any]]:
def _get_backends() -> dict[str, BottleBackend[Any, Any, Any]]:
"""Return the registry of all backend instances, loading lazily on first call."""
global _backends # pylint: disable=global-statement
if _backends is None:
@@ -48,7 +48,7 @@ def get_bottle_backend(
name: str | None = None,
*,
prompt: bool = True,
) -> BottleBackend[Any, Any]:
) -> BottleBackend[Any, Any, Any]:
"""Resolve the bottle backend.
`name` precedence: