refactor: remove leftovers from the orchestrator/gateway consolidation
Sweep for vestiges of the old combined-plane model and the pre-split shared rootfs. Two are load-bearing, the rest are stale docs/comments: - Bug: macOS `enumerate_active` only excluded the gateway container from the agent list, so after the split the orchestrator container (`bot-bottle-mac-orchestrator`, also `bot-bottle-`-prefixed) was enumerated as a phantom agent. Exclude both infra containers; test covers it. - Dead code: the gateway `bootstrap.py` still carried an `orchestrator` daemon spec + `_OPT_IN_DAEMONS` + a signing-key/JWT env branch, all for the old combined container where the gateway process could also run the control plane. No backend ever requests it now — removed; the key-stripping stays as defense-in-depth. Stale-comment reframes: "the/single infra container" -> the orchestrator + gateway pair (or the specific plane); "shared rootfs / bb_role init / one published rootfs" -> the per-plane rootfs + `role_init`; the deleted Dockerfile.infra references in Dockerfile.orchestrator/.gateway; and the macOS "one infra container ... same address" docstring + its now-false share-one-address test (the planes are distinct containers with distinct addresses). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
@@ -11,7 +11,7 @@ The bulk of the implementation lives in sibling modules:
|
||||
- launch: bring-up + teardown context manager
|
||||
- cleanup: orphan enumeration, removal, active listing
|
||||
- backend: DockerBottleBackend façade wiring the above
|
||||
- infra: DockerInfraService (the per-host infra container)
|
||||
- infra: DockerInfraService (the per-host orchestrator + gateway pair)
|
||||
|
||||
Thin by design: the public names are re-exported lazily via `__getattr__`, so
|
||||
importing a leaf like `backend.docker.util` doesn't drag `DockerBottleBackend`
|
||||
|
||||
@@ -2,7 +2,7 @@
|
||||
|
||||
Composes the orchestrator primitives into the register/teardown sequence:
|
||||
|
||||
1. ensure the single infra container (control plane + gateway) is up;
|
||||
1. ensure the per-host pair (orchestrator + gateway containers) is up;
|
||||
2. allocate the bottle a pinned source IP on the gateway network;
|
||||
3. register it and provision its git-gate repos/creds into the gateway.
|
||||
|
||||
@@ -62,7 +62,7 @@ def _network_cidr(network: str) -> str:
|
||||
|
||||
def _network_container_ips(network: str) -> list[str]:
|
||||
"""Every address currently assigned on the gateway network — the ground
|
||||
truth for "in use": the infra container and every live agent. Read from
|
||||
truth for "in use": the gateway container and every live agent. Read from
|
||||
the network so a new bottle can't collide with anything actually attached."""
|
||||
proc = run_docker([
|
||||
"docker", "network", "inspect", "--format",
|
||||
@@ -80,7 +80,7 @@ def _reprovision_running_bottles(
|
||||
infra_name: str = INFRA_NAME,
|
||||
) -> None:
|
||||
"""Re-inject egress tokens for any registered bottles that lost their
|
||||
in-memory tokens (e.g., after an infra container restart).
|
||||
in-memory tokens (e.g., after an orchestrator restart).
|
||||
|
||||
For each registered bottle whose source IP maps to a live container on the
|
||||
gateway network, reads ENV_VAR_SECRET via ``docker exec … printenv`` and
|
||||
@@ -89,7 +89,7 @@ def _reprovision_running_bottles(
|
||||
container exec failure never blocks a new bottle launch."""
|
||||
client = OrchestratorClient(orchestrator_url)
|
||||
# Build {source_ip: container_name} from live containers on the gateway
|
||||
# network, excluding the infra container itself.
|
||||
# network, excluding the gateway container itself.
|
||||
try:
|
||||
proc = run_docker([
|
||||
"docker", "network", "inspect",
|
||||
@@ -133,11 +133,11 @@ def launch_consolidated(
|
||||
infra_name: str = INFRA_NAME,
|
||||
network: str = GATEWAY_NETWORK,
|
||||
) -> LaunchContext:
|
||||
"""Ensure the infra container is up, allocate + register the bottle, and
|
||||
"""Ensure the orchestrator + gateway pair is up, allocate + register the bottle, and
|
||||
provision its git-gate state. Returns the agent's attach context.
|
||||
|
||||
Also reprovisiones egress tokens for any already-running bottles that lost
|
||||
their in-memory credentials (e.g. after an infra container restart), so
|
||||
their in-memory credentials (e.g. after an orchestrator restart), so
|
||||
they regain egress access before the new bottle is registered."""
|
||||
service = service or DockerInfraService()
|
||||
url = service.ensure_running()
|
||||
|
||||
@@ -69,8 +69,8 @@ def _boot_args(
|
||||
pub_b64 = base64.b64encode(pubkey.encode()).decode()
|
||||
args = f"{_BASE_BOOT_ARGS} {ip_arg} bb_pubkey={pub_b64}"
|
||||
# `extra` carries caller-supplied cmdline params the guest init reads
|
||||
# (e.g. `bb_role=orchestrator|gateway` selecting which infra plane a
|
||||
# shared-rootfs infra VM runs). Agent VMs pass nothing.
|
||||
# (e.g. the gateway VM's `bb_orch=<orchestrator guest IP>`). Agent VMs pass
|
||||
# nothing.
|
||||
return f"{args} {extra}".rstrip() if extra else args
|
||||
|
||||
|
||||
|
||||
@@ -8,14 +8,13 @@ gateway VM never opens `bot-bottle.db` (#469), so it holds no signing key —
|
||||
only the token the host hands it via `connect_to_orchestrator`.
|
||||
|
||||
What deliberately stays in `infra_vm` (not moved here): the plane-agnostic VM
|
||||
substrate the orchestrator VM shares — booting a VM from the shared rootfs
|
||||
substrate the orchestrator VM also uses — booting a VM from a per-plane rootfs
|
||||
(`boot_vm`), the stable SSH keypair, the secret-push retry loop, and the PID
|
||||
lifecycle — plus the pair coordinator (`ensure_running`: orchestrator-first
|
||||
health gate, singleton lock, adoption/version marker). The guest-side gateway
|
||||
daemon startup lives in the shared `bb_role=gateway` init branch baked into the
|
||||
one published rootfs both VMs boot, so it can't live in a host-side method
|
||||
either. These are the shared seam that moves to a neutral module when the
|
||||
Orchestrator service lands and `infra_vm` is dissolved.
|
||||
daemon startup lives in the gateway rootfs's guest init (`_gateway_init` in
|
||||
`infra_vm`), so it can't live in a host-side method either. These are the shared
|
||||
seam.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
@@ -33,18 +32,19 @@ from .. import util as backend_util
|
||||
from . import infra_vm, netpool, util
|
||||
from .gateway_transport import FirecrackerGatewayTransport
|
||||
|
||||
# The gateway microVM's name (the `bb_role=gateway` boot tag / run dir). Fixed
|
||||
# per host — one gateway VM, shared by every agent VM.
|
||||
# The gateway microVM's name (its run dir). Fixed per host — one gateway VM,
|
||||
# shared by every agent VM.
|
||||
GATEWAY_NAME = "bot-bottle-gateway"
|
||||
|
||||
# The gateway VM's slim memory ceiling (buildah is present on the shared rootfs
|
||||
# but unused on the data plane) — PRD 0070 "Memory: fixed ceilings".
|
||||
# The gateway VM's slim memory ceiling — the data plane carries no build
|
||||
# tooling (buildah lives only on the orchestrator rootfs) — PRD 0070
|
||||
# "Memory: fixed ceilings".
|
||||
_GW_MEM_MIB = 2048
|
||||
|
||||
# The pre-minted `gateway` JWT path in the guest. The *shared* init (both planes
|
||||
# boot one rootfs) waits for it before starting the data plane, so its canonical
|
||||
# definition lives with that init in `infra_vm`; imported here for the push so
|
||||
# the load-bearing path isn't duplicated.
|
||||
# The pre-minted `gateway` JWT path in the guest. The gateway init (in
|
||||
# `infra_vm`) waits for it before starting the data plane, so its canonical
|
||||
# definition lives with that init; imported here for the push so the
|
||||
# load-bearing path isn't duplicated.
|
||||
_GUEST_GATEWAY_JWT_PATH = infra_vm._GUEST_GATEWAY_JWT_PATH
|
||||
# mitmproxy writes its CA here a beat after start; agents install it to trust
|
||||
# the gateway's TLS interception. Host-side only (SSH cat), so it lives here.
|
||||
@@ -56,7 +56,7 @@ _CA_FETCH_TIMEOUT_SECONDS = 15.0
|
||||
class FirecrackerGateway(Gateway):
|
||||
"""The consolidated gateway as a Firecracker microVM on the gateway link.
|
||||
|
||||
The shared rootfs is built/downloaded by `infra_vm.ensure_built` (the ABC's
|
||||
The gateway rootfs is built/downloaded by `infra_vm.ensure_built` (the ABC's
|
||||
`ensure_built` no-op here); `connect_to_orchestrator` boots the VM resolving
|
||||
policy against the orchestrator and seeds the pre-minted `gateway` token."""
|
||||
|
||||
@@ -94,8 +94,8 @@ class FirecrackerGateway(Gateway):
|
||||
raise GatewayError(
|
||||
f"cannot resolve orchestrator guest IP from {self._orchestrator_url!r}"
|
||||
)
|
||||
# Boot on the gateway link from the shared rootfs (bb_role=gateway), then
|
||||
# push the token the init waits for before starting the data plane.
|
||||
# Boot on the gateway link from the gateway rootfs, then push the token
|
||||
# the init waits for before starting the data plane.
|
||||
vm = infra_vm.boot_vm(
|
||||
name=GATEWAY_NAME, slot=netpool.gw_slot(), run_dir=infra_vm._gw_dir(),
|
||||
role="gateway", mem_mib=_GW_MEM_MIB,
|
||||
|
||||
@@ -8,11 +8,11 @@ signing key over SSH, and waiting for `/health`. The host CLI reaches it at its
|
||||
guest IP, and so does the gateway (`bb_orch` cmdline), so `url()` == `gateway_url()`.
|
||||
|
||||
What deliberately stays in `infra_vm` (not moved here): the plane-agnostic VM
|
||||
substrate the gateway VM also shares — booting a VM from the shared rootfs
|
||||
substrate the gateway VM also uses — booting a VM from a per-plane rootfs
|
||||
(`boot_vm`), the stable SSH keypair, the secret-push retry, the PID lifecycle —
|
||||
plus the pair coordinator (`ensure_running`: adopt-or-boot-both under a singleton
|
||||
lock with a shared version marker) and the single shared `bb_role`-branched init
|
||||
baked into the one published rootfs. Those are the shared seam.
|
||||
lock with a combined version marker) and the per-plane guest inits (`role_init`).
|
||||
Those are the shared seam.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
@@ -30,8 +30,8 @@ from ...orchestrator.lifecycle import (
|
||||
from . import firecracker_vm, infra_vm, netpool
|
||||
from .infra_vm import ORCHESTRATOR_PORT
|
||||
|
||||
# The orchestrator microVM's name (the `bb_role=orchestrator` boot tag / run
|
||||
# dir). Fixed per host — one control-plane VM.
|
||||
# The orchestrator microVM's name (its run dir). Fixed per host — one
|
||||
# control-plane VM.
|
||||
ORCHESTRATOR_NAME = "bot-bottle-orchestrator"
|
||||
|
||||
# Memory ceiling (fixed at boot, demand-paged). The orchestrator keeps the build
|
||||
@@ -58,9 +58,9 @@ def registry_volume_path() -> Path:
|
||||
class FirecrackerOrchestrator(Orchestrator):
|
||||
"""The control plane as a Firecracker microVM on the orchestrator link.
|
||||
|
||||
The shared rootfs is built/downloaded by `infra_vm.ensure_built` (the ABC's
|
||||
`ensure_built` no-op here); `ensure_running` boots the VM, seeds the signing
|
||||
key, and blocks until `/health` answers."""
|
||||
The orchestrator rootfs is built/downloaded by `infra_vm.ensure_built` (the
|
||||
ABC's `ensure_built` no-op here); `ensure_running` boots the VM, seeds the
|
||||
signing key, and blocks until `/health` answers."""
|
||||
|
||||
name = ORCHESTRATOR_NAME
|
||||
|
||||
|
||||
@@ -258,8 +258,8 @@ def build_committed_rootfs_dir(tar_path: Path) -> Path:
|
||||
|
||||
def inject_guest_boot(rootfs: Path, init_script: str | None = None) -> None:
|
||||
"""Drop the static dropbear and the PID-1 init into the rootfs.
|
||||
`init_script` defaults to the SSH-only agent init; the infra VM
|
||||
passes its own (control plane + gateway) init.
|
||||
`init_script` defaults to the SSH-only agent init; each infra VM
|
||||
passes its own per-plane init (orchestrator or gateway).
|
||||
|
||||
A committed snapshot is guest-controlled, so `bb-dropbear`/`bb-init`
|
||||
may already exist as symlinks aimed at a host file (e.g. bb-init ->
|
||||
|
||||
@@ -101,10 +101,10 @@ class MacosContainerBottleBackend(
|
||||
yield bottle
|
||||
|
||||
def ensure_orchestrator(self) -> str:
|
||||
"""Bring up the per-host infra container (control plane + gateway) and
|
||||
"""Bring up the per-host pair (orchestrator + gateway containers) and
|
||||
return its control-plane URL — the on-demand entry point operator tools
|
||||
(`supervise`) call when no control plane is running yet. Mirrors
|
||||
firecracker's infra-VM bring-up."""
|
||||
firecracker's infra bring-up."""
|
||||
from .infra import MacosInfraService
|
||||
return MacosInfraService().ensure_running().orchestrator_url
|
||||
|
||||
|
||||
@@ -15,8 +15,10 @@ caller has to start the agent in between. `ensure_gateway` runs first because
|
||||
the agent's proxy env needs the gateway's address at `container run` time; the
|
||||
agent's *own* address (the attribution key) only exists afterwards.
|
||||
|
||||
The control plane and the gateway are one **infra container** here (see
|
||||
`infra`), so `gateway_ip` and the control-plane host are the same address.
|
||||
The control plane and the gateway are **separate containers** here (see
|
||||
`infra`): the orchestrator on the host-only control network, the gateway on the
|
||||
agent network — `gateway_ip` is the gateway container's agent-network address,
|
||||
distinct from the orchestrator's control-network host.
|
||||
|
||||
The consequence for the identity token: it is minted by registration, i.e.
|
||||
*after* the agent container exists, so it cannot be baked into the run-time
|
||||
@@ -54,9 +56,9 @@ class ConsolidatedLaunchError(RuntimeError):
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class GatewayEndpoint:
|
||||
"""What the agent `container run` needs to reach the shared gateway (the
|
||||
infra container). `gateway_ip` is that container's host-only address, the
|
||||
same host the control-plane URL points at."""
|
||||
"""What the agent `container run` needs to reach the shared gateway.
|
||||
`gateway_ip` is the gateway container's agent-network address (the agent's
|
||||
proxy target); `orchestrator_url` points at the separate control plane."""
|
||||
|
||||
orchestrator_url: str
|
||||
gateway_ip: str # the gateway's address — the agent's proxy target
|
||||
@@ -80,10 +82,10 @@ class LaunchContext:
|
||||
def ensure_gateway(
|
||||
*, service: MacosInfraService | None = None,
|
||||
) -> GatewayEndpoint:
|
||||
"""Ensure the per-host infra container (control plane + gateway) is up and
|
||||
report how to reach it. Idempotent — one singleton, so N bottle launches
|
||||
share it. Call before starting the agent container: the agent's proxy env
|
||||
needs `gateway_ip` at run time."""
|
||||
"""Ensure the per-host pair (orchestrator + gateway containers) is up and
|
||||
report how to reach the gateway. Idempotent — one singleton pair, so N bottle
|
||||
launches share it. Call before starting the agent container: the agent's
|
||||
proxy env needs `gateway_ip` at run time."""
|
||||
service = service or MacosInfraService()
|
||||
infra = service.ensure_running()
|
||||
endpoint = GatewayEndpoint(
|
||||
|
||||
@@ -6,17 +6,18 @@ import subprocess
|
||||
|
||||
from ...bottle_state import read_metadata
|
||||
from .. import ActiveAgent
|
||||
from .infra import INFRA_NAME
|
||||
from .infra import INFRA_NAME, ORCHESTRATOR_NAME
|
||||
|
||||
# The name every agent container carries: `bot-bottle-<slug>`. Exported
|
||||
# because callers that act on a running bottle (gateway-host rewrites,
|
||||
# registry reconciliation) have to map an enumerated slug back to a
|
||||
# container name.
|
||||
CONTAINER_NAME_PREFIX = "bot-bottle-"
|
||||
# The shared per-host infra container carries the same prefix as agent
|
||||
# containers but is infrastructure, not a bottle — one control plane + gateway
|
||||
# serves every agent, so listing it as an agent would invent one per host.
|
||||
_INFRA_NAMES = frozenset({INFRA_NAME})
|
||||
# The two shared per-host infra containers (orchestrator + gateway) carry the
|
||||
# same `bot-bottle-` prefix as agent containers but are infrastructure, not
|
||||
# bottles — one pair serves every agent, so enumerating either as an agent would
|
||||
# invent a phantom bottle per host.
|
||||
_INFRA_NAMES = frozenset({INFRA_NAME, ORCHESTRATOR_NAME})
|
||||
|
||||
|
||||
class EnumerationError(RuntimeError):
|
||||
|
||||
@@ -1,7 +1,7 @@
|
||||
"""Stable gateway name for macOS agents, via each bottle's `/etc/hosts`.
|
||||
|
||||
The shared gateway's address is assigned by vmnet's DHCP and changes whenever
|
||||
the infra container is recreated — a source-hash bump, an image upgrade, a
|
||||
the gateway container is recreated — a source-hash bump, an image upgrade, a
|
||||
crash. Every agent-facing URL (egress proxy, git-http, supervise) embeds that
|
||||
address, and the proxy URL reaches the agent as **process environment** at
|
||||
`container exec` time. A running process's `environ` cannot be rewritten from
|
||||
|
||||
Reference in New Issue
Block a user