refactor: remove leftovers from the orchestrator/gateway consolidation
Sweep for vestiges of the old combined-plane model and the pre-split shared rootfs. Two are load-bearing, the rest are stale docs/comments: - Bug: macOS `enumerate_active` only excluded the gateway container from the agent list, so after the split the orchestrator container (`bot-bottle-mac-orchestrator`, also `bot-bottle-`-prefixed) was enumerated as a phantom agent. Exclude both infra containers; test covers it. - Dead code: the gateway `bootstrap.py` still carried an `orchestrator` daemon spec + `_OPT_IN_DAEMONS` + a signing-key/JWT env branch, all for the old combined container where the gateway process could also run the control plane. No backend ever requests it now — removed; the key-stripping stays as defense-in-depth. Stale-comment reframes: "the/single infra container" -> the orchestrator + gateway pair (or the specific plane); "shared rootfs / bb_role init / one published rootfs" -> the per-plane rootfs + `role_init`; the deleted Dockerfile.infra references in Dockerfile.orchestrator/.gateway; and the macOS "one infra container ... same address" docstring + its now-false share-one-address test (the planes are distinct containers with distinct addresses). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
+3
-6
@@ -37,12 +37,9 @@
|
|||||||
# 9100 supervise (MCP HTTP)
|
# 9100 supervise (MCP HTTP)
|
||||||
|
|
||||||
# Based on `python:3.12-slim` (Debian trixie) rather than the
|
# Based on `python:3.12-slim` (Debian trixie) rather than the
|
||||||
# `mitmproxy/mitmproxy` image (Debian bookworm) so the whole stack —
|
# `mitmproxy/mitmproxy` image (Debian bookworm), matching the trixie base the
|
||||||
# gateway here, and the firecracker infra image that builds FROM this —
|
# orchestrator image needs for buildah (Dockerfile.orchestrator.fc). mitmproxy
|
||||||
# lands on trixie, whose buildah (1.39) can build agent Dockerfiles that
|
# is pip-installed to the same effect as the upstream image.
|
||||||
# use heredocs. mitmproxy is pip-installed to the same effect as the
|
|
||||||
# upstream image. (bookworm's buildah is 1.28, which can't parse
|
|
||||||
# `RUN ... <<EOF`; see the infra image + PR discussion.)
|
|
||||||
FROM python:3.12-slim
|
FROM python:3.12-slim
|
||||||
|
|
||||||
# Runtime system deps:
|
# Runtime system deps:
|
||||||
|
|||||||
+8
-10
@@ -4,21 +4,19 @@
|
|||||||
# `bot_bottle` package baked onto a Python runtime — referenced by BOTH:
|
# `bot_bottle` package baked onto a Python runtime — referenced by BOTH:
|
||||||
# * the docker backend, which runs this image directly as the lean
|
# * the docker backend, which runs this image directly as the lean
|
||||||
# control-plane container; and
|
# control-plane container; and
|
||||||
# * the firecracker infra image (Dockerfile.infra), which `COPY --from`s
|
# * the firecracker orchestrator VM image (Dockerfile.orchestrator.fc),
|
||||||
# this image's `/app/bot_bottle` so the single infra VM runs the same
|
# which is `FROM` this image and adds buildah for in-VM agent builds.
|
||||||
# control plane. Keeping it in one place means future orchestrator deps
|
# Keeping the content in one place means future orchestrator deps (e.g.
|
||||||
# (e.g. iroh) are added here once, not duplicated per backend.
|
# iroh) are added here once, not duplicated per backend.
|
||||||
#
|
#
|
||||||
# It stays deliberately lean: the control plane is **stdlib-only** today, so
|
# It stays deliberately lean: the control plane is **stdlib-only** today, so
|
||||||
# no third-party payload — none of the gateway's mitmproxy/git/gitleaks
|
# no third-party payload — none of the gateway's mitmproxy/git/gitleaks
|
||||||
# (that's Dockerfile.gateway) and no buildah (that's the firecracker
|
# (that's Dockerfile.gateway) and no buildah (that's the firecracker
|
||||||
# builder, and lives only in Dockerfile.infra). Keeping the secret-dense
|
# builder, and lives only in Dockerfile.orchestrator.fc). Keeping the
|
||||||
# control plane on a minimal dependency surface is the point (PRD 0070's
|
# secret-dense control plane on a minimal dependency surface is the point
|
||||||
# "secret concentration").
|
# (PRD 0070's "secret concentration").
|
||||||
#
|
#
|
||||||
# Shares the trixie `python:3.12-slim` base with the gateway image, so when
|
# Shares the trixie `python:3.12-slim` base with the gateway image.
|
||||||
# the orchestrator grows real deps they can be `COPY --from`'d into the
|
|
||||||
# infra image cleanly (same base/python — installed packages copy safely).
|
|
||||||
|
|
||||||
FROM python:3.12-slim
|
FROM python:3.12-slim
|
||||||
|
|
||||||
|
|||||||
@@ -11,7 +11,7 @@ The bulk of the implementation lives in sibling modules:
|
|||||||
- launch: bring-up + teardown context manager
|
- launch: bring-up + teardown context manager
|
||||||
- cleanup: orphan enumeration, removal, active listing
|
- cleanup: orphan enumeration, removal, active listing
|
||||||
- backend: DockerBottleBackend façade wiring the above
|
- backend: DockerBottleBackend façade wiring the above
|
||||||
- infra: DockerInfraService (the per-host infra container)
|
- infra: DockerInfraService (the per-host orchestrator + gateway pair)
|
||||||
|
|
||||||
Thin by design: the public names are re-exported lazily via `__getattr__`, so
|
Thin by design: the public names are re-exported lazily via `__getattr__`, so
|
||||||
importing a leaf like `backend.docker.util` doesn't drag `DockerBottleBackend`
|
importing a leaf like `backend.docker.util` doesn't drag `DockerBottleBackend`
|
||||||
|
|||||||
@@ -2,7 +2,7 @@
|
|||||||
|
|
||||||
Composes the orchestrator primitives into the register/teardown sequence:
|
Composes the orchestrator primitives into the register/teardown sequence:
|
||||||
|
|
||||||
1. ensure the single infra container (control plane + gateway) is up;
|
1. ensure the per-host pair (orchestrator + gateway containers) is up;
|
||||||
2. allocate the bottle a pinned source IP on the gateway network;
|
2. allocate the bottle a pinned source IP on the gateway network;
|
||||||
3. register it and provision its git-gate repos/creds into the gateway.
|
3. register it and provision its git-gate repos/creds into the gateway.
|
||||||
|
|
||||||
@@ -62,7 +62,7 @@ def _network_cidr(network: str) -> str:
|
|||||||
|
|
||||||
def _network_container_ips(network: str) -> list[str]:
|
def _network_container_ips(network: str) -> list[str]:
|
||||||
"""Every address currently assigned on the gateway network — the ground
|
"""Every address currently assigned on the gateway network — the ground
|
||||||
truth for "in use": the infra container and every live agent. Read from
|
truth for "in use": the gateway container and every live agent. Read from
|
||||||
the network so a new bottle can't collide with anything actually attached."""
|
the network so a new bottle can't collide with anything actually attached."""
|
||||||
proc = run_docker([
|
proc = run_docker([
|
||||||
"docker", "network", "inspect", "--format",
|
"docker", "network", "inspect", "--format",
|
||||||
@@ -80,7 +80,7 @@ def _reprovision_running_bottles(
|
|||||||
infra_name: str = INFRA_NAME,
|
infra_name: str = INFRA_NAME,
|
||||||
) -> None:
|
) -> None:
|
||||||
"""Re-inject egress tokens for any registered bottles that lost their
|
"""Re-inject egress tokens for any registered bottles that lost their
|
||||||
in-memory tokens (e.g., after an infra container restart).
|
in-memory tokens (e.g., after an orchestrator restart).
|
||||||
|
|
||||||
For each registered bottle whose source IP maps to a live container on the
|
For each registered bottle whose source IP maps to a live container on the
|
||||||
gateway network, reads ENV_VAR_SECRET via ``docker exec … printenv`` and
|
gateway network, reads ENV_VAR_SECRET via ``docker exec … printenv`` and
|
||||||
@@ -89,7 +89,7 @@ def _reprovision_running_bottles(
|
|||||||
container exec failure never blocks a new bottle launch."""
|
container exec failure never blocks a new bottle launch."""
|
||||||
client = OrchestratorClient(orchestrator_url)
|
client = OrchestratorClient(orchestrator_url)
|
||||||
# Build {source_ip: container_name} from live containers on the gateway
|
# Build {source_ip: container_name} from live containers on the gateway
|
||||||
# network, excluding the infra container itself.
|
# network, excluding the gateway container itself.
|
||||||
try:
|
try:
|
||||||
proc = run_docker([
|
proc = run_docker([
|
||||||
"docker", "network", "inspect",
|
"docker", "network", "inspect",
|
||||||
@@ -133,11 +133,11 @@ def launch_consolidated(
|
|||||||
infra_name: str = INFRA_NAME,
|
infra_name: str = INFRA_NAME,
|
||||||
network: str = GATEWAY_NETWORK,
|
network: str = GATEWAY_NETWORK,
|
||||||
) -> LaunchContext:
|
) -> LaunchContext:
|
||||||
"""Ensure the infra container is up, allocate + register the bottle, and
|
"""Ensure the orchestrator + gateway pair is up, allocate + register the bottle, and
|
||||||
provision its git-gate state. Returns the agent's attach context.
|
provision its git-gate state. Returns the agent's attach context.
|
||||||
|
|
||||||
Also reprovisiones egress tokens for any already-running bottles that lost
|
Also reprovisiones egress tokens for any already-running bottles that lost
|
||||||
their in-memory credentials (e.g. after an infra container restart), so
|
their in-memory credentials (e.g. after an orchestrator restart), so
|
||||||
they regain egress access before the new bottle is registered."""
|
they regain egress access before the new bottle is registered."""
|
||||||
service = service or DockerInfraService()
|
service = service or DockerInfraService()
|
||||||
url = service.ensure_running()
|
url = service.ensure_running()
|
||||||
|
|||||||
@@ -69,8 +69,8 @@ def _boot_args(
|
|||||||
pub_b64 = base64.b64encode(pubkey.encode()).decode()
|
pub_b64 = base64.b64encode(pubkey.encode()).decode()
|
||||||
args = f"{_BASE_BOOT_ARGS} {ip_arg} bb_pubkey={pub_b64}"
|
args = f"{_BASE_BOOT_ARGS} {ip_arg} bb_pubkey={pub_b64}"
|
||||||
# `extra` carries caller-supplied cmdline params the guest init reads
|
# `extra` carries caller-supplied cmdline params the guest init reads
|
||||||
# (e.g. `bb_role=orchestrator|gateway` selecting which infra plane a
|
# (e.g. the gateway VM's `bb_orch=<orchestrator guest IP>`). Agent VMs pass
|
||||||
# shared-rootfs infra VM runs). Agent VMs pass nothing.
|
# nothing.
|
||||||
return f"{args} {extra}".rstrip() if extra else args
|
return f"{args} {extra}".rstrip() if extra else args
|
||||||
|
|
||||||
|
|
||||||
|
|||||||
@@ -8,14 +8,13 @@ gateway VM never opens `bot-bottle.db` (#469), so it holds no signing key —
|
|||||||
only the token the host hands it via `connect_to_orchestrator`.
|
only the token the host hands it via `connect_to_orchestrator`.
|
||||||
|
|
||||||
What deliberately stays in `infra_vm` (not moved here): the plane-agnostic VM
|
What deliberately stays in `infra_vm` (not moved here): the plane-agnostic VM
|
||||||
substrate the orchestrator VM shares — booting a VM from the shared rootfs
|
substrate the orchestrator VM also uses — booting a VM from a per-plane rootfs
|
||||||
(`boot_vm`), the stable SSH keypair, the secret-push retry loop, and the PID
|
(`boot_vm`), the stable SSH keypair, the secret-push retry loop, and the PID
|
||||||
lifecycle — plus the pair coordinator (`ensure_running`: orchestrator-first
|
lifecycle — plus the pair coordinator (`ensure_running`: orchestrator-first
|
||||||
health gate, singleton lock, adoption/version marker). The guest-side gateway
|
health gate, singleton lock, adoption/version marker). The guest-side gateway
|
||||||
daemon startup lives in the shared `bb_role=gateway` init branch baked into the
|
daemon startup lives in the gateway rootfs's guest init (`_gateway_init` in
|
||||||
one published rootfs both VMs boot, so it can't live in a host-side method
|
`infra_vm`), so it can't live in a host-side method either. These are the shared
|
||||||
either. These are the shared seam that moves to a neutral module when the
|
seam.
|
||||||
Orchestrator service lands and `infra_vm` is dissolved.
|
|
||||||
"""
|
"""
|
||||||
|
|
||||||
from __future__ import annotations
|
from __future__ import annotations
|
||||||
@@ -33,18 +32,19 @@ from .. import util as backend_util
|
|||||||
from . import infra_vm, netpool, util
|
from . import infra_vm, netpool, util
|
||||||
from .gateway_transport import FirecrackerGatewayTransport
|
from .gateway_transport import FirecrackerGatewayTransport
|
||||||
|
|
||||||
# The gateway microVM's name (the `bb_role=gateway` boot tag / run dir). Fixed
|
# The gateway microVM's name (its run dir). Fixed per host — one gateway VM,
|
||||||
# per host — one gateway VM, shared by every agent VM.
|
# shared by every agent VM.
|
||||||
GATEWAY_NAME = "bot-bottle-gateway"
|
GATEWAY_NAME = "bot-bottle-gateway"
|
||||||
|
|
||||||
# The gateway VM's slim memory ceiling (buildah is present on the shared rootfs
|
# The gateway VM's slim memory ceiling — the data plane carries no build
|
||||||
# but unused on the data plane) — PRD 0070 "Memory: fixed ceilings".
|
# tooling (buildah lives only on the orchestrator rootfs) — PRD 0070
|
||||||
|
# "Memory: fixed ceilings".
|
||||||
_GW_MEM_MIB = 2048
|
_GW_MEM_MIB = 2048
|
||||||
|
|
||||||
# The pre-minted `gateway` JWT path in the guest. The *shared* init (both planes
|
# The pre-minted `gateway` JWT path in the guest. The gateway init (in
|
||||||
# boot one rootfs) waits for it before starting the data plane, so its canonical
|
# `infra_vm`) waits for it before starting the data plane, so its canonical
|
||||||
# definition lives with that init in `infra_vm`; imported here for the push so
|
# definition lives with that init; imported here for the push so the
|
||||||
# the load-bearing path isn't duplicated.
|
# load-bearing path isn't duplicated.
|
||||||
_GUEST_GATEWAY_JWT_PATH = infra_vm._GUEST_GATEWAY_JWT_PATH
|
_GUEST_GATEWAY_JWT_PATH = infra_vm._GUEST_GATEWAY_JWT_PATH
|
||||||
# mitmproxy writes its CA here a beat after start; agents install it to trust
|
# mitmproxy writes its CA here a beat after start; agents install it to trust
|
||||||
# the gateway's TLS interception. Host-side only (SSH cat), so it lives here.
|
# the gateway's TLS interception. Host-side only (SSH cat), so it lives here.
|
||||||
@@ -56,7 +56,7 @@ _CA_FETCH_TIMEOUT_SECONDS = 15.0
|
|||||||
class FirecrackerGateway(Gateway):
|
class FirecrackerGateway(Gateway):
|
||||||
"""The consolidated gateway as a Firecracker microVM on the gateway link.
|
"""The consolidated gateway as a Firecracker microVM on the gateway link.
|
||||||
|
|
||||||
The shared rootfs is built/downloaded by `infra_vm.ensure_built` (the ABC's
|
The gateway rootfs is built/downloaded by `infra_vm.ensure_built` (the ABC's
|
||||||
`ensure_built` no-op here); `connect_to_orchestrator` boots the VM resolving
|
`ensure_built` no-op here); `connect_to_orchestrator` boots the VM resolving
|
||||||
policy against the orchestrator and seeds the pre-minted `gateway` token."""
|
policy against the orchestrator and seeds the pre-minted `gateway` token."""
|
||||||
|
|
||||||
@@ -94,8 +94,8 @@ class FirecrackerGateway(Gateway):
|
|||||||
raise GatewayError(
|
raise GatewayError(
|
||||||
f"cannot resolve orchestrator guest IP from {self._orchestrator_url!r}"
|
f"cannot resolve orchestrator guest IP from {self._orchestrator_url!r}"
|
||||||
)
|
)
|
||||||
# Boot on the gateway link from the shared rootfs (bb_role=gateway), then
|
# Boot on the gateway link from the gateway rootfs, then push the token
|
||||||
# push the token the init waits for before starting the data plane.
|
# the init waits for before starting the data plane.
|
||||||
vm = infra_vm.boot_vm(
|
vm = infra_vm.boot_vm(
|
||||||
name=GATEWAY_NAME, slot=netpool.gw_slot(), run_dir=infra_vm._gw_dir(),
|
name=GATEWAY_NAME, slot=netpool.gw_slot(), run_dir=infra_vm._gw_dir(),
|
||||||
role="gateway", mem_mib=_GW_MEM_MIB,
|
role="gateway", mem_mib=_GW_MEM_MIB,
|
||||||
|
|||||||
@@ -8,11 +8,11 @@ signing key over SSH, and waiting for `/health`. The host CLI reaches it at its
|
|||||||
guest IP, and so does the gateway (`bb_orch` cmdline), so `url()` == `gateway_url()`.
|
guest IP, and so does the gateway (`bb_orch` cmdline), so `url()` == `gateway_url()`.
|
||||||
|
|
||||||
What deliberately stays in `infra_vm` (not moved here): the plane-agnostic VM
|
What deliberately stays in `infra_vm` (not moved here): the plane-agnostic VM
|
||||||
substrate the gateway VM also shares — booting a VM from the shared rootfs
|
substrate the gateway VM also uses — booting a VM from a per-plane rootfs
|
||||||
(`boot_vm`), the stable SSH keypair, the secret-push retry, the PID lifecycle —
|
(`boot_vm`), the stable SSH keypair, the secret-push retry, the PID lifecycle —
|
||||||
plus the pair coordinator (`ensure_running`: adopt-or-boot-both under a singleton
|
plus the pair coordinator (`ensure_running`: adopt-or-boot-both under a singleton
|
||||||
lock with a shared version marker) and the single shared `bb_role`-branched init
|
lock with a combined version marker) and the per-plane guest inits (`role_init`).
|
||||||
baked into the one published rootfs. Those are the shared seam.
|
Those are the shared seam.
|
||||||
"""
|
"""
|
||||||
|
|
||||||
from __future__ import annotations
|
from __future__ import annotations
|
||||||
@@ -30,8 +30,8 @@ from ...orchestrator.lifecycle import (
|
|||||||
from . import firecracker_vm, infra_vm, netpool
|
from . import firecracker_vm, infra_vm, netpool
|
||||||
from .infra_vm import ORCHESTRATOR_PORT
|
from .infra_vm import ORCHESTRATOR_PORT
|
||||||
|
|
||||||
# The orchestrator microVM's name (the `bb_role=orchestrator` boot tag / run
|
# The orchestrator microVM's name (its run dir). Fixed per host — one
|
||||||
# dir). Fixed per host — one control-plane VM.
|
# control-plane VM.
|
||||||
ORCHESTRATOR_NAME = "bot-bottle-orchestrator"
|
ORCHESTRATOR_NAME = "bot-bottle-orchestrator"
|
||||||
|
|
||||||
# Memory ceiling (fixed at boot, demand-paged). The orchestrator keeps the build
|
# Memory ceiling (fixed at boot, demand-paged). The orchestrator keeps the build
|
||||||
@@ -58,9 +58,9 @@ def registry_volume_path() -> Path:
|
|||||||
class FirecrackerOrchestrator(Orchestrator):
|
class FirecrackerOrchestrator(Orchestrator):
|
||||||
"""The control plane as a Firecracker microVM on the orchestrator link.
|
"""The control plane as a Firecracker microVM on the orchestrator link.
|
||||||
|
|
||||||
The shared rootfs is built/downloaded by `infra_vm.ensure_built` (the ABC's
|
The orchestrator rootfs is built/downloaded by `infra_vm.ensure_built` (the
|
||||||
`ensure_built` no-op here); `ensure_running` boots the VM, seeds the signing
|
ABC's `ensure_built` no-op here); `ensure_running` boots the VM, seeds the
|
||||||
key, and blocks until `/health` answers."""
|
signing key, and blocks until `/health` answers."""
|
||||||
|
|
||||||
name = ORCHESTRATOR_NAME
|
name = ORCHESTRATOR_NAME
|
||||||
|
|
||||||
|
|||||||
@@ -258,8 +258,8 @@ def build_committed_rootfs_dir(tar_path: Path) -> Path:
|
|||||||
|
|
||||||
def inject_guest_boot(rootfs: Path, init_script: str | None = None) -> None:
|
def inject_guest_boot(rootfs: Path, init_script: str | None = None) -> None:
|
||||||
"""Drop the static dropbear and the PID-1 init into the rootfs.
|
"""Drop the static dropbear and the PID-1 init into the rootfs.
|
||||||
`init_script` defaults to the SSH-only agent init; the infra VM
|
`init_script` defaults to the SSH-only agent init; each infra VM
|
||||||
passes its own (control plane + gateway) init.
|
passes its own per-plane init (orchestrator or gateway).
|
||||||
|
|
||||||
A committed snapshot is guest-controlled, so `bb-dropbear`/`bb-init`
|
A committed snapshot is guest-controlled, so `bb-dropbear`/`bb-init`
|
||||||
may already exist as symlinks aimed at a host file (e.g. bb-init ->
|
may already exist as symlinks aimed at a host file (e.g. bb-init ->
|
||||||
|
|||||||
@@ -101,10 +101,10 @@ class MacosContainerBottleBackend(
|
|||||||
yield bottle
|
yield bottle
|
||||||
|
|
||||||
def ensure_orchestrator(self) -> str:
|
def ensure_orchestrator(self) -> str:
|
||||||
"""Bring up the per-host infra container (control plane + gateway) and
|
"""Bring up the per-host pair (orchestrator + gateway containers) and
|
||||||
return its control-plane URL — the on-demand entry point operator tools
|
return its control-plane URL — the on-demand entry point operator tools
|
||||||
(`supervise`) call when no control plane is running yet. Mirrors
|
(`supervise`) call when no control plane is running yet. Mirrors
|
||||||
firecracker's infra-VM bring-up."""
|
firecracker's infra bring-up."""
|
||||||
from .infra import MacosInfraService
|
from .infra import MacosInfraService
|
||||||
return MacosInfraService().ensure_running().orchestrator_url
|
return MacosInfraService().ensure_running().orchestrator_url
|
||||||
|
|
||||||
|
|||||||
@@ -15,8 +15,10 @@ caller has to start the agent in between. `ensure_gateway` runs first because
|
|||||||
the agent's proxy env needs the gateway's address at `container run` time; the
|
the agent's proxy env needs the gateway's address at `container run` time; the
|
||||||
agent's *own* address (the attribution key) only exists afterwards.
|
agent's *own* address (the attribution key) only exists afterwards.
|
||||||
|
|
||||||
The control plane and the gateway are one **infra container** here (see
|
The control plane and the gateway are **separate containers** here (see
|
||||||
`infra`), so `gateway_ip` and the control-plane host are the same address.
|
`infra`): the orchestrator on the host-only control network, the gateway on the
|
||||||
|
agent network — `gateway_ip` is the gateway container's agent-network address,
|
||||||
|
distinct from the orchestrator's control-network host.
|
||||||
|
|
||||||
The consequence for the identity token: it is minted by registration, i.e.
|
The consequence for the identity token: it is minted by registration, i.e.
|
||||||
*after* the agent container exists, so it cannot be baked into the run-time
|
*after* the agent container exists, so it cannot be baked into the run-time
|
||||||
@@ -54,9 +56,9 @@ class ConsolidatedLaunchError(RuntimeError):
|
|||||||
|
|
||||||
@dataclass(frozen=True)
|
@dataclass(frozen=True)
|
||||||
class GatewayEndpoint:
|
class GatewayEndpoint:
|
||||||
"""What the agent `container run` needs to reach the shared gateway (the
|
"""What the agent `container run` needs to reach the shared gateway.
|
||||||
infra container). `gateway_ip` is that container's host-only address, the
|
`gateway_ip` is the gateway container's agent-network address (the agent's
|
||||||
same host the control-plane URL points at."""
|
proxy target); `orchestrator_url` points at the separate control plane."""
|
||||||
|
|
||||||
orchestrator_url: str
|
orchestrator_url: str
|
||||||
gateway_ip: str # the gateway's address — the agent's proxy target
|
gateway_ip: str # the gateway's address — the agent's proxy target
|
||||||
@@ -80,10 +82,10 @@ class LaunchContext:
|
|||||||
def ensure_gateway(
|
def ensure_gateway(
|
||||||
*, service: MacosInfraService | None = None,
|
*, service: MacosInfraService | None = None,
|
||||||
) -> GatewayEndpoint:
|
) -> GatewayEndpoint:
|
||||||
"""Ensure the per-host infra container (control plane + gateway) is up and
|
"""Ensure the per-host pair (orchestrator + gateway containers) is up and
|
||||||
report how to reach it. Idempotent — one singleton, so N bottle launches
|
report how to reach the gateway. Idempotent — one singleton pair, so N bottle
|
||||||
share it. Call before starting the agent container: the agent's proxy env
|
launches share it. Call before starting the agent container: the agent's
|
||||||
needs `gateway_ip` at run time."""
|
proxy env needs `gateway_ip` at run time."""
|
||||||
service = service or MacosInfraService()
|
service = service or MacosInfraService()
|
||||||
infra = service.ensure_running()
|
infra = service.ensure_running()
|
||||||
endpoint = GatewayEndpoint(
|
endpoint = GatewayEndpoint(
|
||||||
|
|||||||
@@ -6,17 +6,18 @@ import subprocess
|
|||||||
|
|
||||||
from ...bottle_state import read_metadata
|
from ...bottle_state import read_metadata
|
||||||
from .. import ActiveAgent
|
from .. import ActiveAgent
|
||||||
from .infra import INFRA_NAME
|
from .infra import INFRA_NAME, ORCHESTRATOR_NAME
|
||||||
|
|
||||||
# The name every agent container carries: `bot-bottle-<slug>`. Exported
|
# The name every agent container carries: `bot-bottle-<slug>`. Exported
|
||||||
# because callers that act on a running bottle (gateway-host rewrites,
|
# because callers that act on a running bottle (gateway-host rewrites,
|
||||||
# registry reconciliation) have to map an enumerated slug back to a
|
# registry reconciliation) have to map an enumerated slug back to a
|
||||||
# container name.
|
# container name.
|
||||||
CONTAINER_NAME_PREFIX = "bot-bottle-"
|
CONTAINER_NAME_PREFIX = "bot-bottle-"
|
||||||
# The shared per-host infra container carries the same prefix as agent
|
# The two shared per-host infra containers (orchestrator + gateway) carry the
|
||||||
# containers but is infrastructure, not a bottle — one control plane + gateway
|
# same `bot-bottle-` prefix as agent containers but are infrastructure, not
|
||||||
# serves every agent, so listing it as an agent would invent one per host.
|
# bottles — one pair serves every agent, so enumerating either as an agent would
|
||||||
_INFRA_NAMES = frozenset({INFRA_NAME})
|
# invent a phantom bottle per host.
|
||||||
|
_INFRA_NAMES = frozenset({INFRA_NAME, ORCHESTRATOR_NAME})
|
||||||
|
|
||||||
|
|
||||||
class EnumerationError(RuntimeError):
|
class EnumerationError(RuntimeError):
|
||||||
|
|||||||
@@ -1,7 +1,7 @@
|
|||||||
"""Stable gateway name for macOS agents, via each bottle's `/etc/hosts`.
|
"""Stable gateway name for macOS agents, via each bottle's `/etc/hosts`.
|
||||||
|
|
||||||
The shared gateway's address is assigned by vmnet's DHCP and changes whenever
|
The shared gateway's address is assigned by vmnet's DHCP and changes whenever
|
||||||
the infra container is recreated — a source-hash bump, an image upgrade, a
|
the gateway container is recreated — a source-hash bump, an image upgrade, a
|
||||||
crash. Every agent-facing URL (egress proxy, git-http, supervise) embeds that
|
crash. Every agent-facing URL (egress proxy, git-http, supervise) embeds that
|
||||||
address, and the proxy URL reaches the agent as **process environment** at
|
address, and the proxy URL reaches the agent as **process environment** at
|
||||||
`container exec` time. A running process's `environ` cannot be rewritten from
|
`container exec` time. A running process's `environ` cannot be rewritten from
|
||||||
|
|||||||
@@ -54,19 +54,15 @@ class _DaemonSpec:
|
|||||||
_EGRESS_ONLY_ENV_PREFIXES: tuple[str, ...] = ("EGRESS_TOKEN_",)
|
_EGRESS_ONLY_ENV_PREFIXES: tuple[str, ...] = ("EGRESS_TOKEN_",)
|
||||||
_READY_GATED_DAEMONS: tuple[str, ...] = ("git-gate", "git-http")
|
_READY_GATED_DAEMONS: tuple[str, ...] = ("git-gate", "git-http")
|
||||||
|
|
||||||
# The control-plane signing key is the orchestrator's alone — verifying tokens.
|
# The control-plane signing key is the orchestrator's alone (it verifies
|
||||||
# The data-plane daemons instead hold the pre-minted `gateway` JWT they present.
|
# tokens), and the orchestrator runs in a separate container/VM — the gateway
|
||||||
# Scoping each to its process (even in the combined infra container) keeps a
|
# only ever holds the pre-minted `gateway` JWT its daemons present. Strip the
|
||||||
# compromised data-plane daemon from reading the key and minting a `cli` token
|
# key from every daemon's env as defense-in-depth, so a compromised data-plane
|
||||||
# (issue #469 review). Values match paths.ORCHESTRATOR_TOKEN_ENV /
|
# daemon can't read it and mint a `cli` token even if it somehow leaked into the
|
||||||
# ORCHESTRATOR_AUTH_JWT_ENV; hardcoded here so this supervisor stays import-light.
|
# gateway container (issue #469 review). Value matches
|
||||||
|
# paths.ORCHESTRATOR_TOKEN_ENV; hardcoded here so this supervisor stays
|
||||||
|
# import-light.
|
||||||
_SIGNING_KEY_ENV = "BOT_BOTTLE_ORCHESTRATOR_TOKEN"
|
_SIGNING_KEY_ENV = "BOT_BOTTLE_ORCHESTRATOR_TOKEN"
|
||||||
_GATEWAY_JWT_ENV = "BOT_BOTTLE_ORCHESTRATOR_AUTH_JWT"
|
|
||||||
|
|
||||||
# Daemons that must be requested explicitly via BOT_BOTTLE_GATEWAY_DAEMONS
|
|
||||||
# and are NOT started in the default (env-var-unset) case. The orchestrator
|
|
||||||
# only runs in the combined infra container, never in a standalone gateway.
|
|
||||||
_OPT_IN_DAEMONS: frozenset[str] = frozenset({"orchestrator"})
|
|
||||||
|
|
||||||
|
|
||||||
def _env_for_daemon(name: str, base_env: dict[str, str]) -> dict[str, str]:
|
def _env_for_daemon(name: str, base_env: dict[str, str]) -> dict[str, str]:
|
||||||
@@ -74,30 +70,20 @@ def _env_for_daemon(name: str, base_env: dict[str, str]) -> dict[str, str]:
|
|||||||
Returns a fresh dict — callers can mutate without affecting `base_env`.
|
Returns a fresh dict — callers can mutate without affecting `base_env`.
|
||||||
|
|
||||||
* `EGRESS_TOKEN_*` upstream-auth slots go to egress only.
|
* `EGRESS_TOKEN_*` upstream-auth slots go to egress only.
|
||||||
* the control-plane signing key goes to the orchestrator only.
|
* the control-plane signing key is stripped from every daemon — the
|
||||||
* the pre-minted `gateway` JWT goes to the data-plane daemons only (the
|
gateway never holds it (it's the orchestrator's); the data-plane daemons
|
||||||
orchestrator verifies tokens, it never presents one)."""
|
present the pre-minted `gateway` JWT instead."""
|
||||||
env = dict(base_env)
|
env = dict(base_env)
|
||||||
if name != "egress":
|
if name != "egress":
|
||||||
env = {
|
env = {
|
||||||
k: v for k, v in env.items()
|
k: v for k, v in env.items()
|
||||||
if not any(k.startswith(p) for p in _EGRESS_ONLY_ENV_PREFIXES)
|
if not any(k.startswith(p) for p in _EGRESS_ONLY_ENV_PREFIXES)
|
||||||
}
|
}
|
||||||
if name == "orchestrator":
|
env.pop(_SIGNING_KEY_ENV, None)
|
||||||
env.pop(_GATEWAY_JWT_ENV, None)
|
|
||||||
else:
|
|
||||||
env.pop(_SIGNING_KEY_ENV, None)
|
|
||||||
return env
|
return env
|
||||||
|
|
||||||
|
|
||||||
# The orchestrator is listed first so it starts before the gateway daemons,
|
|
||||||
# giving the control plane a head start to accept /resolve calls. The gateway
|
|
||||||
# daemons tolerate early /resolve failures and retry per-request.
|
|
||||||
_DAEMONS: tuple[_DaemonSpec, ...] = (
|
_DAEMONS: tuple[_DaemonSpec, ...] = (
|
||||||
_DaemonSpec("orchestrator", (
|
|
||||||
"python3", "-m", "bot_bottle.orchestrator",
|
|
||||||
"--host", "0.0.0.0", "--port", "8099", "--broker", "stub",
|
|
||||||
)),
|
|
||||||
_DaemonSpec("egress", ("/bin/sh", "/app/egress-entrypoint.sh")),
|
_DaemonSpec("egress", ("/bin/sh", "/app/egress-entrypoint.sh")),
|
||||||
_DaemonSpec("git-gate", ("/bin/sh", "/git-gate-entrypoint.sh")),
|
_DaemonSpec("git-gate", ("/bin/sh", "/git-gate-entrypoint.sh")),
|
||||||
_DaemonSpec("git-http", ("python3", "-m", "bot_bottle.gateway.git_gate.http_backend")),
|
_DaemonSpec("git-http", ("python3", "-m", "bot_bottle.gateway.git_gate.http_backend")),
|
||||||
@@ -127,10 +113,8 @@ def _selected_daemons(
|
|||||||
) -> tuple[_DaemonSpec, ...]:
|
) -> tuple[_DaemonSpec, ...]:
|
||||||
"""Filter the daemon set by the BOT_BOTTLE_GATEWAY_DAEMONS env var.
|
"""Filter the daemon set by the BOT_BOTTLE_GATEWAY_DAEMONS env var.
|
||||||
|
|
||||||
When the var is unset/empty, return all non-opt-in daemons (the
|
When the var is unset/empty, return all daemons (the standard gateway
|
||||||
standard gateway subset). Opt-in daemons (e.g. `orchestrator`) only
|
subset). Unknown names are ignored.
|
||||||
run when explicitly named — they never start in a plain gateway
|
|
||||||
container that doesn't set the env var. Unknown names are ignored.
|
|
||||||
|
|
||||||
`all_daemons` defaults to `_DAEMONS` resolved at call time (not at
|
`all_daemons` defaults to `_DAEMONS` resolved at call time (not at
|
||||||
definition time), so tests can pass a custom list."""
|
definition time), so tests can pass a custom list."""
|
||||||
@@ -138,7 +122,7 @@ def _selected_daemons(
|
|||||||
all_daemons = _DAEMONS
|
all_daemons = _DAEMONS
|
||||||
raw = env.get("BOT_BOTTLE_GATEWAY_DAEMONS", "").strip()
|
raw = env.get("BOT_BOTTLE_GATEWAY_DAEMONS", "").strip()
|
||||||
if not raw:
|
if not raw:
|
||||||
return tuple(d for d in all_daemons if d.name not in _OPT_IN_DAEMONS)
|
return tuple(all_daemons)
|
||||||
wanted = {n.strip() for n in raw.split(",") if n.strip()}
|
wanted = {n.strip() for n in raw.split(",") if n.strip()}
|
||||||
return tuple(d for d in all_daemons if d.name in wanted)
|
return tuple(d for d in all_daemons if d.name in wanted)
|
||||||
|
|
||||||
|
|||||||
@@ -182,7 +182,7 @@ class OrchestratorClient:
|
|||||||
"""Drop registry rows for bottles that are no longer running
|
"""Drop registry rows for bottles that are no longer running
|
||||||
(`POST /reconcile`), returning the reaped bottle ids. `live_source_ips`
|
(`POST /reconcile`), returning the reaped bottle ids. `live_source_ips`
|
||||||
is the caller's enumeration of its live bottles — the orchestrator
|
is the caller's enumeration of its live bottles — the orchestrator
|
||||||
can't see the backend from inside the infra container."""
|
can't see the backend from inside the orchestrator container/VM."""
|
||||||
body: dict[str, object] = {"live_source_ips": list(live_source_ips)}
|
body: dict[str, object] = {"live_source_ips": list(live_source_ips)}
|
||||||
if grace_seconds is not None:
|
if grace_seconds is not None:
|
||||||
body["grace_seconds"] = grace_seconds
|
body["grace_seconds"] = grace_seconds
|
||||||
@@ -257,7 +257,7 @@ def discover_orchestrator_url(*, timeout: float = 2.0) -> str:
|
|||||||
f"http://{netpool.orch_slot().guest_ip}:{ORCHESTRATOR_PORT}")
|
f"http://{netpool.orch_slot().guest_ip}:{ORCHESTRATOR_PORT}")
|
||||||
except Exception: # noqa: BLE001 — backend optional / not firecracker
|
except Exception: # noqa: BLE001 — backend optional / not firecracker
|
||||||
pass
|
pass
|
||||||
try: # macOS: infra container control plane on its host-only address
|
try: # macOS: orchestrator container on its host-only address
|
||||||
from ..backend.macos_container.infra import probe_orchestrator_url
|
from ..backend.macos_container.infra import probe_orchestrator_url
|
||||||
url = probe_orchestrator_url()
|
url = probe_orchestrator_url()
|
||||||
if url:
|
if url:
|
||||||
|
|||||||
@@ -64,11 +64,11 @@ class TestEnvForDaemon(unittest.TestCase):
|
|||||||
self.assertNotIn("X", self._BASE)
|
self.assertNotIn("X", self._BASE)
|
||||||
|
|
||||||
|
|
||||||
class TestOrchestratorEnvScoping(unittest.TestCase):
|
class TestSigningKeyScoping(unittest.TestCase):
|
||||||
"""The control-plane signing key stays with the orchestrator; the pre-minted
|
"""The gateway never holds the control-plane signing key — it's the
|
||||||
`gateway` JWT goes to the data-plane daemons (issue #469 review). Scoping
|
orchestrator's, which runs in a separate container/VM. Its daemons present
|
||||||
them per-process keeps a compromised data-plane daemon from reading the key
|
the pre-minted `gateway` JWT instead; the key is stripped from every daemon's
|
||||||
and minting a higher-privilege token, even in the combined infra container."""
|
env as defense-in-depth (issue #469 review)."""
|
||||||
|
|
||||||
_BASE = {
|
_BASE = {
|
||||||
"PATH": "/usr/bin",
|
"PATH": "/usr/bin",
|
||||||
@@ -76,11 +76,6 @@ class TestOrchestratorEnvScoping(unittest.TestCase):
|
|||||||
"BOT_BOTTLE_ORCHESTRATOR_AUTH_JWT": "gw-jwt",
|
"BOT_BOTTLE_ORCHESTRATOR_AUTH_JWT": "gw-jwt",
|
||||||
}
|
}
|
||||||
|
|
||||||
def test_orchestrator_gets_key_not_jwt(self):
|
|
||||||
env = _env_for_daemon("orchestrator", self._BASE)
|
|
||||||
self.assertEqual("sk-x", env["BOT_BOTTLE_ORCHESTRATOR_TOKEN"])
|
|
||||||
self.assertNotIn("BOT_BOTTLE_ORCHESTRATOR_AUTH_JWT", env)
|
|
||||||
|
|
||||||
def test_data_plane_daemons_get_jwt_not_key(self):
|
def test_data_plane_daemons_get_jwt_not_key(self):
|
||||||
for name in ("egress", "git-gate", "git-http", "supervise"):
|
for name in ("egress", "git-gate", "git-http", "supervise"):
|
||||||
env = _env_for_daemon(name, self._BASE)
|
env = _env_for_daemon(name, self._BASE)
|
||||||
|
|||||||
@@ -57,9 +57,11 @@ class TestEnsureGateway(unittest.TestCase):
|
|||||||
def _service(self) -> MagicMock:
|
def _service(self) -> MagicMock:
|
||||||
from bot_bottle.backend.macos_container.infra import InfraEndpoint
|
from bot_bottle.backend.macos_container.infra import InfraEndpoint
|
||||||
service = MagicMock()
|
service = MagicMock()
|
||||||
|
# Two containers now: the orchestrator on the control network, the
|
||||||
|
# gateway on the agent network — distinct addresses.
|
||||||
service.ensure_running.return_value = InfraEndpoint(
|
service.ensure_running.return_value = InfraEndpoint(
|
||||||
orchestrator_url="http://192.168.128.2:8099",
|
orchestrator_url="http://192.168.128.2:8099",
|
||||||
gateway_ip="192.168.128.2",
|
gateway_ip="192.168.128.3",
|
||||||
)
|
)
|
||||||
service.network = "bot-bottle-mac-gateway"
|
service.network = "bot-bottle-mac-gateway"
|
||||||
service.ca_cert_pem.return_value = "PEM"
|
service.ca_cert_pem.return_value = "PEM"
|
||||||
@@ -68,18 +70,16 @@ class TestEnsureGateway(unittest.TestCase):
|
|||||||
def test_reports_gateway_endpoint(self) -> None:
|
def test_reports_gateway_endpoint(self) -> None:
|
||||||
endpoint = self._run(self._service())
|
endpoint = self._run(self._service())
|
||||||
self.assertEqual("http://192.168.128.2:8099", endpoint.orchestrator_url)
|
self.assertEqual("http://192.168.128.2:8099", endpoint.orchestrator_url)
|
||||||
self.assertEqual("192.168.128.2", endpoint.gateway_ip)
|
self.assertEqual("192.168.128.3", endpoint.gateway_ip)
|
||||||
self.assertEqual("PEM", endpoint.gateway_ca_pem)
|
self.assertEqual("PEM", endpoint.gateway_ca_pem)
|
||||||
self.assertEqual("bot-bottle-mac-gateway", endpoint.network)
|
self.assertEqual("bot-bottle-mac-gateway", endpoint.network)
|
||||||
|
|
||||||
def test_orchestrator_and_gateway_share_one_address(self) -> None:
|
def test_gateway_ip_is_distinct_from_the_control_plane(self) -> None:
|
||||||
"""One infra container hosts both, so the gateway IP and the
|
# The planes are separate containers, so the agent's proxy target (the
|
||||||
control-plane host are the same."""
|
# gateway) is a different address from the control-plane host.
|
||||||
endpoint = self._run(self._service())
|
endpoint = self._run(self._service())
|
||||||
self.assertEqual(
|
orch_host = endpoint.orchestrator_url.split("://")[1].split(":")[0]
|
||||||
endpoint.gateway_ip,
|
self.assertNotEqual(orch_host, endpoint.gateway_ip)
|
||||||
endpoint.orchestrator_url.split("://")[1].split(":")[0],
|
|
||||||
)
|
|
||||||
|
|
||||||
|
|
||||||
class TestRegisterAgent(unittest.TestCase):
|
class TestRegisterAgent(unittest.TestCase):
|
||||||
|
|||||||
@@ -60,10 +60,12 @@ class TestMacosContainerEnumerate(unittest.TestCase):
|
|||||||
self.assertEqual(["dev-abc"], [a.slug for a in agents])
|
self.assertEqual(["dev-abc"], [a.slug for a in agents])
|
||||||
self.assertEqual(["macos-container"], [a.backend_name for a in agents])
|
self.assertEqual(["macos-container"], [a.backend_name for a in agents])
|
||||||
|
|
||||||
def test_excludes_the_infra_singleton(self):
|
def test_excludes_both_infra_containers(self):
|
||||||
"""The infra container shares the bot-bottle- prefix but is
|
"""Both infra containers (orchestrator + gateway) share the bot-bottle-
|
||||||
infrastructure — listing it would invent an agent per host."""
|
prefix but are infrastructure — listing either would invent a phantom
|
||||||
agents = self._enumerate("bot-bottle-mac-infra\nbot-bottle-dev-abc\n")
|
agent per host."""
|
||||||
|
agents = self._enumerate(
|
||||||
|
"bot-bottle-mac-infra\nbot-bottle-mac-orchestrator\nbot-bottle-dev-abc\n")
|
||||||
self.assertEqual(["dev-abc"], [a.slug for a in agents])
|
self.assertEqual(["dev-abc"], [a.slug for a in agents])
|
||||||
|
|
||||||
def test_raises_when_the_cli_fails(self):
|
def test_raises_when_the_cli_fails(self):
|
||||||
|
|||||||
Reference in New Issue
Block a user