feat(firecracker): launch through the infra VM, not Docker (Stage B, 5/n)
consolidated_launch now drives the persistent infra VM instead of the `_FirecrackerOrchestratorService` Docker containers — the point Docker leaves the Firecracker launch path. - launch_consolidated: `infra_vm.ensure_running()` (singleton) for the control plane + gateway; register the bottle over HTTP at the infra VM's guest IP; provision git-gate via `SshGatewayTransport` (over SSH into the gateway VM); fetch the gateway CA via `InfraVm.gateway_ca_pem()`. - teardown_consolidated deregisters + deprovisions but does NOT stop the infra VM (persistent per-host singleton shared by every bottle). - new `infra_vm.SshGatewayTransport` + `gateway_transport()` (built from the stable key + orchestrator link IP, so teardown needs no live handle). - drop the DockerGateway/OrchestratorService machinery from the firecracker consolidated path (still used by the docker backend). launch.py already calls launch_consolidated, so the whole firecracker launch path now uses the VM. pyright + unit suite green. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WBMWTEtQdJ4W5UrWuLHCck
This commit is contained in:
@@ -1,27 +1,23 @@
|
|||||||
"""Consolidated bottle launch sequence for the Firecracker backend (PRD 0070).
|
"""Consolidated bottle launch sequence for the Firecracker backend
|
||||||
|
(PRD 0070, Stage B).
|
||||||
|
|
||||||
Mirrors bot_bottle.backend.docker.consolidated_launch but wired for
|
The shared gateway + orchestrator control plane run in a single persistent
|
||||||
Firecracker's TAP-based network topology instead of a shared Docker bridge.
|
per-host **infra VM** (`infra_vm.py`), not Docker containers. Agent VMs reach
|
||||||
|
the gateway's egress / supervise / git-http ports at the infra VM via a
|
||||||
|
PREROUTING DNAT on their own host-side TAP IP (see
|
||||||
|
`scripts/firecracker-netpool.sh`), and the host CLI reaches the control plane
|
||||||
|
over HTTP at the infra VM's guest IP.
|
||||||
|
|
||||||
The per-bottle sidecar bundle (one `docker run` per bottle, published on the
|
Attribution is by the agent VM's guest IP, unspoofable by construction: the
|
||||||
slot's host-side TAP IP) is replaced by a single persistent gateway that
|
/31 point-to-point TAP + the `bot_bottle_fc` nft table ensure only the
|
||||||
every Firecracker VM shares. The gateway runs as a Docker container in the
|
expected VM can source-IP that address.
|
||||||
dev-harness (a Firecracker VM is stage B per PRD 0070), with its ports
|
|
||||||
published on the host (`0.0.0.0:PORT`). VMs reach it at their slot's
|
|
||||||
host-side TAP IP because Docker's iptables PREROUTING DNAT redirects
|
|
||||||
port 9099/9100/9420 traffic to the gateway container — a path the nft
|
|
||||||
isolation table already allows via `ct status dnat accept` in the forward
|
|
||||||
chain.
|
|
||||||
|
|
||||||
Attribution is by the VM's guest IP, which is unspoofable by construction:
|
|
||||||
the /31 point-to-point TAP topology + the `bot_bottle_fc` nft table ensure
|
|
||||||
that only the expected VM can source-IP that address.
|
|
||||||
|
|
||||||
Sequence:
|
Sequence:
|
||||||
1. ensure the Firecracker-flavoured orchestrator + gateway are up;
|
1. ensure the infra VM (control plane + gateway) is up (a singleton — a
|
||||||
|
prior launcher may already have booted it);
|
||||||
2. register the bottle by its guest IP (attribution key) → bottle id +
|
2. register the bottle by its guest IP (attribution key) → bottle id +
|
||||||
identity token;
|
identity token;
|
||||||
3. provision its git-gate repos/creds into the running gateway;
|
3. provision its git-gate repos/creds into the gateway VM (over SSH);
|
||||||
4. fetch the shared gateway CA for the provisioner to install in the rootfs.
|
4. fetch the shared gateway CA for the provisioner to install in the rootfs.
|
||||||
|
|
||||||
The TAP slot allocation, rootfs build, and VM boot are the caller's job.
|
The TAP slot allocation, rootfs build, and VM boot are the caller's job.
|
||||||
@@ -34,34 +30,12 @@ from dataclasses import dataclass
|
|||||||
from ...egress import EgressPlan
|
from ...egress import EgressPlan
|
||||||
from ...git_gate import GitGatePlan
|
from ...git_gate import GitGatePlan
|
||||||
from ...orchestrator.client import OrchestratorClient
|
from ...orchestrator.client import OrchestratorClient
|
||||||
from ...orchestrator.gateway import (
|
|
||||||
GATEWAY_NETWORK,
|
|
||||||
DockerGateway,
|
|
||||||
)
|
|
||||||
from ...orchestrator.lifecycle import (
|
from ...orchestrator.lifecycle import (
|
||||||
OrchestratorService,
|
|
||||||
OrchestratorStartError, # re-exported so callers can catch it
|
OrchestratorStartError, # re-exported so callers can catch it
|
||||||
)
|
)
|
||||||
from ...orchestrator.registration import registration_inputs
|
from ...orchestrator.registration import registration_inputs
|
||||||
from ..docker.egress import EGRESS_PORT
|
from ..docker.gateway_provision import deprovision_git_gate, provision_git_gate
|
||||||
from ..docker.gateway_provision import (
|
from . import infra_vm
|
||||||
DockerGatewayTransport,
|
|
||||||
deprovision_git_gate,
|
|
||||||
provision_git_gate,
|
|
||||||
)
|
|
||||||
from ...supervise import SUPERVISE_PORT
|
|
||||||
|
|
||||||
_GIT_HTTP_PORT = 9420
|
|
||||||
|
|
||||||
# Separate names from the Docker gateway so both backends can coexist on one
|
|
||||||
# host (and for clarity in `docker ps` output).
|
|
||||||
_FC_GATEWAY_NAME = "bot-bottle-fc-gateway"
|
|
||||||
_FC_ORCHESTRATOR_NAME = "bot-bottle-fc-orchestrator"
|
|
||||||
_FC_ORCHESTRATOR_LABEL = "bot-bottle-fc-orchestrator=1"
|
|
||||||
# Ports the gateway publishes on the host so Firecracker VMs can reach it
|
|
||||||
# via their TAP link. Docker's PREROUTING DNAT + nft's `ct status dnat
|
|
||||||
# accept` in the forward chain route the traffic.
|
|
||||||
_FC_GATEWAY_HOST_PORTS = (EGRESS_PORT, SUPERVISE_PORT, _GIT_HTTP_PORT)
|
|
||||||
|
|
||||||
|
|
||||||
class ConsolidatedLaunchError(RuntimeError):
|
class ConsolidatedLaunchError(RuntimeError):
|
||||||
@@ -79,33 +53,6 @@ class LaunchContext:
|
|||||||
orchestrator_url: str
|
orchestrator_url: str
|
||||||
|
|
||||||
|
|
||||||
class _FirecrackerOrchestratorService(OrchestratorService):
|
|
||||||
"""Dev-harness orchestrator for the Firecracker backend.
|
|
||||||
|
|
||||||
Uses a gateway that publishes its ports on the host so Firecracker VMs can
|
|
||||||
reach it via their TAP link. The gateway and orchestrator containers use
|
|
||||||
`*-fc-*` names so both backends can run independently on the same host.
|
|
||||||
"""
|
|
||||||
|
|
||||||
def __init__(self, **kwargs: object) -> None:
|
|
||||||
super().__init__(
|
|
||||||
orchestrator_name=_FC_ORCHESTRATOR_NAME,
|
|
||||||
orchestrator_label=_FC_ORCHESTRATOR_LABEL,
|
|
||||||
**kwargs, # type: ignore[arg-type]
|
|
||||||
)
|
|
||||||
|
|
||||||
def _gateway(self) -> DockerGateway:
|
|
||||||
# The heavy data-plane image (#384 split it from the lean control-plane
|
|
||||||
# `image` this service's orchestrator container runs).
|
|
||||||
return DockerGateway(
|
|
||||||
self._gateway_image,
|
|
||||||
name=_FC_GATEWAY_NAME,
|
|
||||||
network=self.network,
|
|
||||||
orchestrator_url=self.internal_url,
|
|
||||||
host_port_bindings=_FC_GATEWAY_HOST_PORTS,
|
|
||||||
)
|
|
||||||
|
|
||||||
|
|
||||||
def launch_consolidated(
|
def launch_consolidated(
|
||||||
egress_plan: EgressPlan,
|
egress_plan: EgressPlan,
|
||||||
git_gate_plan: GitGatePlan,
|
git_gate_plan: GitGatePlan,
|
||||||
@@ -113,15 +60,12 @@ def launch_consolidated(
|
|||||||
guest_ip: str,
|
guest_ip: str,
|
||||||
image_ref: str = "",
|
image_ref: str = "",
|
||||||
tokens: dict[str, str] | None = None,
|
tokens: dict[str, str] | None = None,
|
||||||
service: OrchestratorService | None = None,
|
|
||||||
gateway_name: str = _FC_GATEWAY_NAME,
|
|
||||||
) -> LaunchContext:
|
) -> LaunchContext:
|
||||||
"""Ensure the orchestrator + Firecracker gateway are up, register the
|
"""Ensure the infra VM is up, register the bottle by its guest IP, and
|
||||||
bottle by its guest IP, and provision its git-gate state. Returns the
|
provision its git-gate state into the gateway VM. Returns the context the
|
||||||
context the VM launch needs. Raises `ConsolidatedLaunchError` (or the
|
agent-VM launch needs. Raises on failure — the caller tears down."""
|
||||||
primitives' own errors) on failure — the caller tears down on failure."""
|
infra = infra_vm.ensure_running()
|
||||||
service = service or _FirecrackerOrchestratorService()
|
url = infra.control_plane_url
|
||||||
url = service.ensure_running()
|
|
||||||
client = OrchestratorClient(url)
|
client = OrchestratorClient(url)
|
||||||
|
|
||||||
inputs = registration_inputs(egress_plan)
|
inputs = registration_inputs(egress_plan)
|
||||||
@@ -131,33 +75,29 @@ def launch_consolidated(
|
|||||||
)
|
)
|
||||||
try:
|
try:
|
||||||
provision_git_gate(
|
provision_git_gate(
|
||||||
DockerGatewayTransport(gateway_name), reg.bottle_id, git_gate_plan)
|
infra_vm.gateway_transport(), reg.bottle_id, git_gate_plan)
|
||||||
except Exception:
|
except Exception:
|
||||||
client.teardown_bottle(reg.bottle_id)
|
client.teardown_bottle(reg.bottle_id)
|
||||||
raise
|
raise
|
||||||
|
|
||||||
# Fetch the shared gateway CA here so the caller can install it in the
|
# The shared gateway CA every agent on this host trusts for TLS
|
||||||
# rootfs (the same CA every agent on this host trusts for TLS interception).
|
# interception — fetched from the infra VM over SSH.
|
||||||
gateway_ca_pem = DockerGateway(
|
|
||||||
name=gateway_name, network=GATEWAY_NETWORK,
|
|
||||||
).ca_cert_pem()
|
|
||||||
|
|
||||||
return LaunchContext(
|
return LaunchContext(
|
||||||
bottle_id=reg.bottle_id,
|
bottle_id=reg.bottle_id,
|
||||||
identity_token=reg.identity_token,
|
identity_token=reg.identity_token,
|
||||||
source_ip=guest_ip,
|
source_ip=guest_ip,
|
||||||
gateway_ca_pem=gateway_ca_pem,
|
gateway_ca_pem=infra.gateway_ca_pem(),
|
||||||
orchestrator_url=url,
|
orchestrator_url=url,
|
||||||
)
|
)
|
||||||
|
|
||||||
|
|
||||||
def teardown_consolidated(
|
def teardown_consolidated(bottle_id: str, *, orchestrator_url: str) -> None:
|
||||||
bottle_id: str, *, orchestrator_url: str, gateway_name: str = _FC_GATEWAY_NAME,
|
"""Deregister the bottle and remove its git-gate state from the gateway
|
||||||
) -> None:
|
VM. Both steps are idempotent so this is safe from a cleanup trap. Does
|
||||||
"""Deregister the bottle and remove its git-gate state from the gateway.
|
NOT stop the infra VM — it's a persistent per-host singleton shared by
|
||||||
Both steps are idempotent so this is safe from a cleanup trap."""
|
every bottle."""
|
||||||
OrchestratorClient(orchestrator_url).teardown_bottle(bottle_id)
|
OrchestratorClient(orchestrator_url).teardown_bottle(bottle_id)
|
||||||
deprovision_git_gate(DockerGatewayTransport(gateway_name), bottle_id)
|
deprovision_git_gate(infra_vm.gateway_transport(), bottle_id)
|
||||||
|
|
||||||
|
|
||||||
__all__ = [
|
__all__ = [
|
||||||
|
|||||||
@@ -18,6 +18,7 @@ surface.
|
|||||||
from __future__ import annotations
|
from __future__ import annotations
|
||||||
|
|
||||||
import os
|
import os
|
||||||
|
import shlex
|
||||||
import signal
|
import signal
|
||||||
import subprocess
|
import subprocess
|
||||||
import time
|
import time
|
||||||
@@ -28,6 +29,7 @@ from pathlib import Path
|
|||||||
|
|
||||||
from ...log import die, info
|
from ...log import die, info
|
||||||
from ..docker import util as docker_mod
|
from ..docker import util as docker_mod
|
||||||
|
from ..docker.gateway_provision import GatewayProvisionError
|
||||||
from . import firecracker_vm, netpool, util
|
from . import firecracker_vm, netpool, util
|
||||||
|
|
||||||
# The single infra-VM image: gateway data plane + baked control-plane source
|
# The single infra-VM image: gateway data plane + baked control-plane source
|
||||||
@@ -228,6 +230,42 @@ def _health_ok(url: str) -> bool:
|
|||||||
return False
|
return False
|
||||||
|
|
||||||
|
|
||||||
|
class SshGatewayTransport:
|
||||||
|
"""`GatewayTransport` for the gateway running in the infra VM — the docker
|
||||||
|
exec/cp equivalents over SSH (dropbear + the stable infra key)."""
|
||||||
|
|
||||||
|
def __init__(self, private_key: Path, guest_ip: str) -> None:
|
||||||
|
self._key = private_key
|
||||||
|
self._ip = guest_ip
|
||||||
|
|
||||||
|
def exec(self, argv: list[str]) -> None:
|
||||||
|
proc = subprocess.run(
|
||||||
|
util.ssh_base_argv(self._key, self._ip) + [shlex.join(argv)],
|
||||||
|
capture_output=True, text=True, timeout=60, check=False,
|
||||||
|
)
|
||||||
|
if proc.returncode != 0:
|
||||||
|
raise GatewayProvisionError(
|
||||||
|
f"infra gateway exec {argv!r} failed: {proc.stderr.strip()}")
|
||||||
|
|
||||||
|
def cp_into(self, src: str, dest: str) -> None:
|
||||||
|
proc = subprocess.run(
|
||||||
|
util.ssh_base_argv(self._key, self._ip) + [f"cat > {shlex.quote(dest)}"],
|
||||||
|
input=Path(src).read_bytes(), capture_output=True, timeout=30, check=False,
|
||||||
|
)
|
||||||
|
if proc.returncode != 0:
|
||||||
|
raise GatewayProvisionError(
|
||||||
|
f"infra gateway cp {src} -> {dest} failed: "
|
||||||
|
f"{proc.stderr.decode(errors='replace').strip()}")
|
||||||
|
|
||||||
|
|
||||||
|
def gateway_transport() -> SshGatewayTransport:
|
||||||
|
"""git-gate provisioning transport for the gateway in the infra VM, built
|
||||||
|
from the stable key + the orchestrator link's guest IP. Needs no live VM
|
||||||
|
handle, so teardown can use it too."""
|
||||||
|
return SshGatewayTransport(
|
||||||
|
_infra_dir() / "id_ed25519", netpool.orch_slot().guest_ip)
|
||||||
|
|
||||||
|
|
||||||
def wait_for_health(
|
def wait_for_health(
|
||||||
infra: InfraVm, *, timeout: float = _HEALTH_TIMEOUT_SECONDS,
|
infra: InfraVm, *, timeout: float = _HEALTH_TIMEOUT_SECONDS,
|
||||||
) -> None:
|
) -> None:
|
||||||
|
|||||||
Reference in New Issue
Block a user