feat(firecracker): launch through the infra VM, not Docker (Stage B, 5/n)

consolidated_launch now drives the persistent infra VM instead of the
`_FirecrackerOrchestratorService` Docker containers — the point Docker
leaves the Firecracker launch path.

- launch_consolidated: `infra_vm.ensure_running()` (singleton) for the
  control plane + gateway; register the bottle over HTTP at the infra VM's
  guest IP; provision git-gate via `SshGatewayTransport` (over SSH into the
  gateway VM); fetch the gateway CA via `InfraVm.gateway_ca_pem()`.
- teardown_consolidated deregisters + deprovisions but does NOT stop the
  infra VM (persistent per-host singleton shared by every bottle).
- new `infra_vm.SshGatewayTransport` + `gateway_transport()` (built from the
  stable key + orchestrator link IP, so teardown needs no live handle).
- drop the DockerGateway/OrchestratorService machinery from the firecracker
  consolidated path (still used by the docker backend).

launch.py already calls launch_consolidated, so the whole firecracker
launch path now uses the VM. pyright + unit suite green.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WBMWTEtQdJ4W5UrWuLHCck
This commit is contained in:
2026-07-16 13:12:48 -04:00
parent 5dfb9b0d75
commit 957eb19368
2 changed files with 69 additions and 91 deletions
@@ -1,27 +1,23 @@
"""Consolidated bottle launch sequence for the Firecracker backend (PRD 0070). """Consolidated bottle launch sequence for the Firecracker backend
(PRD 0070, Stage B).
Mirrors bot_bottle.backend.docker.consolidated_launch but wired for The shared gateway + orchestrator control plane run in a single persistent
Firecracker's TAP-based network topology instead of a shared Docker bridge. per-host **infra VM** (`infra_vm.py`), not Docker containers. Agent VMs reach
the gateway's egress / supervise / git-http ports at the infra VM via a
PREROUTING DNAT on their own host-side TAP IP (see
`scripts/firecracker-netpool.sh`), and the host CLI reaches the control plane
over HTTP at the infra VM's guest IP.
The per-bottle sidecar bundle (one `docker run` per bottle, published on the Attribution is by the agent VM's guest IP, unspoofable by construction: the
slot's host-side TAP IP) is replaced by a single persistent gateway that /31 point-to-point TAP + the `bot_bottle_fc` nft table ensure only the
every Firecracker VM shares. The gateway runs as a Docker container in the expected VM can source-IP that address.
dev-harness (a Firecracker VM is stage B per PRD 0070), with its ports
published on the host (`0.0.0.0:PORT`). VMs reach it at their slot's
host-side TAP IP because Docker's iptables PREROUTING DNAT redirects
port 9099/9100/9420 traffic to the gateway container — a path the nft
isolation table already allows via `ct status dnat accept` in the forward
chain.
Attribution is by the VM's guest IP, which is unspoofable by construction:
the /31 point-to-point TAP topology + the `bot_bottle_fc` nft table ensure
that only the expected VM can source-IP that address.
Sequence: Sequence:
1. ensure the Firecracker-flavoured orchestrator + gateway are up; 1. ensure the infra VM (control plane + gateway) is up (a singleton — a
prior launcher may already have booted it);
2. register the bottle by its guest IP (attribution key) → bottle id + 2. register the bottle by its guest IP (attribution key) → bottle id +
identity token; identity token;
3. provision its git-gate repos/creds into the running gateway; 3. provision its git-gate repos/creds into the gateway VM (over SSH);
4. fetch the shared gateway CA for the provisioner to install in the rootfs. 4. fetch the shared gateway CA for the provisioner to install in the rootfs.
The TAP slot allocation, rootfs build, and VM boot are the caller's job. The TAP slot allocation, rootfs build, and VM boot are the caller's job.
@@ -34,34 +30,12 @@ from dataclasses import dataclass
from ...egress import EgressPlan from ...egress import EgressPlan
from ...git_gate import GitGatePlan from ...git_gate import GitGatePlan
from ...orchestrator.client import OrchestratorClient from ...orchestrator.client import OrchestratorClient
from ...orchestrator.gateway import (
GATEWAY_NETWORK,
DockerGateway,
)
from ...orchestrator.lifecycle import ( from ...orchestrator.lifecycle import (
OrchestratorService,
OrchestratorStartError, # re-exported so callers can catch it OrchestratorStartError, # re-exported so callers can catch it
) )
from ...orchestrator.registration import registration_inputs from ...orchestrator.registration import registration_inputs
from ..docker.egress import EGRESS_PORT from ..docker.gateway_provision import deprovision_git_gate, provision_git_gate
from ..docker.gateway_provision import ( from . import infra_vm
DockerGatewayTransport,
deprovision_git_gate,
provision_git_gate,
)
from ...supervise import SUPERVISE_PORT
_GIT_HTTP_PORT = 9420
# Separate names from the Docker gateway so both backends can coexist on one
# host (and for clarity in `docker ps` output).
_FC_GATEWAY_NAME = "bot-bottle-fc-gateway"
_FC_ORCHESTRATOR_NAME = "bot-bottle-fc-orchestrator"
_FC_ORCHESTRATOR_LABEL = "bot-bottle-fc-orchestrator=1"
# Ports the gateway publishes on the host so Firecracker VMs can reach it
# via their TAP link. Docker's PREROUTING DNAT + nft's `ct status dnat
# accept` in the forward chain route the traffic.
_FC_GATEWAY_HOST_PORTS = (EGRESS_PORT, SUPERVISE_PORT, _GIT_HTTP_PORT)
class ConsolidatedLaunchError(RuntimeError): class ConsolidatedLaunchError(RuntimeError):
@@ -79,33 +53,6 @@ class LaunchContext:
orchestrator_url: str orchestrator_url: str
class _FirecrackerOrchestratorService(OrchestratorService):
"""Dev-harness orchestrator for the Firecracker backend.
Uses a gateway that publishes its ports on the host so Firecracker VMs can
reach it via their TAP link. The gateway and orchestrator containers use
`*-fc-*` names so both backends can run independently on the same host.
"""
def __init__(self, **kwargs: object) -> None:
super().__init__(
orchestrator_name=_FC_ORCHESTRATOR_NAME,
orchestrator_label=_FC_ORCHESTRATOR_LABEL,
**kwargs, # type: ignore[arg-type]
)
def _gateway(self) -> DockerGateway:
# The heavy data-plane image (#384 split it from the lean control-plane
# `image` this service's orchestrator container runs).
return DockerGateway(
self._gateway_image,
name=_FC_GATEWAY_NAME,
network=self.network,
orchestrator_url=self.internal_url,
host_port_bindings=_FC_GATEWAY_HOST_PORTS,
)
def launch_consolidated( def launch_consolidated(
egress_plan: EgressPlan, egress_plan: EgressPlan,
git_gate_plan: GitGatePlan, git_gate_plan: GitGatePlan,
@@ -113,15 +60,12 @@ def launch_consolidated(
guest_ip: str, guest_ip: str,
image_ref: str = "", image_ref: str = "",
tokens: dict[str, str] | None = None, tokens: dict[str, str] | None = None,
service: OrchestratorService | None = None,
gateway_name: str = _FC_GATEWAY_NAME,
) -> LaunchContext: ) -> LaunchContext:
"""Ensure the orchestrator + Firecracker gateway are up, register the """Ensure the infra VM is up, register the bottle by its guest IP, and
bottle by its guest IP, and provision its git-gate state. Returns the provision its git-gate state into the gateway VM. Returns the context the
context the VM launch needs. Raises `ConsolidatedLaunchError` (or the agent-VM launch needs. Raises on failure — the caller tears down."""
primitives' own errors) on failure — the caller tears down on failure.""" infra = infra_vm.ensure_running()
service = service or _FirecrackerOrchestratorService() url = infra.control_plane_url
url = service.ensure_running()
client = OrchestratorClient(url) client = OrchestratorClient(url)
inputs = registration_inputs(egress_plan) inputs = registration_inputs(egress_plan)
@@ -131,33 +75,29 @@ def launch_consolidated(
) )
try: try:
provision_git_gate( provision_git_gate(
DockerGatewayTransport(gateway_name), reg.bottle_id, git_gate_plan) infra_vm.gateway_transport(), reg.bottle_id, git_gate_plan)
except Exception: except Exception:
client.teardown_bottle(reg.bottle_id) client.teardown_bottle(reg.bottle_id)
raise raise
# Fetch the shared gateway CA here so the caller can install it in the # The shared gateway CA every agent on this host trusts for TLS
# rootfs (the same CA every agent on this host trusts for TLS interception). # interception — fetched from the infra VM over SSH.
gateway_ca_pem = DockerGateway(
name=gateway_name, network=GATEWAY_NETWORK,
).ca_cert_pem()
return LaunchContext( return LaunchContext(
bottle_id=reg.bottle_id, bottle_id=reg.bottle_id,
identity_token=reg.identity_token, identity_token=reg.identity_token,
source_ip=guest_ip, source_ip=guest_ip,
gateway_ca_pem=gateway_ca_pem, gateway_ca_pem=infra.gateway_ca_pem(),
orchestrator_url=url, orchestrator_url=url,
) )
def teardown_consolidated( def teardown_consolidated(bottle_id: str, *, orchestrator_url: str) -> None:
bottle_id: str, *, orchestrator_url: str, gateway_name: str = _FC_GATEWAY_NAME, """Deregister the bottle and remove its git-gate state from the gateway
) -> None: VM. Both steps are idempotent so this is safe from a cleanup trap. Does
"""Deregister the bottle and remove its git-gate state from the gateway. NOT stop the infra VM — it's a persistent per-host singleton shared by
Both steps are idempotent so this is safe from a cleanup trap.""" every bottle."""
OrchestratorClient(orchestrator_url).teardown_bottle(bottle_id) OrchestratorClient(orchestrator_url).teardown_bottle(bottle_id)
deprovision_git_gate(DockerGatewayTransport(gateway_name), bottle_id) deprovision_git_gate(infra_vm.gateway_transport(), bottle_id)
__all__ = [ __all__ = [
@@ -18,6 +18,7 @@ surface.
from __future__ import annotations from __future__ import annotations
import os import os
import shlex
import signal import signal
import subprocess import subprocess
import time import time
@@ -28,6 +29,7 @@ from pathlib import Path
from ...log import die, info from ...log import die, info
from ..docker import util as docker_mod from ..docker import util as docker_mod
from ..docker.gateway_provision import GatewayProvisionError
from . import firecracker_vm, netpool, util from . import firecracker_vm, netpool, util
# The single infra-VM image: gateway data plane + baked control-plane source # The single infra-VM image: gateway data plane + baked control-plane source
@@ -228,6 +230,42 @@ def _health_ok(url: str) -> bool:
return False return False
class SshGatewayTransport:
"""`GatewayTransport` for the gateway running in the infra VM — the docker
exec/cp equivalents over SSH (dropbear + the stable infra key)."""
def __init__(self, private_key: Path, guest_ip: str) -> None:
self._key = private_key
self._ip = guest_ip
def exec(self, argv: list[str]) -> None:
proc = subprocess.run(
util.ssh_base_argv(self._key, self._ip) + [shlex.join(argv)],
capture_output=True, text=True, timeout=60, check=False,
)
if proc.returncode != 0:
raise GatewayProvisionError(
f"infra gateway exec {argv!r} failed: {proc.stderr.strip()}")
def cp_into(self, src: str, dest: str) -> None:
proc = subprocess.run(
util.ssh_base_argv(self._key, self._ip) + [f"cat > {shlex.quote(dest)}"],
input=Path(src).read_bytes(), capture_output=True, timeout=30, check=False,
)
if proc.returncode != 0:
raise GatewayProvisionError(
f"infra gateway cp {src} -> {dest} failed: "
f"{proc.stderr.decode(errors='replace').strip()}")
def gateway_transport() -> SshGatewayTransport:
"""git-gate provisioning transport for the gateway in the infra VM, built
from the stable key + the orchestrator link's guest IP. Needs no live VM
handle, so teardown can use it too."""
return SshGatewayTransport(
_infra_dir() / "id_ed25519", netpool.orch_slot().guest_ip)
def wait_for_health( def wait_for_health(
infra: InfraVm, *, timeout: float = _HEALTH_TIMEOUT_SECONDS, infra: InfraVm, *, timeout: float = _HEALTH_TIMEOUT_SECONDS,
) -> None: ) -> None: