feat(firecracker): split orchestrator and gateway into separate VMs (PRD 0070)
tracker-policy-pr / check-pr (pull_request) Successful in 12s
test / integration-docker (pull_request) Successful in 19s
lint / lint (push) Successful in 53s
test / unit (pull_request) Failing after 1m46s
test / integration-firecracker (pull_request) Failing after 2m42s
test / coverage (pull_request) Has been skipped
test / publish-infra (pull_request) Has been skipped

Now that #469 got the DB off the data plane, the Firecracker infra runs as
two microVMs instead of one — mirroring the docker/macos plane split:

  * orchestrator VM (ORCH_IFACE) — control plane + buildah image builds; sole
    DB opener; host-seeded signing key. No gateway daemons.
  * gateway VM (new GW_IFACE) — egress / git-http / supervise data plane;
    mitmproxy CA + a host-minted `gateway` JWT (never the key). Reaches the
    orchestrator only over the one nft forward rule its link allows.

Both boot the SAME shared infra rootfs; a `bb_role=` kernel-cmdline arg
selects which plane a VM's PID-1 init starts, so there is still one published
artifact. The gateway learns the orchestrator's address via `bb_orch=` on the
cmdline (no IP baked into the artifact).

Isolation is nearly free: agents were already nft-dropped except the DNAT'd
gateway ports, so re-pointing that single DNAT rule at the gateway VM
(`dnat to gw_guest`) severs every agent's L3 route to the control plane. The
only added nft is the second infra link's mirror block (masquerade egress +
forward accept, which subsumes gateway->orchestrator) in the shared shell
script and the NixOS module.

netpool gains GW_IFACE + gw_slot() (the /31 above the orch link);
firecracker_vm.boot gains extra_boot_args for the role cmdline; infra_vm
ensure_running() boots + adopts the pair (orchestrator first, then the gateway
that resolves policy against it) and returns an InfraEndpoint mirroring the
docker/macos shape. Builds stay in the orchestrator (PRD 0070 v1); the gateway
is the slim unit.

Unit-tested (test_firecracker_infra_vm rewritten for two VMs; gw_slot helper
test added); the KVM boot / L3-isolation checks are validated on a Firecracker
host.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
2026-07-25 03:44:15 -04:00
parent a74894c6f6
commit 18d9b81add
11 changed files with 819 additions and 355 deletions
@@ -60,12 +60,18 @@ class VmHandle:
self.process.wait(timeout=5)
def _boot_args(guest_ip: str, host_ip: str, pubkey: str) -> str:
def _boot_args(
guest_ip: str, host_ip: str, pubkey: str, extra: str = "",
) -> str:
# ip=<client>::<gw>:<netmask>::<dev>:off — /31 point-to-point link,
# so netmask is 255.255.255.254 and the gateway is the host TAP IP.
ip_arg = f"ip={guest_ip}::{host_ip}:255.255.255.254::eth0:off"
pub_b64 = base64.b64encode(pubkey.encode()).decode()
return f"{_BASE_BOOT_ARGS} {ip_arg} bb_pubkey={pub_b64}"
args = f"{_BASE_BOOT_ARGS} {ip_arg} bb_pubkey={pub_b64}"
# `extra` carries caller-supplied cmdline params the guest init reads
# (e.g. `bb_role=orchestrator|gateway` selecting which infra plane a
# shared-rootfs infra VM runs). Agent VMs pass nothing.
return f"{args} {extra}".rstrip() if extra else args
def _config(
@@ -79,6 +85,7 @@ def _config(
mem_mib: int,
guest_mac: str,
data_drive: Path | None = None,
extra_boot_args: str = "",
) -> dict[str, object]:
drives: list[dict[str, object]] = [
{
@@ -101,7 +108,7 @@ def _config(
return {
"boot-source": {
"kernel_image_path": str(util.kernel_path()),
"boot_args": _boot_args(guest_ip, host_ip, pubkey),
"boot_args": _boot_args(guest_ip, host_ip, pubkey, extra_boot_args),
},
"drives": drives,
"network-interfaces": [
@@ -132,6 +139,7 @@ def boot(
guest_mac: str = "06:00:AC:10:00:02",
detached: bool = False,
data_drive: Path | None = None,
extra_boot_args: str = "",
) -> VmHandle:
"""Write the config and launch the VMM. Returns once the process is
spawned; callers wait for SSH readiness separately.
@@ -147,7 +155,7 @@ def boot(
_config(
rootfs=rootfs, tap=tap, guest_ip=guest_ip, host_ip=host_ip,
pubkey=pubkey, vcpus=vcpus, mem_mib=mem_mib, guest_mac=guest_mac,
data_drive=data_drive,
data_drive=data_drive, extra_boot_args=extra_boot_args,
),
indent=2,
))