feat(firecracker): split orchestrator and gateway into separate VMs (PRD 0070)
tracker-policy-pr / check-pr (pull_request) Successful in 12s
test / integration-docker (pull_request) Successful in 19s
lint / lint (push) Successful in 53s
test / unit (pull_request) Failing after 1m46s
test / integration-firecracker (pull_request) Failing after 2m42s
test / coverage (pull_request) Has been skipped
test / publish-infra (pull_request) Has been skipped

Now that #469 got the DB off the data plane, the Firecracker infra runs as
two microVMs instead of one — mirroring the docker/macos plane split:

  * orchestrator VM (ORCH_IFACE) — control plane + buildah image builds; sole
    DB opener; host-seeded signing key. No gateway daemons.
  * gateway VM (new GW_IFACE) — egress / git-http / supervise data plane;
    mitmproxy CA + a host-minted `gateway` JWT (never the key). Reaches the
    orchestrator only over the one nft forward rule its link allows.

Both boot the SAME shared infra rootfs; a `bb_role=` kernel-cmdline arg
selects which plane a VM's PID-1 init starts, so there is still one published
artifact. The gateway learns the orchestrator's address via `bb_orch=` on the
cmdline (no IP baked into the artifact).

Isolation is nearly free: agents were already nft-dropped except the DNAT'd
gateway ports, so re-pointing that single DNAT rule at the gateway VM
(`dnat to gw_guest`) severs every agent's L3 route to the control plane. The
only added nft is the second infra link's mirror block (masquerade egress +
forward accept, which subsumes gateway->orchestrator) in the shared shell
script and the NixOS module.

netpool gains GW_IFACE + gw_slot() (the /31 above the orch link);
firecracker_vm.boot gains extra_boot_args for the role cmdline; infra_vm
ensure_running() boots + adopts the pair (orchestrator first, then the gateway
that resolves policy against it) and returns an InfraEndpoint mirroring the
docker/macos shape. Builds stay in the orchestrator (PRD 0070 v1); the gateway
is the slim unit.

Unit-tested (test_firecracker_infra_vm rewritten for two VMs; gw_slot helper
test added); the KVM boot / L3-isolation checks are validated on a Firecracker
host.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
2026-07-25 03:44:15 -04:00
parent a74894c6f6
commit 18d9b81add
11 changed files with 819 additions and 355 deletions
+20 -5
View File
@@ -67,6 +67,7 @@ let
BOT_BOTTLE_FC_IFACE_PREFIX = cfg.ifacePrefix;
BOT_BOTTLE_FC_NFT_TABLE = cfg.tableName;
BOT_BOTTLE_FC_ORCH_IFACE = cfg.orchIface;
BOT_BOTTLE_FC_GW_IFACE = cfg.gwIface;
} // ownEnv;
in
{
@@ -135,11 +136,24 @@ in
default = defaults.BOT_BOTTLE_FC_ORCH_IFACE;
defaultText = lib.literalMD "the shared `netpool.defaults.env` value";
description = ''
TAP name for the orchestrator/gateway VM's dedicated link. Unlike
the isolated bbfc* agent pool, this link is NAT'd to the internet
(the orchestrator is trusted infra that builds agent images in-VM
and forwards agent egress upstream). Must match
BOT_BOTTLE_FC_ORCH_IFACE / netpool.ORCH_IFACE.
TAP name for the orchestrator VM's dedicated link. Unlike the
isolated bbfc* agent pool, this link is NAT'd to the internet
(the orchestrator is trusted infra that builds agent images
in-VM). Must match BOT_BOTTLE_FC_ORCH_IFACE / netpool.ORCH_IFACE.
'';
};
gwIface = lib.mkOption {
type = lib.types.str;
default = defaults.BOT_BOTTLE_FC_GW_IFACE;
defaultText = lib.literalMD "the shared `netpool.defaults.env` value";
description = ''
TAP name for the gateway (data-plane) VM's dedicated link the
second infra link, split from the orchestrator per PRD 0070. Like
the orchestrator link it is NAT'd to the internet (the gateway
forwards agent egress upstream); agents DNAT here, never to the
orchestrator, so a breached agent has no L3 route to the control
plane. Must match BOT_BOTTLE_FC_GW_IFACE / netpool.GW_IFACE.
'';
};
@@ -193,6 +207,7 @@ in
BOT_BOTTLE_FC_IFACE_PREFIX=${cfg.ifacePrefix}
BOT_BOTTLE_FC_NFT_TABLE=${cfg.tableName}
BOT_BOTTLE_FC_ORCH_IFACE=${cfg.orchIface}
BOT_BOTTLE_FC_GW_IFACE=${cfg.gwIface}
'';
};
};