feat(firecracker): split orchestrator and gateway into separate VMs (PRD 0070)
tracker-policy-pr / check-pr (pull_request) Successful in 12s
test / integration-docker (pull_request) Successful in 19s
lint / lint (push) Successful in 53s
test / unit (pull_request) Failing after 1m46s
test / integration-firecracker (pull_request) Failing after 2m42s
test / coverage (pull_request) Has been skipped
test / publish-infra (pull_request) Has been skipped
tracker-policy-pr / check-pr (pull_request) Successful in 12s
test / integration-docker (pull_request) Successful in 19s
lint / lint (push) Successful in 53s
test / unit (pull_request) Failing after 1m46s
test / integration-firecracker (pull_request) Failing after 2m42s
test / coverage (pull_request) Has been skipped
test / publish-infra (pull_request) Has been skipped
Now that #469 got the DB off the data plane, the Firecracker infra runs as two microVMs instead of one — mirroring the docker/macos plane split: * orchestrator VM (ORCH_IFACE) — control plane + buildah image builds; sole DB opener; host-seeded signing key. No gateway daemons. * gateway VM (new GW_IFACE) — egress / git-http / supervise data plane; mitmproxy CA + a host-minted `gateway` JWT (never the key). Reaches the orchestrator only over the one nft forward rule its link allows. Both boot the SAME shared infra rootfs; a `bb_role=` kernel-cmdline arg selects which plane a VM's PID-1 init starts, so there is still one published artifact. The gateway learns the orchestrator's address via `bb_orch=` on the cmdline (no IP baked into the artifact). Isolation is nearly free: agents were already nft-dropped except the DNAT'd gateway ports, so re-pointing that single DNAT rule at the gateway VM (`dnat to gw_guest`) severs every agent's L3 route to the control plane. The only added nft is the second infra link's mirror block (masquerade egress + forward accept, which subsumes gateway->orchestrator) in the shared shell script and the NixOS module. netpool gains GW_IFACE + gw_slot() (the /31 above the orch link); firecracker_vm.boot gains extra_boot_args for the role cmdline; infra_vm ensure_running() boots + adopts the pair (orchestrator first, then the gateway that resolves policy against it) and returns an InfraEndpoint mirroring the docker/macos shape. Builds stay in the orchestrator (PRD 0070 v1); the gateway is the slim unit. Unit-tested (test_firecracker_infra_vm rewritten for two VMs; gw_slot helper test added); the KVM boot / L3-isolation checks are validated on a Firecracker host. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
@@ -1,22 +1,23 @@
|
||||
"""Consolidated bottle launch sequence for the Firecracker backend
|
||||
(PRD 0070, Stage B).
|
||||
|
||||
The shared gateway + orchestrator control plane run in a single persistent
|
||||
per-host **infra VM** (`infra_vm.py`), not Docker containers. Agent VMs reach
|
||||
the gateway's egress / supervise / git-http ports at the infra VM via a
|
||||
PREROUTING DNAT on their own host-side TAP IP (see
|
||||
`scripts/firecracker-netpool.sh`), and the host CLI reaches the control plane
|
||||
over HTTP at the infra VM's guest IP.
|
||||
The orchestrator control plane and the shared gateway run in **two** persistent
|
||||
per-host **infra VMs** (`infra_vm.py`), split per PRD 0070 — not Docker
|
||||
containers. Agent VMs reach the gateway's egress / supervise / git-http ports
|
||||
via a PREROUTING DNAT on their own host-side TAP IP that redirects to the
|
||||
**gateway** VM (see `scripts/firecracker-netpool.sh`) — never the orchestrator,
|
||||
so a breached agent has no L3 route to the control plane. The host CLI reaches
|
||||
the control plane over HTTP at the orchestrator VM's guest IP.
|
||||
|
||||
Attribution is by the agent VM's guest IP, unspoofable by construction: the
|
||||
/31 point-to-point TAP + the `bot_bottle_fc` nft table ensure only the
|
||||
expected VM can source-IP that address.
|
||||
|
||||
Sequence:
|
||||
1. ensure the infra VM (control plane + gateway) is up (a singleton — a
|
||||
prior launcher may already have booted it);
|
||||
2. register the bottle by its guest IP (attribution key) → bottle id +
|
||||
identity token;
|
||||
1. ensure the infra VMs (orchestrator control plane + gateway data plane)
|
||||
are up (a singleton pair — a prior launcher may already have booted them);
|
||||
2. register the bottle on the orchestrator by its guest IP (attribution key)
|
||||
→ bottle id + identity token;
|
||||
3. provision its git-gate repos/creds into the gateway VM (over SSH);
|
||||
4. fetch the shared gateway CA for the provisioner to install in the rootfs.
|
||||
|
||||
@@ -150,7 +151,7 @@ def teardown_consolidated(
|
||||
) -> None:
|
||||
"""Deregister the bottle and remove its git-gate state from the gateway
|
||||
VM. Both steps are idempotent so this is safe from a cleanup trap. Does
|
||||
NOT stop the infra VM — it's a persistent per-host singleton shared by
|
||||
NOT stop the infra VMs — they're persistent per-host singletons shared by
|
||||
every bottle."""
|
||||
_teardown_util(bottle_id, infra_vm.gateway_transport(),
|
||||
orchestrator_url=orchestrator_url, timeout=timeout)
|
||||
|
||||
Reference in New Issue
Block a user