Now that #469 got the DB off the data plane, the Firecracker infra runs as
two microVMs instead of one — mirroring the docker/macos plane split:
* orchestrator VM (ORCH_IFACE) — control plane + buildah image builds; sole
DB opener; host-seeded signing key. No gateway daemons.
* gateway VM (new GW_IFACE) — egress / git-http / supervise data plane;
mitmproxy CA + a host-minted `gateway` JWT (never the key). Reaches the
orchestrator only over the one nft forward rule its link allows.
Both boot the SAME shared infra rootfs; a `bb_role=` kernel-cmdline arg
selects which plane a VM's PID-1 init starts, so there is still one published
artifact. The gateway learns the orchestrator's address via `bb_orch=` on the
cmdline (no IP baked into the artifact).
Isolation is nearly free: agents were already nft-dropped except the DNAT'd
gateway ports, so re-pointing that single DNAT rule at the gateway VM
(`dnat to gw_guest`) severs every agent's L3 route to the control plane. The
only added nft is the second infra link's mirror block (masquerade egress +
forward accept, which subsumes gateway->orchestrator) in the shared shell
script and the NixOS module.
netpool gains GW_IFACE + gw_slot() (the /31 above the orch link);
firecracker_vm.boot gains extra_boot_args for the role cmdline; infra_vm
ensure_running() boots + adopts the pair (orchestrator first, then the gateway
that resolves policy against it) and returns an InfraEndpoint mirroring the
docker/macos shape. Builds stay in the orchestrator (PRD 0070 v1); the gateway
is the slim unit.
Unit-tested (test_firecracker_infra_vm rewritten for two VMs; gw_slot helper
test added); the KVM boot / L3-isolation checks are validated on a Firecracker
host.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The /31 point-to-point TAP does NOT make a guest source address
unspoofable: root in an agent VM can source another bottle's guest IP on
its own bbfc TAP. The isolation table only matched iifname class + port
(DNAT) and never bound iifname to its assigned ip saddr — so a spoofed
source was DNAT'd to the gateway and attributed to the *victim* bottle,
getting the victim's policy/tokens. Source-IP attribution was therefore
not actually sound.
Add one anti-spoof rule per slot in the isolation forward chain, before
the established/DNAT accepts: `iifname bbfcN ip saddr != <guestN> drop`.
Generated in the existing setup loop — no new dependency, ~pool_size
lines. Legit traffic (correct saddr) is unchanged; a spoofed saddr on any
bbfc TAP is dropped before it can be attributed.
Apply with a nixos-rebuild (the systemd unit re-runs this script). Codex
review blocker; the app-layer identity token (defense-in-depth) is wired
separately.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WBMWTEtQdJ4W5UrWuLHCck
Each nft table (isolation, orchestrator-egress nat, agent->gateway route)
was re-applied as a plain `table {...}` block, which on a bare `up`
re-apply (not the systemd down->up path) would append duplicate rules or
error on the existing base chains. Use the standard delete-first pattern
(create empty, delete, recreate) so `up` lands identical state regardless
of history — the setup reproduces cleanly on a fresh install and on
re-apply, not just via a full down->up cycle.
No functional change to the resulting ruleset; only its idempotency.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WBMWTEtQdJ4W5UrWuLHCck
Agent VMs must reach the shared gateway that now runs in the infra VM
(egress:9099 / supervise:9100 / git-http:9420 at the orchestrator link's
guest IP). Add a PREROUTING DNAT: agents keep addressing their own
host-side TAP IP on the gateway ports, and the rule redirects that to the
infra VM. The isolation table's existing `ct status dnat accept` forward
rule lets the DNAT'd traffic through; every other agent egress stays
dropped, so a bottle still reaches only the gateway and nothing else.
Source IP is deliberately NOT masqueraded: the gateway attributes each
request to the originating bottle by its guest IP, which the /31 TAP + the
bot_bottle_fc nft table make unspoofable. Keeping the agent addressed at
its own host TAP IP means no per-bottle config change vs the docker-DNAT
path it replaces.
- scripts/firecracker-netpool.sh: `_install_gateway_route` adds
`table ip <table>_gw` (prerouting dstnat -> orch_guest on the gateway
ports); wired into up/down/status. The nix module needs no change — it
runs this script, and the ports are baked in.
Verified on a KVM host: an agent VM's `curl -x http://<its-host-tap>:9099`
reaches mitmproxy in the infra VM and gets a 403 (correct policy denial
for an unregistered bottle) — i.e. the route lands end-to-end. Persist
with a nixos-rebuild; the imperative rule holds until then.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WBMWTEtQdJ4W5UrWuLHCck
The orchestrator/gateway VM is trusted infra, not an isolated agent: it
builds agent images in-VM (buildah must FROM-pull + apt/npm) and, in the
Stage B cutover, forwards agent egress upstream. Give it a dedicated TAP
(`bborch0`) on a /31 at the top of the IP_BASE /16 (clear of the bbfc*
agent pool at the bottom), NAT'd out the host uplink — while agent VMs
keep their fail-closed, gateway-only isolation table.
- netpool.defaults.env / netpool.py: new BOT_BOTTLE_FC_ORCH_IFACE +
`orch_slot()` (index -1 sentinel; host x.y.255.0 / guest x.y.255.1).
- scripts/firecracker-netpool.sh: create + address the orchestrator TAP;
`bot_bottle_fc_nat` table masquerades its /31 out the uplink and
accepts its forward path. Because bootstrap still runs Docker (whose
FORWARD policy is DROP), a best-effort, guarded, idempotent DOCKER-USER
ACCEPT is added too (skipped once Docker is gone). down/status updated.
- nix/firecracker-netpool.nix: mirror the option, pass it via the unit
Environment= (the store-copied script can't read the defaults file),
and add iptables to the unit path for the DOCKER-USER step.
Agent isolation is unchanged: the new rules only ever accept/masquerade
the orchestrator link and never drop, so they can't weaken the bbfc*
drops. Applied by re-running `sudo ./scripts/firecracker-netpool.sh up`.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WBMWTEtQdJ4W5UrWuLHCck
Completes the de-sidecar cleanup: no live code, test, current doc,
script, or nix file mentions or is named "sidecar" any more. Only the
dated PRD/research docs keep the term as historical record (agreed on
the #385 thread).
- Rename `sidecar_init.py`→`gateway_init.py` was done earlier; this pass
sweeps the remaining descriptive uses: the egress / git-gate / supervise
components are the gateway's *daemons*, the shared container is the
*gateway*, the old per-bottle container was the *companion container*.
- Rename `tests/integration/test_sidecar_bundle_image.py`→`test_gateway_image.py`
and its class; update `docs/ci.md` + `tests/README.md` for the renamed/
removed integration tests.
- `SIDECAR_PORTS` shell var in `scripts/firecracker-netpool.sh`→`GATEWAY_PORTS`.
Full unit suite green (bar the pre-existing `/bin/sleep`-missing env
errors in test_gateway_init); docker integration — gateway singleton,
broker, real two-bottle multitenant isolation, and the gateway-image
build — all pass.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WBMWTEtQdJ4W5UrWuLHCck
The pool params (size, IP base, iface prefix, nft table) were triplicated
— hardcoded in netpool.py, scripts/firecracker-netpool.sh, and
nix/firecracker-netpool.nix — plus the IP math (3x) and the nft ruleset
(2x). Nothing enforced agreement; changing the base (ce3fad9, off CGNAT)
forced a coordinated three-file edit, and a missed one would silently
provision a range the launcher doesn't expect.
Collapse to one source of truth:
* netpool.defaults.env — a plain KEY=VALUE file (bash-sourceable,
systemd EnvironmentFile-compatible, Python- and Nix-parseable) holding
the four defaults. A BOT_BOTTLE_FC_* env var still overrides any key.
* netpool.py reads it for the Python defaults (missing file = hard
error, not confusing empty defaults).
* the shell script falls back to it (no literal `:-8` / `10.243.0.0`),
and its `up` is now non-destructive/idempotent (only creates a
missing TAP), so re-running never cuts a live VM.
* the Nix module readFile-parses it for its option defaults and
delegates bring-up to the SAME shell script (dropping its duplicate
IP math, nft ruleset, and TAP loop) — passing every value as
Environment= so the store-detached script never needs the file.
Net: defaults 3x -> 1x, nft ruleset 2x -> 1x, TAP loop 2x -> 1x. The one
remaining IP-math dup (Python launch-addressing vs bash bring-up) is
justified — different runtimes. Tests now guard the invariant (Python
reads the shared file; the script/module hold no literals) instead of
pinning duplicated strings.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WBMWTEtQdJ4W5UrWuLHCck
Add a `group` option to the netpool NixOS module + BOT_BOTTLE_FC_GROUP to
the shell script: when set, the pool's TAP devices are owned by a group
instead of a single user, so any group member can open them (the kernel
lets a TAP's owning-group members attach). This lets an interactive user
and, say, a CI-runner user share one pool. `owner`/`group` are mutually
exclusive (asserted). Single-user `owner` remains the default.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WBMWTEtQdJ4W5UrWuLHCck
The default TAP-pool base was 100.64.0.0/10 (RFC-6598 CGNAT) — chosen to
dodge RFC-1918, but that's exactly the range Tailscale assigns node
addresses from, so on a Tailscale host it's the worst pick. Move the
default to 10.243.0.0/16, an obscure RFC-1918 block that steers clear of
docker/libvirt/k8s/LAN and Tailscale.
No default is collision-proof, so add netpool.overlapping_routes(): it
parses `ip -json route show table all` and flags any route intersecting
the pool range (excluding our own bbfc* TAPs and the default route). The
launch preflight warns on overlap; `backend status` reports it.
Distribute the NixOS host setup as a flake module instead of a
copy-pasted blob: nix/firecracker-netpool.nix computes the taps / nft
table from typed options (poolSize, ipBase, ifacePrefix, owner) with a
/31-alignment assertion, and flake.nix exposes it as
nixosModules.firecracker-netpool. Defaults mirror the backend constants;
writeEnvFile emits the matching BOT_BOTTLE_FC_* so the host pool and the
launcher can't drift.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WBMWTEtQdJ4W5UrWuLHCck
Adds a Firecracker-based backend for Linux, providing mature KVM-based
microVM isolation to replace smolmachines/libkrun (issue #342, closes
the dead-end tracked in #332).
Architecture:
- Guest control over SSH (dropbear injected into the rootfs) on a
point-to-point TAP link. `ssh -t` forwards SIGWINCH natively, so no
resize bridge is needed.
- Networking: a one-time, root-provisioned pool of user-owned TAP
devices (no shared bridge → no docker0/virbr0/cni0 collisions) plus a
dedicated `table inet bot_bottle_fc` nftables table (independent of
Docker/ufw/firewalld rules). `./cli.py firecracker setup` prints the
host-appropriate config (NixOS module or sudo script).
- Rootfs: `docker export` → ext4 via `mke2fs -d` (rootless, no mount),
cached by image digest; per-bottle SSH pubkey + IP passed via the
kernel cmdline.
- Sidecar: reuses the Docker bundle, published on the slot's host TAP IP.
- Fail-closed isolation: TAP pool verified at preflight; the egress
boundary is proven empirically post-boot (before the agent runs) by a
canary probe — the VM must fail to reach the host directly, or launch
is refused.
Linux hosts with Firecracker + KVM now default to this backend;
macOS stays on macos-container.
Not yet validated end-to-end on live hardware (requires the one-time
network pool). Unit tests + pyright pass.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01G8p32HJgPoS1hLPWubbftM
didericis