didericis
3cde6eb3da
The pool params (size, IP base, iface prefix, nft table) were triplicated
— hardcoded in netpool.py, scripts/firecracker-netpool.sh, and
nix/firecracker-netpool.nix — plus the IP math (3x) and the nft ruleset
(2x). Nothing enforced agreement; changing the base (ce3fad9, off CGNAT)
forced a coordinated three-file edit, and a missed one would silently
provision a range the launcher doesn't expect.
Collapse to one source of truth:
* netpool.defaults.env — a plain KEY=VALUE file (bash-sourceable,
systemd EnvironmentFile-compatible, Python- and Nix-parseable) holding
the four defaults. A BOT_BOTTLE_FC_* env var still overrides any key.
* netpool.py reads it for the Python defaults (missing file = hard
error, not confusing empty defaults).
* the shell script falls back to it (no literal `:-8` / `10.243.0.0`),
and its `up` is now non-destructive/idempotent (only creates a
missing TAP), so re-running never cuts a live VM.
* the Nix module readFile-parses it for its option defaults and
delegates bring-up to the SAME shell script (dropping its duplicate
IP math, nft ruleset, and TAP loop) — passing every value as
Environment= so the store-detached script never needs the file.
Net: defaults 3x -> 1x, nft ruleset 2x -> 1x, TAP loop 2x -> 1x. The one
remaining IP-math dup (Python launch-addressing vs bash bring-up) is
justified — different runtimes. Tests now guard the invariant (Python
reads the shared file; the script/module hold no literals) instead of
pinning duplicated strings.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WBMWTEtQdJ4W5UrWuLHCck
196 lines
7.9 KiB
Bash
Executable File
196 lines
7.9 KiB
Bash
Executable File
#!/usr/bin/env bash
|
|
# One-time privileged network setup for the Firecracker backend.
|
|
#
|
|
# Creates a pool of point-to-point TAP devices (owned by the invoking
|
|
# user so the backend can open them without root at launch) and a
|
|
# dedicated nftables table that isolates every VM: a bottle VM can
|
|
# reach only its own sidecar (published on the host-side TAP IP) and
|
|
# nothing else on the host or network.
|
|
#
|
|
# Why a pool + one-time setup: creating a TAP and assigning it an IP
|
|
# needs CAP_NET_ADMIN. Pre-creating user-owned, pre-addressed TAPs
|
|
# means `./cli.py start` never needs root. The nft table is static
|
|
# (keyed on the `bbfc*` interface wildcard), so it covers every slot
|
|
# without per-launch changes.
|
|
#
|
|
# Design notes:
|
|
# * No shared bridge — each slot is an isolated /31 host<->guest link,
|
|
# so there are no bridge name/subnet collisions with docker0,
|
|
# virbr0 (libvirt), cni0 (k8s) or br-* (docker networks).
|
|
# * Own nftables table `bot_bottle_fc` — independent of the iptables
|
|
# filter/nat tables Docker/ufw/firewalld use, so nothing is stomped.
|
|
# * Default IP block is an obscure RFC-1918 /16 (10.243.0.0/16),
|
|
# chosen to dodge the usual occupants (docker 172.17-31, libvirt
|
|
# 192.168.122, k8s 10.42/10.244, home LANs). NOT 100.64.0.0/10 —
|
|
# that's RFC-6598 CGNAT, which Tailscale hands node addresses from.
|
|
#
|
|
# NixOS: imperative rules here do NOT survive nixos-rebuild. Use the
|
|
# declarative module in nix/firecracker-netpool.nix instead (exposed as
|
|
# the flake output nixosModules.firecracker-netpool).
|
|
#
|
|
# Usage:
|
|
# sudo ./scripts/firecracker-netpool.sh up
|
|
# sudo ./scripts/firecracker-netpool.sh down
|
|
# ./scripts/firecracker-netpool.sh status
|
|
#
|
|
# Pool params default to the shared single-source file
|
|
# (bot_bottle/backend/firecracker/netpool.defaults.env); a matching
|
|
# env var overrides its key:
|
|
# BOT_BOTTLE_FC_POOL_SIZE number of slots
|
|
# BOT_BOTTLE_FC_IP_BASE base IPv4 of the /31 pool
|
|
# BOT_BOTTLE_FC_IFACE_PREFIX TAP name prefix
|
|
# BOT_BOTTLE_FC_NFT_TABLE isolation table name
|
|
# BOT_BOTTLE_FC_OWNER owning user (default $SUDO_USER or $USER)
|
|
# BOT_BOTTLE_FC_GROUP owning group; if set, TAPs are group-owned
|
|
# instead of user-owned, so any group member
|
|
# (e.g. an interactive user + a CI runner
|
|
# user) can open the pool. Overrides OWNER.
|
|
|
|
set -euo pipefail
|
|
|
|
# The single source of the pool defaults, shared with netpool.py and the
|
|
# Nix module. The Nix path passes every value as env (the script runs
|
|
# from the store, detached from this file), so this lookup only matters
|
|
# on the direct/sudo path where the script sits in the repo tree.
|
|
_SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
|
_DEFAULTS="$_SCRIPT_DIR/../bot_bottle/backend/firecracker/netpool.defaults.env"
|
|
_default() { # value of KEY=... from the shared file; empty if unavailable
|
|
[ -f "$_DEFAULTS" ] || return 0
|
|
sed -n "s/^$1=//p" "$_DEFAULTS" | tail -1
|
|
}
|
|
|
|
POOL_SIZE="${BOT_BOTTLE_FC_POOL_SIZE:-$(_default BOT_BOTTLE_FC_POOL_SIZE)}"
|
|
IP_BASE="${BOT_BOTTLE_FC_IP_BASE:-$(_default BOT_BOTTLE_FC_IP_BASE)}"
|
|
PREFIX="${BOT_BOTTLE_FC_IFACE_PREFIX:-$(_default BOT_BOTTLE_FC_IFACE_PREFIX)}"
|
|
TABLE="${BOT_BOTTLE_FC_NFT_TABLE:-$(_default BOT_BOTTLE_FC_NFT_TABLE)}"
|
|
OWNER="${BOT_BOTTLE_FC_OWNER:-${SUDO_USER:-$USER}}"
|
|
GROUP="${BOT_BOTTLE_FC_GROUP:-}"
|
|
|
|
# Fail loudly rather than provisioning a half/empty range if a value
|
|
# resolved to nothing (env unset AND the shared file unreadable).
|
|
for _v in POOL_SIZE IP_BASE PREFIX TABLE; do
|
|
[ -n "${!_v}" ] || { echo "error: $_v unresolved (set BOT_BOTTLE_FC_* or fix $_DEFAULTS)" >&2; exit 1; }
|
|
done
|
|
|
|
# Sidecar ports (must match the backend). egress=9099, supervise=9100,
|
|
# git-http=9420. Reached by the VM at its host-side TAP IP.
|
|
SIDECAR_PORTS="9099,9100,9420"
|
|
|
|
# --- IP math ---------------------------------------------------------
|
|
# Slot i occupies the /31 {base+2i, base+2i+1}: host = base+2i (the
|
|
# gateway the VM routes through), guest = base+2i+1 (the VM's address).
|
|
_ip_to_int() {
|
|
local IFS=. ; read -r a b c d <<<"$1" ; echo $(( (a<<24) + (b<<16) + (c<<8) + d ))
|
|
}
|
|
_int_to_ip() {
|
|
local n=$1 ; echo "$(( (n>>24)&255 )).$(( (n>>16)&255 )).$(( (n>>8)&255 )).$(( n&255 ))"
|
|
}
|
|
host_ip() { _int_to_ip $(( $(_ip_to_int "$IP_BASE") + 2*$1 )); }
|
|
guest_ip() { _int_to_ip $(( $(_ip_to_int "$IP_BASE") + 2*$1 + 1 )); }
|
|
iface() { echo "${PREFIX}$1"; }
|
|
|
|
require_root() {
|
|
if [ "$(id -u)" -ne 0 ]; then
|
|
echo "error: '$1' needs root (run under sudo)" >&2
|
|
exit 1
|
|
fi
|
|
}
|
|
|
|
cmd_up() {
|
|
require_root up
|
|
# Group ownership (any member can open the pool) overrides single-user
|
|
# ownership. The kernel lets a TAP's owning-group members attach.
|
|
if [ -n "$GROUP" ]; then
|
|
own_args=(group "$GROUP") ; own_desc="group=$GROUP"
|
|
else
|
|
own_args=(user "$OWNER") ; own_desc="owner=$OWNER"
|
|
fi
|
|
echo "firecracker net pool: $POOL_SIZE slots, base $IP_BASE, $own_desc"
|
|
|
|
# VM->sidecar traffic is DNAT'd to the sidecar container and
|
|
# forwarded, so forwarding must be enabled (Docker also sets this).
|
|
sysctl -qw net.ipv4.ip_forward=1
|
|
|
|
for i in $(seq 0 $((POOL_SIZE-1))); do
|
|
local dev host
|
|
dev="$(iface "$i")" ; host="$(host_ip "$i")"
|
|
# Non-destructive + idempotent: only create a missing TAP (tearing
|
|
# an existing one down would cut a running VM), and `addr replace`
|
|
# is safe to re-run. Ownership is fixed at creation, so to change
|
|
# owner/group run `down` then `up`.
|
|
ip link show "$dev" >/dev/null 2>&1 \
|
|
|| ip tuntap add dev "$dev" mode tap "${own_args[@]}"
|
|
ip addr replace "$host/31" dev "$dev"
|
|
ip link set "$dev" up
|
|
echo " $dev host=$host guest=$(guest_ip "$i") $own_desc"
|
|
done
|
|
|
|
_install_nft
|
|
echo "nftables table inet $TABLE installed (fail-closed boundary)"
|
|
echo "done."
|
|
}
|
|
|
|
_install_nft() {
|
|
# Own table: dropping only matches our bbfc* interfaces, so no other
|
|
# tool's traffic is affected. Priority -10 runs before Docker's
|
|
# filter hooks (priority 0); a drop here is terminal for the packet.
|
|
#
|
|
# forward: VM egress is DNAT'd to the sidecar (established via
|
|
# `ct status dnat`); return traffic via `ct state established`.
|
|
# Anything else from a VM is dropped -> no route to the internet
|
|
# or the rest of the host except through the sidecar proxy.
|
|
# input: a VM never needs host-local delivery (its sidecar is
|
|
# reached via DNAT->forward), so drop all direct input from VMs
|
|
# -> host services bound on 0.0.0.0 are unreachable from the VM.
|
|
nft -f - <<EOF
|
|
table inet $TABLE {
|
|
chain forward {
|
|
type filter hook forward priority -10; policy accept;
|
|
iifname != "${PREFIX}*" return
|
|
ct state established,related accept
|
|
ct status dnat accept
|
|
drop
|
|
}
|
|
chain input {
|
|
type filter hook input priority -10; policy accept;
|
|
iifname != "${PREFIX}*" return
|
|
ct state established,related accept
|
|
drop
|
|
}
|
|
}
|
|
EOF
|
|
}
|
|
|
|
cmd_down() {
|
|
require_root down
|
|
nft delete table inet "$TABLE" 2>/dev/null || true
|
|
for i in $(seq 0 $((POOL_SIZE-1))); do
|
|
local dev ; dev="$(iface "$i")"
|
|
if ip link show "$dev" >/dev/null 2>&1; then
|
|
ip link set "$dev" down 2>/dev/null || true
|
|
ip tuntap del dev "$dev" mode tap 2>/dev/null || true
|
|
echo " removed $dev"
|
|
fi
|
|
done
|
|
echo "done."
|
|
}
|
|
|
|
cmd_status() {
|
|
echo "table inet $TABLE:"
|
|
nft list table inet "$TABLE" 2>/dev/null || echo " (absent)"
|
|
echo "taps:"
|
|
for i in $(seq 0 $((POOL_SIZE-1))); do
|
|
local dev ; dev="$(iface "$i")"
|
|
if ip -brief addr show "$dev" >/dev/null 2>&1; then
|
|
ip -brief addr show "$dev" | sed 's/^/ /'
|
|
fi
|
|
done
|
|
}
|
|
|
|
case "${1:-}" in
|
|
up) cmd_up ;;
|
|
down) cmd_down ;;
|
|
status) cmd_status ;;
|
|
*) echo "usage: $0 {up|down|status}" >&2 ; exit 2 ;;
|
|
esac
|