didericis/bot-bottle
1
0
Fork 0
Code Issues 29 Pull Requests 4 Actions Packages Releases Wiki Activity

refactor(firecracker): split the combined rootfs into per-plane artifacts

Browse Source 
Each infra VM now boots its own rootfs instead of a shared combined image, so
the exposed gateway VM no longer carries buildah + the control-plane code it
never runs (a fatter, less-isolated exposed surface — the opposite of the plane
split's intent). The combined image bought only "one artifact"; each VM already
kept a full copy of the shared rootfs, so nothing was saved at boot.

  * orchestrator rootfs — control plane + buildah (Dockerfile.orchestrator.fc,
    FROM orchestrator); the slim gateway rootfs boots bot-bottle-gateway:latest
    directly. Dockerfile.infra.fc (the combined image) is deleted.
  * `_infra_init` splits into `_orchestrator_init` / `_gateway_init` (shared
    preamble via `_init_head`); each per-plane rootfs bakes only its role init,
    so the `bb_role` cmdline branch is gone.
  * infra_artifact is role-parametrized: a per-role package
    (bot-bottle-firecracker-<role>), version hash, URL, cache dir, and candidate
    subdir. `ensure_built` pulls both; `_expected_version` combines both markers.
  * publish_infra builds the images once, then builds + publishes an
    orchestrator and a gateway artifact under DIR/<role>/.

coverage.sh's candidate-dir plumbing is layout-agnostic (publish_infra --output
now fills DIR/<role>/, which ensure_artifact_gz reads). Full suite green.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
didericis
2026-07-25 15:21:03 -04:00
parent b7599ed146
commit d0d1da612e
8 changed files with 497 additions and 430 deletions
Download Patch File Download Diff File Expand all files Collapse all files
bot_bottle/backend/firecracker/infra_artifact.py
+74 -51
View File
@@ -1,22 +1,26 @@
"""Prebuilt infra-VM rootfs, pulled as an artifact (PRD 0069 Stage 2).
"""Prebuilt infra-VM rootfs images, pulled as artifacts (PRD 0069 Stage 2).
The Firecracker infra VM boots a fixed rootfs (orchestrator control plane +
gateway + buildah, control-plane init as PID 1) that does not vary per launch —
the per-boot bits (authorized_keys, guest IP) ride the kernel cmdline, so one
rootfs boots on any host. Instead of building that rootfs on the launch host
with Docker, we build it **off-host** and publish it as a versioned, ready-to-
boot ext4 (gzip-compressed) to a Gitea **generic package**; the launch host
downloads + verifies + boots it. No Docker, no image tooling on the launch
host — just an HTTP fetch and gunzip.
The two Firecracker infra VMs each boot a fixed per-plane rootfs that does not
vary per launch — the per-boot bits (authorized_keys, guest IP) ride the kernel
cmdline, so one rootfs boots on any host:
* `orchestrator` — the control plane + buildah (in-VM agent-image builds);
* `gateway` — the slim data plane (no build tooling on the exposed VM).
Instead of building them on the launch host with Docker, we build them
**off-host** and publish each as a versioned, ready-to-boot ext4 (gzip-
compressed) to a per-role Gitea **generic package**; the launch host downloads +
verifies + boots them. No Docker, no image tooling on the launch host — just an
HTTP fetch and gunzip.
publish (off-host, see publish_infra.py):
docker build -> rootfs dir -> mke2fs -> gzip -> PUT generic package
pull (this module, launch host):
GET .../rootfs.ext4.gz (+ .sha256) -> verify -> gunzip -> boot
The artifact **version** is a content hash of everything baked into the rootfs
(the shipped bot_bottle package, the three Dockerfiles, and the init), so a
launch host always pulls the artifact matching its code and a content change
Each artifact **version** is a content hash of everything baked into that rootfs
(the shipped bot_bottle package, the role's Dockerfiles, and the role init), so
a launch host always pulls the artifact matching its code and a content change
can't silently boot a stale rootfs. A checksum mismatch fails closed.
Set `BOT_BOTTLE_INFRA_BUILD=local` to skip the pull and build the rootfs
@@ -41,11 +45,23 @@ from . import util
_ARTIFACT_FORMAT = "1"
_REPO_ROOT = Path(__file__).resolve().parents[3]
_DOCKERFILES = ("Dockerfile.orchestrator", "Dockerfile.gateway", "Dockerfile.infra.fc")
# The two per-plane infra VM roles. Each publishes/pulls its own rootfs artifact
# from its own generic package; the Dockerfiles baked into each differ (only the
# orchestrator rootfs carries buildah), so the versions are hashed separately.
ROLES = ("orchestrator", "gateway")
_DOCKERFILES = {
"orchestrator": ("Dockerfile.orchestrator", "Dockerfile.orchestrator.fc"),
"gateway": ("Dockerfile.gateway",),
}
_DEFAULT_BASE = "https://gitea.dideric.is"
_DEFAULT_OWNER = "didericis"
_PACKAGE = "bot-bottle-firecracker-infra"
def _package(role: str) -> str:
return f"bot-bottle-firecracker-{role}"
# Streaming copy chunk for the (hundreds-of-MB) download.
_CHUNK = 1 << 20
@@ -57,21 +73,24 @@ def local_build_requested() -> bool:
return os.environ.get("BOT_BOTTLE_INFRA_BUILD", "").strip().lower() == "local"
def infra_artifact_version(init_script: str, *, repo_root: Path = _REPO_ROOT) -> str:
"""Content hash (16 hex) of everything baked into the infra rootfs: the
whole shipped `bot_bottle` package, the three fixed Dockerfiles, and the
guest init. Deterministic across the publish host and the launch host when
both run the same checkout, so the tag the launch host pulls is exactly the
tag publish produced.
def infra_artifact_version(
init_script: str, role: str, *, repo_root: Path = _REPO_ROOT,
) -> str:
"""Content hash (16 hex) of everything baked into `role`'s infra rootfs: the
whole shipped `bot_bottle` package, that role's Dockerfiles, and its guest
init. Deterministic across the publish host and the launch host when both run
the same checkout, so the tag the launch host pulls is exactly the tag
publish produced.
The package is `COPY bot_bottle /app/bot_bottle`'d wholesale into the image,
so hash *every* regular file under it — not just `*.py`. Non-Python inputs
(e.g. `gateway/egress/entrypoint.sh`, `netpool.defaults.env`) are baked in too, and
a change to one must bump the version or a launch host could boot a stale
rootfs whose code differs from its checkout. `__pycache__`/`.pyc` are the
only exclusions — build artifacts, never copied."""
The package is baked into both images wholesale (orchestrator `COPY`s it,
gateway `pip install`s it), so hash *every* regular file under it — not just
`*.py`. Non-Python inputs (e.g. `gateway/egress/entrypoint.sh`,
`netpool.defaults.env`) are baked in too, and a change to one must bump the
version or a launch host could boot a stale rootfs whose code differs from
its checkout. `__pycache__`/`.pyc` are the only exclusions — build artifacts,
never copied."""
h = hashlib.sha256()
h.update(f"format={_ARTIFACT_FORMAT}\n".encode())
h.update(f"format={_ARTIFACT_FORMAT}\nrole={role}\n".encode())
pkg = repo_root / "bot_bottle"
for path in sorted(pkg.rglob("*")):
if not path.is_file():
@@ -81,7 +100,7 @@ def infra_artifact_version(init_script: str, *, repo_root: Path = _REPO_ROOT) ->
h.update(str(path.relative_to(repo_root)).encode())
h.update(b"\0")
h.update(path.read_bytes())
for name in _DOCKERFILES:
for name in _DOCKERFILES[role]:
h.update(name.encode())
h.update(b"\0")
h.update((repo_root / name).read_bytes())
@@ -107,11 +126,11 @@ def _config() -> tuple[str, str, str]:
return base, owner, token
def artifact_url(version: str, filename: str) -> str:
"""The generic-package download URL for one file of this version's
artifact (`rootfs.ext4.gz` / `rootfs.ext4.gz.sha256`)."""
def artifact_url(version: str, filename: str, *, role: str) -> str:
"""The generic-package download URL for one file of `role`'s artifact at
`version` (`rootfs.ext4.gz` / `rootfs.ext4.gz.sha256`)."""
base, owner, _ = _config()
return f"{base}/api/packages/{owner}/generic/{_PACKAGE}/{version}/{filename}"
return f"{base}/api/packages/{owner}/generic/{_package(role)}/{version}/{filename}"
_GZ_NAME = "rootfs.ext4.gz"
@@ -119,8 +138,8 @@ _SHA_NAME = "rootfs.ext4.gz.sha256"
_CANDIDATE_DIR_ENV = "BOT_BOTTLE_INFRA_ARTIFACT_DIR"
def _cache_root(version: str) -> Path:
return util.cache_dir() / "infra-artifact" / version
def _cache_root(version: str, role: str) -> Path:
return util.cache_dir() / "infra-artifact" / role / version
def _open(url: str) -> urllib.request.Request:
@@ -162,13 +181,17 @@ def _sha256_file(path: Path) -> str:
return h.hexdigest()
def ensure_artifact_gz(version: str) -> Path:
"""The verified, cached `rootfs.ext4.gz` for `version` — downloading it (and
its `.sha256`) once, then reusing it. Fail-closed on a checksum mismatch:
the partial is removed and we die rather than boot an unverified rootfs."""
def ensure_artifact_gz(version: str, *, role: str) -> Path:
"""The verified, cached `rootfs.ext4.gz` for `role` at `version` —
downloading it (and its `.sha256`) once, then reusing it. Fail-closed on a
checksum mismatch: the partial is removed and we die rather than boot an
unverified rootfs.
A pre-staged candidate bundle (`BOT_BOTTLE_INFRA_ARTIFACT_DIR`) holds each
role under its own `<dir>/<role>/` subdir."""
candidate_dir = os.environ.get(_CANDIDATE_DIR_ENV, "").strip()
if candidate_dir:
root = Path(candidate_dir)
root = Path(candidate_dir) / role
version_file = root / "version.txt"
# Guard the read so a missing version.txt is a clean error, not a raw
# FileNotFoundError.
@@ -177,7 +200,7 @@ def ensure_artifact_gz(version: str) -> Path:
declared = version_file.read_text(encoding="utf-8").strip()
if declared != version:
die(
f"infra candidate version mismatch: expected {version}, "
f"infra candidate version mismatch ({role}): expected {version}, "
f"bundle contains {declared or '<empty>'}"
)
gz = root / _GZ_NAME
@@ -188,22 +211,22 @@ def ensure_artifact_gz(version: str) -> Path:
actual = _sha256_file(gz)
if actual != expected:
die(
f"infra candidate checksum mismatch for {version}:\n"
f"infra candidate checksum mismatch ({role}) for {version}:\n"
f" expected {expected}\n actual {actual}"
)
return gz
root = _cache_root(version)
root = _cache_root(version, role)
root.mkdir(parents=True, exist_ok=True)
gz = root / _GZ_NAME
ok = root / ".verified"
if gz.is_file() and ok.is_file():
return gz
info(f"pulling infra rootfs artifact {_PACKAGE}/{version}")
_download(artifact_url(version, _GZ_NAME), gz)
info(f"pulling infra rootfs artifact {_package(role)}/{version}")
_download(artifact_url(version, _GZ_NAME, role=role), gz)
sha = root / _SHA_NAME
_download(artifact_url(version, _SHA_NAME), sha)
_download(artifact_url(version, _SHA_NAME, role=role), sha)
expected = sha.read_text().split()[0].strip().lower()
actual = _sha256_file(gz)
@@ -211,7 +234,7 @@ def ensure_artifact_gz(version: str) -> Path:
gz.unlink(missing_ok=True)
sha.unlink(missing_ok=True)
die(
f"infra artifact checksum mismatch for {version}:\n"
f"infra artifact checksum mismatch ({role}) for {version}:\n"
f" expected {expected}\n"
f" actual {actual}\n"
f" refusing to boot an unverified rootfs."
@@ -220,13 +243,13 @@ def ensure_artifact_gz(version: str) -> Path:
return gz
def materialize_ext4(version: str, dest: Path) -> None:
"""Ensure the verified artifact is cached, then gunzip it to `dest` — a
fresh, writable per-boot rootfs (the VM mutates it; the cached `.gz` stays
def materialize_ext4(version: str, dest: Path, *, role: str) -> None:
"""Ensure the verified `role` artifact is cached, then gunzip it to `dest` —
a fresh, writable per-boot rootfs (the VM mutates it; the cached `.gz` stays
pristine). Atomic via a `.part` sibling."""
gz = ensure_artifact_gz(version)
gz = ensure_artifact_gz(version, role=role)
tmp = dest.with_suffix(dest.suffix + ".part")
info(f"expanding infra rootfs -> {dest}")
info(f"expanding {role} infra rootfs -> {dest}")
with gzip.open(gz, "rb") as src, open(tmp, "wb") as out:
shutil.copyfileobj(src, out, _CHUNK)
tmp.replace(dest)
bot_bottle/backend/firecracker/infra_vm.py
+125 -112
View File
@@ -17,11 +17,11 @@ Two persistent microVMs, split now that #469 got the DB off the data plane
key); reaches the orchestrator's control plane at `orch_guest:8099` over
the one nft forward rule that link allows.
Both VMs boot the **same** shared infra rootfs (gateway + orchestrator +
buildah); a `bb_role=` kernel-cmdline arg selects which plane a VM's PID-1
init starts, so there is still a single published artifact to build/pull. The
gateway VM's slimmer memory ceiling (buildah is present on disk but unused
there) is set at boot.
Each VM boots its **own** per-plane rootfs — the orchestrator rootfs carries the
control plane + buildah (in-VM agent builds), the gateway rootfs is the slim
data plane with no build tooling on the exposed VM. Two artifacts, built/pulled
per role (`infra_artifact`); each rootfs bakes only its own role init as PID 1.
The gateway VM also runs a slimmer memory ceiling.
SSH is left enabled for debugging + provisioning; the control plane is the
load-bearing surface.
@@ -61,15 +61,22 @@ _GUEST_SIGNING_KEY_PATH = "/var/lib/bot-bottle/orchestrator-token"
# the gateway daemons present it to the orchestrator, and never see the key.
_GUEST_GATEWAY_JWT_PATH = "/var/lib/bot-bottle/gateway-jwt"
# The single shared infra image: gateway data plane + COPY'd control-plane
# `bot_bottle` package + buildah (Dockerfile.infra.fc, FROM gateway). Built from
# source by default; a pull-from-registry mode lands later. Both VMs boot from
# it, the `bb_role` cmdline selecting the plane.
_INFRA_IMAGE = "bot-bottle-infra:latest"
# The two per-plane rootfs source images. The orchestrator VM boots a control
# plane + buildah rootfs (Dockerfile.orchestrator.fc, FROM orchestrator); the
# gateway VM boots the slim data-plane image directly (no build tooling on the
# exposed VM). Built from source by default; the launch host pulls prebuilt
# artifacts instead (`infra_artifact`).
_GATEWAY_IMAGE = "bot-bottle-gateway:latest"
_ORCHESTRATOR_IMAGE = "bot-bottle-orchestrator:latest"
_ORCHESTRATOR_FC_IMAGE = "bot-bottle-orchestrator-fc:latest"
_REPO_ROOT = Path(__file__).resolve().parents[3]
# Per-role rootfs source image + the extra free space `mke2fs` leaves for the
# guest to grow into. The orchestrator keeps buildah's large build slack; the
# gateway carries no build tooling, so its rootfs is much smaller.
_ROOTFS_IMAGE = {"orchestrator": _ORCHESTRATOR_FC_IMAGE, "gateway": _GATEWAY_IMAGE}
_ROOTFS_SLACK_MIB = {"orchestrator": 8192, "gateway": 1024}
ORCHESTRATOR_PORT = 8099
# Gateway data-plane ports (agent-facing): egress proxy, supervise MCP,
# git-http. Reached by agent VMs via the PREROUTING DNAT to the gateway VM.
@@ -119,45 +126,54 @@ class InfraEndpoint:
return self.gateway.ca_cert_pem(timeout=timeout)
def role_init(role: str) -> str:
"""The guest PID-1 init for `role` (each per-plane rootfs bakes only its
own — no `bb_role` branch, since the rootfs *is* the role)."""
return _orchestrator_init() if role == "orchestrator" else _gateway_init()
def _role_version(role: str) -> str:
return infra_artifact.infra_artifact_version(role_init(role), role)
def ensure_built() -> None:
"""Ensure the infra rootfs is available before boot.
"""Ensure both infra rootfs artifacts are available before boot.
Default (docker-free, PRD 0069 Stage 2): download + verify the prebuilt
rootfs artifact matching this code version (see `infra_artifact`); the
launch host needs no Docker. `BOT_BOTTLE_INFRA_BUILD=local` instead builds
the fixed images from source with host Docker — the infra image is `FROM`
the gateway image and `COPY --from`s the orchestrator image, so both must
exist first — for iterating on the Dockerfiles."""
orchestrator + gateway rootfs artifacts matching this code version (see
`infra_artifact`); the launch host needs no Docker.
`BOT_BOTTLE_INFRA_BUILD=local` instead builds the images from source with
host Docker (the orchestrator-fc image is `FROM` the orchestrator image, so
it must exist first) — for iterating on the Dockerfiles."""
if infra_artifact.local_build_requested():
build_infra_images_with_docker()
return
infra_artifact.ensure_artifact_gz(
infra_artifact.infra_artifact_version(_infra_init()))
for role in infra_artifact.ROLES:
infra_artifact.ensure_artifact_gz(_role_version(role), role=role)
def build_infra_images_with_docker() -> None:
"""Build the three fixed images from source with host Docker: orchestrator,
gateway, then the Firecracker infra image (Dockerfile.infra.fc: FROM gateway
+ COPY --from orchestrator + buildah). The launch host uses this only in
`BOT_BOTTLE_INFRA_BUILD=local` mode; `publish_infra` uses it off-host to
produce the published artifact."""
"""Build the fixed images from source with host Docker: orchestrator,
gateway, then the orchestrator-fc image (Dockerfile.orchestrator.fc: FROM
orchestrator + buildah). The gateway VM boots the gateway image directly.
The launch host uses this only in `BOT_BOTTLE_INFRA_BUILD=local` mode;
`publish_infra` uses it off-host to produce the published artifacts."""
docker_mod.build_image(
_ORCHESTRATOR_IMAGE, str(_REPO_ROOT), dockerfile="Dockerfile.orchestrator")
docker_mod.build_image(
_GATEWAY_IMAGE, str(_REPO_ROOT), dockerfile="Dockerfile.gateway")
docker_mod.build_image(
_INFRA_IMAGE, str(_REPO_ROOT), dockerfile="Dockerfile.infra.fc")
_ORCHESTRATOR_FC_IMAGE, str(_REPO_ROOT), dockerfile="Dockerfile.orchestrator.fc")
def build_infra_rootfs_dir() -> Path:
"""The infra VMs' shared base rootfs: the infra image prepared with the
role-branched init as PID 1. The init's content is folded into the cache
key so an init change rebuilds the rootfs (the base image digest alone
wouldn't catch it)."""
init = _infra_init()
def build_rootfs_dir(role: str) -> Path:
"""`role`'s base rootfs dir: its source image prepared with the role init as
PID 1. The init's content is folded into the cache key so an init change
rebuilds the rootfs (the base image digest alone wouldn't catch it)."""
init = role_init(role)
tag = hashlib.sha256(init.encode()).hexdigest()[:8]
return util.build_base_rootfs_dir(
_INFRA_IMAGE, variant=f"-infra-{tag}", init_script=init,
_ROOTFS_IMAGE[role], variant=f"-{role}-{tag}", init_script=init,
)
@@ -259,8 +275,8 @@ def boot_vm(
data_drive: Path | None = None,
extra_boot_args: str = "",
) -> InfraVm:
"""Boot one infra VM from the shared rootfs on `slot`'s link, tagged with
its `bb_role` so the guest init starts the right plane. Records the PID."""
"""Boot the `role` infra VM from its per-plane rootfs on `slot`'s link.
Records the PID."""
if not netpool.tap_present(slot.iface):
die(f"infra link {slot.iface} not present.\n"
f" ./cli.py backend setup --backend=firecracker")
@@ -268,18 +284,16 @@ def boot_vm(
run_dir.mkdir(parents=True, exist_ok=True)
rootfs = run_dir / "rootfs.ext4"
if infra_artifact.local_build_requested():
util.build_rootfs_ext4(build_infra_rootfs_dir(), rootfs, slack_mib=8192)
util.build_rootfs_ext4(
build_rootfs_dir(role), rootfs, slack_mib=_ROOTFS_SLACK_MIB[role])
else:
# Prebuilt artifact already carries the buildah build slack; expand it
# to a fresh writable rootfs for this boot.
infra_artifact.materialize_ext4(
infra_artifact.infra_artifact_version(_infra_init()), rootfs)
# Prebuilt artifact already carries the role's build slack; expand it to
# a fresh writable rootfs for this boot.
infra_artifact.materialize_ext4(_role_version(role), rootfs, role=role)
private_key, pubkey = _stable_keypair()
info(f"booting {role} VM on {slot.iface} (guest {slot.guest_ip})")
boot_args = f"bb_role={role}"
if extra_boot_args:
boot_args = f"{boot_args} {extra_boot_args}"
boot_args = extra_boot_args
vm = firecracker_vm.boot(
name=name, rootfs=rootfs, tap=slot.iface,
guest_ip=slot.guest_ip, host_ip=slot.host_ip, pubkey=pubkey,
@@ -323,7 +337,9 @@ def _version_file() -> Path:
def _expected_version() -> str:
return infra_artifact.infra_artifact_version(_infra_init())
"""The combined marker for the running pair: both per-plane artifact
versions, so a change to either rootfs dislodges the adopted pair."""
return " ".join(f"{role}={_role_version(role)}" for role in infra_artifact.ROLES)
def _adoptable(key: Path, url: str, want: str) -> bool:
@@ -455,19 +471,12 @@ def _health_ok(url: str) -> bool:
return False
def _infra_init() -> str:
"""PID-1 init for the infra VMs. Shared setup (pseudo-filesystems, PATH,
resolver, debug SSH), then a `bb_role` cmdline branch selecting the plane:
* orchestrator — mount the persistent registry volume, wait for the
host-seeded signing key, start ONLY the control plane;
* gateway — wait for the host-seeded `gateway` JWT, start ONLY the
data-plane daemons (multi-tenant against the orchestrator at the
`bb_orch` cmdline address).
Both VMs boot this same init (one published artifact); the cmdline selects
the role, so no orchestrator IP is baked in (an IP_BASE override doesn't
change the artifact version)."""
def _init_head() -> str:
"""The shared PID-1 preamble both role inits open with: mount the pseudo-
filesystems, export a real PATH (a bare-init shell's built-in exec path
isn't in the *environment*, so backgrounded `python3 ...` children would
find no PATH), set the direct upstream resolver, install the per-boot SSH
pubkey from the cmdline, and start dropbear for debug/provisioning."""
return f"""#!/bin/sh
# bot-bottle Firecracker infra VM init (PID 1).
mount -t proc proc /proc 2>/dev/null
@@ -476,10 +485,6 @@ mount -t devtmpfs dev /dev 2>/dev/null
mkdir -p /dev/pts && mount -t devpts devpts /dev/pts 2>/dev/null
mount -o remount,rw / 2>/dev/null
# Export a real PATH: a bare-init shell resolves its own execs via a
# built-in default path, but that isn't in the *environment*, so
# gateway_init's subprocess daemons (spawned as `python3 ...`) would
# inherit no PATH and fail to find python3. Export it for all children.
export PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
# Direct upstream resolver (control-plane / gateway egress + buildah).
@@ -497,62 +502,70 @@ mkdir -p /etc/dropbear /run /var/lib/bot-bottle
/bb-dropbear -R -E -p 22 &
ROLE=$(sed -n 's/.*bb_role=\\([^ ]*\\).*/\\1/p' /proc/cmdline)
cd /app
"""
if [ "$ROLE" = gateway ]; then
# Gateway data plane, multi-tenant: each request resolves source-IP ->
# policy against the orchestrator VM (its guest IP is on the cmdline as
# bb_orch). The VM backend reaches git over git-http (9420), so the git://
# daemon (git-gate, needs a per-bottle entrypoint the consolidated model
# doesn't use) is left out. No SUPERVISE_DB_PATH: the data plane reaches the
# supervise queue over the control-plane RPC and never opens bot-bottle.db
# (PRD 0070 / #469). It presents the pre-minted `gateway` JWT the launcher
# pushed; if it never arrives, REFUSE to start rather than run without auth.
ORCH=$(sed -n 's/.*bb_orch=\\([^ ]*\\).*/\\1/p' /proc/cmdline)
GW_JWT=""
i=0
while [ "$i" -lt 600 ]; do
GW_JWT=$(cat {_GUEST_GATEWAY_JWT_PATH} 2>/dev/null)
[ -n "$GW_JWT" ] && break
i=$((i + 1))
sleep 0.1
done
if [ -z "$GW_JWT" ]; then
echo "infra gateway: gateway JWT never arrived; refusing to start the data plane" >&2
else
chmod 600 {_GUEST_GATEWAY_JWT_PATH} 2>/dev/null || true
BOT_BOTTLE_GATEWAY_DAEMONS=egress,git-http,supervise \\
BOT_BOTTLE_ORCHESTRATOR_URL=http://$ORCH:{ORCHESTRATOR_PORT} \\
BOT_BOTTLE_ORCHESTRATOR_AUTH_JWT="$GW_JWT" \\
python3 -m bot_bottle.gateway.bootstrap &
fi
else
# Control plane. Source is baked at /app; the package is stdlib-only.
# Persistent registry volume (second virtio-block device, /dev/vdb) mounted
# at the DB dir, so bot-bottle.db survives orchestrator-VM restarts.
mount -t ext4 /dev/vdb /var/lib/bot-bottle 2>/dev/null || true
# Wait for the launcher to push the host-canonical signing key over SSH,
# then hand it ONLY to the orchestrator (to verify tokens). If it never
# arrives, REFUSE to start rather than run OPEN — open mode would grant
# every unauthenticated caller the `cli` role (issue #469).
CP_KEY=""
i=0
while [ "$i" -lt 600 ]; do
CP_KEY=$(cat {_GUEST_SIGNING_KEY_PATH} 2>/dev/null)
[ -n "$CP_KEY" ] && break
i=$((i + 1))
sleep 0.1
done
if [ -z "$CP_KEY" ]; then
echo "infra: control-plane signing key never arrived; refusing to start the control plane (would run OPEN)" >&2
else
chmod 600 {_GUEST_SIGNING_KEY_PATH} 2>/dev/null || true
BOT_BOTTLE_ROOT=/var/lib/bot-bottle BOT_BOTTLE_ORCHESTRATOR_TOKEN="$CP_KEY" python3 -m bot_bottle.orchestrator \\
--host 0.0.0.0 --port {ORCHESTRATOR_PORT} --broker stub &
fi
fi
_INIT_TAIL = """
# Reap as PID 1; children are backgrounded, so `wait` blocks.
while : ; do wait ; done
"""
def _gateway_init() -> str:
"""PID-1 init for the gateway (data-plane) VM. Waits for the host-seeded
`gateway` JWT, then starts ONLY the data-plane daemons, multi-tenant against
the orchestrator at the `bb_orch` cmdline address. The VM backend reaches git
over git-http (9420), so the git:// daemon (a per-bottle entrypoint the
consolidated model doesn't use) is left out. No SUPERVISE_DB_PATH: the data
plane reaches the supervise queue over the control-plane RPC and never opens
bot-bottle.db (PRD 0070 / #469). If the JWT never arrives, REFUSE to start
rather than run without auth."""
return _init_head() + f"""
ORCH=$(sed -n 's/.*bb_orch=\\([^ ]*\\).*/\\1/p' /proc/cmdline)
GW_JWT=""
i=0
while [ "$i" -lt 600 ]; do
GW_JWT=$(cat {_GUEST_GATEWAY_JWT_PATH} 2>/dev/null)
[ -n "$GW_JWT" ] && break
i=$((i + 1))
sleep 0.1
done
if [ -z "$GW_JWT" ]; then
echo "infra gateway: gateway JWT never arrived; refusing to start the data plane" >&2
else
chmod 600 {_GUEST_GATEWAY_JWT_PATH} 2>/dev/null || true
BOT_BOTTLE_GATEWAY_DAEMONS=egress,git-http,supervise \\
BOT_BOTTLE_ORCHESTRATOR_URL=http://$ORCH:{ORCHESTRATOR_PORT} \\
BOT_BOTTLE_ORCHESTRATOR_AUTH_JWT="$GW_JWT" \\
python3 -m bot_bottle.gateway.bootstrap &
fi
""" + _INIT_TAIL
def _orchestrator_init() -> str:
"""PID-1 init for the orchestrator (control-plane) VM. Mounts the persistent
registry volume (/dev/vdb — bot-bottle.db survives a VM restart), waits for
the host-seeded signing key, then starts ONLY the control plane. If the key
never arrives, REFUSE to start rather than run OPEN — open mode would grant
every unauthenticated caller the `cli` role (#469)."""
return _init_head() + f"""
# Persistent registry volume (second virtio-block device, /dev/vdb) mounted at
# the DB dir, so bot-bottle.db survives orchestrator-VM restarts.
mount -t ext4 /dev/vdb /var/lib/bot-bottle 2>/dev/null || true
CP_KEY=""
i=0
while [ "$i" -lt 600 ]; do
CP_KEY=$(cat {_GUEST_SIGNING_KEY_PATH} 2>/dev/null)
[ -n "$CP_KEY" ] && break
i=$((i + 1))
sleep 0.1
done
if [ -z "$CP_KEY" ]; then
echo "infra: control-plane signing key never arrived; refusing to start the control plane (would run OPEN)" >&2
else
chmod 600 {_GUEST_SIGNING_KEY_PATH} 2>/dev/null || true
BOT_BOTTLE_ROOT=/var/lib/bot-bottle BOT_BOTTLE_ORCHESTRATOR_TOKEN="$CP_KEY" python3 -m bot_bottle.orchestrator \\
--host 0.0.0.0 --port {ORCHESTRATOR_PORT} --broker stub &
fi
""" + _INIT_TAIL
bot_bottle/backend/firecracker/publish_infra.py
+96 -78
View File
@@ -1,20 +1,22 @@
"""Build the infra rootfs and publish it as a Gitea generic package.
"""Build the infra rootfs artifacts and publish them as Gitea generic packages.
The off-host (build / CI) half of PRD 0069 Stage 2: this DOES use Docker, but
never on the launch host. It runs the same pipeline the launch host used to run
locally — `docker build` the three fixed images, export to a rootfs dir, inject
the guest boot, `mke2fs` to an ext4 with the buildah build slack — then gzips
the ext4 and PUTs it (plus a `.sha256`) to
`…/api/packages/<owner>/generic/bot-bottle-firecracker-infra/<version>/`.
locally — `docker build` the fixed images, export each per-plane rootfs, inject
the guest boot, `mke2fs` to an ext4 — then gzips each and PUTs it (plus a
`.sha256`) to `…/api/packages/<owner>/generic/bot-bottle-firecracker-<role>/<version>/`.
The `<version>` is `infra_artifact.infra_artifact_version(...)`, the content
hash of the rootfs inputs, so a launch host at the same code checkout resolves
the exact artifact this produced.
There are two artifacts, one per plane (`orchestrator`, `gateway`); the
orchestrator rootfs carries buildah, the gateway rootfs is slim. Each
`<version>` is `infra_artifact.infra_artifact_version(...)`, the content hash of
that rootfs's inputs, so a launch host at the same code checkout resolves the
exact artifacts this produced.
python3 -m bot_bottle.backend.firecracker.publish_infra --output DIR
python3 -m bot_bottle.backend.firecracker.publish_infra --publish-dir DIR
Auth: a token with `write:package` on the target owner, from
A candidate bundle holds each role under its own `DIR/<role>/` subdir. Auth: a
token with `write:package` on the target owner, from
`BOT_BOTTLE_INFRA_ARTIFACT_TOKEN`.
"""
@@ -33,17 +35,27 @@ from . import infra_artifact, infra_vm, util
_CHUNK = 1 << 20
# A human-readable description shipped alongside the artifact — generic packages
_GZ_NAME = "rootfs.ext4.gz"
_SHA_NAME = "rootfs.ext4.gz.sha256"
# A human-readable description shipped alongside each artifact — generic packages
# have no description field, so this file *is* the description on the package
# page. Uploaded on every publish so it never goes stale.
_ABOUT_NAME = "about.txt"
_ABOUT_TEXT = (
"bot-bottle infra rootfs for the Firecracker backend (PRD 0069 Stage 2, "
"#348): the per-host infra VM (orchestrator control plane + gateway + "
"buildah). Prebuilt off-host, gzip ext4; the launch host downloads + "
"sha256-verifies + boots it, no host Docker. The version tag is a content "
"hash of the rootfs inputs. Files: rootfs.ext4.gz + rootfs.ext4.gz.sha256.\n"
)
def _about_text(role: str) -> str:
return (
f"bot-bottle firecracker {role} rootfs (PRD 0069 Stage 2 / PRD 0070): "
f"the per-host {role} infra VM. Prebuilt off-host, gzip ext4; the launch "
f"host downloads + sha256-verifies + boots it, no host Docker. The "
f"version tag is a content hash of the rootfs inputs. Files: "
f"{_GZ_NAME} + {_SHA_NAME}.\n"
)
def _role_version(role: str) -> str:
return infra_artifact.infra_artifact_version(infra_vm.role_init(role), role)
def _gzip(src: Path, dest: Path) -> None:
@@ -109,34 +121,35 @@ def _delete(url: str, token: str) -> None:
raise SystemExit(f"registry unreachable: {url} ({e.reason})")
def build_artifact(out_dir: Path) -> tuple[str, Path, Path]:
"""Build the infra rootfs ext4, gzip it, and write the checksum. Returns
`(version, gz_path, sha_path)`. Uses host Docker (off-host / CI)."""
version = infra_artifact.infra_artifact_version(infra_vm._infra_init())
print(f"building infra rootfs artifact {version} (docker)")
infra_vm.build_infra_images_with_docker()
base = infra_vm.build_infra_rootfs_dir()
def build_role_artifact(role: str, role_dir: Path) -> str:
"""Build `role`'s rootfs ext4, gzip it, and write the checksum + version into
`role_dir`. Returns the version. Assumes the docker images are already
built (`infra_vm.build_infra_images_with_docker`). Uses host Docker."""
version = _role_version(role)
print(f"building {role} rootfs artifact {version} (docker)")
base = infra_vm.build_rootfs_dir(role)
ext4 = out_dir / "rootfs.ext4"
util.build_rootfs_ext4(base, ext4, slack_mib=8192)
gz = out_dir / "rootfs.ext4.gz"
print("compressing rootfs")
ext4 = role_dir / "rootfs.ext4"
util.build_rootfs_ext4(base, ext4, slack_mib=infra_vm._ROOTFS_SLACK_MIB[role])
gz = role_dir / _GZ_NAME
print(f"compressing {role} rootfs")
_gzip(ext4, gz)
ext4.unlink(missing_ok=True)
sha = out_dir / "rootfs.ext4.gz.sha256"
sha = role_dir / _SHA_NAME
digest = _sha256(gz)
sha.write_text(f"{digest} rootfs.ext4.gz\n")
print(f" {gz.name}: {gz.stat().st_size / 1e6:.0f} MB sha256={digest}")
return version, gz, sha
sha.write_text(f"{digest} {_GZ_NAME}\n")
(role_dir / "version.txt").write_text(version + "\n", encoding="utf-8")
print(f" {role}/{gz.name}: {gz.stat().st_size / 1e6:.0f} MB sha256={digest}")
return version
def _try_download_published(out_dir: Path) -> tuple[str, Path, Path] | None:
"""If this version's artifact is already in the registry, download the gz
and sha to out_dir and return (version, gz_path, sha_path). Returns None
when not yet published."""
version = infra_artifact.infra_artifact_version(infra_vm._infra_init())
sha_url = infra_artifact.artifact_url(version, "rootfs.ext4.gz.sha256")
def _try_download_published(role: str, role_dir: Path) -> str | None:
"""If `role`'s artifact for this version is already in the registry, download
the gz + sha into `role_dir` and return the version. None when not yet
published."""
version = _role_version(role)
sha_url = infra_artifact.artifact_url(version, _SHA_NAME, role=role)
try:
with urllib.request.urlopen(infra_artifact._open(sha_url)):
pass
@@ -146,70 +159,70 @@ def _try_download_published(out_dir: Path) -> tuple[str, Path, Path] | None:
raise SystemExit(f"registry check failed (HTTP {e.code}): {sha_url}")
except urllib.error.URLError as e:
raise SystemExit(f"registry unreachable: {sha_url} ({e.reason})")
print(f"infra rootfs {version} already published — downloading instead of building")
gz = out_dir / "rootfs.ext4.gz"
sha = out_dir / "rootfs.ext4.gz.sha256"
infra_artifact._download(infra_artifact.artifact_url(version, "rootfs.ext4.gz"), gz)
infra_artifact._download(infra_artifact.artifact_url(version, "rootfs.ext4.gz.sha256"), sha)
return version, gz, sha
print(f"{role} rootfs {version} already published — downloading instead of building")
infra_artifact._download(
infra_artifact.artifact_url(version, _GZ_NAME, role=role), role_dir / _GZ_NAME)
infra_artifact._download(sha_url, role_dir / _SHA_NAME)
(role_dir / "version.txt").write_text(version + "\n", encoding="utf-8")
return version
def _publish_bundle(root: Path, token: str) -> str:
version_file = root / "version.txt"
def _publish_bundle(role: str, role_dir: Path, token: str) -> str:
version_file = role_dir / "version.txt"
# Guard the read so a missing version.txt is a clean error, not a raw
# FileNotFoundError.
if not version_file.is_file():
raise SystemExit(f"incomplete artifact bundle: {root}")
raise SystemExit(f"incomplete {role} artifact bundle: {role_dir}")
version = version_file.read_text(encoding="utf-8").strip()
expected = infra_artifact.infra_artifact_version(infra_vm._infra_init())
expected = _role_version(role)
if version != expected:
raise SystemExit(
f"artifact bundle version {version!r} does not match checkout {expected!r}"
f"{role} artifact bundle version {version!r} does not match checkout {expected!r}"
)
gz = root / "rootfs.ext4.gz"
sha = root / "rootfs.ext4.gz.sha256"
gz = role_dir / _GZ_NAME
sha = role_dir / _SHA_NAME
if not gz.is_file() or not sha.is_file():
raise SystemExit(f"incomplete artifact bundle: {root}")
raise SystemExit(f"incomplete {role} artifact bundle: {role_dir}")
expected_sha = sha.read_text().split()[0].strip().lower()
if _sha256(gz) != expected_sha:
raise SystemExit("artifact bundle checksum mismatch")
raise SystemExit(f"{role} artifact bundle checksum mismatch")
gz_url = infra_artifact.artifact_url(version, gz.name)
sha_url = infra_artifact.artifact_url(version, sha.name)
about_url = infra_artifact.artifact_url(version, _ABOUT_NAME)
gz_url = infra_artifact.artifact_url(version, _GZ_NAME, role=role)
sha_url = infra_artifact.artifact_url(version, _SHA_NAME, role=role)
about_url = infra_artifact.artifact_url(version, _ABOUT_NAME, role=role)
# Publishing is idempotent. If this exact complete artifact is already
# present, a test-only main commit is a no-op. Otherwise clear any partial
# upload left by an interrupted prior attempt and upload the complete set.
# present, a re-publish is a no-op. Otherwise clear any partial upload left
# by an interrupted prior attempt and upload the complete set.
try:
with urllib.request.urlopen(infra_artifact._open(sha_url)) as resp:
remote_sha = resp.read().decode("utf-8").split()[0].strip().lower()
except urllib.error.HTTPError as e:
if e.code != 404:
raise SystemExit(f"checking existing artifact failed (HTTP {e.code})")
raise SystemExit(f"checking existing {role} artifact failed (HTTP {e.code})")
remote_sha = ""
except urllib.error.URLError as e:
raise SystemExit(f"registry unreachable: {sha_url} ({e.reason})")
if remote_sha == expected_sha:
print(f"infra rootfs {version} already published")
print(f"{role} rootfs {version} already published")
return version
for url in (gz_url, sha_url, about_url):
_delete(url, token)
_put(gz_url, gz, token)
_put(sha_url, sha.read_bytes(), token)
_put(about_url, _ABOUT_TEXT.encode(), token)
_put(about_url, _about_text(role).encode(), token)
return version
def main(argv: list[str] | None = None) -> int:
parser = argparse.ArgumentParser(
prog="publish_infra", description="Build + publish the infra rootfs artifact.")
prog="publish_infra", description="Build + publish the infra rootfs artifacts.")
mode = parser.add_mutually_exclusive_group(required=True)
mode.add_argument("--output", type=Path,
help="build a candidate bundle in DIR without publishing")
help="build candidate bundles in DIR/<role>/ without publishing")
mode.add_argument("--publish-dir", type=Path,
help="publish an already-built and tested candidate bundle")
help="publish already-built + tested candidate bundles under DIR")
parser.add_argument("--reuse-published", action="store_true",
help="with --output: download from registry if already published instead of building")
args = parser.parse_args(argv)
@@ -221,23 +234,28 @@ def main(argv: list[str] | None = None) -> int:
"with write:package")
if args.output is not None:
args.output.mkdir(parents=True, exist_ok=True)
reused = None
if args.reuse_published:
reused = _try_download_published(args.output)
if reused is not None:
version, _, _ = reused
(args.output / "version.txt").write_text(version + "\n", encoding="utf-8")
print(f"reused published infra rootfs candidate {version}")
return 0
version, _gz, _sha = build_artifact(args.output)
(args.output / "version.txt").write_text(version + "\n", encoding="utf-8")
print(f"built infra rootfs candidate {version}")
# Build (or reuse) all roles. Images are built once, up front, only when
# something actually needs building.
pending = []
for role in infra_artifact.ROLES:
role_dir = args.output / role
role_dir.mkdir(parents=True, exist_ok=True)
if args.reuse_published and _try_download_published(role, role_dir):
print(f"reused published {role} rootfs candidate")
continue
pending.append(role)
if pending:
print("building infra images (docker)")
infra_vm.build_infra_images_with_docker()
for role in pending:
build_role_artifact(role, args.output / role)
print(f"built {role} rootfs candidate")
return 0
assert args.publish_dir is not None
version = _publish_bundle(args.publish_dir, token)
print(f"published infra rootfs {version}")
for role in infra_artifact.ROLES:
version = _publish_bundle(role, args.publish_dir / role, token)
print(f"published {role} rootfs {version}")
return 0