didericis/bot-bottle
1
0
Fork 0
Code Issues 32 Pull Requests 8 Actions Packages Releases Wiki Activity

refactor(firecracker): split the combined rootfs into per-plane artifacts

Browse Source 
Each infra VM now boots its own rootfs instead of a shared combined image, so
the exposed gateway VM no longer carries buildah + the control-plane code it
never runs (a fatter, less-isolated exposed surface — the opposite of the plane
split's intent). The combined image bought only "one artifact"; each VM already
kept a full copy of the shared rootfs, so nothing was saved at boot.

  * orchestrator rootfs — control plane + buildah (Dockerfile.orchestrator.fc,
    FROM orchestrator); the slim gateway rootfs boots bot-bottle-gateway:latest
    directly. Dockerfile.infra.fc (the combined image) is deleted.
  * `_infra_init` splits into `_orchestrator_init` / `_gateway_init` (shared
    preamble via `_init_head`); each per-plane rootfs bakes only its role init,
    so the `bb_role` cmdline branch is gone.
  * infra_artifact is role-parametrized: a per-role package
    (bot-bottle-firecracker-<role>), version hash, URL, cache dir, and candidate
    subdir. `ensure_built` pulls both; `_expected_version` combines both markers.
  * publish_infra builds the images once, then builds + publishes an
    orchestrator and a gateway artifact under DIR/<role>/.

coverage.sh's candidate-dir plumbing is layout-agnostic (publish_infra --output
now fills DIR/<role>/, which ensure_artifact_gz reads). Full suite green.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
didericis
2026-07-25 15:21:03 -04:00
parent b7599ed146
commit d0d1da612e
8 changed files with 497 additions and 430 deletions
Download Patch File Download Diff File Expand all files Collapse all files
bot_bottle/backend/firecracker/infra_artifact.py
+74 -51
View File
@@ -1,22 +1,26 @@
"""Prebuilt infra-VM rootfs, pulled as an artifact (PRD 0069 Stage 2).
"""Prebuilt infra-VM rootfs images, pulled as artifacts (PRD 0069 Stage 2).
The Firecracker infra VM boots a fixed rootfs (orchestrator control plane +
gateway + buildah, control-plane init as PID 1) that does not vary per launch —
the per-boot bits (authorized_keys, guest IP) ride the kernel cmdline, so one
rootfs boots on any host. Instead of building that rootfs on the launch host
with Docker, we build it **off-host** and publish it as a versioned, ready-to-
boot ext4 (gzip-compressed) to a Gitea **generic package**; the launch host
downloads + verifies + boots it. No Docker, no image tooling on the launch
host — just an HTTP fetch and gunzip.
The two Firecracker infra VMs each boot a fixed per-plane rootfs that does not
vary per launch — the per-boot bits (authorized_keys, guest IP) ride the kernel
cmdline, so one rootfs boots on any host:
* `orchestrator` — the control plane + buildah (in-VM agent-image builds);
* `gateway` — the slim data plane (no build tooling on the exposed VM).
Instead of building them on the launch host with Docker, we build them
**off-host** and publish each as a versioned, ready-to-boot ext4 (gzip-
compressed) to a per-role Gitea **generic package**; the launch host downloads +
verifies + boots them. No Docker, no image tooling on the launch host — just an
HTTP fetch and gunzip.
publish (off-host, see publish_infra.py):
docker build -> rootfs dir -> mke2fs -> gzip -> PUT generic package
pull (this module, launch host):
GET .../rootfs.ext4.gz (+ .sha256) -> verify -> gunzip -> boot
The artifact **version** is a content hash of everything baked into the rootfs
(the shipped bot_bottle package, the three Dockerfiles, and the init), so a
launch host always pulls the artifact matching its code and a content change
Each artifact **version** is a content hash of everything baked into that rootfs
(the shipped bot_bottle package, the role's Dockerfiles, and the role init), so
a launch host always pulls the artifact matching its code and a content change
can't silently boot a stale rootfs. A checksum mismatch fails closed.
Set `BOT_BOTTLE_INFRA_BUILD=local` to skip the pull and build the rootfs
@@ -41,11 +45,23 @@ from . import util
_ARTIFACT_FORMAT = "1"
_REPO_ROOT = Path(__file__).resolve().parents[3]
_DOCKERFILES = ("Dockerfile.orchestrator", "Dockerfile.gateway", "Dockerfile.infra.fc")
# The two per-plane infra VM roles. Each publishes/pulls its own rootfs artifact
# from its own generic package; the Dockerfiles baked into each differ (only the
# orchestrator rootfs carries buildah), so the versions are hashed separately.
ROLES = ("orchestrator", "gateway")
_DOCKERFILES = {
"orchestrator": ("Dockerfile.orchestrator", "Dockerfile.orchestrator.fc"),
"gateway": ("Dockerfile.gateway",),
}
_DEFAULT_BASE = "https://gitea.dideric.is"
_DEFAULT_OWNER = "didericis"
_PACKAGE = "bot-bottle-firecracker-infra"
def _package(role: str) -> str:
return f"bot-bottle-firecracker-{role}"
# Streaming copy chunk for the (hundreds-of-MB) download.
_CHUNK = 1 << 20
@@ -57,21 +73,24 @@ def local_build_requested() -> bool:
return os.environ.get("BOT_BOTTLE_INFRA_BUILD", "").strip().lower() == "local"
def infra_artifact_version(init_script: str, *, repo_root: Path = _REPO_ROOT) -> str:
"""Content hash (16 hex) of everything baked into the infra rootfs: the
whole shipped `bot_bottle` package, the three fixed Dockerfiles, and the
guest init. Deterministic across the publish host and the launch host when
both run the same checkout, so the tag the launch host pulls is exactly the
tag publish produced.
def infra_artifact_version(
init_script: str, role: str, *, repo_root: Path = _REPO_ROOT,
) -> str:
"""Content hash (16 hex) of everything baked into `role`'s infra rootfs: the
whole shipped `bot_bottle` package, that role's Dockerfiles, and its guest
init. Deterministic across the publish host and the launch host when both run
the same checkout, so the tag the launch host pulls is exactly the tag
publish produced.
The package is `COPY bot_bottle /app/bot_bottle`'d wholesale into the image,
so hash *every* regular file under it — not just `*.py`. Non-Python inputs
(e.g. `gateway/egress/entrypoint.sh`, `netpool.defaults.env`) are baked in too, and
a change to one must bump the version or a launch host could boot a stale
rootfs whose code differs from its checkout. `__pycache__`/`.pyc` are the
only exclusions — build artifacts, never copied."""
The package is baked into both images wholesale (orchestrator `COPY`s it,
gateway `pip install`s it), so hash *every* regular file under it — not just
`*.py`. Non-Python inputs (e.g. `gateway/egress/entrypoint.sh`,
`netpool.defaults.env`) are baked in too, and a change to one must bump the
version or a launch host could boot a stale rootfs whose code differs from
its checkout. `__pycache__`/`.pyc` are the only exclusions — build artifacts,
never copied."""
h = hashlib.sha256()
h.update(f"format={_ARTIFACT_FORMAT}\n".encode())
h.update(f"format={_ARTIFACT_FORMAT}\nrole={role}\n".encode())
pkg = repo_root / "bot_bottle"
for path in sorted(pkg.rglob("*")):
if not path.is_file():
@@ -81,7 +100,7 @@ def infra_artifact_version(init_script: str, *, repo_root: Path = _REPO_ROOT) ->
h.update(str(path.relative_to(repo_root)).encode())
h.update(b"\0")
h.update(path.read_bytes())
for name in _DOCKERFILES:
for name in _DOCKERFILES[role]:
h.update(name.encode())
h.update(b"\0")
h.update((repo_root / name).read_bytes())
@@ -107,11 +126,11 @@ def _config() -> tuple[str, str, str]:
return base, owner, token
def artifact_url(version: str, filename: str) -> str:
"""The generic-package download URL for one file of this version's
artifact (`rootfs.ext4.gz` / `rootfs.ext4.gz.sha256`)."""
def artifact_url(version: str, filename: str, *, role: str) -> str:
"""The generic-package download URL for one file of `role`'s artifact at
`version` (`rootfs.ext4.gz` / `rootfs.ext4.gz.sha256`)."""
base, owner, _ = _config()
return f"{base}/api/packages/{owner}/generic/{_PACKAGE}/{version}/{filename}"
return f"{base}/api/packages/{owner}/generic/{_package(role)}/{version}/{filename}"
_GZ_NAME = "rootfs.ext4.gz"
@@ -119,8 +138,8 @@ _SHA_NAME = "rootfs.ext4.gz.sha256"
_CANDIDATE_DIR_ENV = "BOT_BOTTLE_INFRA_ARTIFACT_DIR"
def _cache_root(version: str) -> Path:
return util.cache_dir() / "infra-artifact" / version
def _cache_root(version: str, role: str) -> Path:
return util.cache_dir() / "infra-artifact" / role / version
def _open(url: str) -> urllib.request.Request:
@@ -162,13 +181,17 @@ def _sha256_file(path: Path) -> str:
return h.hexdigest()
def ensure_artifact_gz(version: str) -> Path:
"""The verified, cached `rootfs.ext4.gz` for `version` — downloading it (and
its `.sha256`) once, then reusing it. Fail-closed on a checksum mismatch:
the partial is removed and we die rather than boot an unverified rootfs."""
def ensure_artifact_gz(version: str, *, role: str) -> Path:
"""The verified, cached `rootfs.ext4.gz` for `role` at `version` —
downloading it (and its `.sha256`) once, then reusing it. Fail-closed on a
checksum mismatch: the partial is removed and we die rather than boot an
unverified rootfs.
A pre-staged candidate bundle (`BOT_BOTTLE_INFRA_ARTIFACT_DIR`) holds each
role under its own `<dir>/<role>/` subdir."""
candidate_dir = os.environ.get(_CANDIDATE_DIR_ENV, "").strip()
if candidate_dir:
root = Path(candidate_dir)
root = Path(candidate_dir) / role
version_file = root / "version.txt"
# Guard the read so a missing version.txt is a clean error, not a raw
# FileNotFoundError.
@@ -177,7 +200,7 @@ def ensure_artifact_gz(version: str) -> Path:
declared = version_file.read_text(encoding="utf-8").strip()
if declared != version:
die(
f"infra candidate version mismatch: expected {version}, "
f"infra candidate version mismatch ({role}): expected {version}, "
f"bundle contains {declared or '<empty>'}"
)
gz = root / _GZ_NAME
@@ -188,22 +211,22 @@ def ensure_artifact_gz(version: str) -> Path:
actual = _sha256_file(gz)
if actual != expected:
die(
f"infra candidate checksum mismatch for {version}:\n"
f"infra candidate checksum mismatch ({role}) for {version}:\n"
f" expected {expected}\n actual {actual}"
)
return gz
root = _cache_root(version)
root = _cache_root(version, role)
root.mkdir(parents=True, exist_ok=True)
gz = root / _GZ_NAME
ok = root / ".verified"
if gz.is_file() and ok.is_file():
return gz
info(f"pulling infra rootfs artifact {_PACKAGE}/{version}")
_download(artifact_url(version, _GZ_NAME), gz)
info(f"pulling infra rootfs artifact {_package(role)}/{version}")
_download(artifact_url(version, _GZ_NAME, role=role), gz)
sha = root / _SHA_NAME
_download(artifact_url(version, _SHA_NAME), sha)
_download(artifact_url(version, _SHA_NAME, role=role), sha)
expected = sha.read_text().split()[0].strip().lower()
actual = _sha256_file(gz)
@@ -211,7 +234,7 @@ def ensure_artifact_gz(version: str) -> Path:
gz.unlink(missing_ok=True)
sha.unlink(missing_ok=True)
die(
f"infra artifact checksum mismatch for {version}:\n"
f"infra artifact checksum mismatch ({role}) for {version}:\n"
f" expected {expected}\n"
f" actual {actual}\n"
f" refusing to boot an unverified rootfs."
@@ -220,13 +243,13 @@ def ensure_artifact_gz(version: str) -> Path:
return gz
def materialize_ext4(version: str, dest: Path) -> None:
"""Ensure the verified artifact is cached, then gunzip it to `dest` — a
fresh, writable per-boot rootfs (the VM mutates it; the cached `.gz` stays
def materialize_ext4(version: str, dest: Path, *, role: str) -> None:
"""Ensure the verified `role` artifact is cached, then gunzip it to `dest` —
a fresh, writable per-boot rootfs (the VM mutates it; the cached `.gz` stays
pristine). Atomic via a `.part` sibling."""
gz = ensure_artifact_gz(version)
gz = ensure_artifact_gz(version, role=role)
tmp = dest.with_suffix(dest.suffix + ".part")
info(f"expanding infra rootfs -> {dest}")
info(f"expanding {role} infra rootfs -> {dest}")
with gzip.open(gz, "rb") as src, open(tmp, "wb") as out:
shutil.copyfileobj(src, out, _CHUNK)
tmp.replace(dest)