feat(firecracker): pull the infra rootfs as a prebuilt artifact (PRD 0069 Stage 2) #395
Reference in New Issue
Block a user
Delete Branch "nix-fixed-images-stage2"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
What
Stage 2 of the docker-free Firecracker backend (#348, PRD 0069): the launch host stops building the fixed infra image with Docker. Instead the infra VM's rootfs is prebuilt off-host and pulled as a ready-to-boot ext4 artifact — the launch host does
HTTP GET → sha256 verify → gunzip → boot, no Docker, no OCI/rootfs tooling.This works because the infra rootfs is already host- and bottle-agnostic:
authorized_keysand the guest IP ride the kernel cmdline, not the rootfs, so one published ext4 boots on any host.Approach (decisions made during design)
…/api/packages/<owner>/generic/bot-bottle-infra/<version>/rootfs.ext4.gz(+.sha256). Release attachments were ruled out — this instance caps them at 100 MB with a filetype allowlist that excludes.ext4.bot_bottlepackage + the three Dockerfiles + the init), so a launch host pulls the artifact matching its code and a content change can't silently boot a stale rootfs. Checksum mismatch fails closed.Pieces
firecracker/infra_artifact.py— version hash, pull + verify + gunzip, config (BOT_BOTTLE_INFRA_ARTIFACT_BASE/OWNER/TOKEN, default this Gitea; reuses the shared Gitea token).firecracker/infra_vm.py—ensure_built/bootdefault to pull;BOT_BOTTLE_INFRA_BUILD=localkeeps the docker build-from-source path for iterating on Dockerfiles. Docker-build trio extracted tobuild_infra_images_with_docker().firecracker/publish_infra.py— the off-host half:python3 -m bot_bottle.backend.firecracker.publish_infra [--dry-run|--force]builds the images with Docker, mke2fs's the rootfs (with the buildah build slack), gzips, and PUTs it to the generic package.test_infra_artifact.py(version determinism/sensitivity, download+verify, checksum fail-closed, 404→publish pointer, config overrides) + updatedtest_firecracker_infra_vm.py(default pulls without Docker; local mode still builds deps-before-infra).⚠️ Rollout (not done in this PR)
write:package(the generic/container package API returns 403 today).python3 -m …publish_infra) — until then the default pull path 404s;BOT_BOTTLE_INFRA_BUILD=localpreserves today's behavior.Freeze/migrate's remaining host-Docker use is a separate PR (as agreed).
🤖 Generated with Claude Code
https://claude.ai/code/session_01UoEZHDjv84ChoZbozQERhJ
Two blocking findings in the artifact identity and publishing path. The focused unit suite passes (22 tests), but it does not exercise these cases.
@@ -0,0 +66,4 @@h = hashlib.sha256()h.update(f"format={_ARTIFACT_FORMAT}\n".encode())pkg = _REPO_ROOT / "bot_bottle"for path in sorted(pkg.rglob("*.py")):[P1] Include every copied runtime file in the artifact version. This loop hashes only
*.py, butDockerfile.gatewayalso copiesbot_bottle/egress_entrypoint.shinto/app/egress-entrypoint.sh, andDockerfile.orchestratorcopies the package directory wholesale. A change to that shell entrypoint therefore leaves the version unchanged, so a launch host can fetch and boot an older rootfs even though its checkout contains different runtime code. Hash the actual Docker build inputs (respecting.dockerignore) or at minimum all files explicitly copied into these images, and add a regression test showing that changing the shell script changes the version.@@ -0,0 +127,4 @@if args.force:_delete(gz_url, token)_delete(sha_url, token)_put(gz_url, gz.read_bytes(), token)[P1] Stream the rootfs upload instead of reading it into memory.
gz.read_bytes()materializes the entire compressed ext4 in Python beforeurlopensends it. This image is built with 8 GiB of writable slack and is explicitly expected to be hundreds of MB, so publishing can require another artifact-sized allocation and OOM a modest CI/build host._putshould accept a path/file and stream it (with an explicit content length if Gitea requires one); the checksum file can remain a small in-memory upload.@@ -0,0 +54,4 @@def local_build_requested() -> bool:"""True when the operator opted into the dev Docker-build path instead ofpulling the published artifact (`BOT_BOTTLE_INFRA_BUILD=local`)."""return os.environ.get("BOT_BOTTLE_INFRA_BUILD", "").strip().lower() == "local"If this isn't set, should fallback to a config setting from the sqlite db... that will I think require having the host do a request to the orchestrator via HTTP, since the db now lives in the infra vm. Evaluate the merits of that decision/help me determine whether it's worth it.
I don’t think an orchestrator/SQLite fallback is worth adding for
BOT_BOTTLE_INFRA_BUILD. This setting is needed while deciding how to obtain the rootfs, before the infra VM can be booted; on a cold start there is no orchestrator to query until after that decision succeeds, so an HTTP fallback creates a bootstrap cycle. It would also move a host bootstrap choice into the runtime-state database, while the registry module explicitly says declarative config stays under~/.bot-bottle/.My recommendation: keep the environment variable as the immediate override in this PR. If we want a persistent default later, read it from a small host-side declarative config file (env > host config >
pull), not from the infra VM database. That works for first boot, recovery when the VM is unhealthy, and offline/local-build operation without introducing a new unauthenticated pre-boot control-plane endpoint.@@ -0,0 +54,4 @@def local_build_requested() -> bool:"""True when the operator opted into the dev Docker-build path instead ofpulling the published artifact (`BOT_BOTTLE_INFRA_BUILD=local`)."""return os.environ.get("BOT_BOTTLE_INFRA_BUILD", "").strip().lower() == "local"I don’t think an orchestrator/SQLite fallback is worth adding for
BOT_BOTTLE_INFRA_BUILD. This setting is needed while deciding how to obtain the rootfs, before the infra VM can be booted; on a cold start there is no orchestrator to query until after that decision succeeds, so an HTTP fallback creates a bootstrap cycle. It would also move a host bootstrap choice into the runtime-state database, while the registry module explicitly says declarative config stays under~/.bot-bottle/.My recommendation: keep the environment variable as the immediate override in this PR. If we want a persistent default later, read it from a small host-side declarative config file (env > host config >
pull), not from the infra VM database. That works for first boot, recovery when the VM is unhealthy, and offline/local-build operation without introducing a new unauthenticated pre-boot control-plane endpoint.@didericis-codex duh, right, good call. My bad. That said, it's unfortunate how the db can't be mounted on the host. A host mounted db would be a great way to centralize both state and config, think it'd be ideal. Env vars work fine for now, but it'd be a bit more versatile to drive this from a db. Would also be cool if we could have a global edit panel and view into this from the console. That way the console can drive headless servers more easily/without needing to go onto the host.
@@ -0,0 +54,4 @@def local_build_requested() -> bool:"""True when the operator opted into the dev Docker-build path instead ofpulling the published artifact (`BOT_BOTTLE_INFRA_BUILD=local`)."""return os.environ.get("BOT_BOTTLE_INFRA_BUILD", "").strip().lower() == "local"Agreed on the product goal: the console should have a global settings view and be able to manage a headless host without SSH. I’d separate that from making the SQLite file host-mounted, though. The registry ext4 is a block device attached read/write to the VM; mounting it concurrently on the host would give ext4 two independent writers and risks filesystem/DB corruption. Moving SQLite onto a shared filesystem also makes its locking/durability semantics the hard part.
A cleaner split seems to be:
pullvslocal, artifact registry location/auth) in host-owned config/env.That preserves the centralized console experience for nearly everything while keeping the unavoidable pre-boot layer explicit and small. I’d treat the config API/console panel as a follow-up rather than expand this artifact PR.
a73aba935fto8d8a88aeeb@didericis-claude address the findings of the review
Addressed both blocking findings in
f2891a1.[P1] Artifact version now hashes every baked-in file, not just
*.py. You're right —Dockerfile.orchestratorCOPY bot_bottle's the whole package, soegress_entrypoint.sh(andnetpool.defaults.env, the contrib Dockerfiles) are baked in but didn't move the version.infra_artifact_versionnow hashes every regular file underbot_bottle/(excluding only__pycache__/.pyc, which are never copied) plus the three Dockerfiles and the init. Regression tests added: changing the shell entrypoint bumps the version; adding.pyc/__pycache__does not. (infra_artifact_versiongained arepo_rootkwarg so the test can hash a synthetic tree.)[P1] Upload streams from disk.
_putno longer doesgz.read_bytes(); it opens the.ext4.gzand passes the file object as the request body with an explicitContent-Length, sourlopensends it in blocks instead of materializing a second artifact-sized buffer. The tiny.sha256stays in-memory. Test asserts the body is the file object (hasread, notbytes) with the rightContent-Length.On the
BOT_BOTTLE_INFRA_BUILD/ config-DB discussion (thanks for the detailed reasoning): agreed, and I've kept it as a plain env override in this PR — no orchestrator/SQLite fallback, no pre-boot control-plane endpoint. The "bootstrap settings in host-owned config vs. global runtime config in the orchestrator DB" split sounds right as a follow-up, out of scope here.Also note: the full publish pipeline is now validated end-to-end via
--dry-run— builds all three images, exports,mke2fs, gzips to a 195 MBrootfs.ext4.gz; only the uploadPUTs remain (pending your go-ahead to actually publish).@didericis-claude go ahead with the actual publish
The two blocking findings are resolved in
f2891a1. The artifact identity now covers every regular file copied into the image (while excluding generated bytecode), and the large rootfs upload is streamed from an open file with an explicit Content-Length. Focused tests pass (26/26), the full unit suite passes (1779/1779), and diff --check is clean.