feat(firecracker): pull the infra rootfs as a prebuilt artifact (PRD 0069 Stage 2)
Stage 2 of the docker-free Firecracker backend (#348): stop building the fixed infra image on the launch host. The infra VM's rootfs is host- and bottle-agnostic (authorized_keys + guest IP ride the kernel cmdline, not the rootfs), so it's built once off-host and published as a versioned, ready-to- boot ext4; the launch host downloads + verifies + boots it — no Docker, no image tooling, just HTTP + gunzip. - infra_artifact.py: version = content hash of the rootfs inputs (the shipped bot_bottle package + the three Dockerfiles + the init), so a launch host pulls the artifact matching its code and a content change can't silently boot a stale rootfs. Pull + sha256-verify (fail-closed) + gunzip from a Gitea generic package; base/owner/token configurable, default this Gitea. - infra_vm.ensure_built/boot default to the pull path; BOT_BOTTLE_INFRA_BUILD= local keeps the docker build-from-source path for iterating on Dockerfiles. - publish_infra.py: the off-host half — builds the images with Docker, mke2fs the rootfs (with buildah slack), gzips, and PUTs it to the generic package. Rollout note: default=pull means a launch 404s until an artifact is published; until the Gitea packages endpoint is enabled + an artifact published, use BOT_BOTTLE_INFRA_BUILD=local. Freeze/migrate's remaining docker use is a separate PR. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UoEZHDjv84ChoZbozQERhJ
This commit is contained in:
@@ -8,16 +8,20 @@
|
||||
> **Superseded in part by [PRD 0070](0070-per-host-orchestrator.md) (#351):**
|
||||
> the sidecar-consolidation framing here (Stage 1, per-host sidecar; Stage 4,
|
||||
> sidecar-as-VM) is taken over by 0070's per-host orchestrator. This PRD still
|
||||
> owns the docker-free **image-building** work — Stage 2 (nix-built fixed
|
||||
> images, a dependency of 0070) and Stage 3 (in-VM Dockerfile builder).
|
||||
> owns the docker-free **image-provisioning** work — Stage 2 (pull the fixed
|
||||
> images from an OCI registry instead of building them with host Docker, a
|
||||
> dependency of 0070) and Stage 3 (in-VM Dockerfile builder).
|
||||
|
||||
## Summary
|
||||
|
||||
Make the Firecracker backend depend on **firecracker + KVM only**, removing
|
||||
Docker from the host. Two moves get us there: run the **sidecar bundle as a
|
||||
persistent, per-host service** (eventually a Firecracker VM) instead of a
|
||||
per-bottle container, and **build agent rootfs images without a host Docker
|
||||
daemon** (nix for the fixed images; an in-VM builder for user Dockerfiles).
|
||||
per-bottle container, and **provision rootfs images without a host Docker
|
||||
daemon** — pull the fixed images (orchestrator/gateway/infra) from an OCI
|
||||
registry and unpack them daemonlessly, and build user Dockerfiles in an in-VM
|
||||
builder. The images are still *built* with Docker, but off the launch host
|
||||
(CI / a publish step) and pushed to the registry; the launch host only pulls.
|
||||
|
||||
## Motivation
|
||||
|
||||
@@ -93,13 +97,51 @@ torn down at exit."
|
||||
Can ship as a container first (quick resource/ops win) and become a VM in
|
||||
Stage 4.
|
||||
|
||||
### Stage 2 — Fixed images built with nix (no Docker)
|
||||
### Stage 2 — Fixed rootfs prebuilt + pulled as an artifact (no host Docker)
|
||||
|
||||
The images bot-bottle *ships* — the sidecar, the agent base, and the builder
|
||||
(Stage 3) — are built declaratively with nix (`nixos-generators` /
|
||||
`make-ext4-fs` / `pkgs.dockerTools` for the rootfs), producing an ext4 or
|
||||
tar with correct ownership. Removes Docker for everything we own and gives
|
||||
the rootless-rootfs correctness (#347) for free on these images.
|
||||
The one fixed image the Firecracker backend needs at launch — the combined
|
||||
**infra** rootfs the infra VM boots (orchestrator control plane + gateway +
|
||||
buildah, with the control-plane init as PID 1) — is **prebuilt end-to-end off
|
||||
the launch host and published as a versioned, ready-to-boot ext4 artifact**.
|
||||
The launch host **downloads the `.ext4` and boots it directly** — no
|
||||
`docker build`, no `docker export`, no `mke2fs`, no image tooling at all.
|
||||
|
||||
This is possible because the infra rootfs is already **host- and
|
||||
bottle-agnostic**: the per-boot bits (authorized_keys, guest IP) arrive on the
|
||||
**kernel cmdline**, not in the rootfs (see `build_base_rootfs_dir`). So one
|
||||
published ext4 boots on any launch host.
|
||||
|
||||
- **Artifact.** `rootfs.ext4` + a `rootfs.ext4.sha256`, published as a Gitea
|
||||
**generic package** (`bot-bottle-infra/<tag>`) — generic packages take
|
||||
arbitrary large binaries (no attachment size cap / file-type allowlist that
|
||||
release attachments impose). The matching `vmlinux` kernel can ship the same
|
||||
way, so the whole VM is fetchable.
|
||||
- **Pull.** The launch host `GET`s
|
||||
`…/api/packages/<owner>/generic/bot-bottle-infra/<tag>/rootfs.ext4` (+
|
||||
`.sha256`) for its pinned tag, verifies the checksum, caches it under the
|
||||
tag, and attaches it as the infra VM's root disk. Host prerequisite is an
|
||||
HTTP client — nothing else. Public packages need no auth to pull; a token
|
||||
with `read:package` covers a private instance.
|
||||
- **Registry.** The artifact base URL + owner are configurable, defaulting to
|
||||
this deployment's Gitea (`https://gitea.dideric.is` / `didericis`);
|
||||
overridable via env for other deployments / air-gapped mirrors.
|
||||
- **Versioning.** A pinned tag bumped when the infra rootfs contents change
|
||||
(bot_bottle's shipped files, the base deps, or the init), so a launch host
|
||||
pulls the artifact matching its code and a content change can't silently
|
||||
boot a stale rootfs. A checksum mismatch fails closed.
|
||||
- **Publish.** A `publish` step (CLI subcommand / CI job) runs the full
|
||||
pipeline **on a build/CI host** — `docker build` the three Dockerfiles →
|
||||
export → inject guest boot → `mke2fs` → upload the `.ext4` + `.sha256`.
|
||||
Building still uses Docker, but never on the launch/runner host, which is
|
||||
the one #348 needs unprivileged.
|
||||
- **Dev escape hatch.** An explicit opt-in still builds the rootfs locally
|
||||
with Docker (for iterating on the Dockerfiles without a publish
|
||||
round-trip); it is never the default path.
|
||||
|
||||
Removes Docker from the launch host entirely for the fixed image, and the
|
||||
launch host needs no OCI/rootfs tooling — just fetch + boot. The build-time
|
||||
cache / build-time-egress open problems a from-scratch build would face don't
|
||||
arise: the launch host never builds, it downloads a finished disk.
|
||||
|
||||
### Stage 3 — User Dockerfiles built in a builder VM (the unlock)
|
||||
|
||||
|
||||
Reference in New Issue
Block a user