Compare commits

..

11 Commits

Author SHA1 Message Date
didericis dfc693e0b6 fix(firecracker): keep the snapshot partial private even if one was left behind
test / unit (pull_request) Successful in 1m9s
test / integration (pull_request) Successful in 21s
test / coverage (pull_request) Successful in 1m14s
lint / lint (push) Successful in 2m21s
test / unit (push) Successful in 1m18s
test / integration (push) Successful in 30s
test / coverage (push) Successful in 1m27s
Update Quality Badges / update-badges (push) Successful in 1m17s
Follow-up to the codex review on #398. os.open's mode arg only applies on
creation, so a committed-rootfs.tar.partial left 0644 by an interrupted run
would be opened/truncated (not re-moded) and stay world-readable for the
whole SSH stream. Unlink any leftover and exclusively recreate it
(O_EXCL|O_NOFOLLOW), then fchmod 0600 immediately so umask can't loosen it.

Test pre-creates a 0644 partial and asserts the fd is 0600 mid-stream.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UoEZHDjv84ChoZbozQERhJ
2026-07-17 01:02:19 -04:00
didericis 39d47b8108 fix(firecracker): harden committed-snapshot resume against guest-controlled data
lint / lint (push) Successful in 2m14s
test / unit (pull_request) Successful in 1m7s
test / integration (pull_request) Successful in 23s
test / coverage (pull_request) Successful in 1m17s
Address the codex review on #398:

- P1: inject_guest_boot no longer follows a symlink at bb-init/bb-dropbear.
  A committed snapshot is guest-controlled and could plant those paths as
  symlinks aimed at a host file (e.g. bb-init -> ~/.bashrc); write_text /
  copy2 would then overwrite the target as the host user during resume.
  Replace any pre-existing entry and create the files with
  O_EXCL|O_NOFOLLOW so the write stays inside the staging tree.

- P2: write the snapshot tar owner-only (0600). It can contain the bottle's
  private workspace; it was being created world-readable (0644).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UoEZHDjv84ChoZbozQERhJ
2026-07-17 00:44:06 -04:00
didericis d0a0ce8d60 test(firecracker): satisfy pyright strict + line length in committed-rootfs tests
lint / lint (push) Successful in 2m22s
test / unit (pull_request) Successful in 1m20s
test / integration (pull_request) Successful in 28s
test / coverage (pull_request) Successful in 1m20s
Annotate the counting_run subprocess.run wrapper (reportMissingParameterType)
and wrap an over-long patch target line.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UoEZHDjv84ChoZbozQERhJ
2026-07-17 00:36:43 -04:00
didericis 5c08701983 feat(firecracker): port freeze/migrate off host Docker (PRD 0069 / #397)
lint / lint (push) Failing after 2m9s
test / unit (pull_request) Successful in 1m7s
test / integration (pull_request) Successful in 24s
test / coverage (pull_request) Successful in 1m22s
The last host-Docker dependency in the Firecracker launch path. Freeze
and resume no longer touch the docker daemon, so the backend needs
firecracker + KVM only — completing #348.

Freeze: stream the guest rootfs over SSH straight into a persistent
committed-rootfs.tar (the resumable/migratable artifact) instead of
round-tripping through `docker build` from a scratch image. Written to
a .partial sibling and atomically renamed so a failed freeze leaves no
truncated artifact.

Resume: extract the snapshot tar into a cached base dir and feed it to
the existing rootless `mke2fs -d` pipeline, replacing the
`docker create` + `docker export | tar` path. Recreate the
proc/sys/dev/run mount points the freezer excludes so the guest init
can mount them.

`util.build_base_rootfs_dir` / `docker_image_id` stay — they still back
the opt-in BOT_BOTTLE_INFRA_BUILD=local dev path and off-host
publish_infra, which are out of scope.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UoEZHDjv84ChoZbozQERhJ
2026-07-17 00:31:00 -04:00
didericis e3e195f866 chore(firecracker): name the artifact package bot-bottle-firecracker-infra
test / unit (pull_request) Successful in 1m19s
test / integration (pull_request) Successful in 23s
test / coverage (pull_request) Successful in 1m18s
lint / lint (push) Successful in 2m17s
test / unit (push) Successful in 1m18s
test / integration (push) Successful in 32s
test / coverage (push) Successful in 1m21s
Update Quality Badges / update-badges (push) Successful in 1m17s
Rename the Gitea generic package from bot-bottle-infra to
bot-bottle-firecracker-infra so it's self-evident in the package list which
backend it serves (and leaves room for other artifacts, e.g. a shipped
kernel). The version slot stays the content hash — "firecracker" belongs in
the package name, not the version. Docker image / VM names are unchanged.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UoEZHDjv84ChoZbozQERhJ
2026-07-17 00:04:48 -04:00
didericis e3d24b7e41 chore(firecracker): ship an about.txt description with the infra artifact
lint / lint (push) Successful in 2m55s
test / unit (pull_request) Successful in 1m47s
test / integration (pull_request) Successful in 38s
test / coverage (pull_request) Successful in 1m28s
Generic packages have no description field, so publish_infra now uploads a
short about.txt alongside the rootfs on every publish — it's what identifies
the package as the Firecracker backend's infra rootfs on the package page.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UoEZHDjv84ChoZbozQERhJ
2026-07-16 23:59:38 -04:00
didericis f2891a1634 fix(firecracker): hash all baked-in files + stream the artifact upload
lint / lint (push) Successful in 2m26s
test / unit (pull_request) Successful in 1m44s
test / integration (pull_request) Successful in 40s
test / coverage (pull_request) Successful in 1m32s
Address PR #395 review (two P1s):

- Version hash covered only `bot_bottle/**.py`, but the image `COPY`s the
  whole package — non-Python inputs baked in (egress_entrypoint.sh,
  netpool.defaults.env) didn't change the version, so a launch host could
  boot a stale rootfs whose code differs from its checkout. Hash every
  regular file under bot_bottle/ (excluding __pycache__/.pyc). Regression
  tests: a shell-script change bumps the version; .pyc/__pycache__ don't.

- publish_infra `_put` read the whole (hundreds-of-MB) gz into memory via
  read_bytes(). Stream it from disk with an explicit Content-Length; the
  tiny .sha256 stays in-memory. Test asserts the body is the file object,
  not bytes.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UoEZHDjv84ChoZbozQERhJ
2026-07-16 23:46:46 -04:00
didericis 8d8a88aeeb fix(firecracker): read only BOT_BOTTLE_INFRA_ARTIFACT_TOKEN for the artifact
lint / lint (push) Successful in 2m37s
test / unit (pull_request) Successful in 1m25s
test / integration (pull_request) Successful in 31s
test / coverage (pull_request) Successful in 1m31s
Drop the fallback to the general-purpose BOT_BOTTLE_CLAUDE_GITEA_TOKEN so
the artifact pull/publish uses a dedicated, package-scoped token that can
be granted (or revoked) independently of the general Gitea token.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UoEZHDjv84ChoZbozQERhJ
2026-07-16 23:41:53 -04:00
didericis 18f190b7e3 feat(firecracker): pull the infra rootfs as a prebuilt artifact (PRD 0069 Stage 2)
Stage 2 of the docker-free Firecracker backend (#348): stop building the
fixed infra image on the launch host. The infra VM's rootfs is host- and
bottle-agnostic (authorized_keys + guest IP ride the kernel cmdline, not the
rootfs), so it's built once off-host and published as a versioned, ready-to-
boot ext4; the launch host downloads + verifies + boots it — no Docker, no
image tooling, just HTTP + gunzip.

- infra_artifact.py: version = content hash of the rootfs inputs (the shipped
  bot_bottle package + the three Dockerfiles + the init), so a launch host
  pulls the artifact matching its code and a content change can't silently
  boot a stale rootfs. Pull + sha256-verify (fail-closed) + gunzip from a
  Gitea generic package; base/owner/token configurable, default this Gitea.
- infra_vm.ensure_built/boot default to the pull path; BOT_BOTTLE_INFRA_BUILD=
  local keeps the docker build-from-source path for iterating on Dockerfiles.
- publish_infra.py: the off-host half — builds the images with Docker, mke2fs
  the rootfs (with buildah slack), gzips, and PUTs it to the generic package.

Rollout note: default=pull means a launch 404s until an artifact is published;
until the Gitea packages endpoint is enabled + an artifact published, use
BOT_BOTTLE_INFRA_BUILD=local. Freeze/migrate's remaining docker use is a
separate PR.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UoEZHDjv84ChoZbozQERhJ
2026-07-16 23:41:53 -04:00
didericis-claude bbb8913382 refactor(git-gate): centralize hostname qualification in globalize_slug
test / unit (pull_request) Successful in 1m19s
test / integration (pull_request) Successful in 24s
test / coverage (pull_request) Successful in 1m24s
lint / lint (push) Successful in 2m37s
test / unit (push) Successful in 1m27s
test / integration (push) Successful in 35s
test / coverage (push) Successful in 1m36s
Update Quality Badges / update-badges (push) Successful in 1m29s
Adds globalize_slug(slug) to bottle_state alongside bottle_identity.
git_gate_provision now calls globalize_slug(slug) instead of inlining
socket.gethostname(), so the hostname-qualification logic has a single,
named home. Assumes slug is a mint_slug output.

Title format changes from bot-bottle:{host}:{slug}:{name}
to bot-bottle:{host}-{slug}:{name} to match the globalize_slug contract.
2026-07-16 23:07:12 -04:00
didericis-claude 59be808ab1 feat(git-gate): include hostname in deploy key title
Closes #388 (part 1 of 3). Deploy key titles now carry the machine
hostname so keys provisioned on different hosts don't collide with
each other on the forge when a prior bottle was never torn down.

Title format: bot-bottle:<hostname>:<slug>:<repo-name>
2026-07-16 23:07:12 -04:00
13 changed files with 494 additions and 76 deletions
+44 -30
View File
@@ -1,9 +1,12 @@
"""FirecrackerFreezer — snapshot a running microVM to a Docker image.
"""FirecrackerFreezer — snapshot a running microVM to a rootfs tar.
The VM is live and can't be block-copied safely, so — like the macOS
backend — we stream the guest root filesystem out over the control
channel (SSH here) and rebuild an image from it. The bottle keeps
running after the snapshot.
channel (SSH here). Unlike the other backends this needs no Docker: the
tar *is* the resumable artifact. `resume` extracts it and rebuilds a
fresh per-bottle ext4 with `mke2fs -d` (see `util.build_committed_rootfs_dir`
and `launch._build_agent_base`). The bottle keeps running after the
snapshot.
"""
from __future__ import annotations
@@ -11,9 +14,9 @@ from __future__ import annotations
import json
import os
import subprocess
import tempfile
from pathlib import Path
from ...bottle_state import committed_rootfs_path
from ...log import die, info
from .. import ActiveAgent
from ..freeze import Freezer
@@ -30,14 +33,13 @@ class FirecrackerFreezer(Freezer):
if not private_key.is_file() or not guest_ip:
die(f"cannot freeze {agent.slug}: run dir {run_dir} is missing the "
f"SSH key or VM config (is the bottle still running?)")
image_tag = f"bot-bottle-committed-{agent.slug}:latest"
_commit_via_ssh(private_key, guest_ip, image_tag)
info(f"committed {agent.slug} -> {image_tag!r}")
return image_tag
tar_path = committed_rootfs_path(agent.slug)
_commit_rootfs_via_ssh(private_key, guest_ip, tar_path)
info(f"committed {agent.slug} -> {tar_path}")
return str(tar_path)
def _export_hint(self, slug: str, image_ref: str) -> None:
info(f"to export for migration: docker image save {image_ref} "
f"-o {slug}.tar")
info(f"to export for migration: cp {image_ref} {slug}.tar")
def _guest_ip_from_config(config_path: Path) -> str:
@@ -53,24 +55,36 @@ def _guest_ip_from_config(config_path: Path) -> str:
return ""
def _commit_via_ssh(private_key: Path, guest_ip: str, image_tag: str) -> None:
with tempfile.TemporaryDirectory(prefix="bot-bottle-fc-commit.") as tmp:
rootfs_tar = os.path.join(tmp, "rootfs.tar")
ssh = util.ssh_base_argv(private_key, guest_ip)
with open(rootfs_tar, "wb") as tar_out:
result = subprocess.run(
[*ssh, "--", "tar", "--create", "--one-file-system",
"--exclude=./proc", "--exclude=./sys", "--exclude=./dev",
"--exclude=./run", "--file=-", "--directory=/", "."],
stdout=tar_out, stderr=subprocess.PIPE, check=False,
)
if result.returncode != 0:
die(f"ssh tar for {guest_ip} failed: "
f"{(result.stderr or b'').decode().strip() or '<no stderr>'}")
with open(os.path.join(tmp, "Dockerfile"), "w", encoding="utf-8") as f:
f.write("FROM scratch\nADD rootfs.tar /\nUSER node\nWORKDIR /home/node\n")
build = subprocess.run(
["docker", "build", "-t", image_tag, tmp], check=False,
def _commit_rootfs_via_ssh(private_key: Path, guest_ip: str, tar_path: Path) -> None:
"""Stream the guest rootfs out over SSH into `tar_path`. Excludes the
virtual/live mounts (proc/sys/dev/run) — resume recreates those empty
mount points. Written to a `.partial` sibling and renamed on success so
a failed freeze never leaves a truncated artifact in its place."""
tar_path.parent.mkdir(parents=True, exist_ok=True)
partial = tar_path.with_name(tar_path.name + ".partial")
ssh = util.ssh_base_argv(private_key, guest_ip)
# The snapshot can contain the bottle's private workspace, so keep it
# owner-only (0600) for the whole stream. The `os.open` mode only applies
# on *creation*, so unlink any leftover partial (a prior interrupted run
# could have left it world-readable, or something could swap in a symlink
# at this predictable name) and exclusively recreate it — O_EXCL|O_NOFOLLOW
# — then fchmod immediately so umask can't loosen it. Re-assert after the
# rename too (os.replace carries the source mode, but be explicit).
partial.unlink(missing_ok=True)
fd = os.open(
partial, os.O_WRONLY | os.O_CREAT | os.O_EXCL | os.O_NOFOLLOW, 0o600
)
os.fchmod(fd, 0o600)
with os.fdopen(fd, "wb") as tar_out:
result = subprocess.run(
[*ssh, "--", "tar", "--create", "--one-file-system",
"--exclude=./proc", "--exclude=./sys", "--exclude=./dev",
"--exclude=./run", "--file=-", "--directory=/", "."],
stdout=tar_out, stderr=subprocess.PIPE, check=False,
)
if build.returncode != 0:
die(f"docker build for {image_tag!r} failed")
if result.returncode != 0:
partial.unlink(missing_ok=True)
die(f"ssh tar for {guest_ip} failed: "
f"{(result.stderr or b'').decode().strip() or '<no stderr>'}")
os.replace(partial, tar_path)
os.chmod(tar_path, 0o600)
@@ -45,7 +45,7 @@ _DOCKERFILES = ("Dockerfile.orchestrator", "Dockerfile.gateway", "Dockerfile.inf
_DEFAULT_BASE = "https://gitea.dideric.is"
_DEFAULT_OWNER = "didericis"
_PACKAGE = "bot-bottle-infra"
_PACKAGE = "bot-bottle-firecracker-infra"
# Streaming copy chunk for the (hundreds-of-MB) download.
_CHUNK = 1 << 20
@@ -57,24 +57,34 @@ def local_build_requested() -> bool:
return os.environ.get("BOT_BOTTLE_INFRA_BUILD", "").strip().lower() == "local"
def infra_artifact_version(init_script: str) -> str:
def infra_artifact_version(init_script: str, *, repo_root: Path = _REPO_ROOT) -> str:
"""Content hash (16 hex) of everything baked into the infra rootfs: the
whole shipped `bot_bottle` package (the infra image `COPY`s it wholesale),
the three fixed Dockerfiles, and the guest init. Deterministic across the
publish host and the launch host when both run the same checkout, so the
tag the launch host pulls is exactly the tag publish produced."""
whole shipped `bot_bottle` package, the three fixed Dockerfiles, and the
guest init. Deterministic across the publish host and the launch host when
both run the same checkout, so the tag the launch host pulls is exactly the
tag publish produced.
The package is `COPY bot_bottle /app/bot_bottle`'d wholesale into the image,
so hash *every* regular file under it — not just `*.py`. Non-Python inputs
(e.g. `egress_entrypoint.sh`, `netpool.defaults.env`) are baked in too, and
a change to one must bump the version or a launch host could boot a stale
rootfs whose code differs from its checkout. `__pycache__`/`.pyc` are the
only exclusions — build artifacts, never copied."""
h = hashlib.sha256()
h.update(f"format={_ARTIFACT_FORMAT}\n".encode())
pkg = _REPO_ROOT / "bot_bottle"
for path in sorted(pkg.rglob("*.py")):
if "__pycache__" in path.parts:
pkg = repo_root / "bot_bottle"
for path in sorted(pkg.rglob("*")):
if not path.is_file():
continue
h.update(str(path.relative_to(_REPO_ROOT)).encode())
if "__pycache__" in path.parts or path.suffix == ".pyc":
continue
h.update(str(path.relative_to(repo_root)).encode())
h.update(b"\0")
h.update(path.read_bytes())
for name in _DOCKERFILES:
p = _REPO_ROOT / name
h.update(name.encode())
h.update(p.read_bytes())
h.update(b"\0")
h.update((repo_root / name).read_bytes())
h.update(b"init\0")
h.update(init_script.encode())
return h.hexdigest()[:16]
+9 -11
View File
@@ -1,7 +1,8 @@
"""Launch flow for the Firecracker backend (PRD 0070, consolidated).
Per bottle:
1. build the agent image (docker), export it to a cached ext4 rootfs;
1. build the agent rootfs in a builder VM (buildah, no host docker), or
resume a frozen bottle from its committed rootfs tar; cache the ext4;
2. ensure the per-host orchestrator + shared gateway are up;
3. claim a free TAP pool slot (rootless flock);
4. register the bottle on the orchestrator by the VM's guest IP (the
@@ -31,6 +32,7 @@ from typing import Callable, Generator
from ...agent_provider import runtime_for
from ...bottle_state import (
committed_rootfs_path,
egress_state_dir,
git_gate_state_dir,
read_committed_image,
@@ -45,7 +47,6 @@ from ...git_gate import (
)
from ...log import info, warn
from ...supervise import SUPERVISE_PORT
from ..docker import util as docker_mod
from ..docker.egress import EGRESS_PORT
from ..util import AGENT_CA_BUNDLE, AGENT_CA_PATH
from . import firecracker_vm, image_builder, isolation_probe, netpool, util
@@ -210,16 +211,13 @@ def _build_agent_base(
) -> tuple[FirecrackerBottlePlan, Path]:
"""Produce the agent's base rootfs dir. Primary path: build the Dockerfile
inside a Firecracker builder VM (buildah, no host docker), smoke-testing
the image before export. A committed snapshot (freeze/migrate) is still
exported via the host docker path until that is ported too."""
the image before export. A committed snapshot (freeze/migrate) is resumed
directly from the rootfs tar the freezer wrote — no host docker either."""
committed = read_committed_image(plan.slug)
if committed and docker_mod.image_exists(committed):
info(f"using committed image {committed!r}")
plan = dataclasses.replace(
plan,
agent_provision=dataclasses.replace(plan.agent_provision, image=committed),
)
return plan, util.build_base_rootfs_dir(committed)
committed_tar = committed_rootfs_path(plan.slug)
if committed and committed_tar.is_file():
info(f"resuming from committed rootfs {committed_tar}")
return plan, util.build_committed_rootfs_dir(committed_tar)
base = image_builder.build_agent_rootfs_dir(
Path(plan.dockerfile_path),
image_tag=plan.image,
@@ -5,7 +5,7 @@ never on the launch host. It runs the same pipeline the launch host used to run
locally — `docker build` the three fixed images, export to a rootfs dir, inject
the guest boot, `mke2fs` to an ext4 with the buildah build slack — then gzips
the ext4 and PUTs it (plus a `.sha256`) to
`…/api/packages/<owner>/generic/bot-bottle-infra/<version>/`.
`…/api/packages/<owner>/generic/bot-bottle-firecracker-infra/<version>/`.
The `<version>` is `infra_artifact.infra_artifact_version(...)`, the content
hash of the rootfs inputs, so a launch host at the same code checkout resolves
@@ -33,6 +33,18 @@ from . import infra_artifact, infra_vm, util
_CHUNK = 1 << 20
# A human-readable description shipped alongside the artifact — generic packages
# have no description field, so this file *is* the description on the package
# page. Uploaded on every publish so it never goes stale.
_ABOUT_NAME = "about.txt"
_ABOUT_TEXT = (
"bot-bottle infra rootfs for the Firecracker backend (PRD 0069 Stage 2, "
"#348): the per-host infra VM (orchestrator control plane + gateway + "
"buildah). Prebuilt off-host, gzip ext4; the launch host downloads + "
"sha256-verifies + boots it, no host Docker. The version tag is a content "
"hash of the rootfs inputs. Files: rootfs.ext4.gz + rootfs.ext4.gz.sha256.\n"
)
def _gzip(src: Path, dest: Path) -> None:
with open(src, "rb") as fh, gzip.open(dest, "wb") as out:
@@ -47,8 +59,22 @@ def _sha256(path: Path) -> str:
return h.hexdigest()
def _put(url: str, body: bytes, token: str) -> None:
req = urllib.request.Request(url, data=body, method="PUT")
def _put(url: str, body: "bytes | Path", token: str) -> None:
"""PUT `body` (raw bytes, or a Path streamed from disk) to `url`. The rootfs
is hundreds of MB, so it is passed as a Path and streamed — `urlopen` reads
the open file in blocks rather than materializing it in memory (with an
explicit Content-Length, which Gitea requires and which also stops urllib
from `len()`-ing a non-bytes body)."""
handle = None
if isinstance(body, Path):
length = body.stat().st_size
handle = open(body, "rb")
data: object = handle
else:
length = len(body)
data = body
req = urllib.request.Request(url, data=data, method="PUT") # type: ignore[arg-type]
req.add_header("Content-Length", str(length))
if token:
req.add_header("Authorization", f"token {token}")
req.add_header("Content-Type", "application/octet-stream")
@@ -64,6 +90,9 @@ def _put(url: str, body: bytes, token: str) -> None:
raise SystemExit(f"upload failed (HTTP {e.code}): {url}\n{e.read().decode(errors='replace')}")
except urllib.error.URLError as e:
raise SystemExit(f"registry unreachable: {url} ({e.reason})")
finally:
if handle is not None:
handle.close()
def _delete(url: str, token: str) -> None:
@@ -121,14 +150,17 @@ def main(argv: list[str] | None = None) -> int:
version, gz, sha = build_artifact(Path(tmp))
gz_url = infra_artifact.artifact_url(version, gz.name)
sha_url = infra_artifact.artifact_url(version, sha.name)
about_url = infra_artifact.artifact_url(version, _ABOUT_NAME)
if args.dry_run:
print(f"dry-run: would upload -> {gz_url}")
return 0
if args.force:
_delete(gz_url, token)
_delete(sha_url, token)
_put(gz_url, gz.read_bytes(), token)
_put(sha_url, sha.read_bytes(), token)
_delete(about_url, token)
_put(gz_url, gz, token) # streamed from disk (hundreds of MB)
_put(sha_url, sha.read_bytes(), token) # tiny, in-memory is fine
_put(about_url, _ABOUT_TEXT.encode(), token) # package description
print(f"published infra rootfs {version}")
return 0
+72 -6
View File
@@ -14,6 +14,7 @@ and `./cli.py backend setup --backend=firecracker`.
from __future__ import annotations
import hashlib
import os
import platform
import shutil
@@ -212,15 +213,80 @@ def build_base_rootfs_dir(
return base
def build_committed_rootfs_dir(tar_path: Path) -> Path:
"""Prepare a base rootfs dir from a frozen-bottle snapshot tar (the
freeze/resume path — no Docker). Extracts the snapshot, recreates the
virtual mount points the freezer excluded, and injects the guest init +
static dropbear, mirroring `build_base_rootfs_dir` but sourced from a tar
we control rather than a Docker image.
Cached under the rootfs cache, keyed by the tar's size+mtime so a
re-freeze re-extracts but repeated resumes of the same snapshot don't.
Returns the prepared directory (read as the `mke2fs -d` source)."""
st = tar_path.stat()
fingerprint = hashlib.sha256(
f"{tar_path}:{st.st_size}:{st.st_mtime_ns}".encode()
).hexdigest()[:16]
base = cache_dir() / "rootfs" / f"committed-{fingerprint}"
ready = base / ".bb-ready"
if ready.is_file():
return base
if base.exists():
shutil.rmtree(base, ignore_errors=True)
base.mkdir(parents=True)
info(f"extracting committed rootfs {tar_path} -> {base}")
result = subprocess.run(
["tar", "-x", "-f", str(tar_path), "-C", str(base)],
capture_output=True, text=True, check=False,
)
if result.returncode != 0:
die(f"extracting committed rootfs {tar_path} failed: "
f"{result.stderr.strip() or '<no stderr>'}")
# The freezer excludes the live/virtual filesystems from the snapshot;
# recreate them as empty mount points so the guest init can mount
# proc/sys/dev and dropbear has a writable /run.
for mount_point in ("proc", "sys", "dev", "run"):
(base / mount_point).mkdir(mode=0o755, exist_ok=True)
inject_guest_boot(base)
ready.write_text("ok\n")
return base
def inject_guest_boot(rootfs: Path, init_script: str | None = None) -> None:
"""Drop the static dropbear and the PID-1 init into the rootfs.
`init_script` defaults to the SSH-only agent init; the infra VM
passes its own (control plane + gateway) init."""
shutil.copy2(dropbear_path(), rootfs / "bb-dropbear")
os.chmod(rootfs / "bb-dropbear", 0o755)
init = rootfs / "bb-init"
init.write_text(init_script or _GUEST_INIT)
os.chmod(init, 0o755)
passes its own (control plane + gateway) init.
A committed snapshot is guest-controlled, so `bb-dropbear`/`bb-init`
may already exist as symlinks aimed at a host file (e.g. bb-init ->
~/.bashrc). Replace whatever is there and create the files with
O_EXCL|O_NOFOLLOW so the write always lands a fresh regular file in
the staging tree and never follows a planted symlink out of it."""
_write_staged_file(rootfs / "bb-dropbear", dropbear_path().read_bytes())
_write_staged_file(rootfs / "bb-init", (init_script or _GUEST_INIT).encode())
def _write_staged_file(path: Path, data: bytes) -> None:
"""Write `data` to `path` (mode 0755) as a fresh regular file inside a
staging rootfs, replacing any pre-existing entry without following a
symlink at `path`. Fails closed on anything unexpected there."""
if path.is_symlink() or path.exists():
if path.is_dir() and not path.is_symlink():
shutil.rmtree(path)
else:
path.unlink()
fd = os.open(
path, os.O_WRONLY | os.O_CREAT | os.O_EXCL | os.O_NOFOLLOW, 0o755
)
try:
os.write(fd, data)
finally:
os.close(fd)
os.chmod(path, 0o755)
def build_rootfs_ext4(base_dir: Path, out_path: Path, *, slack_mib: int = 1024) -> None:
+21
View File
@@ -31,6 +31,7 @@ from __future__ import annotations
import dataclasses
import json
import secrets
import socket
import string
from dataclasses import dataclass
from pathlib import Path
@@ -43,6 +44,7 @@ from .paths import bot_bottle_root
_STATE_SUBDIR = "state"
_PER_BOTTLE_DOCKERFILE_NAME = "Dockerfile"
_COMMITTED_IMAGE_NAME = "committed-image"
_COMMITTED_ROOTFS_NAME = "committed-rootfs.tar"
_TRANSCRIPT_SUBDIR = "transcript"
# Per-daemon scratch subdirs. PRD 0018 chunk 2: bind-mount sources
# live here so chunk 3's `docker compose up` can find them at stable
@@ -87,6 +89,14 @@ def bottle_identity(agent_name: str) -> str:
return f"{slug}-{suffix}"
def globalize_slug(slug: str) -> str:
"""Return a globally-unique slug qualified with the current hostname.
Assumes slug is a value returned from mint_slug. Use wherever a slug
must be unique across hosts (e.g. deploy-key titles)."""
return f"{socket.gethostname()}-{slug}"
@dataclass(frozen=True)
class BottleMetadata:
"""Persistent record of how a bottle was launched, written at
@@ -191,6 +201,15 @@ def committed_image_path(identity: str) -> Path:
return bottle_state_dir(identity) / _COMMITTED_IMAGE_NAME
def committed_rootfs_path(identity: str) -> Path:
"""Where the Firecracker freezer stores a snapshot of the bottle's
guest rootfs (a plain tar). This is the resumable/migratable artifact
the Firecracker backend boots from — no Docker image involved. The
matching `committed-image` state file records that a snapshot exists
(and its path); `resume` boots from this tar when both are present."""
return bottle_state_dir(identity) / _COMMITTED_ROOTFS_NAME
def write_committed_image(identity: str, image_tag: str) -> Path:
"""Persist the committed image tag for `identity`. The next
`cli.py resume <identity>` will boot from this image instead of
@@ -340,10 +359,12 @@ __all__ = [
"BottleMetadata",
"agent_state_dir",
"bottle_identity",
"globalize_slug",
"bottle_state_dir",
"cleanup_state",
"clear_preserve_marker",
"committed_image_path",
"committed_rootfs_path",
"egress_state_dir",
"git_gate_state_dir",
"is_preserved",
+2 -1
View File
@@ -13,6 +13,7 @@ import dataclasses
from pathlib import Path
from typing import TYPE_CHECKING
from .bottle_state import globalize_slug
from .errors import MissingEnvVarError
from .log import info
from .manifest import ManifestBottle, ManifestGitEntry
@@ -46,7 +47,7 @@ def _provision_dynamic_key(
owner_repo = entry.UpstreamPath
if owner_repo.endswith(".git"):
owner_repo = owner_repo[:-4]
title = f"bot-bottle:{slug}:{entry.Name}"
title = f"bot-bottle:{globalize_slug(slug)}:{entry.Name}"
info(f"provisioning deploy key for git-gate.repos[{entry.Name!r}]")
key_id, private_key_bytes = provisioner.create(owner_repo, title)
@@ -112,12 +112,12 @@ bottle-agnostic**: the per-boot bits (authorized_keys, guest IP) arrive on the
published ext4 boots on any launch host.
- **Artifact.** `rootfs.ext4` + a `rootfs.ext4.sha256`, published as a Gitea
**generic package** (`bot-bottle-infra/<tag>`) — generic packages take
**generic package** (`bot-bottle-firecracker-infra/<tag>`) — generic packages take
arbitrary large binaries (no attachment size cap / file-type allowlist that
release attachments impose). The matching `vmlinux` kernel can ship the same
way, so the whole VM is fetchable.
- **Pull.** The launch host `GET`s
`…/api/packages/<owner>/generic/bot-bottle-infra/<tag>/rootfs.ext4` (+
`…/api/packages/<owner>/generic/bot-bottle-firecracker-infra/<tag>/rootfs.ext4` (+
`.sha256`) for its pinned tag, verifies the checksum, caches it under the
tag, and attaches it as the infra VM's root disk. Host prerequisite is an
HTTP client — nothing else. Public packages need no auth to pull; a token
+10 -6
View File
@@ -205,25 +205,29 @@ class TestFirecrackerFreezer(_FakeHomeMixin, unittest.TestCase):
)
def test_snapshots_running_vm_without_stopping(self):
"""Commit should tar the running guest rootfs over SSH, not stop it."""
"""Commit should tar the running guest rootfs over SSH into the
committed-rootfs artifact (no Docker), not stop the VM."""
slug = "dev-abc12"
self._write_meta(slug)
self._stage_run_dir(slug)
freezer = FirecrackerFreezer()
agent = _make_agent(slug, "firecracker")
with patch("bot_bottle.backend.firecracker.freezer._commit_via_ssh") as mock_commit, \
commit_fn = "bot_bottle.backend.firecracker.freezer._commit_rootfs_via_ssh"
with patch(commit_fn) as mock_commit, \
patch("bot_bottle.backend.freeze.info"), \
patch("bot_bottle.backend.firecracker.freezer.info"):
freezer.commit(agent)
image_tag = f"bot-bottle-committed-{slug}:latest"
tar_path = bottle_state.committed_rootfs_path(slug)
self.assertEqual(1, mock_commit.call_count)
# (private_key, guest_ip, image_tag) — guest_ip parsed from config.
# (private_key, guest_ip, tar_path) — guest_ip parsed from config.
args = mock_commit.call_args.args
self.assertEqual("100.64.0.1", args[1])
self.assertEqual(image_tag, args[2])
self.assertEqual(image_tag, bottle_state.read_committed_image(slug))
self.assertEqual(tar_path, args[2])
# The committed-image state records the artifact path; resume boots
# from the tar rather than a Docker image.
self.assertEqual(str(tar_path), bottle_state.read_committed_image(slug))
self.assertTrue(bottle_state.is_preserved(slug))
+170
View File
@@ -6,10 +6,13 @@ branches. Mock subprocess/os so nothing needs KVM or a live VM.
from __future__ import annotations
import json
import os
import stat
import subprocess
import tempfile
import unittest
from pathlib import Path
from typing import Any
from unittest.mock import patch
from bot_bottle.backend.firecracker import firecracker_vm, freezer, netpool, util
@@ -128,5 +131,172 @@ class TestRequireFirecracker(unittest.TestCase):
util.require_firecracker()
class TestBuildCommittedRootfsDir(unittest.TestCase):
"""Resume prepares the base rootfs dir from the freezer's snapshot tar
with no Docker: extract, recreate the excluded mount points, inject the
guest boot bits."""
def _make_tar(self, tmp: Path) -> Path:
import tarfile
src = tmp / "src"
(src / "home" / "node").mkdir(parents=True)
(src / "home" / "node" / "hello").write_text("hi")
tar_path = tmp / "rootfs.tar"
with tarfile.open(tar_path, "w") as tar:
tar.add(src, arcname=".")
return tar_path
def test_extracts_recreates_mountpoints_and_injects_boot(self):
with tempfile.TemporaryDirectory(prefix="fc-committed.") as d:
tmp = Path(d)
tar_path = self._make_tar(tmp)
# Stand in for the static dropbear that inject_guest_boot copies.
dropbear = tmp / "dropbear"
dropbear.write_text("#!/bin/true\n")
cache = tmp / "cache"
cache.mkdir()
with patch.object(util, "cache_dir", return_value=cache), \
patch.object(util, "dropbear_path", return_value=dropbear), \
patch.object(util, "info"):
base = util.build_committed_rootfs_dir(tar_path)
self.assertEqual("hi", (base / "home" / "node" / "hello").read_text())
for mount_point in ("proc", "sys", "dev", "run"):
self.assertTrue((base / mount_point).is_dir(),
f"missing recreated mount point /{mount_point}")
self.assertTrue((base / "bb-dropbear").is_file())
self.assertTrue((base / "bb-init").is_file())
self.assertTrue((base / ".bb-ready").is_file())
def test_caches_on_repeat_and_reextracts_after_refreeze(self):
with tempfile.TemporaryDirectory(prefix="fc-committed.") as d:
tmp = Path(d)
tar_path = self._make_tar(tmp)
dropbear = tmp / "dropbear"
dropbear.write_text("#!/bin/true\n")
cache = tmp / "cache"
cache.mkdir()
ctx = [
patch.object(util, "cache_dir", return_value=cache),
patch.object(util, "dropbear_path", return_value=dropbear),
patch.object(util, "info"),
]
for c in ctx:
c.start()
self.addCleanup(lambda: [c.stop() for c in ctx])
real_run = subprocess.run
calls = {"n": 0}
def counting_run(argv: list[str], *a: Any, **k: Any) -> Any:
if argv and argv[0] == "tar":
calls["n"] += 1
return real_run(argv, *a, **k)
with patch.object(util.subprocess, "run", side_effect=counting_run):
first = util.build_committed_rootfs_dir(tar_path)
second = util.build_committed_rootfs_dir(tar_path)
self.assertEqual(first, second)
self.assertEqual(1, calls["n"]) # cached — no re-extract
# A re-freeze rewrites the tar; a new size/mtime -> new cache
# key -> re-extract. Force a distinct mtime so the test isn't
# at the mercy of filesystem timestamp granularity.
import tarfile
extra = tmp / "extra"
extra.mkdir()
(extra / "note").write_text("v2")
with tarfile.open(tar_path, "w") as tar:
tar.add(extra, arcname=".")
st = tar_path.stat()
os.utime(tar_path, ns=(st.st_atime_ns, st.st_mtime_ns + 1_000_000_000))
third = util.build_committed_rootfs_dir(tar_path)
self.assertNotEqual(first, third)
self.assertEqual(2, calls["n"])
class TestInjectGuestBootSymlinkSafe(unittest.TestCase):
"""A committed snapshot is guest-controlled: inject_guest_boot must not
follow a planted symlink and overwrite a host file during resume."""
def test_planted_symlink_does_not_escape_staging_tree(self):
with tempfile.TemporaryDirectory(prefix="fc-inject.") as d:
tmp = Path(d)
dropbear = tmp / "dropbear"
dropbear.write_bytes(b"DROPBEAR")
# A host file the malicious snapshot tries to clobber.
victim = tmp / "victim"
victim.write_text("original")
rootfs = tmp / "rootfs"
rootfs.mkdir()
# The snapshot planted bb-init/bb-dropbear as symlinks to it.
(rootfs / "bb-init").symlink_to(victim)
(rootfs / "bb-dropbear").symlink_to(victim)
with patch.object(util, "dropbear_path", return_value=dropbear):
util.inject_guest_boot(rootfs, init_script="#!/bin/sh\nreal\n")
# Host file untouched; the staged paths are fresh regular files.
self.assertEqual("original", victim.read_text())
self.assertFalse((rootfs / "bb-init").is_symlink())
self.assertFalse((rootfs / "bb-dropbear").is_symlink())
self.assertEqual("#!/bin/sh\nreal\n", (rootfs / "bb-init").read_text())
self.assertEqual(b"DROPBEAR", (rootfs / "bb-dropbear").read_bytes())
class TestCommitRootfsPermissions(unittest.TestCase):
"""The snapshot tar can hold the bottle's private workspace, so the
freezer must write it owner-only (0600)."""
def _commit(self, tar_path: Path) -> int:
"""Run _commit_rootfs_via_ssh with a stubbed ssh|tar pipe; return the
mode of the open partial observed mid-stream (from subprocess.run)."""
key = tar_path.parent.parent / "key"
key.write_text("K")
seen: dict[str, int] = {}
def fake_run(argv: list[str], *a: Any, **k: Any) -> Any:
out = k["stdout"]
seen["mode"] = stat.S_IMODE(os.fstat(out.fileno()).st_mode)
out.write(b"TARDATA")
return subprocess.CompletedProcess(argv, 0, b"", b"")
with patch.object(freezer.util, "ssh_base_argv", return_value=["ssh"]), \
patch.object(freezer.subprocess, "run", side_effect=fake_run):
freezer._commit_rootfs_via_ssh(key, "10.0.0.1", tar_path)
return seen["mode"]
def test_snapshot_created_owner_only(self):
with tempfile.TemporaryDirectory(prefix="fc-freeze.") as d:
tar_path = Path(d) / "state" / "committed-rootfs.tar"
tar_path.parent.mkdir()
stream_mode = self._commit(tar_path)
self.assertEqual(0o600, stream_mode) # private during the stream
self.assertEqual(b"TARDATA", tar_path.read_bytes())
self.assertEqual(0o600, stat.S_IMODE(tar_path.stat().st_mode))
def test_leftover_world_readable_partial_is_recreated_private(self):
"""A partial left 0644 by an interrupted prior run must not keep the
new snapshot world-readable while it streams."""
with tempfile.TemporaryDirectory(prefix="fc-freeze.") as d:
tar_path = Path(d) / "state" / "committed-rootfs.tar"
tar_path.parent.mkdir()
partial = tar_path.with_name(tar_path.name + ".partial")
partial.write_bytes(b"stale")
os.chmod(partial, 0o644)
stream_mode = self._commit(tar_path)
self.assertEqual(0o600, stream_mode)
self.assertEqual(b"TARDATA", tar_path.read_bytes())
self.assertEqual(0o600, stat.S_IMODE(tar_path.stat().st_mode))
if __name__ == "__main__":
unittest.main()
+4 -2
View File
@@ -6,6 +6,7 @@ Covers the pure `git_gate_render_gitconfig` renderer and the dynamic
from __future__ import annotations
import socket
import tempfile
import types
import unittest
@@ -126,8 +127,9 @@ class TestProvisionDynamicKey(unittest.TestCase):
self.assertEqual(b"PRIVATE-KEY-BYTES", key_file.read_bytes())
id_file = Path(d) / "repo-deploy-key-id"
self.assertEqual("kid123", id_file.read_text())
# owner_repo had .git stripped; title carries slug + name
self.assertEqual([("o/r", "bot-bottle:myslug:repo")], fake.created)
# owner_repo had .git stripped; title carries globalize_slug(slug) + name
hostname = socket.gethostname()
self.assertEqual([("o/r", f"bot-bottle:{hostname}-myslug:repo")], fake.created)
def test_missing_token_raises(self) -> None:
with tempfile.TemporaryDirectory() as d, \
+39 -1
View File
@@ -79,6 +79,44 @@ class TestVersion(unittest.TestCase):
ia.infra_artifact_version("a"), ia.infra_artifact_version("b"))
class TestVersionInputs(unittest.TestCase):
"""The hash must cover *every* file baked into the rootfs, not just `*.py`
(`COPY bot_bottle` is wholesale) else a non-Python change (e.g. the egress
entrypoint shell script) leaves the version unchanged and a launch host
boots a rootfs whose code differs from its checkout."""
def _fake_repo(self, root: Path) -> None:
pkg = root / "bot_bottle"
pkg.mkdir()
(pkg / "app.py").write_text("print('hi')\n")
(pkg / "egress_entrypoint.sh").write_text("#!/bin/sh\nexec mitmdump\n")
(pkg / "netpool.defaults.env").write_text("FOO=1\n")
for name in ("Dockerfile.orchestrator", "Dockerfile.gateway", "Dockerfile.infra"):
(root / name).write_text(f"FROM scratch # {name}\n")
def test_non_python_file_change_bumps_version(self) -> None:
with tempfile.TemporaryDirectory() as d:
root = Path(d)
self._fake_repo(root)
before = ia.infra_artifact_version("init", repo_root=root)
(root / "bot_bottle" / "egress_entrypoint.sh").write_text(
"#!/bin/sh\nexec mitmdump --different\n")
after = ia.infra_artifact_version("init", repo_root=root)
self.assertNotEqual(before, after)
def test_pyc_and_pycache_ignored(self) -> None:
with tempfile.TemporaryDirectory() as d:
root = Path(d)
self._fake_repo(root)
before = ia.infra_artifact_version("init", repo_root=root)
cache = root / "bot_bottle" / "__pycache__"
cache.mkdir()
(cache / "app.cpython-312.pyc").write_bytes(b"\x00bytecode")
(root / "bot_bottle" / "app.pyc").write_bytes(b"\x00bytecode")
after = ia.infra_artifact_version("init", repo_root=root)
self.assertEqual(before, after)
class TestEnsureArtifact(_CacheMixin):
def test_downloads_verifies_and_caches(self) -> None:
version = "deadbeef00000000"
@@ -131,7 +169,7 @@ class TestConfig(unittest.TestCase):
url = ia.artifact_url("v1", "rootfs.ext4.gz")
self.assertEqual(
"https://mirror.example/api/packages/acme/generic/"
"bot-bottle-infra/v1/rootfs.ext4.gz", url)
"bot-bottle-firecracker-infra/v1/rootfs.ext4.gz", url)
def test_local_build_flag(self) -> None:
with mock.patch.dict(os.environ, {"BOT_BOTTLE_INFRA_BUILD": "local"}):
+62
View File
@@ -0,0 +1,62 @@
"""Unit: the infra-artifact publisher's upload path (PRD 0069 Stage 2).
The rootfs is hundreds of MB, so `_put` must stream it from disk rather than
read it into memory. Network is mocked; no Docker, no real build.
"""
from __future__ import annotations
import tempfile
import unittest
import urllib.request
from pathlib import Path
from unittest import mock
from bot_bottle.backend.firecracker import publish_infra as pub
class _Resp:
status = 201
def __enter__(self) -> "_Resp":
return self
def __exit__(self, *a: object) -> bool:
return False
class TestPut(unittest.TestCase):
def test_streams_file_body_with_content_length(self) -> None:
captured: list[urllib.request.Request] = []
def fake_urlopen(req: urllib.request.Request, *a: object, **k: object) -> _Resp:
captured.append(req)
return _Resp()
with tempfile.TemporaryDirectory() as d:
f = Path(d) / "rootfs.ext4.gz"
payload = b"x" * 4096
f.write_bytes(payload)
with mock.patch.object(pub.urllib.request, "urlopen", fake_urlopen):
pub._put("https://reg/pkg", f, token="t")
req = captured[0]
# Body is the open file object (streamed), never the bytes in memory.
self.assertTrue(hasattr(req.data, "read"))
self.assertNotIsInstance(req.data, (bytes, bytearray))
self.assertEqual(str(len(payload)), req.get_header("Content-length"))
def test_small_bytes_body_still_works(self) -> None:
captured: list[urllib.request.Request] = []
def fake_urlopen(req: urllib.request.Request, *a: object, **k: object) -> _Resp:
captured.append(req)
return _Resp()
with mock.patch.object(pub.urllib.request, "urlopen", fake_urlopen):
pub._put("https://reg/sha", b"abc123 rootfs\n", token="")
self.assertEqual(b"abc123 rootfs\n", captured[0].data)
if __name__ == "__main__":
unittest.main()