refactor(firecracker): single infra VM builds too — buildah in one image (PR #354 review)

Addresses the review finding that buildah lived only in the orchestrator
image, so the persistent infra VM wasn't the builder — a separate throwaway
builder VM contended with it for the orchestrator TAP. Consolidate:

- **Rebase the gateway (and thus infra) on `python:3.12-slim` = Debian
  trixie**, pip-installing mitmproxy instead of `FROM mitmproxy/mitmproxy`
  (Debian bookworm). trixie ships buildah 1.39, which can build agent
  Dockerfiles that use heredocs; bookworm's 1.28 can't (`Unknown
  instruction: "{"`). CA path is unchanged (set via `--set confdir=`).
- **buildah lives only in `Dockerfile.infra`** now (removed from the
  orchestrator image, which is lean/stdlib-only again).
- **Shared orchestrator content**: `Dockerfile.orchestrator` is the single
  definition of the control-plane payload; the infra image `COPY --from`s
  it (same trixie base → clean copy, and future deps like iroh are added
  once). The docker backend runs the orchestrator image directly.
- **`image_builder` builds inside the infra VM** (which now has buildah)
  over SSH — no throwaway builder VM, so the `bborch0` contention is gone.
  `ensure_built` builds orchestrator + gateway before infra (FROM gateway,
  COPY --from orchestrator).

Verified on a KVM host: images build (buildah 1.39 in infra), the agent
image builds *inside* the infra VM (heredoc Dockerfile and all), the infra
VM stays healthy, the agent boots and `claude --version` = 2.1.172. The
rebased gateway still starts as a docker container and generates its CA
(docker backend unaffected).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WBMWTEtQdJ4W5UrWuLHCck

Dockerfile.gateway

@@ -39,28 +39,32 @@
 # with Dockerfile.git-gate's prior base (now deleted at chunk 3).
 FROM zricethezav/gitleaks@sha256:c00b6bd0aeb3071cbcb79009cb16a60dd9e0a7c60e2be9ab65d25e6bc8abbb7f AS gitleaks-src
 
-# Stage 2: assembly. mitmproxy/mitmproxy is debian-slim-based with
-# Python + mitmdump pre-installed — heavier than the others, so
-# this stage starts there and pulls the standalone binaries in.
-FROM mitmproxy/mitmproxy:11.1.3
-
-# Run as root inside the bundle. The bundle is the isolation
-# boundary; per-daemon user separation inside it is not load-bearing
-# and complicates the supervisor's spawn path.
-USER root
+# Stage 2: assembly. Based on `python:3.12-slim` (Debian trixie) rather
+# than the `mitmproxy/mitmproxy` image (Debian bookworm) so the whole
+# stack — gateway here, and the firecracker infra image that builds
+# FROM this — lands on trixie, whose buildah (1.39) can build agent
+# Dockerfiles that use heredocs. mitmproxy is pip-installed to the same
+# effect as the upstream image. (bookworm's buildah is 1.28, which can't
+# parse `RUN ... <<EOF`; see the infra image + PR discussion.)
+FROM python:3.12-slim
 
 # Runtime system deps:
 #   git supplies the `git daemon` subcommand (no separate package)
 #     plus the core `git` binary the pre-receive hook invokes.
 #   openssh-client supplies the upstream SSH transport the
 #     pre-receive hook uses to forward accepted refs.
-#   ca-certificates is needed for mitmdump upstream TLS (the
-#     base image already has it; listed for explicitness).
+#   ca-certificates is needed for mitmdump upstream TLS.
 RUN apt-get update \
  && apt-get install -y --no-install-recommends \
       git openssh-client ca-certificates \
  && rm -rf /var/lib/apt/lists/*
 
+# mitmdump (the egress data plane). The upstream mitmproxy image baked
+# this in; on the plain python base we pip-install the same pinned
+# version. Its CA dir is set explicitly via `--set confdir=` in
+# egress-entrypoint.sh, so it doesn't depend on a `mitmproxy` home user.
+RUN pip install --no-cache-dir mitmproxy==11.1.3
+
 # Pull the standalone binaries into the final image.
 COPY --from=gitleaks-src /usr/bin/gitleaks /usr/bin/gitleaks
 

Dockerfile.infra

@@ -1,21 +1,45 @@
 # Firecracker single infra-VM image (PRD 0070 Stage B).
 #
-# The per-host infra VM runs BOTH the orchestrator control plane and the
-# gateway data plane in one microVM (see backend/firecracker/infra_vm.py).
-# It layers the stdlib-only control-plane source onto the gateway
-# data-plane image, so the single VM has mitmproxy / git / gitleaks /
-# supervise (gateway) AND `bot_bottle.orchestrator` (control plane) —
-# reusing the gateway payload rather than copying it into a third image.
+# The per-host infra VM runs the orchestrator control plane, the gateway
+# data plane, AND builds agent images (buildah) — all in one microVM (see
+# backend/firecracker/infra_vm.py). It composes:
+#   * FROM the gateway image (mitmproxy / git / gitleaks / supervise + the
+#     flat daemon modules) — now trixie-based, so buildah 1.39 is available;
+#   * `COPY --from` the orchestrator image's content (the single definition
+#     of the control-plane payload — see Dockerfile.orchestrator), so this
+#     VM and the docker backend share one orchestrator definition; and
+#   * buildah, installed HERE only (the docker orchestrator/gateway images
+#     never carry it).
 #
-# The docker backend keeps the two images separate (a lean orchestrator
-# container + a gateway container); this combined image exists only for
-# the Firecracker single-VM cut. If the egress blast radius ever warrants
-# it, splitting egress back into its own VM is a routing change, not a
-# repackaging (see PRD 0070's "secret concentration").
+# multi-`FROM` can't union two bases (that's multi-stage, not multiple
+# inheritance), so the orchestrator content is pulled in via `COPY --from`
+# rather than a second base. Both images share the trixie `python:3.12-slim`
+# base, so the copy is clean (same python; future installed deps copy too).
+#
+# The docker backend keeps orchestrator + gateway as separate images; this
+# combined image exists only for the Firecracker single-VM cut. Splitting a
+# service back into its own VM later is a routing change, not a repackaging
+# (PRD 0070's "secret concentration"; a disposable builder can boot from
+# this same image on its own TAP).
 FROM bot-bottle-gateway:latest
 
-# The gateway image copies only its daemon modules flat under /app; the
-# control plane needs the whole (stdlib-only) package. Both coexist:
-# `/app/gateway_init.py` keeps its flat-sibling imports, and
-# `python3 -m bot_bottle.orchestrator` resolves /app/bot_bottle.
-COPY bot_bottle /app/bot_bottle
+# --- in-VM agent-image builder (PRD 0069 Stage 3) -------------------
+# The Firecracker backend builds users' agent Dockerfiles *inside this VM*
+# with buildah (rootless, daemonless) instead of on the host — no host
+# Docker daemon, no root-equivalent `docker` group. `crun` is the OCI
+# runtime; `netavark` + `aardvark-dns` are the network backend for `FROM`
+# pulls + `RUN` egress. Requires the trixie base (buildah 1.39: bookworm's
+# 1.28 can't parse Dockerfile heredocs that agent images use).
+RUN apt-get update \
+ && apt-get install -y --no-install-recommends \
+      buildah crun netavark aardvark-dns \
+ && rm -rf /var/lib/apt/lists/*
+# vfs + chroot: buildah works as root in the bare microVM (no
+# fuse-overlayfs / overlay module / subuid maps). Matches image_builder.
+ENV STORAGE_DRIVER=vfs \
+    BUILDAH_ISOLATION=chroot
+
+# The orchestrator content, pulled from its single definition. The gateway
+# image already has the flat daemon modules under /app; this adds the full
+# `bot_bottle` package so `python3 -m bot_bottle.orchestrator` resolves.
+COPY --from=bot-bottle-orchestrator:latest /app/bot_bottle /app/bot_bottle

Dockerfile.orchestrator

@@ -1,60 +1,34 @@
-# Orchestrator control-plane image (PRD 0070, #384) + in-VM agent-image
-# builder (PRD 0069 Stage 3).
+# Orchestrator control-plane image (PRD 0070, #384).
 #
-# The per-host orchestrator runs `python3 -m bot_bottle.orchestrator`.
-# The `bot_bottle` package is **stdlib-only** by design, so the control
-# plane itself needs nothing but a Python runtime — none of the
-# gateway's mitmproxy / git / gitleaks payload (that is the separate
-# `bot-bottle-gateway` image, Dockerfile.gateway). Splitting them keeps
-# the secret-dense control plane (it concentrates every bottle's egress
-# tokens — see PRD 0070's "secret concentration") on a minimal
-# dependency surface.
+# This is the **single definition of the orchestrator's content** — the
+# `bot_bottle` package baked onto a Python runtime — referenced by BOTH:
+#   * the docker backend, which runs this image directly as the lean
+#     control-plane container; and
+#   * the firecracker infra image (Dockerfile.infra), which `COPY --from`s
+#     this image's `/app/bot_bottle` so the single infra VM runs the same
+#     control plane. Keeping it in one place means future orchestrator deps
+#     (e.g. iroh) are added here once, not duplicated per backend.
 #
-# The repo is bind-mounted read-only into the container at run time (see
-# `orchestrator/lifecycle.py`), so the source is NOT copied in here: the
-# image is just the runtime. `ensure_running` recreates the container
-# only when the bind-mounted source hash changes (#381), which is why
-# the code stays a mount rather than a baked layer.
+# It stays deliberately lean: the control plane is **stdlib-only** today, so
+# no third-party payload — none of the gateway's mitmproxy/git/gitleaks
+# (that's Dockerfile.gateway) and no buildah (that's the firecracker
+# builder, and lives only in Dockerfile.infra). Keeping the secret-dense
+# control plane on a minimal dependency surface is the point (PRD 0070's
+# "secret concentration").
+#
+# Shares the trixie `python:3.12-slim` base with the gateway image, so when
+# the orchestrator grows real deps they can be `COPY --from`'d into the
+# infra image cleanly (same base/python — installed packages copy safely).
 
 FROM python:3.12-slim
 
-# --- in-VM agent-image builder (PRD 0069 Stage 3) -------------------
-# The Firecracker backend removes the host Docker daemon by building
-# users' agent Dockerfiles *inside the orchestrator VM* with buildah
-# (rootless, daemonless) instead of on the host. The host then needs no
-# Docker daemon and no root-equivalent `docker` group; an untrusted
-# Dockerfile builds inside the confined orchestrator VM, never on the
-# host — strictly more isolated than host `docker build`.
-#
-# `git` lets buildah resolve git-context builds and lets the backend
-# clone/pull; `ca-certificates` is needed for `FROM` pulls over TLS.
-# buildah's runtime helpers (stripped by --no-install-recommends, so
-# listed explicitly): `crun` is the OCI runtime that executes build
-# steps; `netavark` + `aardvark-dns` are the network backend buildah
-# uses to give `FROM` pulls and `RUN` steps network access.
-RUN apt-get update \
- && apt-get install -y --no-install-recommends \
-      buildah crun netavark aardvark-dns git ca-certificates \
- && rm -rf /var/lib/apt/lists/*
-
-# vfs storage + chroot isolation: buildah then needs neither
-# fuse-overlayfs / an overlay kernel module nor configured subuid maps
-# (newuidmap/newgidmap), so it works unconditionally as root in a
-# minimal microVM rootfs. vfs copies layers rather than stacking them —
-# slower and heavier than overlay; a build-cache optimization (overlay
-# where available, a persistent cache disk) is tracked for later.
-ENV STORAGE_DRIVER=vfs \
-    BUILDAH_ISOLATION=chroot
-
-# No third-party *Python* deps — the control plane stays stdlib only.
 WORKDIR /app
 
-# Bake the (stdlib-only) control-plane source so the image is
-# self-contained — it runs from a built image, no runtime bind-mount.
-# This is what the Firecracker infra VM boots (a guest can't bind-mount
-# host source); the docker backend may still bind-mount /app for dev
-# live-reload, which simply overlays this baked copy. `.dockerignore`
-# keeps .git/docs/*.md out of the context.
+# The orchestrator content. Baked so the image is self-contained (runs from
+# a built image, no runtime bind-mount); the docker backend may still
+# bind-mount /app for dev live-reload, which simply overlays this copy.
+# `.dockerignore` keeps .git/docs/*.md out of the context. (Future deps like
+# iroh go here too — a shared requirements installed on this same base.)
 COPY bot_bottle /app/bot_bottle
 
 # Documentation only; lifecycle.py overrides the entrypoint to

bot_bottle/backend/firecracker/image_builder.py

@@ -1,23 +1,17 @@
 """Docker-free agent-image builds for the Firecracker backend (PRD 0069 Stage 3).
 
-Instead of `docker build` on the host, an agent's Dockerfile is built inside a
-throwaway Firecracker *builder VM* running buildah (rootless, daemonless): no
-host Docker daemon, no root-equivalent `docker` group, and the untrusted
-Dockerfile executes in a confined microVM rather than on the host.
+Agent Dockerfiles build **inside the persistent per-host infra VM**
+(`infra_vm.py`), which carries buildah (rootless, daemonless): no host Docker
+daemon, no root-equivalent `docker` group. The build runs over SSH against the
+infra VM and its rootfs streams back to the host, where the existing
+`mke2fs -d` path (`util.build_rootfs_ext4`) turns it into a bootable ext4.
 
-Flow (on a cache miss, keyed by Dockerfile content):
-  1. boot the builder VM (the orchestrator image, which carries buildah) on
-     the NAT'd orchestrator link (`netpool.orch_slot()`), so the Dockerfile's
-     `FROM` pulls + apt/npm reach the internet;
-  2. send the Dockerfile over SSH and `buildah build` it;
-  3. `buildah mount` the built image and stream its rootfs tar back to the
-     host, extracting it into the cache dir;
-  4. inject the guest boot bits + mark the cache ready — the same base-dir
-     shape `firecracker.util` turns into a bootable ext4 with `mke2fs -d`.
-
-The builder shares the orchestrator's TAP; once the orchestrator is a
-persistent VM (Stage B) these builds move onto its control plane instead of a
-throwaway boot.
+Building in the infra VM — rather than a throwaway builder VM — means there is
+one buildah image (`bot-bottle-infra`) and no contention for the orchestrator
+TAP. Tradeoff: an untrusted Dockerfile's `RUN` steps share the VM with the
+control plane + gateway (buildah `--isolation chroot` isn't a hard boundary) —
+the accepted single-VM blast-radius tradeoff, re-splittable into a disposable
+builder (booted from this same image on its own TAP) later.
 """
 
 from __future__ import annotations
@@ -28,28 +22,15 @@ import subprocess
 from pathlib import Path
 
 from ...log import die, info
-from . import firecracker_vm, netpool, util
+from . import infra_vm, util
 
-# The builder runs the orchestrator image — it is the component that carries
-# buildah (see Dockerfile.orchestrator). Built from the host docker image via
-# the bootstrap export path until we pull a pre-built image instead.
-_ORCHESTRATOR_IMAGE = "bot-bottle-orchestrator:latest"
-
-# The builder VM makes DIRECT registry/apt connections (not through the
-# gateway proxy the way agents do), and the kernel `ip=` cmdline sets no
-# resolver — so give it one. Public for now; build-time egress through the
-# gateway will remove the need for it.
-_BUILDER_RESOLVER = "1.1.1.1"
-
-# vfs + chroot: buildah works as root in the bare microVM (no fuse-overlayfs /
-# overlay module / subuid maps). Must match Dockerfile.orchestrator's env.
-# `--isolation` is a build/run-only flag; `from`/`mount` take just the store.
+# vfs + chroot: buildah works as root in the microVM (no fuse-overlayfs /
+# overlay module / subuid maps). `--isolation` is a build/run-only flag;
+# `from`/`mount` take just the store.
 _BUILD_FLAGS = "--isolation chroot --storage-driver vfs"
 _STORE_FLAG = "--storage-driver vfs"
 
-_SSH_READY_TIMEOUT = 40.0
 _BUILD_TIMEOUT_SECONDS = 900.0
-_LOCAL_TAG = "bot-bottle-agent-build"
 
 
 def _dockerfile_hash(dockerfile: Path) -> str:
@@ -63,10 +44,10 @@ def _dockerfile_hash(dockerfile: Path) -> str:
 def build_agent_rootfs_dir(
     dockerfile: Path, *, image_tag: str, smoke_test: tuple[str, ...] = (),
 ) -> Path:
-    """Build `dockerfile` in a Firecracker builder VM (buildah, no host
-    docker), export its rootfs, inject the guest boot bits, and return the
-    cached base dir — the same shape `util.build_rootfs_ext4` consumes.
-    Cached by Dockerfile content, so a repeat launch skips the rebuild.
+    """Build `dockerfile` in the infra VM (buildah, no host docker), export its
+    rootfs, inject the guest boot bits, and return the cached base dir — the
+    same shape `util.build_rootfs_ext4` consumes. Cached by Dockerfile content,
+    so a repeat launch skips the rebuild.
 
     `smoke_test` (the provider's declared argv, e.g. `("claude","--version")`)
     is run in the freshly built image before export, catching an npm
@@ -82,47 +63,38 @@ def build_agent_rootfs_dir(
         shutil.rmtree(base, ignore_errors=True)
     base.mkdir(parents=True)
 
-    info(f"building agent image {image_tag!r} in a firecracker builder VM")
-    _build_in_vm(dockerfile, base, smoke_test)
+    info(f"building agent image {image_tag!r} in the infra VM")
+    _build_in_infra(dockerfile, base, smoke_test, digest)
     util.inject_guest_boot(base)
     ready.write_text("ok\n")
     return base
 
 
-def _build_in_vm(dockerfile: Path, base: Path, smoke_test: tuple[str, ...]) -> None:
-    """Boot the builder VM, build the Dockerfile with buildah, and stream the
-    resulting rootfs into `base`. Tears the VM down on every exit path."""
-    slot = netpool.orch_slot()
-    if not netpool.tap_present(slot.iface):
-        die(f"orchestrator/builder link {slot.iface} not present.\n"
-            f"    ./cli.py backend setup --backend=firecracker")
-
-    # The builder runs the orchestrator rootfs (bootstrap: exported from the
-    # host docker image; a pulled image replaces this later).
-    orch_base = util.build_base_rootfs_dir(_ORCHESTRATOR_IMAGE)
-    run_dir = util.cache_dir() / "run" / "builder"
-    run_dir.mkdir(parents=True, exist_ok=True)
-    rootfs = run_dir / "rootfs.ext4"
-    # vfs copies every layer, so the builder disk needs generous headroom.
-    util.build_rootfs_ext4(orch_base, rootfs, slack_mib=8192)
-    private_key, pubkey = util.generate_keypair(run_dir)
-
-    vm = firecracker_vm.boot(
-        name="bot-bottle-builder", rootfs=rootfs, tap=slot.iface,
-        guest_ip=slot.guest_ip, host_ip=slot.host_ip, pubkey=pubkey,
-        run_dir=run_dir, mem_mib=4096,
-    )
+def _build_in_infra(
+    dockerfile: Path, base: Path, smoke_test: tuple[str, ...], digest: str,
+) -> None:
+    """Ensure the infra VM is up, `buildah build` the Dockerfile in it, smoke
+    test the image, and stream its rootfs into `base`. The infra VM persists;
+    only the per-build container/image/context are cleaned up."""
+    infra = infra_vm.ensure_running()
+    key, ip = infra.private_key, infra.guest_ip
+    tag = f"bot-bottle-agent-build-{digest}"
+    ctx = f"/tmp/agent-build-{digest}"
     try:
-        firecracker_vm.wait_for_ssh(vm, private_key, timeout=_SSH_READY_TIMEOUT)
-        _ssh(private_key, slot.guest_ip,
-             f"printf 'nameserver {_BUILDER_RESOLVER}\\n' > /etc/resolv.conf "
-             f"&& mkdir -p /build/ctx")
-        _send_dockerfile(private_key, slot.guest_ip, dockerfile)
-        _buildah_build(private_key, slot.guest_ip)
-        _smoke_test(private_key, slot.guest_ip, smoke_test)
-        _stream_rootfs(private_key, slot.guest_ip, base)
+        prep = _ssh(key, ip, f"rm -rf {ctx}; mkdir -p {ctx}/ctx")
+        if prep.returncode != 0:
+            die(f"preparing build dir in the infra VM failed: {prep.stderr.strip()}")
+        _send_dockerfile(key, ip, dockerfile, ctx)
+        _buildah_build(key, ip, ctx, tag)
+        _smoke_test(key, ip, tag, smoke_test)
+        _stream_rootfs(key, ip, tag, base)
     finally:
-        vm.terminate()
+        # Keep the ephemeral infra rootfs from accumulating build state. Builds
+        # are serial per launch, so removing all working containers is safe.
+        _ssh(key, ip,
+             f"buildah rm {_STORE_FLAG} -a >/dev/null 2>&1; "
+             f"buildah rmi {_STORE_FLAG} {tag} >/dev/null 2>&1; rm -rf {ctx}",
+             timeout=60)
 
 
 def _ssh(private_key: Path, guest_ip: str, script: str,
@@ -133,36 +105,35 @@ def _ssh(private_key: Path, guest_ip: str, script: str,
     )
 
 
-def _send_dockerfile(private_key: Path, guest_ip: str, dockerfile: Path) -> None:
+def _send_dockerfile(private_key: Path, guest_ip: str, dockerfile: Path, ctx: str) -> None:
     proc = subprocess.run(
-        util.ssh_base_argv(private_key, guest_ip) + ["cat > /build/Dockerfile"],
+        util.ssh_base_argv(private_key, guest_ip) + [f"cat > {ctx}/Dockerfile"],
         input=dockerfile.read_bytes(), capture_output=True, timeout=30, check=False,
     )
     if proc.returncode != 0:
-        die(f"sending Dockerfile to the builder VM failed: "
+        die(f"sending Dockerfile to the infra VM failed: "
             f"{proc.stderr.decode(errors='replace').strip()}")
 
 
-def _buildah_build(private_key: Path, guest_ip: str) -> None:
+def _buildah_build(private_key: Path, guest_ip: str, ctx: str, tag: str) -> None:
     build = _ssh(
         private_key, guest_ip,
-        f"buildah build {_BUILD_FLAGS} -t {_LOCAL_TAG} "
-        f"-f /build/Dockerfile /build/ctx",
+        f"buildah build {_BUILD_FLAGS} -t {tag} -f {ctx}/Dockerfile {ctx}/ctx",
         timeout=_BUILD_TIMEOUT_SECONDS,
     )
     if build.returncode != 0:
         tail = "\n".join((build.stdout + build.stderr).strip().splitlines()[-20:])
-        die(f"buildah build in the builder VM failed:\n{tail}")
+        die(f"buildah build in the infra VM failed:\n{tail}")
 
 
-def _smoke_test(private_key: Path, guest_ip: str, argv: tuple[str, ...]) -> None:
+def _smoke_test(private_key: Path, guest_ip: str, tag: str, argv: tuple[str, ...]) -> None:
     """Run the provider's smoke argv inside the freshly built image
     (`buildah run`, which uses the image's own PATH), failing the build
     loudly if the CLI is a broken stub. No-op without a declared test."""
     if not argv:
         return
     cmd = (
-        f"set -e; ctr=$(buildah from {_STORE_FLAG} {_LOCAL_TAG}); "
+        f"set -e; ctr=$(buildah from {_STORE_FLAG} {tag}); "
         f"buildah run {_BUILD_FLAGS} \"$ctr\" -- {' '.join(argv)}; rc=$?; "
         f"buildah rm \"$ctr\" >/dev/null 2>&1 || true; exit $rc"
     )
@@ -173,13 +144,12 @@ def _smoke_test(private_key: Path, guest_ip: str, argv: tuple[str, ...]) -> None
             f"({' '.join(argv)}):\n" + "\n".join(detail))
 
 
-def _stream_rootfs(private_key: Path, guest_ip: str, base: Path) -> None:
-    """`buildah mount` the built image in the VM and pipe its rootfs tar
+def _stream_rootfs(private_key: Path, guest_ip: str, tag: str, base: Path) -> None:
+    """`buildah mount` the built image in the infra VM and pipe its rootfs tar
     straight into `base` on the host (extracted as the non-root host user, so
-    uid 0 isn't preserved — the guest init restores /root ownership, same as
-    the docker-export path)."""
+    uid 0 isn't preserved — the guest init restores /root ownership)."""
     export = (
-        f"set -e; ctr=$(buildah from {_STORE_FLAG} {_LOCAL_TAG}); "
+        f"set -e; ctr=$(buildah from {_STORE_FLAG} {tag}); "
         f"mnt=$(buildah mount {_STORE_FLAG} \"$ctr\"); "
         f"tar -C \"$mnt\" -cf - ."
     )
@@ -196,5 +166,5 @@ def _stream_rootfs(private_key: Path, guest_ip: str, base: Path) -> None:
                if ssh_proc.stderr else "")
     rc = ssh_proc.wait()
     if rc != 0 or untar.returncode != 0:
-        die(f"exporting the built rootfs from the builder VM failed: "
+        die(f"exporting the built rootfs from the infra VM failed: "
             f"{ssh_err.strip() or '<no stderr>'}")

bot_bottle/backend/firecracker/infra_vm.py

@@ -39,6 +39,7 @@ from . import firecracker_vm, netpool, util
 # a pull-from-registry mode lands later.
 _INFRA_IMAGE = "bot-bottle-infra:latest"
 _GATEWAY_IMAGE = "bot-bottle-gateway:latest"
+_ORCHESTRATOR_IMAGE = "bot-bottle-orchestrator:latest"
 _REPO_ROOT = Path(__file__).resolve().parents[3]
 
 CONTROL_PLANE_PORT = 8099
@@ -105,10 +106,12 @@ class InfraVm:
 
 
 def ensure_built() -> None:
-    """Build the infra image from source (bootstrap via host docker): the
-    gateway data-plane image, then the infra image that layers the
-    control-plane source onto it. A pull-from-registry mode replaces this
-    later."""
+    """Build the infra image from source (bootstrap via host docker). The
+    infra image `COPY --from`s the orchestrator image and is `FROM` the
+    gateway image, so both must exist first. A pull-from-registry mode
+    replaces this later."""
+    docker_mod.build_image(
+        _ORCHESTRATOR_IMAGE, str(_REPO_ROOT), dockerfile="Dockerfile.orchestrator")
     docker_mod.build_image(
         _GATEWAY_IMAGE, str(_REPO_ROOT), dockerfile="Dockerfile.gateway")
     docker_mod.build_image(

tests/unit/test_firecracker_image_builder.py

@@ -1,6 +1,6 @@
 """Unit tests for the docker-free Firecracker agent-image builder.
 
-The VM boot / SSH / buildah plumbing (`_build_in_vm`) is integration-tested
+The VM boot / SSH / buildah plumbing (`_build_in_infra`) is integration-tested
 on a KVM host; here we cover the cache decision, the boot-bit injection, and
 the smoke-test no-op — the logic that must hold without a VM.
 """
@@ -29,7 +29,7 @@ class TestBuildAgentRootfsDir(unittest.TestCase):
         base.mkdir(parents=True)
         (base / ".bb-ready").write_text("ok\n")
         with patch.object(image_builder.util, "cache_dir", return_value=self.cache), \
-             patch.object(image_builder, "_build_in_vm") as build:
+             patch.object(image_builder, "_build_in_infra") as build:
             out = image_builder.build_agent_rootfs_dir(
                 self.dockerfile, image_tag="t:latest")
         build.assert_not_called()
@@ -37,7 +37,7 @@ class TestBuildAgentRootfsDir(unittest.TestCase):
 
     def test_cache_miss_builds_injects_and_marks_ready(self):
         with patch.object(image_builder.util, "cache_dir", return_value=self.cache), \
-             patch.object(image_builder, "_build_in_vm") as build, \
+             patch.object(image_builder, "_build_in_infra") as build, \
              patch.object(image_builder.util, "inject_guest_boot") as inject:
             out = image_builder.build_agent_rootfs_dir(
                 self.dockerfile, image_tag="t:latest", smoke_test=("claude", "--version"))
@@ -59,7 +59,7 @@ class TestBuildAgentRootfsDir(unittest.TestCase):
 class TestSmokeTest(unittest.TestCase):
     def test_empty_argv_is_noop(self):
         with patch.object(image_builder, "_ssh") as ssh:
-            image_builder._smoke_test(Path("/k"), "10.0.0.1", ())
+            image_builder._smoke_test(Path("/k"), "10.0.0.1", "tag", ())
         ssh.assert_not_called()
 
     def test_failed_smoke_dies(self):
@@ -67,7 +67,7 @@ class TestSmokeTest(unittest.TestCase):
         result = subprocess.CompletedProcess([], 1, stdout="broken", stderr="")
         with patch.object(image_builder, "_ssh", return_value=result), \
              self.assertRaises(SystemExit):
-            image_builder._smoke_test(Path("/k"), "10.0.0.1", ("claude", "--version"))
+            image_builder._smoke_test(Path("/k"), "10.0.0.1", "tag", ("claude", "--version"))
 
 
 if __name__ == "__main__":

tests/unit/test_firecracker_infra_vm.py

@@ -95,11 +95,14 @@ class TestRegistryVolume(unittest.TestCase):
 
 
 class TestEnsureBuilt(unittest.TestCase):
-    def test_builds_gateway_then_infra(self):
+    def test_builds_deps_before_infra(self):
         with patch.object(infra_vm.docker_mod, "build_image") as build:
             infra_vm.ensure_built()
         tags = [c.args[0] for c in build.call_args_list]
-        self.assertEqual([infra_vm._GATEWAY_IMAGE, infra_vm._INFRA_IMAGE], tags)
+        # infra is FROM gateway and COPY --from orchestrator, so both first.
+        self.assertEqual(infra_vm._INFRA_IMAGE, tags[-1])
+        self.assertIn(infra_vm._ORCHESTRATOR_IMAGE, tags[:-1])
+        self.assertIn(infra_vm._GATEWAY_IMAGE, tags[:-1])
 
 
 class TestWaitForHealth(unittest.TestCase):