refactor(firecracker): single infra VM builds too — buildah in one image (PR #354 review)

Addresses the review finding that buildah lived only in the orchestrator
image, so the persistent infra VM wasn't the builder — a separate throwaway
builder VM contended with it for the orchestrator TAP. Consolidate:

- **Rebase the gateway (and thus infra) on `python:3.12-slim` = Debian
  trixie**, pip-installing mitmproxy instead of `FROM mitmproxy/mitmproxy`
  (Debian bookworm). trixie ships buildah 1.39, which can build agent
  Dockerfiles that use heredocs; bookworm's 1.28 can't (`Unknown
  instruction: "{"`). CA path is unchanged (set via `--set confdir=`).
- **buildah lives only in `Dockerfile.infra`** now (removed from the
  orchestrator image, which is lean/stdlib-only again).
- **Shared orchestrator content**: `Dockerfile.orchestrator` is the single
  definition of the control-plane payload; the infra image `COPY --from`s
  it (same trixie base → clean copy, and future deps like iroh are added
  once). The docker backend runs the orchestrator image directly.
- **`image_builder` builds inside the infra VM** (which now has buildah)
  over SSH — no throwaway builder VM, so the `bborch0` contention is gone.
  `ensure_built` builds orchestrator + gateway before infra (FROM gateway,
  COPY --from orchestrator).

Verified on a KVM host: images build (buildah 1.39 in infra), the agent
image builds *inside* the infra VM (heredoc Dockerfile and all), the infra
VM stays healthy, the agent boots and `claude --version` = 2.1.172. The
rebased gateway still starts as a docker container and generates its CA
(docker backend unaffected).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WBMWTEtQdJ4W5UrWuLHCck

bot_bottle/backend/firecracker/image_builder.py

@@ -1,23 +1,17 @@
 """Docker-free agent-image builds for the Firecracker backend (PRD 0069 Stage 3).
 
-Instead of `docker build` on the host, an agent's Dockerfile is built inside a
-throwaway Firecracker *builder VM* running buildah (rootless, daemonless): no
-host Docker daemon, no root-equivalent `docker` group, and the untrusted
-Dockerfile executes in a confined microVM rather than on the host.
+Agent Dockerfiles build **inside the persistent per-host infra VM**
+(`infra_vm.py`), which carries buildah (rootless, daemonless): no host Docker
+daemon, no root-equivalent `docker` group. The build runs over SSH against the
+infra VM and its rootfs streams back to the host, where the existing
+`mke2fs -d` path (`util.build_rootfs_ext4`) turns it into a bootable ext4.
 
-Flow (on a cache miss, keyed by Dockerfile content):
-  1. boot the builder VM (the orchestrator image, which carries buildah) on
-     the NAT'd orchestrator link (`netpool.orch_slot()`), so the Dockerfile's
-     `FROM` pulls + apt/npm reach the internet;
-  2. send the Dockerfile over SSH and `buildah build` it;
-  3. `buildah mount` the built image and stream its rootfs tar back to the
-     host, extracting it into the cache dir;
-  4. inject the guest boot bits + mark the cache ready — the same base-dir
-     shape `firecracker.util` turns into a bootable ext4 with `mke2fs -d`.
-
-The builder shares the orchestrator's TAP; once the orchestrator is a
-persistent VM (Stage B) these builds move onto its control plane instead of a
-throwaway boot.
+Building in the infra VM — rather than a throwaway builder VM — means there is
+one buildah image (`bot-bottle-infra`) and no contention for the orchestrator
+TAP. Tradeoff: an untrusted Dockerfile's `RUN` steps share the VM with the
+control plane + gateway (buildah `--isolation chroot` isn't a hard boundary) —
+the accepted single-VM blast-radius tradeoff, re-splittable into a disposable
+builder (booted from this same image on its own TAP) later.
 """
 
 from __future__ import annotations
@@ -28,28 +22,15 @@ import subprocess
 from pathlib import Path
 
 from ...log import die, info
-from . import firecracker_vm, netpool, util
+from . import infra_vm, util
 
-# The builder runs the orchestrator image — it is the component that carries
-# buildah (see Dockerfile.orchestrator). Built from the host docker image via
-# the bootstrap export path until we pull a pre-built image instead.
-_ORCHESTRATOR_IMAGE = "bot-bottle-orchestrator:latest"
-
-# The builder VM makes DIRECT registry/apt connections (not through the
-# gateway proxy the way agents do), and the kernel `ip=` cmdline sets no
-# resolver — so give it one. Public for now; build-time egress through the
-# gateway will remove the need for it.
-_BUILDER_RESOLVER = "1.1.1.1"
-
-# vfs + chroot: buildah works as root in the bare microVM (no fuse-overlayfs /
-# overlay module / subuid maps). Must match Dockerfile.orchestrator's env.
-# `--isolation` is a build/run-only flag; `from`/`mount` take just the store.
+# vfs + chroot: buildah works as root in the microVM (no fuse-overlayfs /
+# overlay module / subuid maps). `--isolation` is a build/run-only flag;
+# `from`/`mount` take just the store.
 _BUILD_FLAGS = "--isolation chroot --storage-driver vfs"
 _STORE_FLAG = "--storage-driver vfs"
 
-_SSH_READY_TIMEOUT = 40.0
 _BUILD_TIMEOUT_SECONDS = 900.0
-_LOCAL_TAG = "bot-bottle-agent-build"
 
 
 def _dockerfile_hash(dockerfile: Path) -> str:
@@ -63,10 +44,10 @@ def _dockerfile_hash(dockerfile: Path) -> str:
 def build_agent_rootfs_dir(
     dockerfile: Path, *, image_tag: str, smoke_test: tuple[str, ...] = (),
 ) -> Path:
-    """Build `dockerfile` in a Firecracker builder VM (buildah, no host
-    docker), export its rootfs, inject the guest boot bits, and return the
-    cached base dir — the same shape `util.build_rootfs_ext4` consumes.
-    Cached by Dockerfile content, so a repeat launch skips the rebuild.
+    """Build `dockerfile` in the infra VM (buildah, no host docker), export its
+    rootfs, inject the guest boot bits, and return the cached base dir — the
+    same shape `util.build_rootfs_ext4` consumes. Cached by Dockerfile content,
+    so a repeat launch skips the rebuild.
 
     `smoke_test` (the provider's declared argv, e.g. `("claude","--version")`)
     is run in the freshly built image before export, catching an npm
@@ -82,47 +63,38 @@ def build_agent_rootfs_dir(
         shutil.rmtree(base, ignore_errors=True)
     base.mkdir(parents=True)
 
-    info(f"building agent image {image_tag!r} in a firecracker builder VM")
-    _build_in_vm(dockerfile, base, smoke_test)
+    info(f"building agent image {image_tag!r} in the infra VM")
+    _build_in_infra(dockerfile, base, smoke_test, digest)
     util.inject_guest_boot(base)
     ready.write_text("ok\n")
     return base
 
 
-def _build_in_vm(dockerfile: Path, base: Path, smoke_test: tuple[str, ...]) -> None:
-    """Boot the builder VM, build the Dockerfile with buildah, and stream the
-    resulting rootfs into `base`. Tears the VM down on every exit path."""
-    slot = netpool.orch_slot()
-    if not netpool.tap_present(slot.iface):
-        die(f"orchestrator/builder link {slot.iface} not present.\n"
-            f"    ./cli.py backend setup --backend=firecracker")
-
-    # The builder runs the orchestrator rootfs (bootstrap: exported from the
-    # host docker image; a pulled image replaces this later).
-    orch_base = util.build_base_rootfs_dir(_ORCHESTRATOR_IMAGE)
-    run_dir = util.cache_dir() / "run" / "builder"
-    run_dir.mkdir(parents=True, exist_ok=True)
-    rootfs = run_dir / "rootfs.ext4"
-    # vfs copies every layer, so the builder disk needs generous headroom.
-    util.build_rootfs_ext4(orch_base, rootfs, slack_mib=8192)
-    private_key, pubkey = util.generate_keypair(run_dir)
-
-    vm = firecracker_vm.boot(
-        name="bot-bottle-builder", rootfs=rootfs, tap=slot.iface,
-        guest_ip=slot.guest_ip, host_ip=slot.host_ip, pubkey=pubkey,
-        run_dir=run_dir, mem_mib=4096,
-    )
+def _build_in_infra(
+    dockerfile: Path, base: Path, smoke_test: tuple[str, ...], digest: str,
+) -> None:
+    """Ensure the infra VM is up, `buildah build` the Dockerfile in it, smoke
+    test the image, and stream its rootfs into `base`. The infra VM persists;
+    only the per-build container/image/context are cleaned up."""
+    infra = infra_vm.ensure_running()
+    key, ip = infra.private_key, infra.guest_ip
+    tag = f"bot-bottle-agent-build-{digest}"
+    ctx = f"/tmp/agent-build-{digest}"
     try:
-        firecracker_vm.wait_for_ssh(vm, private_key, timeout=_SSH_READY_TIMEOUT)
-        _ssh(private_key, slot.guest_ip,
-             f"printf 'nameserver {_BUILDER_RESOLVER}\\n' > /etc/resolv.conf "
-             f"&& mkdir -p /build/ctx")
-        _send_dockerfile(private_key, slot.guest_ip, dockerfile)
-        _buildah_build(private_key, slot.guest_ip)
-        _smoke_test(private_key, slot.guest_ip, smoke_test)
-        _stream_rootfs(private_key, slot.guest_ip, base)
+        prep = _ssh(key, ip, f"rm -rf {ctx}; mkdir -p {ctx}/ctx")
+        if prep.returncode != 0:
+            die(f"preparing build dir in the infra VM failed: {prep.stderr.strip()}")
+        _send_dockerfile(key, ip, dockerfile, ctx)
+        _buildah_build(key, ip, ctx, tag)
+        _smoke_test(key, ip, tag, smoke_test)
+        _stream_rootfs(key, ip, tag, base)
     finally:
-        vm.terminate()
+        # Keep the ephemeral infra rootfs from accumulating build state. Builds
+        # are serial per launch, so removing all working containers is safe.
+        _ssh(key, ip,
+             f"buildah rm {_STORE_FLAG} -a >/dev/null 2>&1; "
+             f"buildah rmi {_STORE_FLAG} {tag} >/dev/null 2>&1; rm -rf {ctx}",
+             timeout=60)
 
 
 def _ssh(private_key: Path, guest_ip: str, script: str,
@@ -133,36 +105,35 @@ def _ssh(private_key: Path, guest_ip: str, script: str,
     )
 
 
-def _send_dockerfile(private_key: Path, guest_ip: str, dockerfile: Path) -> None:
+def _send_dockerfile(private_key: Path, guest_ip: str, dockerfile: Path, ctx: str) -> None:
     proc = subprocess.run(
-        util.ssh_base_argv(private_key, guest_ip) + ["cat > /build/Dockerfile"],
+        util.ssh_base_argv(private_key, guest_ip) + [f"cat > {ctx}/Dockerfile"],
         input=dockerfile.read_bytes(), capture_output=True, timeout=30, check=False,
     )
     if proc.returncode != 0:
-        die(f"sending Dockerfile to the builder VM failed: "
+        die(f"sending Dockerfile to the infra VM failed: "
             f"{proc.stderr.decode(errors='replace').strip()}")
 
 
-def _buildah_build(private_key: Path, guest_ip: str) -> None:
+def _buildah_build(private_key: Path, guest_ip: str, ctx: str, tag: str) -> None:
     build = _ssh(
         private_key, guest_ip,
-        f"buildah build {_BUILD_FLAGS} -t {_LOCAL_TAG} "
-        f"-f /build/Dockerfile /build/ctx",
+        f"buildah build {_BUILD_FLAGS} -t {tag} -f {ctx}/Dockerfile {ctx}/ctx",
         timeout=_BUILD_TIMEOUT_SECONDS,
     )
     if build.returncode != 0:
         tail = "\n".join((build.stdout + build.stderr).strip().splitlines()[-20:])
-        die(f"buildah build in the builder VM failed:\n{tail}")
+        die(f"buildah build in the infra VM failed:\n{tail}")
 
 
-def _smoke_test(private_key: Path, guest_ip: str, argv: tuple[str, ...]) -> None:
+def _smoke_test(private_key: Path, guest_ip: str, tag: str, argv: tuple[str, ...]) -> None:
     """Run the provider's smoke argv inside the freshly built image
     (`buildah run`, which uses the image's own PATH), failing the build
     loudly if the CLI is a broken stub. No-op without a declared test."""
     if not argv:
         return
     cmd = (
-        f"set -e; ctr=$(buildah from {_STORE_FLAG} {_LOCAL_TAG}); "
+        f"set -e; ctr=$(buildah from {_STORE_FLAG} {tag}); "
         f"buildah run {_BUILD_FLAGS} \"$ctr\" -- {' '.join(argv)}; rc=$?; "
         f"buildah rm \"$ctr\" >/dev/null 2>&1 || true; exit $rc"
     )
@@ -173,13 +144,12 @@ def _smoke_test(private_key: Path, guest_ip: str, argv: tuple[str, ...]) -> None
             f"({' '.join(argv)}):\n" + "\n".join(detail))
 
 
-def _stream_rootfs(private_key: Path, guest_ip: str, base: Path) -> None:
-    """`buildah mount` the built image in the VM and pipe its rootfs tar
+def _stream_rootfs(private_key: Path, guest_ip: str, tag: str, base: Path) -> None:
+    """`buildah mount` the built image in the infra VM and pipe its rootfs tar
     straight into `base` on the host (extracted as the non-root host user, so
-    uid 0 isn't preserved — the guest init restores /root ownership, same as
-    the docker-export path)."""
+    uid 0 isn't preserved — the guest init restores /root ownership)."""
     export = (
-        f"set -e; ctr=$(buildah from {_STORE_FLAG} {_LOCAL_TAG}); "
+        f"set -e; ctr=$(buildah from {_STORE_FLAG} {tag}); "
         f"mnt=$(buildah mount {_STORE_FLAG} \"$ctr\"); "
         f"tar -C \"$mnt\" -cf - ."
     )
@@ -196,5 +166,5 @@ def _stream_rootfs(private_key: Path, guest_ip: str, base: Path) -> None:
                if ssh_proc.stderr else "")
     rc = ssh_proc.wait()
     if rc != 0 or untar.returncode != 0:
-        die(f"exporting the built rootfs from the builder VM failed: "
+        die(f"exporting the built rootfs from the infra VM failed: "
             f"{ssh_err.strip() or '<no stderr>'}")