ci(coverage): run the diff-coverage gate on a self-hosted KVM runner

Re-land the coverage gate deferred from #343. The Firecracker VM/SSH
orchestration (~230 lines) is only exercised by the integration suite,
which needs /dev/kvm + the provisioned TAP/nft pool — a container runner
skips it and those lines read uncovered, so the 90% diff gate can't pass
on ubuntu-latest. Move the `coverage` job to a self-hosted `kvm` runner
with a firecracker-readiness preflight (binary + /dev/kvm + `backend
status`) so the integration test actually runs. Unit/lint stay on
ubuntu-latest. README documents the runner prerequisites.

Depends on a registered self-hosted runner labelled `kvm`; until one is
provisioned this gate will not run. See PRD 0069 / #348.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WBMWTEtQdJ4W5UrWuLHCck

.gitea/workflows/test.yml

@@ -71,30 +71,45 @@ jobs:
       - name: Run integration tests
         run: python3 -m unittest discover -t . -s tests/integration -v
 
-  # Combined unit+integration coverage report (informational). See
+  # Combined unit+integration coverage + the diff-coverage gate (the hard
-  # docs/decisions/0004-coverage-policy.md.
+  # gate: new/changed lines >= 90%). See docs/decisions/0004-coverage-policy.md.
   #
-  # The hard diff-coverage gate (changed lines >= 90%) is DEFERRED: the
+  # This runs on a self-hosted KVM runner (label `kvm`), NOT ubuntu-latest,
-  # Firecracker backend's VM/SSH orchestration is covered by the integration
+  # because the Firecracker backend's subprocess/VM orchestration
-  # suite, which needs /dev/kvm + the provisioned TAP/nft pool — a
+  # (launch/boot/SSH/isolation-probe) is covered by the integration suite,
-  # container-based runner skips it and those lines read uncovered, so the
+  # and that suite needs `/dev/kvm` + the provisioned TAP/nft pool — which a
-  # gate can't pass here. Re-enabling it on a self-hosted KVM runner is
+  # container-based runner doesn't have. On such a runner the firecracker
-  # tracked separately (see PRD 0069 / #348 and the ci-runner branch).
+  # integration test skips and its ~230 orchestration lines read as
+  # uncovered, so the gate can't pass there.
+  #
+  # Runner prerequisites (provision once on the host; see the README
+  # "Firecracker on Linux" section): the `firecracker` binary on PATH,
+  # `/dev/kvm` accessible to the runner user, Docker, the cached guest
+  # kernel + static dropbear (BOT_BOTTLE_FC_KERNEL / BOT_BOTTLE_FC_DROPBEAR),
+  # and the network pool installed as the persistent systemd unit
+  # (`./cli.py backend setup --backend=firecracker`). The preflight step
+  # below fails fast with instructions if anything is missing.
   coverage:
-    runs-on: ubuntu-latest
+    runs-on: [self-hosted, kvm]
     steps:
       - name: Checkout
         uses: actions/checkout@v4
         with:
           fetch-depth: 0
 
-      - name: Set up Python
+      - name: Preflight — Firecracker host is ready
-        uses: actions/setup-python@v5
+        run: |
-        with:
+          command -v firecracker >/dev/null || {
-          python-version: "3.12"
+            echo "firecracker not on PATH — provision the runner (README: Firecracker on Linux)"; exit 1; }
+          test -e /dev/kvm || { echo "/dev/kvm missing — KVM not available on this runner"; exit 1; }
+          # `backend status` exits non-zero unless the TAP pool is up + no
+          # range overlap; it prints the exact `backend setup` fix.
+          python3 cli.py backend status --backend=firecracker
 
-      - name: Install dev requirements
+      - name: Combined coverage (unit + integration, incl. firecracker)
-        run: python3 -m pip install -r requirements-dev.txt
-
-      - name: Combined coverage report (unit + integration)
         run: PYTHON=python3 bash scripts/coverage.sh critical
+
+      - name: Diff-coverage gate (changed lines >= 90%)
+        run: |
+          git fetch --no-tags origin main:refs/remotes/origin/main
+          python3 scripts/diff_coverage.py --base origin/main --min 90

README.md

@@ -5,7 +5,7 @@
 # bot-bottle
 
 [![test](https://gitea.dideric.is/didericis/bot-bottle/actions/workflows/test.yml/badge.svg?branch=main)](https://gitea.dideric.is/didericis/bot-bottle/actions?workflow=test.yml)
-[![coverage](https://img.shields.io/badge/coverage-82%25-brightgreen)](https://coverage.readthedocs.io/)
+[![coverage](https://img.shields.io/badge/coverage-84%25-brightgreen)](https://coverage.readthedocs.io/)
 [![core coverage](https://img.shields.io/badge/core%20coverage-95%25-brightgreen)](https://gitea.dideric.is/didericis/bot-bottle/src/branch/main/docs/decisions/0004-coverage-policy.md)
 
 **Problem:** Developer wants to run a coding agent without supervision, but they don't want a prompt injected or misbehaving agent wrecking their environment or exfiltrating sensitive data.
@@ -90,6 +90,8 @@ BOT_BOTTLE_BACKEND=firecracker ./cli.py start <agent>
 
 > **NixOS:** enable `virtualisation.docker`, ensure the KVM module is loaded (`boot.kernelModules = [ "kvm-intel" ];` or `kvm-amd`), and add your user to the `kvm` and `docker` groups. For the network pool, consume the flake module — `imports = [ inputs.bot-bottle.nixosModules.firecracker-netpool ]; services.bot-bottle-firecracker = { enable = true; owner = "you"; };` — then `nixos-rebuild switch` (imperative nft/TAP rules don't survive a rebuild; channel users can `imports = [ <bot-bottle>/nix/firecracker-netpool.nix ]`). `firecracker` isn't in nixpkgs by default as a user binary — install the release binary (pin the version) and put it on `PATH`.
 
+> **CI:** the coverage gate (`.gitea/workflows/test.yml` → `coverage` job) runs on a self-hosted runner labelled `kvm`, because the Firecracker backend's VM/SSH orchestration is exercised only by the integration suite, which needs `/dev/kvm` + the provisioned pool (a container runner would skip it and read as uncovered). Provision that runner exactly like a normal Firecracker host — `firecracker` on `PATH`, `/dev/kvm`, Docker, the cached guest kernel + static dropbear, and the pool installed as the persistent systemd unit — then register it with the `kvm` label. The unit/lint jobs still run on `ubuntu-latest`.
+
 ```sh
 ./cli.py start <agent>   # builds the image on first run, drops you into claude
 ```

bot_bottle/agent_provider.py

@@ -61,13 +61,6 @@ class AgentProviderRuntime:
     prompt_mode: PromptMode
     bypass_args: tuple[str, ...]
     resume_args: tuple[str, ...]
-    # argv run inside a throwaway container of a freshly built agent
-    # image, right after `build_image()`, to catch a build that
-    # exited 0 but produced a broken CLI (e.g. an npm
-    # optionalDependencies fetch for a platform-native binary that
-    # silently no-ops on a transient failure). Empty tuple skips the
-    # check — not every provider has opted in yet.
-    smoke_test: tuple[str, ...] = ()
 
 
 @dataclass(frozen=True)

bot_bottle/backend/docker/launch.py

@@ -36,7 +36,6 @@ from contextlib import ExitStack, contextmanager
 from pathlib import Path
 from typing import Callable, Generator
 
-from ...agent_provider import runtime_for
 from ...egress import egress_resolve_token_values
 from ...git_gate import (
     provision_git_gate_dynamic_keys,
@@ -112,9 +111,6 @@ def launch(
                 plan.image, _REPO_DIR,
                 dockerfile=plan.dockerfile_path,
             )
-            docker_mod.verify_agent_image(
-                plan.image, runtime_for(plan.agent_provider_template).smoke_test,
-            )
 
         internal_network = network_mod.network_name_for_slug(plan.slug)
         egress_network = network_mod.network_egress_name_for_slug(plan.slug)

bot_bottle/backend/docker/util.py

@@ -4,7 +4,6 @@ existence, and building images."""
 
 from __future__ import annotations
 
-import os
 import re
 import shutil
 import subprocess
@@ -89,31 +88,6 @@ def docker_exec_root(container: str, argv: list[str]) -> None:
     )
 
 
-def docker_exec(container: str, argv: list[str], *, user: str = "") -> None:
-    """Run `docker exec` in the named container, dying with the
-    command's own stderr on failure. Pass `user=\"0\"` to run as root."""
-    cmd = ["docker", "exec"]
-    if user:
-        cmd += ["-u", user]
-    cmd += [container, *argv]
-    result = subprocess.run(cmd, capture_output=True, text=True, check=False)
-    if result.returncode != 0:
-        die(
-            f"docker exec in {container} failed: "
-            f"{(result.stderr or '').strip() or '<no stderr>'}"
-        )
-
-
-def docker_cp(src: str, dest: str) -> None:
-    """Run `docker cp`, dying with the command's own stderr on failure."""
-    result = subprocess.run(
-        ["docker", "cp", src, dest], capture_output=True, text=True, check=False,
-    )
-    if result.returncode != 0:
-        die(f"docker cp {src} -> {dest} failed: "
-            f"{(result.stderr or '').strip() or '<no stderr>'}")
-
-
 _SLUG_RE = re.compile(r"[^a-z0-9]+")
 
 
@@ -134,45 +108,15 @@ def build_image(ref: str, context: str, *, dockerfile: str = "") -> None:
 
     `dockerfile` is an optional path (relative to `context`, or
     absolute) for callers that need to build from a non-default
-    Dockerfile in the same context — e.g. `Dockerfile.git-gate`.
+    Dockerfile in the same context — e.g. `Dockerfile.git-gate`."""
-
-    Set `BOT_BOTTLE_NO_CACHE=1` (the `start --no-cache` flag) to force
-    `--no-cache`. The npm/curl installers some provider Dockerfiles
-    shell out to can silently no-op on a transient network failure —
-    e.g. an `optionalDependencies` fetch for a platform-native binary —
-    and Docker will then cache that broken layer indefinitely."""
     info(f"building image {ref} from {context} (layer cache keeps repeat builds fast)")
     args = ["docker", "build", "-t", ref]
-    if os.environ.get("BOT_BOTTLE_NO_CACHE") == "1":
-        args.append("--no-cache")
     if dockerfile:
         args.extend(["-f", dockerfile])
     args.append(context)
     subprocess.run(args, check=True)
 
 
-def verify_agent_image(image: str, argv: tuple[str, ...]) -> None:
-    """Run `argv` inside a throwaway container of a freshly built agent
-    image and die loudly if it fails, instead of shipping an image
-    whose CLI only breaks at first real use. No-op when the provider
-    hasn't declared a smoke test (`AgentProviderRuntime.smoke_test`)."""
-    if not argv:
-        return
-    result = subprocess.run(
-        ["docker", "run", "--rm", "--entrypoint", argv[0], image, *argv[1:]],
-        capture_output=True,
-        text=True,
-        check=False,
-    )
-    if result.returncode != 0:
-        detail = (result.stderr or result.stdout or "").strip()
-        die(
-            f"agent image {image!r} failed its post-build smoke test "
-            f"({' '.join(argv)}): {detail}\n"
-            f"Try rebuilding from scratch: bot-bottle start --no-cache"
-        )
-
-
 # def build_image_with_cwd(
 #     derived: str,
 #     base: str,

bot_bottle/backend/firecracker/launch.py

@@ -24,7 +24,6 @@ from contextlib import ExitStack, contextmanager
 from pathlib import Path
 from typing import Callable, Generator
 
-from ...agent_provider import runtime_for
 from ...bottle_state import (
     egress_state_dir,
     git_gate_state_dir,
@@ -43,7 +42,6 @@ from ...git_gate import (
 from ...log import die, info, warn
 from ...supervise import DB_PATH_IN_CONTAINER, SUPERVISE_PORT
 from ...util import expand_tilde
-from ..docker import util as docker_mod
 from ..docker.egress import (
     EGRESS_CA_IN_CONTAINER,
     EGRESS_PORT,
@@ -108,9 +106,9 @@ def launch(
         plan = _provision_git_gate_keys(plan)
 
         sidecar_name = sidecar_container_name(plan.slug)
-        docker_mod.force_remove_container(sidecar_name)
+        _force_remove_container(sidecar_name)
         _start_sidecar_bundle(plan, sidecar_name, slot.host_ip)
-        stack.callback(docker_mod.force_remove_container, sidecar_name)
+        stack.callback(_force_remove_container, sidecar_name)
         _stage_git_gate(plan, sidecar_name)
 
         plan = _stamp_agent_urls(plan, slot.host_ip)
@@ -172,18 +170,15 @@ def _mint_certs(plan: FirecrackerBottlePlan) -> FirecrackerBottlePlan:
 
 
 def _build_agent_image(plan: FirecrackerBottlePlan) -> FirecrackerBottlePlan:
-    docker_mod.build_image(SIDECAR_BUNDLE_IMAGE, _REPO_DIR, dockerfile=SIDECAR_BUNDLE_DOCKERFILE)
+    _docker_build(SIDECAR_BUNDLE_IMAGE, _REPO_DIR, dockerfile=SIDECAR_BUNDLE_DOCKERFILE)
     committed = read_committed_image(plan.slug)
-    if committed and docker_mod.image_exists(committed):
+    if committed and _image_exists(committed):
         info(f"using committed image {committed!r}")
         return dataclasses.replace(
             plan,
             agent_provision=dataclasses.replace(plan.agent_provision, image=committed),
         )
-    docker_mod.build_image(plan.image, _REPO_DIR, dockerfile=plan.dockerfile_path)
+    _docker_build(plan.image, _REPO_DIR, dockerfile=plan.dockerfile_path)
-    docker_mod.verify_agent_image(
-        plan.image, runtime_for(plan.agent_provider_template).smoke_test,
-    )
     return plan
 
 
@@ -296,15 +291,15 @@ def _stage_git_gate(plan: FirecrackerBottlePlan, sidecar_name: str) -> None:
     gp = plan.git_gate_plan
     if not gp.upstreams:
         return
-    docker_mod.docker_exec(sidecar_name, [
+    _docker_exec(sidecar_name, [
         "mkdir", "-p",
         str(Path(GIT_GATE_HOOK_IN_CONTAINER).parent),
         GIT_GATE_CREDS_DIR_IN_CONTAINER, "/git",
         str(Path(_GIT_GATE_READY_FILE).parent),
     ])
     for host_path, container_path in _git_gate_files(plan):
-        docker_mod.docker_cp(host_path, f"{sidecar_name}:{container_path}")
+        _docker_cp(host_path, f"{sidecar_name}:{container_path}")
-    docker_mod.docker_exec(sidecar_name, [
+    _docker_exec(sidecar_name, [
         "sh", "-c",
         f"chmod 755 {GIT_GATE_ENTRYPOINT_IN_CONTAINER} "
         f"{GIT_GATE_HOOK_IN_CONTAINER} {GIT_GATE_ACCESS_HOOK_IN_CONTAINER} && "
@@ -360,3 +355,50 @@ def _agent_guest_env(plan: FirecrackerBottlePlan, host_ip: str) -> dict[str, str
         if value is not None:
             env[name] = value
     return env
+
+
+# --- docker helpers --------------------------------------------------
+
+def _docker_build(ref: str, context: str, *, dockerfile: str = "") -> None:
+    info(f"docker build {ref}")
+    args = ["docker", "build", "-t", ref]
+    if dockerfile:
+        if not os.path.isabs(dockerfile):
+            dockerfile = os.path.join(context, dockerfile)
+        args += ["-f", dockerfile]
+    args.append(context)
+    result = subprocess.run(args, check=False)
+    if result.returncode != 0:
+        die(f"docker build for {ref!r} failed")
+
+
+def _image_exists(ref: str) -> bool:
+    return subprocess.run(
+        ["docker", "image", "inspect", ref],
+        stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL, check=False,
+    ).returncode == 0
+
+
+def _force_remove_container(name: str) -> None:
+    subprocess.run(
+        ["docker", "rm", "-f", name],
+        stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL, check=False,
+    )
+
+
+def _docker_exec(name: str, argv: list[str]) -> None:
+    result = subprocess.run(
+        ["docker", "exec", name, *argv], capture_output=True, text=True, check=False,
+    )
+    if result.returncode != 0:
+        die(f"docker exec in {name} failed: "
+            f"{(result.stderr or '').strip() or '<no stderr>'}")
+
+
+def _docker_cp(host_path: str, dest: str) -> None:
+    result = subprocess.run(
+        ["docker", "cp", host_path, dest], capture_output=True, text=True, check=False,
+    )
+    if result.returncode != 0:
+        die(f"docker cp {host_path} -> {dest} failed: "
+            f"{(result.stderr or '').strip() or '<no stderr>'}")

bot_bottle/backend/macos_container/launch.py

@@ -17,7 +17,6 @@ from contextlib import ExitStack, contextmanager
 from pathlib import Path
 from typing import Callable, Generator
 
-from ...agent_provider import runtime_for
 from ...bottle_state import (
     egress_state_dir,
     git_gate_state_dir,
@@ -170,9 +169,6 @@ def _build_images(plan: MacosContainerBottlePlan) -> MacosContainerBottlePlan:
         _REPO_DIR,
         dockerfile=plan.dockerfile_path,
     )
-    container_mod.verify_agent_image(
-        plan.image, runtime_for(plan.agent_provider_template).smoke_test,
-    )
     return plan
 
 

bot_bottle/backend/macos_container/util.py

@@ -60,21 +60,13 @@ def dns_server() -> str:
 
 
 def build_image(ref: str, context: str, *, dockerfile: str = "") -> None:
-    """Build an OCI image with Apple's BuildKit-backed `container build`.
+    """Build an OCI image with Apple's BuildKit-backed `container build`."""
-
-    Set `BOT_BOTTLE_NO_CACHE=1` (the `start --no-cache` flag) to force
-    `--no-cache`. The npm/curl installers some provider Dockerfiles
-    shell out to can silently no-op on a transient network failure —
-    e.g. an `optionalDependencies` fetch for a platform-native binary —
-    and the builder will then cache that broken layer indefinitely."""
     info(
         f"building image {ref} from {context} with Apple Container "
         "(layer cache keeps repeat builds fast)"
     )
     _ensure_builder_dns()
     args = [_CONTAINER, "build", "-t", ref, "--dns", dns_server()]
-    if os.environ.get("BOT_BOTTLE_NO_CACHE") == "1":
-        args.append("--no-cache")
     if dockerfile:
         # `container build` resolves -f relative to the current working
         # directory, not the build context. Anchor a relative Dockerfile to
@@ -86,28 +78,6 @@ def build_image(ref: str, context: str, *, dockerfile: str = "") -> None:
     subprocess.run(args, check=True)
 
 
-def verify_agent_image(image: str, argv: tuple[str, ...]) -> None:
-    """Run `argv` inside a throwaway container of a freshly built agent
-    image and die loudly if it fails, instead of shipping an image
-    whose CLI only breaks at first real use. No-op when the provider
-    hasn't declared a smoke test (`AgentProviderRuntime.smoke_test`)."""
-    if not argv:
-        return
-    result = subprocess.run(
-        [_CONTAINER, "run", "--rm", "--entrypoint", argv[0], image, *argv[1:]],
-        capture_output=True,
-        text=True,
-        check=False,
-    )
-    if result.returncode != 0:
-        detail = (result.stderr or result.stdout or "").strip()
-        die(
-            f"agent image {image!r} failed its post-build smoke test "
-            f"({' '.join(argv)}): {detail}\n"
-            f"Try rebuilding from scratch: bot-bottle start --no-cache"
-        )
-
-
 def commit_container(container_name: str, image_tag: str) -> None:
     """Snapshot a running Apple Container as a local image.
 

bot_bottle/cli/start.py

@@ -46,17 +46,6 @@ def cmd_start(argv: list[str]) -> int:
     parser = argparse.ArgumentParser(prog=f"{PROG} start", add_help=True)
     parser.add_argument("--dry-run", action="store_true")
     parser.add_argument("--cwd", action="store_true", help="copy host cwd into the running bottle")
-    parser.add_argument(
-        "--no-cache",
-        action="store_true",
-        help=(
-            "rebuild agent/sidecar images from scratch, bypassing the "
-            "build layer cache. Use when an image looks broken after a "
-            "dependency bump — e.g. an installer's optionalDependencies "
-            "fetch silently no-op'd on a transient failure and got baked "
-            "into a cached layer."
-        ),
-    )
     parser.add_argument(
         "--backend",
         choices=known_backend_names(),
@@ -108,11 +97,6 @@ def cmd_start(argv: list[str]) -> int:
     args = parser.parse_args(argv)
 
     dry_run = args.dry_run or os.environ.get("BOT_BOTTLE_DRY_RUN") == "1"
-    if args.no_cache or os.environ.get("BOT_BOTTLE_NO_CACHE") == "1":
-        # Read by build_image() in each backend's util module — set here
-        # so both the interactive and --headless paths pick it up without
-        # threading a no_cache field through every backend's plan dataclass.
-        os.environ["BOT_BOTTLE_NO_CACHE"] = "1"
 
     manifest = ManifestIndex.resolve(USER_CWD)
     backend_name: str | None = args.backend

bot_bottle/contrib/claude/agent_provider.py

@@ -91,7 +91,6 @@ _RUNTIME = AgentProviderRuntime(
     prompt_mode="append_file",
     bypass_args=("--dangerously-skip-permissions",),
     resume_args=("--continue",),
-    smoke_test=("claude", "--version"),
 )
 
 

bot_bottle/contrib/codex/agent_provider.py

@@ -61,7 +61,6 @@ _RUNTIME = AgentProviderRuntime(
     prompt_mode="read_prompt_file",
     bypass_args=("--dangerously-bypass-approvals-and-sandbox",),
     resume_args=("resume", "--last"),
-    smoke_test=(_CODEX_CLI, "--version"),
 )
 
 

tests/unit/test_docker_launch_committed_image.py

@@ -103,8 +103,6 @@ class TestLaunchCommittedImage(unittest.TestCase):
             launch_mod.docker_mod, "image_exists", return_value=image_present,
         ), mock.patch.object(
             launch_mod.docker_mod, "build_image", side_effect=fake_build,
-        ), mock.patch.object(
-            launch_mod.docker_mod, "verify_agent_image",
         ), mock.patch.object(
             launch_mod, "egress_tls_init",
             return_value=(Path("/egress_ca"), Path("/egress_cert")),

tests/unit/test_docker_launch_teardown.py

@@ -87,7 +87,6 @@ class TestTeardownWarning(unittest.TestCase):
         buf = io.StringIO()
 
         with mock.patch.object(launch_mod.docker_mod, "build_image"), \
-             mock.patch.object(launch_mod.docker_mod, "verify_agent_image"), \
              mock.patch.object(
                  launch_mod, "egress_tls_init",
                  return_value=(Path("/egress_ca"), Path("/egress_cert")),

tests/unit/test_macos_container_launch.py

@@ -357,17 +357,12 @@ class TestMacosContainerLaunchCommittedImage(unittest.TestCase):
             launch.container_mod, "image_exists", return_value=False,
         ), patch.object(
             launch.container_mod, "build_image", side_effect=fake_build,
-        ), patch.object(
+        ):
-            launch.container_mod, "verify_agent_image",
-        ) as verify:
             updated = launch._build_images(plan)
 
         self.assertEqual("bot-bottle-agent:latest", updated.image)
         self.assertEqual(2, len(calls))
         self.assertEqual("bot-bottle-agent:latest", calls[1][0])
-        verify.assert_called_once_with(
-            "bot-bottle-agent:latest", ("claude", "--version"),
-        )
 
 
 if __name__ == "__main__":