ci(test): split integration into per-backend jobs

Add separate `integration-docker` and `integration-firecracker` jobs,
each with an explicit BOT_BOTTLE_BACKEND env var, so the backend used
is visible in CI output and skipped backends surface as a distinct job
rather than silent unittest.skip lines.

- integration-docker: ubuntu-latest, BOT_BOTTLE_BACKEND=docker
- integration-firecracker: [self-hosted, kvm], BOT_BOTTLE_BACKEND=firecracker,
  same-repo PRs + push + workflow_dispatch only (untrusted fork PRs do
  not execute on the privileged KVM runner)
- coverage: same same-repo restriction; refs #414 for the planned
  follow-up that moves coverage to ubuntu-latest via artifact combination

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

.gitea/workflows/test.yml

@@ -4,16 +4,15 @@
 # dependencies are required to execute it. Tests are split by directory:
 #
 #   tests/unit/         — pure unit tests; always run
-#   tests/integration/  — need a reachable Docker daemon; skip cleanly
+#   tests/integration/  — need a reachable backend; skip cleanly when
-#                         (via tests/_docker.py:skip_unless_docker) when
+#                         the backend isn't available on the runner
-#                         Docker isn't available on the runner
 #   tests/canaries/     — upstream regression canaries; run on a separate
 #                         schedule (see canaries.yml), not here
 #
-# This workflow assumes the Gitea Actions runner exposes the host Docker
+# Integration tests run once per backend in separate jobs. Each job sets
-# socket to the job container so `docker` commands inside the job can
+# BOT_BOTTLE_BACKEND explicitly so the test suite uses the right backend.
-# reach the daemon. If that's not yet configured on the runner the
+# Backends that aren't available on the runner fail the preflight step
-# integration tests will skip rather than fail.
+# rather than silently skipping inside the test output.
 
 name: test
 
@@ -23,9 +22,16 @@ on:
       - main
     paths:
       - '**.py'
+      - '.gitea/workflows/**.yml'
+      - 'scripts/**'
+      - 'README.md'
   pull_request:
     paths:
       - '**.py'
+      - '.gitea/workflows/**.yml'
+      - 'scripts/**'
+      - 'README.md'
+  workflow_dispatch:
 
 jobs:
   unit:
@@ -48,7 +54,7 @@ jobs:
       - name: Report unit coverage
         run: python3 -m coverage report -m
 
-  integration:
+  integration-docker:
     runs-on: ubuntu-latest
     steps:
       - name: Checkout
@@ -65,33 +71,95 @@ jobs:
             echo "docker not on PATH — integration tests will skip"
           fi
 
-      - name: Run integration tests
+      - name: Run integration tests (docker)
+        env:
+          BOT_BOTTLE_BACKEND: docker
         run: python3 -m unittest discover -t . -s tests/integration -v
 
-  # Combined unit+integration coverage report (informational). See
+  # Integration tests against the Firecracker backend. Runs on a self-hosted
-  # docs/decisions/0004-coverage-policy.md.
+  # KVM runner (label `kvm`) where /dev/kvm and the TAP/nft pool are available.
   #
-  # The hard diff-coverage gate (changed lines >= 90%) is DEFERRED: the
+  # Restricted to same-repo PRs, push to main, and workflow_dispatch — fork
-  # Firecracker backend's VM/SSH orchestration is covered by the integration
+  # PRs don't execute untrusted code on the privileged runner.
-  # suite, which needs /dev/kvm + the provisioned TAP/nft pool — a
+  #
-  # container-based runner skips it and those lines read uncovered, so the
+  # Runner prerequisites (provision once; see README "Firecracker on Linux"):
-  # gate can't pass here. Re-enabling it on a self-hosted KVM runner is
+  # `firecracker` on PATH, `/dev/kvm` accessible, Docker, cached kernel +
-  # tracked separately (see PRD 0069 / #348 and the ci-runner branch).
+  # static dropbear, and the pool as a persistent systemd unit.
+  integration-firecracker:
+    runs-on: [self-hosted, kvm]
+    if: >-
+      github.event_name == 'push' ||
+      github.event_name == 'workflow_dispatch' ||
+      (github.event_name == 'pull_request' &&
+       github.event.pull_request.head.repo.full_name == github.repository)
+    steps:
+      - name: Checkout
+        uses: actions/checkout@v4
+
+      - name: Preflight — Firecracker host is ready
+        run: |
+          command -v firecracker >/dev/null || {
+            echo "firecracker not on PATH — provision the runner (README: Firecracker on Linux)"; exit 1; }
+          test -e /dev/kvm || { echo "/dev/kvm missing — KVM not available on this runner"; exit 1; }
+          # `backend status` exits non-zero unless the TAP pool is up + no
+          # range overlap; it prints the exact `backend setup` fix.
+          python3 cli.py backend status --backend=firecracker
+
+      - name: Install dev requirements
+        run: python3 -m pip install --user -r requirements-dev.txt
+
+      - name: Run integration tests (firecracker)
+        env:
+          BOT_BOTTLE_BACKEND: firecracker
+        run: python3 -m unittest discover -t . -s tests/integration -v
+
+  # Combined unit+integration coverage + the diff-coverage gate (the hard
+  # gate: new/changed lines >= 90%). See docs/decisions/0004-coverage-policy.md.
+  #
+  # This runs on a self-hosted KVM runner (label `kvm`), NOT ubuntu-latest,
+  # because the Firecracker backend's subprocess/VM orchestration
+  # (launch/boot/SSH/isolation-probe) is covered by the integration suite,
+  # and that suite needs `/dev/kvm` + the provisioned TAP/nft pool — which a
+  # container-based runner doesn't have. On such a runner the firecracker
+  # integration test skips and its ~230 orchestration lines read as
+  # uncovered, so the gate can't pass there.
+  #
+  # Restricted to the same events as integration-firecracker (same-repo PRs,
+  # push, workflow_dispatch) for the same security reason.
+  #
+  # See #414 for the planned follow-up: artifact-based coverage combination
+  # (run tests once in their respective jobs, combine .coverage files here).
   coverage:
-    runs-on: ubuntu-latest
+    runs-on: [self-hosted, kvm]
+    if: >-
+      github.event_name == 'push' ||
+      github.event_name == 'workflow_dispatch' ||
+      (github.event_name == 'pull_request' &&
+       github.event.pull_request.head.repo.full_name == github.repository)
     steps:
       - name: Checkout
         uses: actions/checkout@v4
         with:
           fetch-depth: 0
 
-      # No actions/setup-python: the runner image already ships Python 3.12,
+      - name: Preflight — Firecracker host is ready
-      # and older act_runner engines mishandle setup-python's PATH (coverage
+        run: |
-      # lands in one interpreter, `python3` resolves to another). Install
+          command -v firecracker >/dev/null || {
-      # straight into the ephemeral job container's system Python —
+            echo "firecracker not on PATH — provision the runner (README: Firecracker on Linux)"; exit 1; }
-      # --break-system-packages is safe because the container is disposable.
+          test -e /dev/kvm || { echo "/dev/kvm missing — KVM not available on this runner"; exit 1; }
-      - name: Install dev requirements
+          # `backend status` exits non-zero unless the TAP pool is up + no
-        run: python3 -m pip install --break-system-packages -r requirements-dev.txt
+          # range overlap; it prints the exact `backend setup` fix.
+          python3 cli.py backend status --backend=firecracker
 
-      - name: Combined coverage report (unit + integration)
+      - name: Install dev requirements
+        run: python3 -m pip install --user -r requirements-dev.txt
+
+      - name: Combined coverage (unit + integration, incl. firecracker)
+        env:
+          BOT_BOTTLE_BACKEND: firecracker
         run: PYTHON=python3 bash scripts/coverage.sh critical
+
+      - name: Diff-coverage gate (changed lines >= 90%)
+        run: |
+          git fetch --no-tags origin main:refs/remotes/origin/main
+          python3 scripts/diff_coverage.py --base origin/main --min 90

README.md

@@ -90,6 +90,8 @@ BOT_BOTTLE_BACKEND=firecracker ./cli.py start <agent>
 
 > **NixOS:** enable `virtualisation.docker`, ensure the KVM module is loaded (`boot.kernelModules = [ "kvm-intel" ];` or `kvm-amd`), and add your user to the `kvm` and `docker` groups. For the network pool, consume the flake module — `imports = [ inputs.bot-bottle.nixosModules.firecracker-netpool ]; services.bot-bottle-firecracker = { enable = true; owner = "you"; };` — then `nixos-rebuild switch` (imperative nft/TAP rules don't survive a rebuild; channel users can `imports = [ <bot-bottle>/nix/firecracker-netpool.nix ]`). `firecracker` isn't in nixpkgs by default as a user binary — install the release binary (pin the version) and put it on `PATH`.
 
+> **CI:** the coverage gate (`.gitea/workflows/test.yml` → `coverage` job) runs on a self-hosted runner labelled `kvm`, because the Firecracker backend's VM/SSH orchestration is exercised only by the integration suite, which needs `/dev/kvm` + the provisioned pool (a container runner would skip it and read as uncovered). Provision that runner exactly like a normal Firecracker host — `firecracker` on `PATH`, `/dev/kvm`, Docker, the cached guest kernel + static dropbear, and the pool installed as the persistent systemd unit — then register it with the `kvm` label. The unit/lint jobs still run on `ubuntu-latest`.
+
 ```sh
 ./cli.py start <agent>   # builds the image on first run, drops you into claude
 ```

tests/integration/test_sandbox_escape.py

@@ -69,11 +69,12 @@ _DUMMY_HOST_KEY = (
 
 @skip_unless_docker()
 @unittest.skipIf(
-    os.environ.get("GITEA_ACTIONS") == "true",
+    os.environ.get("GITEA_ACTIONS") == "true"
-    "skipped under act_runner: egress_tls_init uses a host bind mount "
+    and os.environ.get("BOT_BOTTLE_BACKEND") != "firecracker",
-    "the runner container can't see, and the network topology hides "
+    "skipped under act_runner unless BOT_BOTTLE_BACKEND=firecracker: "
-    "sibling-gateway visibility — same constraint as the other "
+    "egress_tls_init uses a host bind mount the runner container can't "
-    "bottle-bringup integration tests",
+    "see, and the network topology hides sibling-gateway visibility — "
+    "these constraints don't apply on the self-hosted KVM runner",
 )
 class TestSandboxEscape(unittest.TestCase):
     """End-to-end attacks against a real bottle. The bottle stays