From 671d91070e8cec27075510f16722cb2d49eec2cd Mon Sep 17 00:00:00 2001 From: codex Date: Sun, 26 Jul 2026 08:02:12 +0000 Subject: [PATCH] docs(ci): document the assurance topology --- README.md | 4 +-- docs/ci.md | 83 ++++++++++++++++++++++++------------------------- tests/README.md | 20 +++++++----- 3 files changed, 55 insertions(+), 52 deletions(-) diff --git a/README.md b/README.md index dae91e9b..da908a87 100644 --- a/README.md +++ b/README.md @@ -75,7 +75,7 @@ On compatible macOS hosts, the default backend requires Apple's `container` CLI Use `BOT_BOTTLE_BACKEND=docker ./cli.py start ` on hosts where neither Apple Container nor KVM is available and Docker is the desired backend. -> **CI (macOS Apple Container):** the `integration-macos` job (`.gitea/workflows/test.yml`) runs the integration suite against `BOT_BOTTLE_BACKEND=macos-container` on a self-hosted macOS runner labelled `macos`, because Apple Container needs the host virtualization framework and cannot run in a Linux container (so it can't reuse the `kvm` runner). Provision an Apple Silicon host with the `container` CLI on `PATH` and `container system status` running, then register the runner in **host mode** (not docker mode) with the `macos` label — `brew install gitea-runner` (the `act_runner` rename). Give it a Python ≥ 3.11 with `coverage` importable on the launchd service's `PATH` (a launchd service doesn't inherit your shell profile, so pin `node` and the Python env explicitly). The job is **advisory** — `workflow_dispatch` (manual) only, never triggered by push or PR — since a single laptop that sleeps/roams must not block merges or churn on every push to main; its coverage doesn't feed the gate. The infra container is a singleton (`bot-bottle-mac-infra`), so keep runner concurrency at 1. +> **CI (macOS Apple Container):** the advisory `integration-macos` job in `.gitea/workflows/pre-release-test.yml` runs only on manual dispatch. It targets a self-hosted host-mode runner labelled `macos`; Apple Container cannot run inside the Linux pull-request runner. Provision an Apple Silicon host with the `container` CLI running and Python ≥ 3.11 plus `coverage` on the launchd service's explicit `PATH`. The infra container is a singleton (`bot-bottle-mac-infra`), so keep runner concurrency at 1. Its coverage is reported separately and never feeds the required pull-request gate. ### Containers inside a bottle @@ -174,7 +174,7 @@ BOT_BOTTLE_BACKEND=firecracker ./cli.py start > **NixOS:** enable `virtualisation.docker`, ensure the KVM module is loaded (`boot.kernelModules = [ "kvm-intel" ];` or `kvm-amd`), and add your user to the `kvm` and `docker` groups. For the network pool, consume the flake module — `imports = [ inputs.bot-bottle.nixosModules.firecracker-netpool ]; services.bot-bottle-firecracker = { enable = true; owner = "you"; };` — then `nixos-rebuild switch` (imperative nft/TAP rules don't survive a rebuild; channel users can `imports = [ /nix/firecracker-netpool.nix ]`). `firecracker` isn't in nixpkgs by default as a user binary — install the release binary (pin the version) and put it on `PATH`. -> **CI:** the coverage gate (`.gitea/workflows/test.yml` → `coverage` job) runs on a self-hosted runner labelled `kvm`, because the Firecracker backend's VM/SSH orchestration is exercised only by the integration suite, which needs `/dev/kvm` + the provisioned pool (a container runner would skip it and read as uncovered). Provision that runner exactly like a normal Firecracker host — `firecracker` on `PATH`, `/dev/kvm`, the cached guest kernel + static dropbear, and the pool installed as the persistent systemd unit — then register it with the `kvm` label. A Docker-capable hosted job builds the candidate once; KVM tests boot those exact bytes, and a successful main run publishes them. The unit/lint jobs still run on `ubuntu-latest`. +> **CI:** Firecracker integration runs in the manually dispatched `.gitea/workflows/pre-release-test.yml` on a self-hosted runner labelled `kvm`; privileged KVM hosts never execute unreviewed PR code automatically. Provision it like a normal Firecracker host: `firecracker` on `PATH`, `/dev/kvm`, the cached guest kernel and static dropbear, and the persistent TAP/nft pool. The required pull-request workflow runs unit plus the complete Docker integration suite on `ubuntu-latest`; see `docs/ci.md`. ```sh ./cli.py start # builds the image on first run, drops you into claude diff --git a/docs/ci.md b/docs/ci.md index 64e96c5d..4eeb5ae7 100644 --- a/docs/ci.md +++ b/docs/ci.md @@ -1,50 +1,49 @@ # CI -The test workflow lives at [`.gitea/workflows/test.yml`](../.gitea/workflows/test.yml). -It runs the unit suite plus one integration job per backend -(`integration-docker`, `integration-firecracker`, `integration-macos`) on: +## Required pull-request gate -- every push to a branch with an open pull request, and -- every push to `main`. +[`.gitea/workflows/test.yml`](../.gitea/workflows/test.yml) runs the unit +suite, Docker integration suite, combined coverage report, and diff-coverage +gate when tested package/build inputs change on a pull request or on `main`. -`integration-macos` is the exception: it is **advisory**, running only on -`workflow_dispatch` (manual dispatch), never on push or pull requests. It targets the -Apple Container backend on a self-hosted macOS runner (label `macos`, -registered in host mode — Apple Container can't run in a Linux container, so it -can't reuse the `kvm` runner). A single non-redundant laptop must not be able -to block a PR merge, so the job stays out of the `coverage` job's `needs` and -its coverage never feeds the diff-coverage gate. Because the infra container is -a singleton (`bot-bottle-mac-infra`), the job declares a `concurrency` group -and tears the container down on exit; keep runner concurrency at 1. See the -README "macOS Apple Container" CI note for runner provisioning. +The Docker job preflights the backend before discovery. Gitea's `act_runner` +runs the job in a container with the host Docker socket, so the test process +reaches host-published ports through its bridge gateway and uses named Docker +volumes for orchestrator/CA state the host daemon must mount. The orchestrator +runs the package baked into the image built from the checkout; it does not bind +the job container's invisible workspace into a sibling container. -Each integration job selects its backend via `BOT_BOTTLE_BACKEND` and -runs a **preflight** (`./cli.py backend status --backend=`) that -prints a clear per-check readiness summary and fails the job when the -backend is missing — so absent infrastructure is visible at the job level -rather than hidden among per-test `unittest.skip` lines. The skip guards in -[`tests/_backend.py`](../tests/_backend.py) gate on the same readiness -check (`bot_bottle.backend.has_backend`): backend-agnostic tests use -`skip_unless_selected_backend_available()` and run through whichever -backend is selected (checking, e.g., Linux + `/dev/kvm` for Firecracker -rather than unrelated Docker availability); Docker-implementation tests use -`skip_unless_backend("docker")` and no-op under a non-Docker run. +`scripts.unittest_gate` enforces the Docker job's contract: all 22 integration +tests must execute and none may skip. This includes the real gateway-image, +control-plane authentication, multitenant policy/token isolation, +sandbox-escape, and orphan-network tests. Backend skip decorators remain useful +for local runs, but the CI preflight plus execution-count gate prevents a +missing backend or runner-topology regression from becoming a green job. -A small subset of integration tests skip when running specifically -under Gitea Actions (`GITEA_ACTIONS=true`), because `act_runner` runs -the job inside a container with the host's `/var/run/docker.sock` -mounted in. That topology breaks two assumptions those tests make: +Combined unit + Docker coverage is informational globally. Two focused gates +are enforced: -- networks created via the host daemon aren't always visible to a - same-process `docker network ls` call from inside the job container, - and -- ports published by sibling containers land on the host's loopback, - not on the job container's `127.0.0.1` — so HTTP probes against - `http://127.0.0.1:` from inside the job time out. +- changed executable Python lines must be at least 90% covered; and +- the validated critical security/logic core must remain at least 90% covered. -The affected tests (`test_orphan_cleanup.test_create_and_remove`, -`test_gateway_image.TestGatewayImage`) still run -locally where the test process and Docker daemon share a host. -Making them work in CI is a follow-up: either re-write them to -discover container IPs via `docker inspect`, or reconfigure the -runner with host networking. +## Privileged pre-release matrix + +[`.gitea/workflows/pre-release-test.yml`](../.gitea/workflows/pre-release-test.yml) +is manually dispatched before a release. It repeats unit and Docker integration +coverage, then runs: + +- Firecracker integration on the self-hosted `kvm` runner; and +- advisory Apple Container integration on the self-hosted `macos` runner. + +These privileged host-mode runners never execute unreviewed pull-request code +automatically. Firecracker coverage is combined in the manual pre-release +report; macOS reports advisory coverage in its own job. The macOS infra +container is a singleton, so its job uses a concurrency group and always tears +the service down. + +## Scheduled canary + +[`.gitea/workflows/canaries.yml`](../.gitea/workflows/canaries.yml) runs weekly +and on manual dispatch. It verifies the pinned gitleaks release URL, checksum, +archive shape, and executable. The same unittest execution gate requires at +least one executed canary and rejects skips. diff --git a/tests/README.md b/tests/README.md index 28985cd9..988ffa86 100644 --- a/tests/README.md +++ b/tests/README.md @@ -20,10 +20,11 @@ tests/ ... # many others; see unit/ directory integration/ test_gateway_image.py - test_dry_run_plan.py + test_sandbox_escape.py test_orphan_cleanup.py ... - canaries/ # opt-in; see below (currently empty) + canaries/ + test_gitleaks_release.py # opt-in upstream artifact check ``` Classification falls out of the directory — no hand-maintained list to @@ -43,24 +44,27 @@ Discovery is invoked with `-t .` (top-level dir = repo root) so the ## What the integration tests cover -- `test_dry_run_plan.py` — `cli.py start --dry-run --format=json` emits - a structured plan that contains the resolved egress allowlist and - the bottle's runtime, and creates zero Docker resources. - `test_orphan_cleanup.py` — `network_remove` is idempotent against missing resources, so the EXIT trap can call it unconditionally. - `test_gateway_image.py` — builds Dockerfile.gateway and probes that gitleaks / mitmdump / supervise are all reachable inside the gateway image. +- `test_orchestrator_docker_auth.py` — drives the real control-plane + container and verifies role-scoped authentication. +- `test_multitenant_isolation.py` and `test_sandbox_escape.py` — exercise + token/allowlist separation and end-to-end escape attempts. ## Canaries `tests/canaries/` holds upstream-regression checks gated on `BOT_BOTTLE_RUN_CANARIES=1` and not part of the per-push suite. -They're invoked by the scheduled `canaries` workflow. Currently -no canaries are defined. +They're invoked by the scheduled `canaries` workflow. The gitleaks canary +downloads the exact release archive pinned by `Dockerfile.gateway`, verifies +its architecture-specific checksum, and executes the binary. ```bash -BOT_BOTTLE_RUN_CANARIES=1 python -m unittest discover -t . -s tests/canaries -v +BOT_BOTTLE_RUN_CANARIES=1 python -m scripts.unittest_gate \ + -t . -s tests/canaries -v --minimum-executed 1 --fail-on-skip ``` ## What's NOT covered