refactor: adopt macOS test/test-ready convention for variants
prd-number-check / require-numbered-prds (pull_request) Successful in 16s
tracker-policy-pr / check-pr (pull_request) Failing after 12m35s

Replaces the BB_TEST_PREREQS env toggle with the two subcommands the macOS
harness established, so the two repos read the same:

  test        A bare host — prerequisites NOT set up (the default state of a
              stock cloud image). Exercises install.sh's own prerequisite-guard
              logic. SOUND (PASS) when install.sh either installs cleanly or
              declines with one of its own recognized, actionable errors; a
              crash or unrecognized failure fails.
  test-ready  Prerequisites satisfied — the harness installs python3/git/pipx
              first, then runs install.sh, which must actually land: entry point
              runnable, doctor reporting a usable python and config.

Structure now mirrors scripts/macos-install-test.sh: a shared _test_cycle
driving up -> (prereqs) -> run -> verdict -> down, thin cmd_test / cmd_test_ready
wrappers setting _STEPS / _PASS_CLAIM / _REQUIRE_INSTALL, a standalone `prereqs`
subcommand, and a _test_teardown that prints the PASS/FAIL claim.

The doctor check is now the macOS-style classifier rather than a bare exit-code
read: a Traceback is an install defect (fail), a missing `ok: python:` /
`ok: config:` line is a fail, and a not-ready backend is reported, not fatal
(BB_TEST_REQUIRE_BACKEND=1 makes it fatal for a nested-virt host). This is where
Linux diverges from macOS test-ready — a plain VM has no nested KVM for
Firecracker and the harness doesn't provision the Docker backend, so backend
readiness is never the Linux criterion.

test-all now runs every distro × {test, test-ready}. Research note updated.

Validated with `bash -n` and `shellcheck`.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
2026-07-27 10:07:52 -04:00
parent 84a9fdd457
commit bff06bcedf
2 changed files with 288 additions and 209 deletions
+33 -30
View File
@@ -32,38 +32,42 @@ equivalent of `docker run --rm`, for a whole machine.
| **Throwaway user** (`useradd`/`userdel`) | The macOS pick, but weaker on Linux: it reaches the real host, yet every system package it installs (python, pipx, git via `apt`/`dnf`/…) stays behind, and it can only ever test the *one* distro the host runs. The whole Linux-specific value is the cross-distro matrix. |
| **Disposable KVM VM** (this harness) | A genuine kernel + userland + package-manager boundary that wipes to nothing on teardown, and swaps freely between distro cloud images. The one real cost — nested virtualization for the *backend* — doesn't apply, because we gate the installer, not the runtime (below). |
## Two variants: a ready host and a bare host
## Two variants: `test` (bare host) and `test-ready` (prepared host)
`install.sh` never installs a backend, and never installs its own toolchain
prerequisites (python3, git, pipx) — it installs the `bot-bottle` package and
runs `doctor`, which *reports* what's missing
([`install.sh`](../../install.sh) header,
[`bot_bottle/cli/commands/doctor.py`](../../bot_bottle/cli/commands/doctor.py)).
That leaves two distinct things worth testing, selected with
`BB_TEST_PREREQS`:
That leaves two distinct things worth testing, split into two subcommands that
mirror the macOS harness's `test` / `test-ready` convention (there the split is
the backend service; here it is the toolchain the installer needs):
- **`with`** (default) — the harness installs python3 + git + pipx first, then
runs `install.sh`. This is the *ready-host happy path*: does a clean install
actually land and produce a working CLI?
- **`without`** — `install.sh` runs on the **bare cloud image**, prerequisites
and all left as the vendor ships them. This exercises `install.sh`'s own
- **`test`** `install.sh` runs on the **bare cloud image**, prerequisites and
all left as the vendor ships them. This exercises `install.sh`'s own
prerequisite-guard logic — the entire first half of the script (python
version gate, git-for-git-specs gate, pipx/pip PEP-668 handling).
- **`test-ready`** — the harness installs python3 + git + pipx first (the
`prereqs` step), then runs `install.sh`. This is the *prepared-host happy
path*: does a clean install actually land and produce a working CLI?
**Pass criteria differ by variant:**
| Variant | PASS when |
|---|---|
| `with` | `install.sh` exits 0 **and** a `bot-bottle` entry point is present and `bot-bottle --version` runs. |
| `without` | `install.sh` **either** fully succeeds (the image already carried enough) **or** declines with one of its own recognized, actionable prerequisite errors (missing python3/git, no usable pip, PEP 668). A crash or an *unrecognized* failure is a FAIL. |
| `test` | `install.sh` **either** installs cleanly (the image already carried enough) **or** declines with one of its own recognized, actionable prerequisite errors (missing python3/git, no usable pip, PEP 668). A crash or an *unrecognized* failure is a FAIL. |
| `test-ready` | `install.sh` actually lands: the `bot-bottle` entry point is present and runs, and `doctor` reports a usable python and config without crashing. A graceful decline is no longer good enough. |
Neither variant requires a green `doctor`: inside the VM there is no nested KVM
or Docker, so **every backend is correctly reported not-ready and `doctor`
exits non-zero by design** — and `install.sh` swallows that (its trailing
`if doctor; then … else … fi` leaves the script's exit code at 0). `doctor`'s
full output is still printed whenever an entry point exists, so a genuine
installer regression (a broken shim, an import error, a botched PATH) stays
visible.
or Docker, so **the backend is correctly reported not-ready** — install.sh does
not install a backend and cannot regress one, and this harness does not
provision the Docker backend. This is where Linux necessarily diverges from the
macOS `test-ready`, which reaches the host backend; `BB_TEST_REQUIRE_BACKEND=1`
makes readiness fatal anyway, for a nested-virt host that can satisfy it. The
verdict instead classifies `doctor`'s output the way the macOS harness does — a
`Traceback` is an install defect (fail), a missing `python`/`config` line is a
fail, a not-ready backend is reported — so a genuine installer regression (a
broken shim, an import error, a botched PATH) stays visible.
`test-all` runs the full matrix — every distro × both variants — each cell in
its own throwaway VM, and prints a per-cell PASS/FAIL summary.
@@ -80,15 +84,14 @@ Each distro exercises a different corner of the installer:
| **Alpine** | Alpine "cloud" (cloudinit) image | musl libc + BusyBox `sh` — the harshest POSIX-`sh` host for a `#!/bin/sh` installer. |
| **NixOS** | `channels.nixos.org` OpenStack image | No FHS `~/.local` on PATH by default; `nix-env` user-profile prereqs; pipx laying a self-contained venv on a non-FHS host. |
In the `with` variant the harness installs `python3 + git + pipx` first on
each distro (install.sh installs none of them), so all five drive the
recommended pipx path. The `without` variant then removes that scaffolding and
lets each distro's bare image collide with install.sh's guards — on most cloud
images python3 is present (cloud-init needs it) but git and pipx are not, so
install.sh is expected to decline at the git-for-git-specs gate or the PEP-668
pip check with an actionable message. Both are legitimate, and the two
variants together cover the whole first half of the installer as well as the
happy path.
In `test-ready` the harness installs `python3 + git + pipx` first on each
distro (install.sh installs none of them), so all five drive the recommended
pipx path. `test` then removes that scaffolding and lets each distro's bare
image collide with install.sh's guards — on most cloud images python3 is
present (cloud-init needs it) but git and pipx are not, so install.sh is
expected to decline at the git-for-git-specs gate or the PEP-668 pip check with
an actionable message. Both are legitimate, and the two variants together cover
the whole first half of the installer as well as the happy path.
## What a clean install touches (the footprint that decides "wipeable")
@@ -97,7 +100,7 @@ happy path.
| Config / state / db | `~/.bot-bottle/{agents,bottles,contrib,…}` ([`install.sh`](../../install.sh)) | ✅ | ❌ overlay deleted |
| pipx venv + shim | `~/.local/pipx/venvs/bot-bottle`, shim in `~/.local/bin` | ✅ | ❌ overlay deleted |
| pip `--user` fallback | `~/.local/lib` + `~/.local/bin` | ✅ | ❌ overlay deleted |
| **Distro prerequisites** (python/git/pipx) | system paths via `apt`/`dnf`/`pacman`/`apk`/`nix-env` | ❌ | ❌ **overlay deleted** |
| **Distro prerequisites** (python/git/pipx, `test-ready` only) | system paths via `apt`/`dnf`/`pacman`/`apk`/`nix-env` | ❌ | ❌ **overlay deleted** |
Unlike the macOS throwaway user (whose Homebrew / Apple-Container / Rosetta
footprint *survives*), **every row here dies with the overlay** — that is the
@@ -118,10 +121,10 @@ only thing that persists between runs, on purpose.
download time (GNU `hash file`, bare-hash, and Fedora's BSD
`SHA256 (file) = hash` formats are all handled). The NixOS channel image
ships no stable sums file, so that one needs `BB_TEST_SKIP_VERIFY=1`.
- **`test-all`** runs every distro × both variants, each cell in its own
subshell on its own forwarded port, so one cell's failure (or teardown trap)
can't abort the matrix; it prints a per-cell PASS/FAIL summary. Pin
`BB_TEST_PREREQS` to run just one variant's row.
- **`test-all`** runs every distro × both variants (`test` and `test-ready`),
each cell in its own subshell on its own forwarded port, so one cell's
failure (or teardown trap) can't abort the matrix; it prints a per-cell
PASS/FAIL summary.
## Not wired into PR CI