feat: add bare-host install variant to the Linux harness
prd-number-check / require-numbered-prds (pull_request) Successful in 8s
tracker-policy-pr / check-pr (pull_request) Failing after 8s

Splits the Linux clean-install harness into two variants, selected with
BB_TEST_PREREQS:

  with    (default) — install python3 + git + pipx first, then install.sh.
                      The ready-host happy path; PASS = install.sh exits 0 and
                      leaves a runnable bot-bottle entry point.
  without           — run install.sh on the BARE cloud image. Exercises
                      install.sh's own prerequisite-guard logic (python gate,
                      git-for-git-specs gate, pipx/pip PEP-668 handling). PASS =
                      install.sh either fully succeeds OR declines with one of
                      its own recognized, actionable prerequisite errors; a
                      crash or unrecognized failure is a FAIL.

cmd_run now captures install.sh's exit code + full output (install.rc /
install.log) instead of aborting on non-zero, so the verdict step applies the
variant's criterion. The old assert_installed is replaced by a quiet
entry_point_runnable helper plus classify_outcome, which matches the bare-host
declines against the exact die() messages install.sh prints. doctor still runs
for visibility, but only when an entry point exists.

test-all now runs the full matrix -- every distro x both variants -- each cell
in its own throwaway VM, and can be pinned to one variant via BB_TEST_PREREQS.
Research note updated to document both variants and their pass criteria.

Validated with `bash -n` and `shellcheck`.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
2026-07-27 09:56:05 -04:00
parent 6de3f01d68
commit 84a9fdd457
2 changed files with 182 additions and 92 deletions
+42 -25
View File
@@ -32,28 +32,41 @@ equivalent of `docker run --rm`, for a whole machine.
| **Throwaway user** (`useradd`/`userdel`) | The macOS pick, but weaker on Linux: it reaches the real host, yet every system package it installs (python, pipx, git via `apt`/`dnf`/…) stays behind, and it can only ever test the *one* distro the host runs. The whole Linux-specific value is the cross-distro matrix. |
| **Disposable KVM VM** (this harness) | A genuine kernel + userland + package-manager boundary that wipes to nothing on teardown, and swaps freely between distro cloud images. The one real cost — nested virtualization for the *backend* — doesn't apply, because we gate the installer, not the runtime (below). |
## Scope: installer correctness, not runtime
## Two variants: a ready host and a bare host
`install.sh` never installs a backend — it installs the `bot-bottle` Python
package and runs `doctor`, which *reports* what's missing
`install.sh` never installs a backend, and never installs its own toolchain
prerequisites (python3, git, pipx) — it installs the `bot-bottle` package and
runs `doctor`, which *reports* what's missing
([`install.sh`](../../install.sh) header,
[`bot_bottle/cli/commands/doctor.py`](../../bot_bottle/cli/commands/doctor.py)).
Inside the test VM there is no nested KVM or Docker, so **every backend is
correctly reported not-ready and `doctor` exits non-zero by design**. That is
expected and is *not* a test failure.
That leaves two distinct things worth testing, selected with
`BB_TEST_PREREQS`:
The harness's pass criterion is therefore not a green doctor but:
- **`with`** (default) — the harness installs python3 + git + pipx first, then
runs `install.sh`. This is the *ready-host happy path*: does a clean install
actually land and produce a working CLI?
- **`without`** — `install.sh` runs on the **bare cloud image**, prerequisites
and all left as the vendor ships them. This exercises `install.sh`'s own
prerequisite-guard logic — the entire first half of the script (python
version gate, git-for-git-specs gate, pipx/pip PEP-668 handling).
1. the distro prerequisites install,
2. `install.sh` exits 0,
3. the `bot-bottle` entry point is present on the fresh user's PATH (or at the
pipx/pip location install.sh prints), and
4. `bot-bottle --version` runs — proving the package imports and the shim
works.
**Pass criteria differ by variant:**
`doctor`'s full output is still printed every run, so a genuine installer
regression (a broken shim, an import error, a botched PATH) is visible even
though the backend line is red.
| Variant | PASS when |
|---|---|
| `with` | `install.sh` exits 0 **and** a `bot-bottle` entry point is present and `bot-bottle --version` runs. |
| `without` | `install.sh` **either** fully succeeds (the image already carried enough) **or** declines with one of its own recognized, actionable prerequisite errors (missing python3/git, no usable pip, PEP 668). A crash or an *unrecognized* failure is a FAIL. |
Neither variant requires a green `doctor`: inside the VM there is no nested KVM
or Docker, so **every backend is correctly reported not-ready and `doctor`
exits non-zero by design** — and `install.sh` swallows that (its trailing
`if doctor; then … else … fi` leaves the script's exit code at 0). `doctor`'s
full output is still printed whenever an entry point exists, so a genuine
installer regression (a broken shim, an import error, a botched PATH) stays
visible.
`test-all` runs the full matrix — every distro × both variants — each cell in
its own throwaway VM, and prints a per-cell PASS/FAIL summary.
## The distro matrix is the point
@@ -67,12 +80,15 @@ Each distro exercises a different corner of the installer:
| **Alpine** | Alpine "cloud" (cloudinit) image | musl libc + BusyBox `sh` — the harshest POSIX-`sh` host for a `#!/bin/sh` installer. |
| **NixOS** | `channels.nixos.org` OpenStack image | No FHS `~/.local` on PATH by default; `nix-env` user-profile prereqs; pipx laying a self-contained venv on a non-FHS host. |
Per the decision to have a clean install *succeed* everywhere, the harness
installs `python3 + git + pipx` first on each distro (install.sh installs
none of them), so all five drive the recommended pipx path rather than the
PEP-668-blocked pip fallback. Exercising that fallback deliberately (skip the
pipx prereq, expect the "externally managed" die message) is a natural future
toggle but is out of scope here.
In the `with` variant the harness installs `python3 + git + pipx` first on
each distro (install.sh installs none of them), so all five drive the
recommended pipx path. The `without` variant then removes that scaffolding and
lets each distro's bare image collide with install.sh's guards — on most cloud
images python3 is present (cloud-init needs it) but git and pipx are not, so
install.sh is expected to decline at the git-for-git-specs gate or the PEP-668
pip check with an actionable message. Both are legitimate, and the two
variants together cover the whole first half of the installer as well as the
happy path.
## What a clean install touches (the footprint that decides "wipeable")
@@ -102,9 +118,10 @@ only thing that persists between runs, on purpose.
download time (GNU `hash file`, bare-hash, and Fedora's BSD
`SHA256 (file) = hash` formats are all handled). The NixOS channel image
ships no stable sums file, so that one needs `BB_TEST_SKIP_VERIFY=1`.
- **`test-all`** runs each distro in its own subshell on its own forwarded
port, so one distro's failure (or teardown trap) can't abort the matrix,
and prints a per-distro PASS/FAIL summary.
- **`test-all`** runs every distro × both variants, each cell in its own
subshell on its own forwarded port, so one cell's failure (or teardown trap)
can't abort the matrix; it prints a per-cell PASS/FAIL summary. Pin
`BB_TEST_PREREQS` to run just one variant's row.
## Not wired into PR CI