didericis/bot-bottle
1
0
Fork 0
Code Issues 33 Pull Requests 9 Actions Packages Releases Wiki Activity

refactor: adopt macOS test/test-ready convention for variants
prd-number-check / require-numbered-prds (pull_request) Successful in 16s
Details
tracker-policy-pr / check-pr (pull_request) Failing after 12m35s
Details

Browse Source 
Replaces the BB_TEST_PREREQS env toggle with the two subcommands the macOS
harness established, so the two repos read the same:

  test        A bare host — prerequisites NOT set up (the default state of a
              stock cloud image). Exercises install.sh's own prerequisite-guard
              logic. SOUND (PASS) when install.sh either installs cleanly or
              declines with one of its own recognized, actionable errors; a
              crash or unrecognized failure fails.
  test-ready  Prerequisites satisfied — the harness installs python3/git/pipx
              first, then runs install.sh, which must actually land: entry point
              runnable, doctor reporting a usable python and config.

Structure now mirrors scripts/macos-install-test.sh: a shared _test_cycle
driving up -> (prereqs) -> run -> verdict -> down, thin cmd_test / cmd_test_ready
wrappers setting _STEPS / _PASS_CLAIM / _REQUIRE_INSTALL, a standalone `prereqs`
subcommand, and a _test_teardown that prints the PASS/FAIL claim.

The doctor check is now the macOS-style classifier rather than a bare exit-code
read: a Traceback is an install defect (fail), a missing `ok: python:` /
`ok: config:` line is a fail, and a not-ready backend is reported, not fatal
(BB_TEST_REQUIRE_BACKEND=1 makes it fatal for a nested-virt host). This is where
Linux diverges from macOS test-ready — a plain VM has no nested KVM for
Firecracker and the harness doesn't provision the Docker backend, so backend
readiness is never the Linux criterion.

test-all now runs every distro × {test, test-ready}. Research note updated.

Validated with `bash -n` and `shellcheck`.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
didericis
2026-07-27 10:07:52 -04:00
parent 84a9fdd457
commit bff06bcedf
2 changed files with 288 additions and 209 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/research/testing-clean-install-on-linux.md
+33 -30
View File
@@ -32,38 +32,42 @@ equivalent of `docker run --rm`, for a whole machine.
| **Throwaway user** (`useradd`/`userdel`) | The macOS pick, but weaker on Linux: it reaches the real host, yet every system package it installs (python, pipx, git via `apt`/`dnf`/…) stays behind, and it can only ever test the *one* distro the host runs. The whole Linux-specific value is the cross-distro matrix. | | **Throwaway user** (`useradd`/`userdel`) | The macOS pick, but weaker on Linux: it reaches the real host, yet every system package it installs (python, pipx, git via `apt`/`dnf`/…) stays behind, and it can only ever test the *one* distro the host runs. The whole Linux-specific value is the cross-distro matrix. |
| **Disposable KVM VM** (this harness) | A genuine kernel + userland + package-manager boundary that wipes to nothing on teardown, and swaps freely between distro cloud images. The one real cost — nested virtualization for the *backend* — doesn't apply, because we gate the installer, not the runtime (below). | | **Disposable KVM VM** (this harness) | A genuine kernel + userland + package-manager boundary that wipes to nothing on teardown, and swaps freely between distro cloud images. The one real cost — nested virtualization for the *backend* — doesn't apply, because we gate the installer, not the runtime (below). |
## Two variants: a ready host and a bare host ## Two variants: `test` (bare host) and `test-ready` (prepared host)
`install.sh` never installs a backend, and never installs its own toolchain `install.sh` never installs a backend, and never installs its own toolchain
prerequisites (python3, git, pipx) — it installs the `bot-bottle` package and prerequisites (python3, git, pipx) — it installs the `bot-bottle` package and
runs `doctor`, which *reports* what's missing runs `doctor`, which *reports* what's missing
([`install.sh`](../../install.sh) header, ([`install.sh`](../../install.sh) header,
[`bot_bottle/cli/commands/doctor.py`](../../bot_bottle/cli/commands/doctor.py)). [`bot_bottle/cli/commands/doctor.py`](../../bot_bottle/cli/commands/doctor.py)).
That leaves two distinct things worth testing, selected with That leaves two distinct things worth testing, split into two subcommands that
`BB_TEST_PREREQS`: mirror the macOS harness's `test` / `test-ready` convention (there the split is
the backend service; here it is the toolchain the installer needs):
- **`with`** (default) — the harness installs python3 + git + pipx first, then - **`test`** — `install.sh` runs on the **bare cloud image**, prerequisites and
runs `install.sh`. This is the *ready-host happy path*: does a clean install all left as the vendor ships them. This exercises `install.sh`'s own
actually land and produce a working CLI?
- **`without`** — `install.sh` runs on the **bare cloud image**, prerequisites
and all left as the vendor ships them. This exercises `install.sh`'s own
prerequisite-guard logic — the entire first half of the script (python prerequisite-guard logic — the entire first half of the script (python
version gate, git-for-git-specs gate, pipx/pip PEP-668 handling). version gate, git-for-git-specs gate, pipx/pip PEP-668 handling).
- **`test-ready`** — the harness installs python3 + git + pipx first (the
`prereqs` step), then runs `install.sh`. This is the *prepared-host happy
path*: does a clean install actually land and produce a working CLI?
**Pass criteria differ by variant:** **Pass criteria differ by variant:**
| Variant | PASS when | | Variant | PASS when |
|---|---| |---|---|
| `with` | `install.sh` exits 0 **and** a `bot-bottle` entry point is present and `bot-bottle --version` runs. | | `test` | `install.sh` **either** installs cleanly (the image already carried enough) **or** declines with one of its own recognized, actionable prerequisite errors (missing python3/git, no usable pip, PEP 668). A crash or an *unrecognized* failure is a FAIL. |
| `without` | `install.sh` **either** fully succeeds (the image already carried enough) **or** declines with one of its own recognized, actionable prerequisite errors (missing python3/git, no usable pip, PEP 668). A crash or an *unrecognized* failure is a FAIL. | | `test-ready` | `install.sh` actually lands: the `bot-bottle` entry point is present and runs, and `doctor` reports a usable python and config without crashing. A graceful decline is no longer good enough. |
Neither variant requires a green `doctor`: inside the VM there is no nested KVM Neither variant requires a green `doctor`: inside the VM there is no nested KVM
or Docker, so **every backend is correctly reported not-ready and `doctor` or Docker, so **the backend is correctly reported not-ready** — install.sh does
exits non-zero by design** — and `install.sh` swallows that (its trailing not install a backend and cannot regress one, and this harness does not
`if doctor; then … else … fi` leaves the script's exit code at 0). `doctor`'s provision the Docker backend. This is where Linux necessarily diverges from the
full output is still printed whenever an entry point exists, so a genuine macOS `test-ready`, which reaches the host backend; `BB_TEST_REQUIRE_BACKEND=1`
installer regression (a broken shim, an import error, a botched PATH) stays makes readiness fatal anyway, for a nested-virt host that can satisfy it. The
visible. verdict instead classifies `doctor`'s output the way the macOS harness does — a
`Traceback` is an install defect (fail), a missing `python`/`config` line is a
fail, a not-ready backend is reported — so a genuine installer regression (a
broken shim, an import error, a botched PATH) stays visible.
`test-all` runs the full matrix — every distro × both variants — each cell in `test-all` runs the full matrix — every distro × both variants — each cell in
its own throwaway VM, and prints a per-cell PASS/FAIL summary. its own throwaway VM, and prints a per-cell PASS/FAIL summary.
@@ -80,15 +84,14 @@ Each distro exercises a different corner of the installer:
| **Alpine** | Alpine "cloud" (cloudinit) image | musl libc + BusyBox `sh` — the harshest POSIX-`sh` host for a `#!/bin/sh` installer. | | **Alpine** | Alpine "cloud" (cloudinit) image | musl libc + BusyBox `sh` — the harshest POSIX-`sh` host for a `#!/bin/sh` installer. |
| **NixOS** | `channels.nixos.org` OpenStack image | No FHS `~/.local` on PATH by default; `nix-env` user-profile prereqs; pipx laying a self-contained venv on a non-FHS host. | | **NixOS** | `channels.nixos.org` OpenStack image | No FHS `~/.local` on PATH by default; `nix-env` user-profile prereqs; pipx laying a self-contained venv on a non-FHS host. |
In the `with` variant the harness installs `python3 + git + pipx` first on In `test-ready` the harness installs `python3 + git + pipx` first on each
each distro (install.sh installs none of them), so all five drive the distro (install.sh installs none of them), so all five drive the recommended
recommended pipx path. The `without` variant then removes that scaffolding and pipx path. `test` then removes that scaffolding and lets each distro's bare
lets each distro's bare image collide with install.sh's guards — on most cloud image collide with install.sh's guards — on most cloud images python3 is
images python3 is present (cloud-init needs it) but git and pipx are not, so present (cloud-init needs it) but git and pipx are not, so install.sh is
install.sh is expected to decline at the git-for-git-specs gate or the PEP-668 expected to decline at the git-for-git-specs gate or the PEP-668 pip check with
pip check with an actionable message. Both are legitimate, and the two an actionable message. Both are legitimate, and the two variants together cover
variants together cover the whole first half of the installer as well as the the whole first half of the installer as well as the happy path.
happy path.
## What a clean install touches (the footprint that decides "wipeable") ## What a clean install touches (the footprint that decides "wipeable")
@@ -97,7 +100,7 @@ happy path.
| Config / state / db | `~/.bot-bottle/{agents,bottles,contrib,…}` ([`install.sh`](../../install.sh)) | ✅ | ❌ overlay deleted | | Config / state / db | `~/.bot-bottle/{agents,bottles,contrib,…}` ([`install.sh`](../../install.sh)) | ✅ | ❌ overlay deleted |
| pipx venv + shim | `~/.local/pipx/venvs/bot-bottle`, shim in `~/.local/bin` | ✅ | ❌ overlay deleted | | pipx venv + shim | `~/.local/pipx/venvs/bot-bottle`, shim in `~/.local/bin` | ✅ | ❌ overlay deleted |
| pip `--user` fallback | `~/.local/lib` + `~/.local/bin` | ✅ | ❌ overlay deleted | | pip `--user` fallback | `~/.local/lib` + `~/.local/bin` | ✅ | ❌ overlay deleted |
| **Distro prerequisites** (python/git/pipx) | system paths via `apt`/`dnf`/`pacman`/`apk`/`nix-env` | ❌ | ❌ **overlay deleted** | | **Distro prerequisites** (python/git/pipx, `test-ready` only) | system paths via `apt`/`dnf`/`pacman`/`apk`/`nix-env` | ❌ | ❌ **overlay deleted** |
Unlike the macOS throwaway user (whose Homebrew / Apple-Container / Rosetta Unlike the macOS throwaway user (whose Homebrew / Apple-Container / Rosetta
footprint *survives*), **every row here dies with the overlay** — that is the footprint *survives*), **every row here dies with the overlay** — that is the
@@ -118,10 +121,10 @@ only thing that persists between runs, on purpose.
download time (GNU `hash file`, bare-hash, and Fedora's BSD download time (GNU `hash file`, bare-hash, and Fedora's BSD
`SHA256 (file) = hash` formats are all handled). The NixOS channel image `SHA256 (file) = hash` formats are all handled). The NixOS channel image
ships no stable sums file, so that one needs `BB_TEST_SKIP_VERIFY=1`. ships no stable sums file, so that one needs `BB_TEST_SKIP_VERIFY=1`.
- **`test-all`** runs every distro × both variants, each cell in its own - **`test-all`** runs every distro × both variants (`test` and `test-ready`),
subshell on its own forwarded port, so one cell's failure (or teardown trap) each cell in its own subshell on its own forwarded port, so one cell's
can't abort the matrix; it prints a per-cell PASS/FAIL summary. Pin failure (or teardown trap) can't abort the matrix; it prints a per-cell
`BB_TEST_PREREQS` to run just one variant's row. PASS/FAIL summary.
## Not wired into PR CI ## Not wired into PR CI
scripts/linux-install-test.sh
+255 -179
View File
@@ -6,8 +6,8 @@
# deleted afterward. install.sh's entire footprint is user-home-local (the # deleted afterward. install.sh's entire footprint is user-home-local (the
# pipx venv under ~/.local, the ~/.bot-bottle config dir, and a printed PATH # pipx venv under ~/.local, the ~/.bot-bottle config dir, and a printed PATH
# hint), so a fresh VM's fresh $HOME is the clean surface we want — and unlike # hint), so a fresh VM's fresh $HOME is the clean surface we want — and unlike
# a throwaway user account, tearing the VM down also wipes the OS-level # a throwaway user account, tearing the VM down also wipes any OS-level
# prerequisites we install into it, so the reset is total. Full rationale in # prerequisites installed into it, so the reset is total. Full rationale in
# docs/research/testing-clean-install-on-linux.md. # docs/research/testing-clean-install-on-linux.md.
# #
# Why a VM and not a container or a throwaway user: a container shares the # Why a VM and not a container or a throwaway user: a container shares the
@@ -19,50 +19,57 @@
# per run (`qemu-img create -b base`), deleted on teardown — the Linux # per run (`qemu-img create -b base`), deleted on teardown — the Linux
# equivalent of `docker run --rm`, but for a whole machine. # equivalent of `docker run --rm`, but for a whole machine.
# #
# Two variants, chosen with BB_TEST_PREREQS:
#
# with (default) — install THIS distro's prerequisites (python3, git,
# pipx — see the DISTRO table) FIRST, then run install.sh.
# This is the happy path: a ready host. PASS requires
# install.sh to exit 0 and leave a runnable entry point.
# without — run install.sh on the BARE cloud image, prerequisites
# and all left as the vendor ships them. This exercises
# install.sh's own prerequisite-guard logic. PASS means
# install.sh EITHER fully succeeds (the image already had
# enough) OR declines with one of its own recognized,
# actionable prerequisite errors (missing python3/git,
# no pipx, PEP 668 externally-managed). A crash or an
# unrecognized failure is a FAIL.
#
# What each run does: boot a fresh VM, (optionally) install prerequisites, pipe
# *this checkout's* install.sh into it exactly as `curl … | sh` would, then
# apply the variant's pass criterion and run `bot-bottle doctor` for visibility
# when an entry point exists. doctor reports every backend as not-ready inside
# the VM (there is no nested KVM/Docker), which is expected: this harness gates
# INSTALLER correctness, not runtime, so a green doctor is never the criterion.
#
# Usage: # Usage:
# ./scripts/linux-install-test.sh test # up -> (prereqs) -> run -> verdict -> down (one distro) # ./scripts/linux-install-test.sh test # up -> run -> verdict -> down (bare host)
# ./scripts/linux-install-test.sh test-all # `test` across every distro AND both variants, with a summary # ./scripts/linux-install-test.sh test-ready # ... with prerequisites installed first
# ./scripts/linux-install-test.sh up # fetch base image, boot a fresh VM # ./scripts/linux-install-test.sh test-all # every distro × both variants, with a summary
# ./scripts/linux-install-test.sh run # (prereqs) + install.sh + doctor in the VM # ./scripts/linux-install-test.sh up # fetch base image, boot a fresh VM
# ./scripts/linux-install-test.sh status # VM reachable? entry point installed? doctor # ./scripts/linux-install-test.sh prereqs # install python3 + git + pipx in the VM
# ./scripts/linux-install-test.sh down # kill the VM, delete overlay + seed (the reset) # ./scripts/linux-install-test.sh run # pipe install.sh into the VM
# ./scripts/linux-install-test.sh ssh # open an interactive shell in the running VM # ./scripts/linux-install-test.sh status # VM reachable? is the install sound?
# ./scripts/linux-install-test.sh down # kill the VM, delete overlay + seed (the reset)
# ./scripts/linux-install-test.sh ssh # open an interactive shell in the running VM
#
# TWO TEST VARIANTS, because "does install.sh handle an unprepared host" and
# "does a prepared host get a clean install" are different questions (mirrors
# the macOS harness's test / test-ready split):
#
# test A Linux system WITHOUT the prerequisites set up — the default
# state of a stock cloud image (python3 is usually present for
# cloud-init, but git and pipx are not). This exercises
# install.sh's own prerequisite-guard logic. It is SOUND — a
# PASS — when install.sh EITHER installs cleanly (the image
# already had enough) OR declines with one of its own recognized,
# actionable errors (missing python3/git, no usable pip, PEP 668
# externally-managed). A crash or an unrecognized failure fails.
#
# test-ready A Linux system WITH the prerequisites satisfied — the harness
# installs python3 + git + pipx first (see the DISTRO table),
# then runs install.sh. A graceful decline is no longer good
# enough here: the install MUST land, the entry point must run,
# and doctor must report a usable python and config without
# crashing.
#
# Neither variant requires a green backend. install.sh does not install a
# backend and cannot regress one, and inside a plain VM there is no nested KVM
# for Firecracker; the harness does not provision the Docker backend either. So
# backend readiness is reported, not required (this is where Linux necessarily
# diverges from the macOS test-ready, which reaches the host backend).
# BB_TEST_REQUIRE_BACKEND=1 makes it fatal anyway, for a nested-virt host.
# #
# Config via env: # Config via env:
# BB_TEST_DISTRO ubuntu | fedora | arch | alpine | nixos (default: ubuntu) # BB_TEST_DISTRO ubuntu | fedora | arch | alpine | nixos (default: ubuntu)
# BB_TEST_PREREQS with = install python3/git/pipx first | without = bare image (default: with) # BB_TEST_SSH_PORT host port forwarded to the guest's :22 (default: 2222)
# BB_TEST_SSH_PORT host port forwarded to the guest's :22 (default: 2222) # BB_TEST_CACHE_DIR where base images are cached (default: ~/.cache/bot-bottle-install-test)
# BB_TEST_CACHE_DIR where base images are cached (default: ~/.cache/bot-bottle-install-test) # BB_TEST_RUN_DIR per-run scratch (overlay, seed, key, …) (default: a mktemp dir)
# BB_TEST_RUN_DIR per-run scratch (overlay, seed, key, pid) (default: a mktemp dir) # BB_TEST_MEM_MB guest RAM (default: 2048)
# BB_TEST_MEM_MB guest RAM (default: 2048) # BB_TEST_CPUS guest vCPUs (default: 2)
# BB_TEST_CPUS guest vCPUs (default: 2) # BB_TEST_DISK overlay virtual size (default: 12G)
# BB_TEST_DISK overlay virtual size (default: 12G) # BB_TEST_BOOT_TIMEOUT seconds to wait for SSH after boot (default: 300)
# BB_TEST_BOOT_TIMEOUT seconds to wait for SSH after boot (default: 300) # BB_TEST_KEEP 1 = the test cycles skip teardown, to poke at a failure
# BB_TEST_KEEP 1 = `test`/`test-all` skip teardown, to poke at a failure # BB_TEST_REQUIRE_BACKEND 1 = make a not-ready backend fatal (needs nested virt)
# BB_TEST_INSTALL_URL curl this install.sh in the guest instead of piping the local checkout # BB_TEST_INSTALL_URL curl this install.sh in the guest instead of piping the local checkout
# BB_TEST_SKIP_VERIFY 1 = skip base-image checksum verification (not recommended) # BB_TEST_SKIP_VERIFY 1 = skip base-image checksum verification (not recommended)
# BOT_BOTTLE_INSTALL_SPEC passed through to install.sh (pip / git spec) # BOT_BOTTLE_INSTALL_SPEC passed through to install.sh (pip / git spec)
# #
# Notes: # Notes:
@@ -77,7 +84,6 @@
set -euo pipefail set -euo pipefail
DISTRO="${BB_TEST_DISTRO:-ubuntu}" DISTRO="${BB_TEST_DISTRO:-ubuntu}"
PREREQS="${BB_TEST_PREREQS:-with}" # with | without (see the header)
SSH_PORT="${BB_TEST_SSH_PORT:-2222}" SSH_PORT="${BB_TEST_SSH_PORT:-2222}"
CACHE_DIR="${BB_TEST_CACHE_DIR:-${XDG_CACHE_HOME:-$HOME/.cache}/bot-bottle-install-test}" CACHE_DIR="${BB_TEST_CACHE_DIR:-${XDG_CACHE_HOME:-$HOME/.cache}/bot-bottle-install-test}"
MEM_MB="${BB_TEST_MEM_MB:-2048}" MEM_MB="${BB_TEST_MEM_MB:-2048}"
@@ -88,23 +94,27 @@ BOOT_TIMEOUT="${BB_TEST_BOOT_TIMEOUT:-300}"
_SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" _SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
_REPO_ROOT="$(cd "$_SCRIPT_DIR/.." && pwd)" _REPO_ROOT="$(cd "$_SCRIPT_DIR/.." && pwd)"
# Per-run scratch. Persisted across sub-commands (up/run/status/down) via a # Per-run scratch. Persisted across sub-commands (up/prereqs/run/status/down)
# marker file so `up` in one invocation and `down` in the next find the same # via a marker file so `up` in one invocation and `down` in the next find the
# VM; `test` creates its own and cleans it up itself. # same VM; the test cycles set RUN_DIR themselves and never write the marker.
RUN_DIR="${BB_TEST_RUN_DIR:-}" RUN_DIR="${BB_TEST_RUN_DIR:-}"
# Set by `test`, which chains the steps and suppresses the per-step "next # Set by the test cycles, which chain the steps and suppress the per-step "next
# command" hints. # command" hints. _STEPS / _PASS_CLAIM / _REQUIRE_INSTALL are the per-variant
# knobs the shared cycle and teardown read.
IN_TEST=0 IN_TEST=0
_STEPS=4
_PASS_CLAIM=""
_REQUIRE_INSTALL=0
# --- distro table ---------------------------------------------------- # --- distro table ----------------------------------------------------
# For each distro: cloud-image URL | checksum-file URL | default SSH user | # For each distro: cloud-image URL | checksum-file URL | default SSH user |
# prerequisite-install command (run in the guest; uses sudo when user != root). # prerequisite-install command (run in the guest; uses sudo when user != root).
# #
# The prereq step is deliberate: the user asked that a clean install SUCCEED on # The prerequisite command installs python3 + git + pipx — install.sh installs
# every distro, so we install python3 + git + pipx first (install.sh installs # none of them — so `test-ready` (and the `prereqs` sub-command) drive
# none of them). That drives install.sh down its recommended pipx path on all # install.sh down its recommended pipx path. Bump the URLs here when a distro
# five. Bump the URLs here when a distro publishes a newer build. # publishes a newer build.
declare -A IMAGE_URL SUM_URL SSH_USER PREREQ declare -A IMAGE_URL SUM_URL SSH_USER PREREQ
IMAGE_URL[ubuntu]="https://cloud-images.ubuntu.com/noble/current/noble-server-cloudimg-amd64.img" IMAGE_URL[ubuntu]="https://cloud-images.ubuntu.com/noble/current/noble-server-cloudimg-amd64.img"
@@ -167,16 +177,9 @@ known_distro() {
|| { echo "error: unknown distro '$DISTRO' (known: ${ALL_DISTROS[*]})" >&2; exit 1; } || { echo "error: unknown distro '$DISTRO' (known: ${ALL_DISTROS[*]})" >&2; exit 1; }
} }
known_variant() {
case "$PREREQS" in
with|without) ;;
*) echo "error: BB_TEST_PREREQS must be 'with' or 'without' (got '$PREREQS')" >&2; exit 1 ;;
esac
}
# --- run-dir bookkeeping --------------------------------------------- # --- run-dir bookkeeping ---------------------------------------------
# The marker lets `run`/`status`/`down` in separate invocations find the VM # The marker lets prereqs/run/status/down in separate invocations find the VM
# that `up` started. `test` sets RUN_DIR itself and never writes the marker. # that `up` started. The test cycles set RUN_DIR themselves and never write it.
_marker() { echo "${TMPDIR:-/tmp}/bot-bottle-install-test.$DISTRO.run"; } _marker() { echo "${TMPDIR:-/tmp}/bot-bottle-install-test.$DISTRO.run"; }
_ensure_run_dir() { _ensure_run_dir() {
@@ -344,27 +347,30 @@ cmd_up() {
-netdev "user,id=n0,hostfwd=tcp:127.0.0.1:$SSH_PORT-:22" \ -netdev "user,id=n0,hostfwd=tcp:127.0.0.1:$SSH_PORT-:22" \
-device virtio-net-pci,netdev=n0 -device virtio-net-pci,netdev=n0
# Only publish the marker (so a later `run`/`down` finds this VM) when we # Only publish the marker (so a later prereqs/run/down finds this VM) when
# aren't inside `test`, which manages its own RUN_DIR + teardown. # we aren't inside a test cycle, which manages its own RUN_DIR + teardown.
[ "$IN_TEST" = 1 ] || echo "$RUN_DIR" > "$(_marker)" [ "$IN_TEST" = 1 ] || echo "$RUN_DIR" > "$(_marker)"
wait_for_ssh wait_for_ssh
if [ "$IN_TEST" != 1 ]; then if [ "$IN_TEST" != 1 ]; then
echo "== VM is up. Install into it with: BB_TEST_DISTRO=$DISTRO $0 run ==" echo "== VM is up. Prepare it with: BB_TEST_DISTRO=$DISTRO $0 prereqs (or go straight to 'run') =="
fi fi
} }
cmd_run() { # Install install.sh's toolchain prerequisites (python3 + git + pipx) into the
require_linux; known_variant # running VM. This is what separates `test-ready` from `test`, and it is a
# distinct sub-command so a manual up/prereqs/run/down cycle is possible.
cmd_prereqs() {
require_linux
_load_run_dir || { echo "error: no running VM for $DISTRO; run '$0 up' first" >&2; return 1; } _load_run_dir || { echo "error: no running VM for $DISTRO; run '$0 up' first" >&2; return 1; }
echo "== installing prerequisites (python3 + git + pipx) on $DISTRO =="
# Runs via the guest login shell; PREREQ is a client-side table value.
guest "${PREREQ[$DISTRO]}"
}
if [ "$PREREQS" = with ]; then cmd_run() {
echo "== [prereqs] installing python3 + git + pipx on $DISTRO ==" require_linux
# Runs via the guest's login shell; PREREQ is a client-side table value. _load_run_dir || { echo "error: no running VM for $DISTRO; run '$0 up' first" >&2; return 1; }
guest "${PREREQ[$DISTRO]}"
else
echo "== [prereqs] SKIPPED (BB_TEST_PREREQS=without) — install.sh runs on the bare $DISTRO image =="
fi
local spec_env="" local spec_env=""
[ -n "${BOT_BOTTLE_INSTALL_SPEC:-}" ] \ [ -n "${BOT_BOTTLE_INSTALL_SPEC:-}" ] \
@@ -372,8 +378,8 @@ cmd_run() {
echo "== installing bot-bottle as ${SSH_USER[$DISTRO]} ==" echo "== installing bot-bottle as ${SSH_USER[$DISTRO]} =="
# Capture install.sh's exit code and full output rather than aborting on # Capture install.sh's exit code and full output rather than aborting on
# non-zero: in the `without` variant a clean prerequisite *decline* is a # non-zero: on a bare host a clean prerequisite *decline* is sound, so the
# PASS, so the verdict step — not set -e — decides the outcome. # verdict step — not set -e — decides the outcome.
local rc local rc
set +e set +e
if [ -n "${BB_TEST_INSTALL_URL:-}" ]; then if [ -n "${BB_TEST_INSTALL_URL:-}" ]; then
@@ -386,28 +392,9 @@ cmd_run() {
rc="${PIPESTATUS[0]}" rc="${PIPESTATUS[0]}"
set -e set -e
printf '%s\n' "$rc" > "$RUN_DIR/install.rc" printf '%s\n' "$rc" > "$RUN_DIR/install.rc"
echo "== install.sh exited $rc (BB_TEST_PREREQS=$PREREQS) ==" echo "== install.sh exited $rc =="
[ "$IN_TEST" = 1 ] \ [ "$IN_TEST" = 1 ] \
|| echo "== verdict anytime with: BB_TEST_DISTRO=$DISTRO BB_TEST_PREREQS=$PREREQS $0 status ==" || echo "== verdict anytime with: BB_TEST_DISTRO=$DISTRO $0 status =="
}
# `bot-bottle doctor` in the guest, for visibility. install.sh prints a PATH
# hint rather than editing a profile, so on a fresh login the entry point may
# be installed but not yet on PATH — try the known pipx/pip locations too.
doctor_in_guest() {
# shellcheck disable=SC2016 # $HOME must expand in the GUEST shell.
guest '
for bb in "$HOME/.local/bin/bot-bottle" "$HOME/.bot-bottle/venv/bin/bot-bottle"; do
if [ -x "$bb" ]; then
command -v bot-bottle >/dev/null 2>&1 \
|| echo " (not on PATH — running $bb directly, as install.sh advises)"
exec "$bb" doctor
fi
done
command -v bot-bottle >/dev/null 2>&1 && exec bot-bottle doctor
echo " no bot-bottle entry point found for this user" >&2
exit 1
'
} }
# Quietly report whether a runnable bot-bottle entry point exists for the # Quietly report whether a runnable bot-bottle entry point exists for the
@@ -423,49 +410,104 @@ entry_point_runnable() {
' >/dev/null 2>&1 ' >/dev/null 2>&1
} }
# The prerequisite-decline messages install.sh prints via die(). In the # `bot-bottle doctor` in the guest, classified. doctor's own exit code
# `without` variant, ANY of these means install.sh correctly refused to run on # conflates "is the install sound" with "is a backend ready to run a bottle" —
# an under-provisioned host — an actionable decline, which is a PASS. # and inside a plain VM no backend can be ready (no nested KVM/Docker), so the
# raw exit code is non-zero by design. This separates the two: an unhandled
# traceback, or a missing python/config line, is an install defect and fails;
# a not-ready backend is reported, not fatal (unless BB_TEST_REQUIRE_BACKEND=1,
# for a nested-virt host that can actually satisfy it).
doctor_in_guest() {
local out rc=0 bad=0
out="$(mktemp "${TMPDIR:-/tmp}/bb-doctor.XXXXXX")"
# shellcheck disable=SC2016 # $HOME/$bb must expand in the GUEST shell.
guest '
for bb in bot-bottle "$HOME/.local/bin/bot-bottle" "$HOME/.bot-bottle/venv/bin/bot-bottle"; do
if command -v "$bb" >/dev/null 2>&1; then
case "$bb" in
bot-bottle) : ;;
*) echo " (not on PATH — running $bb directly, as install.sh advises)" ;;
esac
exec "$bb" doctor
fi
done
echo " no bot-bottle entry point found for this user" >&2
exit 1
' >"$out" 2>&1 || rc=$?
cat "$out"
# An unhandled exception is always an install/product defect, never an
# environment fact — doctor's non-zero exit alone would not distinguish it.
if grep -q 'Traceback (most recent call last)' "$out"; then
echo " doctor crashed (traceback above) — a defect, not a missing prerequisite" >&2
bad=1
fi
grep -qE '^ok: +python:' "$out" \
|| { echo " doctor never reported a usable python" >&2; bad=1; }
grep -qE '^ok: +config:' "$out" \
|| { echo " doctor never reported a usable config dir" >&2; bad=1; }
if [ "$rc" -ne 0 ] && ! grep -qE '^(fail|warn): +backend' "$out"; then
echo " doctor failed for something other than backend readiness" >&2
bad=1
fi
local backend_ready=1
grep -qE '^fail: +backend' "$out" && backend_ready=0
rm -f "$out"
[ "$bad" -eq 0 ] || return 1
if [ "${BB_TEST_REQUIRE_BACKEND:-0}" = "1" ] && [ "$backend_ready" -eq 0 ]; then
echo " backend is not ready and BB_TEST_REQUIRE_BACKEND=1 — failing" >&2
return 1
fi
if [ "$backend_ready" -eq 1 ]; then
echo " doctor: install sound; a backend is ready"
else
echo " doctor: install sound; no backend ready (expected in a plain VM — install gate only)"
fi
return 0
}
# The prerequisite-decline messages install.sh prints via die(). On a bare host
# ANY of these means install.sh correctly refused rather than half-installing —
# a sound outcome for `test`.
PREREQ_ERR_RE='is required but was not found|or newer is required|git is required to install from|neither pipx nor a usable|externally managed \(PEP 668\)|is not on PATH' PREREQ_ERR_RE='is required but was not found|or newer is required|git is required to install from|neither pipx nor a usable|externally managed \(PEP 668\)|is not on PATH'
# The teeth of the harness. Applies the variant's pass criterion to the # The verdict: is the install sound? Reads install.sh's captured exit code and
# install.sh exit code (from cmd_run) and the guest's resulting state. It never # output (from cmd_run) plus the guest's resulting state.
# requires a green doctor: no backend can be ready inside the VM (no nested # - installed & runnable -> the verdict is doctor's soundness classification.
# KVM/Docker), so doctor exits non-zero there by design. # - no entry point, but a recognized prerequisite decline, and declines are
classify_outcome() { # allowed (bare `test`, _REQUIRE_INSTALL=0) -> sound.
local rc="" verdict="FAIL" detail # - anything else -> not sound.
install_verdict() {
local rc=""
[ -f "$RUN_DIR/install.rc" ] && rc="$(cat "$RUN_DIR/install.rc")" [ -f "$RUN_DIR/install.rc" ] && rc="$(cat "$RUN_DIR/install.rc")"
if [ "$PREREQS" = with ]; then if [ "${rc:-1}" = 0 ] && entry_point_runnable; then
# Prerequisites were provisioned, so a clean install is mandatory. echo "doctor (in guest):"
if [ "${rc:-1}" = 0 ] && entry_point_runnable; then doctor_in_guest
verdict="PASS"; detail="install.sh succeeded and the entry point runs" return $?
else
detail="expected a clean install on a provisioned host (install.sh rc=${rc:-?})"
fi
else
# Bare image: success OR a recognized, actionable prerequisite decline.
if [ "${rc:-1}" = 0 ] && entry_point_runnable; then
verdict="PASS"; detail="bare image already had prerequisites; install.sh succeeded"
elif [ -n "$rc" ] && [ "$rc" != 0 ] \
&& [ -f "$RUN_DIR/install.log" ] \
&& grep -Eiq "$PREREQ_ERR_RE" "$RUN_DIR/install.log"; then
verdict="PASS"; detail="install.sh declined with an actionable prerequisite error (rc=$rc)"
else
detail="install.sh neither succeeded nor gave a recognized prerequisite error (rc=${rc:-?})"
fi
fi fi
if [ "$verdict" = PASS ]; then if [ "${_REQUIRE_INSTALL:-0}" != "1" ] \
echo "PASS[$DISTRO/$PREREQS]: $detail" && [ -n "$rc" ] && [ "$rc" != 0 ] \
&& [ -f "$RUN_DIR/install.log" ] \
&& grep -Eiq "$PREREQ_ERR_RE" "$RUN_DIR/install.log"; then
echo " install.sh declined with an actionable prerequisite error (rc=$rc)"
echo " — sound on a bare host; run 'test-ready' (or 'prereqs') to install."
return 0 return 0
fi fi
echo "FAIL[$DISTRO/$PREREQS]: $detail" >&2
if [ "${_REQUIRE_INSTALL:-0}" = "1" ]; then
echo " prerequisites were provisioned, but install.sh left no runnable entry point (rc=${rc:-?})" >&2
else
echo " install.sh neither installed nor gave a recognized prerequisite error (rc=${rc:-?})" >&2
fi
return 1 return 1
} }
cmd_status() { cmd_status() {
require_linux; known_variant require_linux
if ! _load_run_dir; then if ! _load_run_dir; then
echo "vm: no running VM for $DISTRO" echo "vm: no running VM for $DISTRO"
return 0 return 0
@@ -476,15 +518,12 @@ cmd_status() {
echo "vm: $DISTRO run-dir present but QEMU not alive" echo "vm: $DISTRO run-dir present but QEMU not alive"
return 1 return 1
fi fi
# doctor is advisory and only meaningful once something installed; skip it if install_verdict; then
# cleanly when install.sh declined (the `without` variant's PASS path). echo "OK[$DISTRO]: the install is sound"
if entry_point_runnable; then return 0
echo "doctor (in guest):"
doctor_in_guest || true # backend not-ready inside the VM is expected
else
echo "doctor: (no entry point installed — skipped)"
fi fi
classify_outcome echo "FAIL[$DISTRO]: the install is not sound (see above)" >&2
return 1
} }
cmd_down() { cmd_down() {
@@ -519,72 +558,107 @@ cmd_ssh() {
exec ssh -t "${opts[@]}" "${SSH_USER[$DISTRO]}@127.0.0.1" exec ssh -t "${opts[@]}" "${SSH_USER[$DISTRO]}@127.0.0.1"
} }
# Teardown half of `test`, armed the moment the VM exists so a failure or a # Teardown half of the test cycles, armed the moment the VM exists so a failure
# Ctrl-C still leaves nothing running. # or a Ctrl-C still leaves nothing running.
_test_teardown() { _test_teardown() {
local rc=$? local rc=$?
trap - EXIT INT TERM trap - EXIT INT TERM
if [ "${BB_TEST_KEEP:-0}" = "1" ]; then if [ "${BB_TEST_KEEP:-0}" = "1" ]; then
echo echo
echo "== down: SKIPPED (BB_TEST_KEEP=1); VM still up, remove with: BB_TEST_DISTRO=$DISTRO $0 down ==" echo "== [$_STEPS/$_STEPS] down: SKIPPED (BB_TEST_KEEP=1) =="
echo " (ssh in with: BB_TEST_RUN_DIR=$RUN_DIR BB_TEST_DISTRO=$DISTRO $0 ssh)" echo " VM still up; remove with: BB_TEST_DISTRO=$DISTRO $0 down"
echo " ssh in with: BB_TEST_RUN_DIR=$RUN_DIR BB_TEST_DISTRO=$DISTRO $0 ssh"
exit "$rc" exit "$rc"
fi fi
echo echo
echo "== down ==" echo "== [$_STEPS/$_STEPS] down =="
cmd_down || rc=1 cmd_down || rc=1
if [ "$rc" -eq 0 ]; then
echo
echo "PASS[$DISTRO]: $_PASS_CLAIM"
else
echo
echo "FAIL[$DISTRO]: see above (the VM was torn down regardless)." >&2
fi
exit "$rc" exit "$rc"
} }
cmd_test() { # The two variants differ only in whether the prerequisites get installed
require_linux; require_kvm; require_tools; known_distro; known_variant # before install.sh runs, which is exactly the question each one asks:
#
# test a bare host — assert install.sh is SOUND (installs cleanly, or
# declines with an actionable prerequisite error).
# test-ready prerequisites satisfied — assert install.sh actually LANDS.
_test_cycle() {
local with_prereqs="$1"
require_linux; require_kvm; require_tools; known_distro
IN_TEST=1 IN_TEST=1
RUN_DIR="$(mktemp -d "${TMPDIR:-/tmp}/bb-install-test.$DISTRO.XXXXXX")" RUN_DIR="$(mktemp -d "${TMPDIR:-/tmp}/bb-install-test.$DISTRO.XXXXXX")"
echo "== [1/4] up ($DISTRO, prereqs=$PREREQS) ==" echo "== [1/$_STEPS] up ($DISTRO) =="
cmd_up cmd_up
trap _test_teardown EXIT INT TERM trap _test_teardown EXIT INT TERM
echo local step=2
echo "== [2/4] run ==" if [ "$with_prereqs" = 1 ]; then
cmd_run echo
echo "== [$step/$_STEPS] prereqs =="
cmd_prereqs || { echo "error: could not install prerequisites (see above)." >&2; return 1; }
step=$(( step + 1 ))
fi
echo echo
echo "== [3/4] verdict ==" echo "== [$step/$_STEPS] run =="
local rc=0 cmd_run
if entry_point_runnable; then step=$(( step + 1 ))
echo "doctor (in guest):"
doctor_in_guest || true # advisory only echo
fi echo "== [$step/$_STEPS] verdict =="
classify_outcome || rc=1 # install.sh exits 0 even when doctor reports unmet prerequisites, so the
# _test_teardown runs on exit and owns the final exit code. # install succeeding is not the verdict — this is.
return "$rc" cmd_status || {
echo "error: the install is not sound for $DISTRO (see above)." >&2
echo " re-run with BB_TEST_KEEP=1 to keep the VM and dig in." >&2
return 1
}
}
cmd_test() {
_STEPS=4
_PASS_CLAIM="on a bare $DISTRO host, install.sh behaves soundly."
_test_cycle 0
}
cmd_test_ready() {
_STEPS=5
_PASS_CLAIM="a $DISTRO host with prerequisites satisfied installs bot-bottle cleanly."
# Prerequisites are provisioned, so a graceful decline is no longer an
# acceptable outcome — the install must actually land.
_REQUIRE_INSTALL=1
_test_cycle 1
} }
cmd_test_all() { cmd_test_all() {
require_linux; require_kvm; require_tools require_linux; require_kvm; require_tools
# Cover both variants by default; honor BB_TEST_PREREQS if the caller pinned
# one (so `BB_TEST_PREREQS=without … test-all` runs just the bare matrix).
local -a variants
if [ -n "${BB_TEST_PREREQS:-}" ]; then variants=("$BB_TEST_PREREQS"); else variants=(with without); fi
local -a passed=() failed=() local -a passed=() failed=()
local port="$SSH_PORT" local port="$SSH_PORT"
for v in "${variants[@]}"; do local script
script="$_SCRIPT_DIR/$(basename "${BASH_SOURCE[0]}")"
# Every distro × both variants. Each cell gets its own forwarded port so a
# leftover from a prior cell can't collide, and its own subshell so one
# cell's failure (or teardown trap) doesn't abort the matrix.
for sub in test test-ready; do
for d in "${ALL_DISTROS[@]}"; do for d in "${ALL_DISTROS[@]}"; do
echo echo
echo "########################################################" echo "########################################################"
echo "# $d (prereqs=$v)" echo "# $d ($sub)"
echo "########################################################" echo "########################################################"
# Each cell gets its own forwarded port so a leftover from a prior if ( DISTRO="$d" SSH_PORT="$port" \
# run can't collide, and its own subshell so a `test` failure (or a BB_TEST_DISTRO="$d" BB_TEST_SSH_PORT="$port" \
# teardown trap) doesn't abort the whole matrix. bash "$script" "$sub" ); then
if ( DISTRO="$d" SSH_PORT="$port" PREREQS="$v" \ passed+=("$d/$sub")
BB_TEST_DISTRO="$d" BB_TEST_SSH_PORT="$port" BB_TEST_PREREQS="$v" \
bash "$_SCRIPT_DIR/$(basename "${BASH_SOURCE[0]}")" test ); then
passed+=("$d/$v")
else else
failed+=("$d/$v") failed+=("$d/$sub")
fi fi
port=$(( port + 1 )) port=$(( port + 1 ))
done done
@@ -597,12 +671,14 @@ cmd_test_all() {
} }
case "${1:-}" in case "${1:-}" in
test) cmd_test ;; test) cmd_test ;;
test-all) cmd_test_all ;; test-ready) cmd_test_ready ;;
up) cmd_up ;; test-all) cmd_test_all ;;
run) cmd_run ;; up) cmd_up ;;
status) cmd_status ;; prereqs) cmd_prereqs ;;
down) cmd_down ;; run) cmd_run ;;
ssh) cmd_ssh ;; status) cmd_status ;;
*) echo "usage: $0 {test|test-all|up|run|status|down|ssh} (distro via BB_TEST_DISTRO)" >&2; exit 2 ;; down) cmd_down ;;
ssh) cmd_ssh ;;
*) echo "usage: $0 {test|test-ready|test-all|up|prereqs|run|status|down|ssh} (distro via BB_TEST_DISTRO)" >&2; exit 2 ;;
esac esac