feat: add Linux clean-install test harness #534

2026-07-27T09:48:21-04:00

didericis-claude commented

2026-07-27 09:48:21 -04:00

What

Adds scripts/linux-install-test.sh, a throwaway-VM harness that exercises install.sh the way a brand-new user would across a Linux distro matrix (Ubuntu, Fedora, Arch, Alpine, NixOS), plus the research note docs/research/testing-clean-install-on-linux.md that motivates the approach. This is the Linux counterpart to the macOS clean-install harness (#529).

Why a disposable VM (not a throwaway user or a container)

On macOS the pick was a throwaway user account (nested virt is gated to M3+). On Linux the calculus flips:

A container shares the host kernel and ships a deliberately minimal userland — it doesn't reproduce the externally-managed Python (PEP 668) that real installs put in front of the user.
A throwaway user reaches the real host, but every system package it installs stays behind, and it can only ever test the one distro the host runs.
A disposable KVM VM is a genuine kernel + userland + package-manager boundary that wipes to nothing on teardown, and a single harness can swap distro cloud images. The host already needs KVM for the Firecracker backend, so a per-run copy-on-write VM (qemu-img create -b base) is cheap — the Linux equivalent of docker run --rm, for a whole machine.

How a run works

Cache one read-only base image → boot a throwaway overlay via QEMU/KVM with user-mode networking (no root, no bridge) → inject an ephemeral SSH key + passwordless login via a cloud-init seed ISO → install the distro's prerequisites (python3 + git + pipx) → pipe this checkout's install.sh into the guest exactly as curl … | sh would → assert the CLI installed → delete the overlay. Because the overlay carries the OS-level prerequisites too, the reset is total.

Subcommands

up — fetch/verify base image (cached), boot a fresh VM
run — install prereqs + install.sh + doctor in the VM
status — VM reachable? entry point installed? runs doctor
down — kill the VM, delete overlay + seed (the reset)
test — up → run → status → down for one distro; arms an EXIT/INT/TERM trap the moment the VM exists, so a failure or Ctrl-C still tears it down
test-all — the matrix, each distro in its own subshell on its own forwarded port, with a PASS/FAIL summary
ssh — an interactive shell in the running guest

Distro selected via BB_TEST_DISTRO; see the header for all env knobs (BB_TEST_KEEP, BB_TEST_INSTALL_URL, BOT_BOTTLE_INSTALL_SPEC, …).

Scope: installer correctness, not runtime

There is no nested KVM/Docker in the VM, so bot-bottle doctor correctly reports every backend not-ready and exits non-zero by design. The pass criterion is therefore install.sh exiting 0, the bot-bottle entry point present on the fresh user's PATH, and bot-bottle --version running — not a green doctor. doctor's full output is still printed so a real installer regression (broken shim, import error) stays visible.

Notes baked into the script

Per the "clean install should succeed everywhere" decision, prerequisites are installed first on each distro (install.sh installs none of them), so all five drive the recommended pipx path.
Base-image downloads are checksum-verified against each vendor's published sums file (GNU, bare-hash, and Fedora BSD SHA256 (file) = hash formats). The NixOS channel image ships no stable sums file → use BB_TEST_SKIP_VERIFY=1.
The cloud-image URLs in the DISTRO table are the one place to bump when a distro cuts a newer build.

Validated with bash -n and shellcheck. Runtime is host-only (needs /dev/kvm, qemu, cloud-localds), so like the macOS harness it isn't exercised by the Linux PR CI.

🤖 Generated with Claude Code

## What Adds `scripts/linux-install-test.sh`, a throwaway-VM harness that exercises `install.sh` the way a brand-new user would across a Linux distro matrix (Ubuntu, Fedora, Arch, Alpine, NixOS), plus the research note `docs/research/testing-clean-install-on-linux.md` that motivates the approach. This is the Linux counterpart to the macOS clean-install harness (#529). ## Why a disposable VM (not a throwaway user or a container) On macOS the pick was a throwaway *user account* (nested virt is gated to M3+). On Linux the calculus flips: - A **container** shares the host kernel and ships a deliberately minimal userland — it doesn't reproduce the externally-managed Python (PEP 668) that real installs put in front of the user. - A **throwaway user** reaches the real host, but every system package it installs stays behind, and it can only ever test the *one* distro the host runs. - A **disposable KVM VM** is a genuine kernel + userland + package-manager boundary that wipes to nothing on teardown, and a single harness can swap distro cloud images. The host already needs KVM for the Firecracker backend, so a per-run copy-on-write VM (`qemu-img create -b base`) is cheap — the Linux equivalent of `docker run --rm`, for a whole machine. ## How a run works Cache one read-only base image → boot a throwaway overlay via QEMU/KVM with user-mode networking (no root, no bridge) → inject an ephemeral SSH key + passwordless login via a cloud-init seed ISO → install the distro's prerequisites (`python3 + git + pipx`) → pipe **this checkout's** `install.sh` into the guest exactly as `curl … | sh` would → assert the CLI installed → delete the overlay. Because the overlay carries the OS-level prerequisites too, the reset is total. ## Subcommands - `up` — fetch/verify base image (cached), boot a fresh VM - `run` — install prereqs + `install.sh` + `doctor` in the VM - `status` — VM reachable? entry point installed? runs `doctor` - `down` — kill the VM, delete overlay + seed (the reset) - `test` — `up → run → status → down` for one distro; arms an EXIT/INT/TERM trap the moment the VM exists, so a failure or Ctrl-C still tears it down - `test-all` — the matrix, each distro in its own subshell on its own forwarded port, with a PASS/FAIL summary - `ssh` — an interactive shell in the running guest Distro selected via `BB_TEST_DISTRO`; see the header for all env knobs (`BB_TEST_KEEP`, `BB_TEST_INSTALL_URL`, `BOT_BOTTLE_INSTALL_SPEC`, …). ## Scope: installer correctness, not runtime There is no nested KVM/Docker in the VM, so `bot-bottle doctor` correctly reports every backend not-ready and exits non-zero **by design**. The pass criterion is therefore `install.sh` exiting 0, the `bot-bottle` entry point present on the fresh user's PATH, and `bot-bottle --version` running — **not** a green doctor. doctor's full output is still printed so a real installer regression (broken shim, import error) stays visible. ## Notes baked into the script - Per the "clean install should succeed everywhere" decision, prerequisites are installed first on each distro (install.sh installs none of them), so all five drive the recommended pipx path. - Base-image downloads are checksum-verified against each vendor's published sums file (GNU, bare-hash, and Fedora BSD `SHA256 (file) = hash` formats). The NixOS channel image ships no stable sums file → use `BB_TEST_SKIP_VERIFY=1`. - The cloud-image URLs in the `DISTRO` table are the one place to bump when a distro cuts a newer build. Validated with `bash -n` and `shellcheck`. Runtime is host-only (needs `/dev/kvm`, `qemu`, `cloud-localds`), so like the macOS harness it isn't exercised by the Linux PR CI. 🤖 Generated with [Claude Code](https://claude.com/claude-code)

didericis-claude added 1 commit 2026-07-27 09:48:23 -04:00

feat: add Linux clean-install test harness

tracker-policy-pr / check-pr (pull_request) Failing after 10s

Details

prd-number-check / require-numbered-prds (pull_request) Failing after 11m51s

Details

6de3f01d68

Adds scripts/linux-install-test.sh, a throwaway-VM harness that exercises
install.sh the way a brand-new user would across a Linux distro matrix
(Ubuntu, Fedora, Arch, Alpine, NixOS), plus the research note
docs/research/testing-clean-install-on-linux.md that motivates the approach.

This is the Linux counterpart to the macOS clean-install harness. On Linux
the boundary of choice flips from a throwaway user account to a disposable
KVM VM: it's a genuine kernel + userland + package-manager boundary that
wipes to nothing on teardown, and a single harness can swap distro cloud
images to cover the several package-management regimes Linux fragments into.
The host already needs KVM for the Firecracker backend, so a per-run,
copy-on-write VM (qemu-img create -b base) is cheap here -- the equivalent
of `docker run --rm`, for a whole machine.

Per run the harness caches one read-only base image, boots a throwaway
overlay via QEMU/KVM with user-mode networking (no root, no bridge), injects
an ephemeral SSH key + passwordless login through a cloud-init seed ISO,
installs the distro's prerequisites (python3 + git + pipx), pipes THIS
checkout's install.sh into the guest exactly as `curl ... | sh` would, and
asserts the CLI installed cleanly. The overlay is deleted on teardown, so
even the OS-level prerequisites are wiped -- unlike a throwaway user, the
reset is total.

Subcommands mirror the macOS harness (up/run/status/down/test) plus test-all
(the matrix) and ssh (an interactive guest shell). test arms an EXIT/INT/TERM
trap the moment the VM exists, so a failure or Ctrl-C still tears it down.

Scope is installer correctness, not runtime: there is no nested KVM/Docker in
the VM, so `bot-bottle doctor` correctly reports every backend not-ready and
exits non-zero by design. The pass criterion is therefore install.sh exiting
0, the bot-bottle entry point being present on the fresh user's PATH, and
`bot-bottle --version` running -- not a green doctor. doctor's output is still
printed so a real installer regression (broken shim, import error) stays
visible.

Validated with `bash -n` and `shellcheck`. Runtime is host-only (needs
/dev/kvm, qemu, cloud-localds), so like the macOS harness it isn't exercised
by the Linux PR CI. The cloud-image URLs in the DISTRO table are the one
place to bump when a distro cuts a newer build.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

didericis added 1 commit 2026-07-27 09:56:12 -04:00

feat: add bare-host install variant to the Linux harness

prd-number-check / require-numbered-prds (pull_request) Successful in 8s

Details

tracker-policy-pr / check-pr (pull_request) Failing after 8s

Details

84a9fdd457

Splits the Linux clean-install harness into two variants, selected with
BB_TEST_PREREQS:

  with    (default) — install python3 + git + pipx first, then install.sh.
                      The ready-host happy path; PASS = install.sh exits 0 and
                      leaves a runnable bot-bottle entry point.
  without           — run install.sh on the BARE cloud image. Exercises
                      install.sh's own prerequisite-guard logic (python gate,
                      git-for-git-specs gate, pipx/pip PEP-668 handling). PASS =
                      install.sh either fully succeeds OR declines with one of
                      its own recognized, actionable prerequisite errors; a
                      crash or unrecognized failure is a FAIL.

cmd_run now captures install.sh's exit code + full output (install.rc /
install.log) instead of aborting on non-zero, so the verdict step applies the
variant's criterion. The old assert_installed is replaced by a quiet
entry_point_runnable helper plus classify_outcome, which matches the bare-host
declines against the exact die() messages install.sh prints. doctor still runs
for visibility, but only when an entry point exists.

test-all now runs the full matrix -- every distro x both variants -- each cell
in its own throwaway VM, and can be pinned to one variant via BB_TEST_PREREQS.
Research note updated to document both variants and their pass criteria.

Validated with `bash -n` and `shellcheck`.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

didericis added 1 commit 2026-07-27 10:07:58 -04:00

refactor: adopt macOS test/test-ready convention for variants

prd-number-check / require-numbered-prds (pull_request) Successful in 16s

Details

tracker-policy-pr / check-pr (pull_request) Failing after 12m35s

Details

bff06bcedf

Replaces the BB_TEST_PREREQS env toggle with the two subcommands the macOS
harness established, so the two repos read the same:

  test        A bare host — prerequisites NOT set up (the default state of a
              stock cloud image). Exercises install.sh's own prerequisite-guard
              logic. SOUND (PASS) when install.sh either installs cleanly or
              declines with one of its own recognized, actionable errors; a
              crash or unrecognized failure fails.
  test-ready  Prerequisites satisfied — the harness installs python3/git/pipx
              first, then runs install.sh, which must actually land: entry point
              runnable, doctor reporting a usable python and config.

Structure now mirrors scripts/macos-install-test.sh: a shared _test_cycle
driving up -> (prereqs) -> run -> verdict -> down, thin cmd_test / cmd_test_ready
wrappers setting _STEPS / _PASS_CLAIM / _REQUIRE_INSTALL, a standalone `prereqs`
subcommand, and a _test_teardown that prints the PASS/FAIL claim.

The doctor check is now the macOS-style classifier rather than a bare exit-code
read: a Traceback is an install defect (fail), a missing `ok: python:` /
`ok: config:` line is a fail, and a not-ready backend is reported, not fatal
(BB_TEST_REQUIRE_BACKEND=1 makes it fatal for a nested-virt host). This is where
Linux diverges from macOS test-ready — a plain VM has no nested KVM for
Firecracker and the harness doesn't provision the Docker backend, so backend
readiness is never the Linux criterion.

test-all now runs every distro × {test, test-ready}. Research note updated.

Validated with `bash -n` and `shellcheck`.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

didericis added the Kind/Testing label 2026-07-27 10:27:00 -04:00

didericis added 1 commit 2026-07-27 10:47:15 -04:00

fix: correct harness defects found running the matrix

prd-number-check / require-numbered-prds (pull_request) Successful in 8s

Details

tracker-policy-pr / check-pr (pull_request) Successful in 10s

Details

39353cefce

First full run on the KVM host surfaced three harness bugs and two stale
image URLs; all fixed here.

Harness bugs:
- Liveness probe used `bot-bottle --version`, which the CLI does not implement
  (unknown args die non-zero) — so every SUCCESSFUL install was misreported as
  "no runnable entry point". Switched to `bot-bottle --help`, which exits 0
  before any DB/migration/network work.
- cmd_down never removed serial.log, so its rmdir failed and every run left an
  orphan scratch dir behind. Added serial.log to the cleanup.
- The teardown trap was armed AFTER cmd_up, but cmd_up's wait_for_ssh can fail
  with QEMU already running (a guest that never opens SSH) — leaking the VM.
  Arm the trap before cmd_up.

Stale image URLs:
- Fedora 41 is EOL and 404s; bumped to Fedora 44 (44-1.7).
- Alpine bumped to 3.21.7; its cloud images ship a .sha512 only (this verifier
  is sha256), so SUM_URL is now empty (skip) with a note.

Validated on delphi (QEMU 11.0.2): ubuntu/fedora/arch pass BOTH test and
test-ready. Alpine (cloud-init seed not applied → no SSH) and NixOS (no
upstream cloud qcow2) remain blocked on image provisioning, documented in the
research note as follow-ups.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

didericis added 1 commit 2026-07-27 11:33:58 -04:00

feat: get Alpine and NixOS cells to green

prd-number-check / require-numbered-prds (pull_request) Successful in 16s

Details

tracker-policy-pr / check-pr (pull_request) Successful in 15s

Details

3e7440de69

Completes the matrix: all five distros now pass both `test` and `test-ready`,
validated end-to-end on the delphi KVM host (QEMU 11.0.2).

Alpine — the generic_ cloud image silently ignores a NoCloud seed, so cloud-init
never ran. Fixes, all confirmed by booting:
- switch to the nocloud_ image variant (built for a local seed),
- start sshd from a runcmd (OpenRC doesn't auto-start it after key injection),
- give the account a throwaway password — Alpine's non-PAM sshd refuses pubkey
  auth for a cloud-init-locked ('!') account, unlike the UsePAM=yes distros,
- install sudo via cloud-init packages: (the minimal image has none, so the
  test-ready `sudo apk add` prereq failed).
Also add instance-id to the NoCloud meta-data, which the nocloud image requires.

NixOS — publishes no downloadable cloud qcow2 (Hydra builds AMIs), so the harness
now BUILDS one with nixos-generators (new scripts/linux-install-test-nixos.nix:
cloud-init for the key, flakes enabled, deliberately no python/git/pipx).
ensure_base_image branches to build_nixos_image for the nixos distro. The
test-ready prereq uses `nix profile install` pinned to nixpkgs/nixos-24.11,
because the guest's default unstable registry builds pipx from source and its
test suite currently fails to build.

Validation: 10/10 cells pass, clean teardown, no leaked VMs or run dirs.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

didericis merged commit 6385752040 into main

2026-07-27 12:24:33 -04:00

didericis deleted branch feat/linux-install-test-harness

2026-07-27 12:24:34 -04:00

Sign in to join this conversation.

2 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: didericis/bot-bottle#534