39353cefce8e678bb51b4f520cd24470b775b150
4 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 didericis
|
39353cefce |
fix: correct harness defects found running the matrix
First full run on the KVM host surfaced three harness bugs and two stale image URLs; all fixed here. Harness bugs: - Liveness probe used `bot-bottle --version`, which the CLI does not implement (unknown args die non-zero) — so every SUCCESSFUL install was misreported as "no runnable entry point". Switched to `bot-bottle --help`, which exits 0 before any DB/migration/network work. - cmd_down never removed serial.log, so its rmdir failed and every run left an orphan scratch dir behind. Added serial.log to the cleanup. - The teardown trap was armed AFTER cmd_up, but cmd_up's wait_for_ssh can fail with QEMU already running (a guest that never opens SSH) — leaking the VM. Arm the trap before cmd_up. Stale image URLs: - Fedora 41 is EOL and 404s; bumped to Fedora 44 (44-1.7). - Alpine bumped to 3.21.7; its cloud images ship a .sha512 only (this verifier is sha256), so SUM_URL is now empty (skip) with a note. Validated on delphi (QEMU 11.0.2): ubuntu/fedora/arch pass BOTH test and test-ready. Alpine (cloud-init seed not applied → no SSH) and NixOS (no upstream cloud qcow2) remain blocked on image provisioning, documented in the research note as follow-ups. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> |
||
|
didericis
|
bff06bcedf |
refactor: adopt macOS test/test-ready convention for variants
Replaces the BB_TEST_PREREQS env toggle with the two subcommands the macOS
harness established, so the two repos read the same:
test A bare host — prerequisites NOT set up (the default state of a
stock cloud image). Exercises install.sh's own prerequisite-guard
logic. SOUND (PASS) when install.sh either installs cleanly or
declines with one of its own recognized, actionable errors; a
crash or unrecognized failure fails.
test-ready Prerequisites satisfied — the harness installs python3/git/pipx
first, then runs install.sh, which must actually land: entry point
runnable, doctor reporting a usable python and config.
Structure now mirrors scripts/macos-install-test.sh: a shared _test_cycle
driving up -> (prereqs) -> run -> verdict -> down, thin cmd_test / cmd_test_ready
wrappers setting _STEPS / _PASS_CLAIM / _REQUIRE_INSTALL, a standalone `prereqs`
subcommand, and a _test_teardown that prints the PASS/FAIL claim.
The doctor check is now the macOS-style classifier rather than a bare exit-code
read: a Traceback is an install defect (fail), a missing `ok: python:` /
`ok: config:` line is a fail, and a not-ready backend is reported, not fatal
(BB_TEST_REQUIRE_BACKEND=1 makes it fatal for a nested-virt host). This is where
Linux diverges from macOS test-ready — a plain VM has no nested KVM for
Firecracker and the harness doesn't provision the Docker backend, so backend
readiness is never the Linux criterion.
test-all now runs every distro × {test, test-ready}. Research note updated.
Validated with `bash -n` and `shellcheck`.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
|
||
|
didericis
|
84a9fdd457 |
feat: add bare-host install variant to the Linux harness
Splits the Linux clean-install harness into two variants, selected with
BB_TEST_PREREQS:
with (default) — install python3 + git + pipx first, then install.sh.
The ready-host happy path; PASS = install.sh exits 0 and
leaves a runnable bot-bottle entry point.
without — run install.sh on the BARE cloud image. Exercises
install.sh's own prerequisite-guard logic (python gate,
git-for-git-specs gate, pipx/pip PEP-668 handling). PASS =
install.sh either fully succeeds OR declines with one of
its own recognized, actionable prerequisite errors; a
crash or unrecognized failure is a FAIL.
cmd_run now captures install.sh's exit code + full output (install.rc /
install.log) instead of aborting on non-zero, so the verdict step applies the
variant's criterion. The old assert_installed is replaced by a quiet
entry_point_runnable helper plus classify_outcome, which matches the bare-host
declines against the exact die() messages install.sh prints. doctor still runs
for visibility, but only when an entry point exists.
test-all now runs the full matrix -- every distro x both variants -- each cell
in its own throwaway VM, and can be pinned to one variant via BB_TEST_PREREQS.
Research note updated to document both variants and their pass criteria.
Validated with `bash -n` and `shellcheck`.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
|
||
|
didericis
|
6de3f01d68 |
feat: add Linux clean-install test harness
Adds scripts/linux-install-test.sh, a throwaway-VM harness that exercises install.sh the way a brand-new user would across a Linux distro matrix (Ubuntu, Fedora, Arch, Alpine, NixOS), plus the research note docs/research/testing-clean-install-on-linux.md that motivates the approach. This is the Linux counterpart to the macOS clean-install harness. On Linux the boundary of choice flips from a throwaway user account to a disposable KVM VM: it's a genuine kernel + userland + package-manager boundary that wipes to nothing on teardown, and a single harness can swap distro cloud images to cover the several package-management regimes Linux fragments into. The host already needs KVM for the Firecracker backend, so a per-run, copy-on-write VM (qemu-img create -b base) is cheap here -- the equivalent of `docker run --rm`, for a whole machine. Per run the harness caches one read-only base image, boots a throwaway overlay via QEMU/KVM with user-mode networking (no root, no bridge), injects an ephemeral SSH key + passwordless login through a cloud-init seed ISO, installs the distro's prerequisites (python3 + git + pipx), pipes THIS checkout's install.sh into the guest exactly as `curl ... | sh` would, and asserts the CLI installed cleanly. The overlay is deleted on teardown, so even the OS-level prerequisites are wiped -- unlike a throwaway user, the reset is total. Subcommands mirror the macOS harness (up/run/status/down/test) plus test-all (the matrix) and ssh (an interactive guest shell). test arms an EXIT/INT/TERM trap the moment the VM exists, so a failure or Ctrl-C still tears it down. Scope is installer correctness, not runtime: there is no nested KVM/Docker in the VM, so `bot-bottle doctor` correctly reports every backend not-ready and exits non-zero by design. The pass criterion is therefore install.sh exiting 0, the bot-bottle entry point being present on the fresh user's PATH, and `bot-bottle --version` running -- not a green doctor. doctor's output is still printed so a real installer regression (broken shim, import error) stays visible. Validated with `bash -n` and `shellcheck`. Runtime is host-only (needs /dev/kvm, qemu, cloud-localds), so like the macOS harness it isn't exercised by the Linux PR CI. The cloud-image URLs in the DISTRO table are the one place to bump when a distro cuts a newer build. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> |