bot-bottle/docs/research/testing-clean-install-on-linux.md

Testing a clean bot-bottle install on Linux

How do you exercise install.sh the way a brand-new user would — on a pristine Linux environment you can throw away afterward — without polluting your daily-driver host, and across the several package-management regimes Linux fragments into? This is the Linux counterpart to testing-clean-install-on-macos.md; the conclusion is different because Linux gives us a boundary macOS doesn't.

Summary

On macOS the honest options were a throwaway user or a VM, and the throwaway user won on pragmatics (nested virtualization is gated to M3+). On Linux the calculus flips: a disposable KVM virtual machine, booted from a distro cloud image and deleted per run, is both the cleanest boundary and the one that lets a single harness cover Ubuntu, Fedora, Arch, Alpine, and NixOS. The host already requires KVM for the Firecracker backend, so the VM is cheap here.

The harness lives at scripts/linux-install-test.sh. Per run it caches one read-only base image, boots a throwaway copy-on-write overlay (qemu-img create -b base), installs the distro's prerequisites, pipes this checkout's install.sh into the guest exactly as curl … | sh would, asserts the CLI installed, and deletes the overlay — the Linux equivalent of docker run --rm, for a whole machine.

Why a VM, not a container or a throwaway user

Mechanism	Why it's the wrong boundary here
Container (docker run --rm)	Shares the host kernel and ships a deliberately minimal userland — no systemd, a stubbed-out package manager story, and (crucially) it doesn't reproduce the externally-managed Python (PEP 668) that real desktop/server installs put in front of the user. It tests "does install.sh run in a container," not "does it run on a real distro."
Throwaway user (useradd/userdel)	The macOS pick, but weaker on Linux: it reaches the real host, yet every system package it installs (python, pipx, git via apt/dnf/…) stays behind, and it can only ever test the one distro the host runs. The whole Linux-specific value is the cross-distro matrix.
Disposable KVM VM (this harness)	A genuine kernel + userland + package-manager boundary that wipes to nothing on teardown, and swaps freely between distro cloud images. The one real cost — nested virtualization for the backend — doesn't apply, because we gate the installer, not the runtime (below).

Two variants: test (bare host) and test-ready (prepared host)

install.sh never installs a backend, and never installs its own toolchain prerequisites (python3, git, pipx) — it installs the bot-bottle package and runs doctor, which reports what's missing (install.sh header, bot_bottle/cli/commands/doctor.py). That leaves two distinct things worth testing, split into two subcommands that mirror the macOS harness's test / test-ready convention (there the split is the backend service; here it is the toolchain the installer needs):

• test — install.sh runs on the bare cloud image, prerequisites and all left as the vendor ships them. This exercises install.sh's own prerequisite-guard logic — the entire first half of the script (python version gate, git-for-git-specs gate, pipx/pip PEP-668 handling).
• test-ready — the harness installs python3 + git + pipx first (the prereqs step), then runs install.sh. This is the prepared-host happy path: does a clean install actually land and produce a working CLI?

Pass criteria differ by variant:

Variant	PASS when
test	install.sh either installs cleanly (the image already carried enough) or declines with one of its own recognized, actionable prerequisite errors (missing python3/git, no usable pip, PEP 668). A crash or an unrecognized failure is a FAIL.
test-ready	install.sh actually lands: the bot-bottle entry point is present and runs, and doctor reports a usable python and config without crashing. A graceful decline is no longer good enough.

Neither variant requires a green doctor: inside the VM there is no nested KVM or Docker, so the backend is correctly reported not-ready — install.sh does not install a backend and cannot regress one, and this harness does not provision the Docker backend. This is where Linux necessarily diverges from the macOS test-ready, which reaches the host backend; BB_TEST_REQUIRE_BACKEND=1 makes readiness fatal anyway, for a nested-virt host that can satisfy it. The verdict instead classifies doctor's output the way the macOS harness does — a Traceback is an install defect (fail), a missing python/config line is a fail, a not-ready backend is reported — so a genuine installer regression (a broken shim, an import error, a botched PATH) stays visible.

test-all runs the full matrix — every distro × both variants — each cell in its own throwaway VM, and prints a per-cell PASS/FAIL summary.

The distro matrix is the point

Each distro exercises a different corner of the installer:

Distro	Cloud image	What it stresses
Ubuntu (noble)	cloud-images.ubuntu.com	The common case; apt's pipx, externally-managed Python (PEP 668) → install.sh's pipx path.
Fedora	Fedora Cloud Base Generic	dnf packaging, a different default Python, BSD-style checksum file.
Arch	geo.mirror.pkgbuild.com/images/latest	Rolling / newest Python; python-pipx.
Alpine	Alpine "cloud" (cloudinit) image	musl libc + BusyBox sh — the harshest POSIX-sh host for a #!/bin/sh installer.
NixOS	channels.nixos.org OpenStack image	No FHS ~/.local on PATH by default; nix-env user-profile prereqs; pipx laying a self-contained venv on a non-FHS host.

Validation run (2026-07-27)

First full run on the delphi KVM host, QEMU 11.0.2:

Distro	test (bare)	test-ready (prepared)
Ubuntu 24.04	✅ declines at git gate	✅ installs, doctor python+config green
Fedora 44	✅ declines at git gate	✅ installs, doctor python+config green
Arch (latest)	✅ declines at git gate	✅ installs, doctor python+config green
Alpine 3.21	⚠️ blocked (boot)	⚠️ blocked (boot)
NixOS 24.11	⚠️ blocked (no image)	⚠️ blocked (no image)

On all three working distros the bare test sees install.sh exit 1 at the git-for-git-specs gate (cloud images ship python3 but not git) — a sound decline — and test-ready installs cleanly with doctor reporting a usable python and config (backends all not-ready, as expected in a plain VM).

Two known gaps, both in image provisioning rather than the installer:

• Alpine boots to a localhost login: prompt but its cloud image does not apply our NoCloud (cloud-localds) seed the way Ubuntu/Fedora/Arch do, so the SSH key is never injected and wait_for_ssh times out. Needs an Alpine-specific seed path (e.g. presenting the seed as a CD-ROM, or Alpine's tiny-cloud expectations) before it can run.
• NixOS publishes no downloadable cloud qcow2 on channels.nixos.org or the release bucket (its cloud images are Hydra-built AMIs). The table URL 404s; a working NixOS cell needs a locally built image (nixos-generators, baking the test key), which is a follow-up.

The run also fixed real defects it surfaced: the pinned Fedora 41 was EOL/404 (bumped to 44), and the liveness probe used bot-bottle --version — which the CLI does not implement, so every successful install was misreported as failed until it was switched to bot-bottle --help.

In test-ready the harness installs python3 + git + pipx first on each distro (install.sh installs none of them), so all five drive the recommended pipx path. test then removes that scaffolding and lets each distro's bare image collide with install.sh's guards — on most cloud images python3 is present (cloud-init needs it) but git and pipx are not, so install.sh is expected to decline at the git-for-git-specs gate or the PEP-668 pip check with an actionable message. Both are legitimate, and the two variants together cover the whole first half of the installer as well as the happy path.

What a clean install touches (the footprint that decides "wipeable")

Artifact	Location	In the guest's $HOME?	Survives VM teardown?
Config / state / db	~/.bot-bottle/{agents,bottles,contrib,…} (install.sh)	✅	❌ overlay deleted
pipx venv + shim	~/.local/pipx/venvs/bot-bottle, shim in ~/.local/bin	✅	❌ overlay deleted
pip --user fallback	~/.local/lib + ~/.local/bin	✅	❌ overlay deleted
Distro prerequisites (python/git/pipx, test-ready only)	system paths via apt/dnf/pacman/apk/nix-env	❌	❌ overlay deleted

Unlike the macOS throwaway user (whose Homebrew / Apple-Container / Rosetta footprint survives), every row here dies with the overlay — that is the VM's whole advantage. The cached base image is read-only backing and is the only thing that persists between runs, on purpose.

Design notes baked into the harness

• User-mode networking (-netdev user,hostfwd=tcp:127.0.0.1:PORT-:22): no root, no bridge, no host network state touched. Only SSH is forwarded.
• cloud-init seed ISO (cloud-localds) injects an ephemeral SSH keypair and a passwordless-sudo login. The keypair is generated per run and deleted on teardown; the guest can't be logged into after it's gone.
• Copy-on-write overlay: the cached base is never mutated, so a corrupt or interrupted run can't poison the cache; downloads land at *.partial and are renamed only after checksum verification.
• Checksums: verified against each vendor's published sums file at download time (GNU hash file, bare-hash, and Fedora's BSD SHA256 (file) = hash formats are all handled). The NixOS channel image ships no stable sums file, so that one needs BB_TEST_SKIP_VERIFY=1.
• test-all runs every distro × both variants (test and test-ready), each cell in its own subshell on its own forwarded port, so one cell's failure (or teardown trap) can't abort the matrix; it prints a per-cell PASS/FAIL summary.

Not wired into PR CI

Like the macOS harness, the runtime is host-specific (needs /dev/kvm, qemu, and cloud-localds) and is not exercised by the Linux pull-request runner. It is validated statically (bash -n, shellcheck) and run by hand on a KVM-capable host. The cloud-image URLs in the DISTRO table are the one place to bump when a distro cuts a newer build.