didericis
39353cefce
First full run on the KVM host surfaced three harness bugs and two stale image URLs; all fixed here. Harness bugs: - Liveness probe used `bot-bottle --version`, which the CLI does not implement (unknown args die non-zero) — so every SUCCESSFUL install was misreported as "no runnable entry point". Switched to `bot-bottle --help`, which exits 0 before any DB/migration/network work. - cmd_down never removed serial.log, so its rmdir failed and every run left an orphan scratch dir behind. Added serial.log to the cleanup. - The teardown trap was armed AFTER cmd_up, but cmd_up's wait_for_ssh can fail with QEMU already running (a guest that never opens SSH) — leaking the VM. Arm the trap before cmd_up. Stale image URLs: - Fedora 41 is EOL and 404s; bumped to Fedora 44 (44-1.7). - Alpine bumped to 3.21.7; its cloud images ship a .sha512 only (this verifier is sha256), so SUM_URL is now empty (skip) with a note. Validated on delphi (QEMU 11.0.2): ubuntu/fedora/arch pass BOTH test and test-ready. Alpine (cloud-init seed not applied → no SSH) and NixOS (no upstream cloud qcow2) remain blocked on image provisioning, documented in the research note as follow-ups. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
170 lines
11 KiB
Markdown
170 lines
11 KiB
Markdown
# Testing a clean bot-bottle install on Linux
|
||
|
||
How do you exercise `install.sh` the way a brand-new user would — on a
|
||
pristine Linux environment you can throw away afterward — *without*
|
||
polluting your daily-driver host, and across the several package-management
|
||
regimes Linux fragments into? This is the Linux counterpart to
|
||
[`testing-clean-install-on-macos.md`](testing-clean-install-on-macos.md);
|
||
the conclusion is different because Linux gives us a boundary macOS doesn't.
|
||
|
||
## Summary
|
||
|
||
On macOS the honest options were a throwaway user or a VM, and the throwaway
|
||
user won on pragmatics (nested virtualization is gated to M3+). On Linux the
|
||
calculus flips: a **disposable KVM virtual machine, booted from a distro
|
||
cloud image and deleted per run, is both the cleanest boundary and the one
|
||
that lets a single harness cover Ubuntu, Fedora, Arch, Alpine, and NixOS**.
|
||
The host already requires KVM for the Firecracker backend, so the VM is cheap
|
||
here.
|
||
|
||
The harness lives at [`scripts/linux-install-test.sh`](../../scripts/linux-install-test.sh).
|
||
Per run it caches one read-only base image, boots a throwaway copy-on-write
|
||
overlay (`qemu-img create -b base`), installs the distro's prerequisites,
|
||
pipes *this checkout's* `install.sh` into the guest exactly as `curl … | sh`
|
||
would, asserts the CLI installed, and deletes the overlay — the Linux
|
||
equivalent of `docker run --rm`, for a whole machine.
|
||
|
||
## Why a VM, not a container or a throwaway user
|
||
|
||
| Mechanism | Why it's the wrong boundary here |
|
||
|---|---|
|
||
| **Container** (`docker run --rm`) | Shares the host kernel and ships a deliberately minimal userland — no systemd, a stubbed-out package manager story, and (crucially) it doesn't reproduce the *externally-managed Python* (PEP 668) that real desktop/server installs put in front of the user. It tests "does install.sh run in a container," not "does it run on a real distro." |
|
||
| **Throwaway user** (`useradd`/`userdel`) | The macOS pick, but weaker on Linux: it reaches the real host, yet every system package it installs (python, pipx, git via `apt`/`dnf`/…) stays behind, and it can only ever test the *one* distro the host runs. The whole Linux-specific value is the cross-distro matrix. |
|
||
| **Disposable KVM VM** (this harness) | A genuine kernel + userland + package-manager boundary that wipes to nothing on teardown, and swaps freely between distro cloud images. The one real cost — nested virtualization for the *backend* — doesn't apply, because we gate the installer, not the runtime (below). |
|
||
|
||
## Two variants: `test` (bare host) and `test-ready` (prepared host)
|
||
|
||
`install.sh` never installs a backend, and never installs its own toolchain
|
||
prerequisites (python3, git, pipx) — it installs the `bot-bottle` package and
|
||
runs `doctor`, which *reports* what's missing
|
||
([`install.sh`](../../install.sh) header,
|
||
[`bot_bottle/cli/commands/doctor.py`](../../bot_bottle/cli/commands/doctor.py)).
|
||
That leaves two distinct things worth testing, split into two subcommands that
|
||
mirror the macOS harness's `test` / `test-ready` convention (there the split is
|
||
the backend service; here it is the toolchain the installer needs):
|
||
|
||
- **`test`** — `install.sh` runs on the **bare cloud image**, prerequisites and
|
||
all left as the vendor ships them. This exercises `install.sh`'s own
|
||
prerequisite-guard logic — the entire first half of the script (python
|
||
version gate, git-for-git-specs gate, pipx/pip PEP-668 handling).
|
||
- **`test-ready`** — the harness installs python3 + git + pipx first (the
|
||
`prereqs` step), then runs `install.sh`. This is the *prepared-host happy
|
||
path*: does a clean install actually land and produce a working CLI?
|
||
|
||
**Pass criteria differ by variant:**
|
||
|
||
| Variant | PASS when |
|
||
|---|---|
|
||
| `test` | `install.sh` **either** installs cleanly (the image already carried enough) **or** declines with one of its own recognized, actionable prerequisite errors (missing python3/git, no usable pip, PEP 668). A crash or an *unrecognized* failure is a FAIL. |
|
||
| `test-ready` | `install.sh` actually lands: the `bot-bottle` entry point is present and runs, and `doctor` reports a usable python and config without crashing. A graceful decline is no longer good enough. |
|
||
|
||
Neither variant requires a green `doctor`: inside the VM there is no nested KVM
|
||
or Docker, so **the backend is correctly reported not-ready** — install.sh does
|
||
not install a backend and cannot regress one, and this harness does not
|
||
provision the Docker backend. This is where Linux necessarily diverges from the
|
||
macOS `test-ready`, which reaches the host backend; `BB_TEST_REQUIRE_BACKEND=1`
|
||
makes readiness fatal anyway, for a nested-virt host that can satisfy it. The
|
||
verdict instead classifies `doctor`'s output the way the macOS harness does — a
|
||
`Traceback` is an install defect (fail), a missing `python`/`config` line is a
|
||
fail, a not-ready backend is reported — so a genuine installer regression (a
|
||
broken shim, an import error, a botched PATH) stays visible.
|
||
|
||
`test-all` runs the full matrix — every distro × both variants — each cell in
|
||
its own throwaway VM, and prints a per-cell PASS/FAIL summary.
|
||
|
||
## The distro matrix is the point
|
||
|
||
Each distro exercises a different corner of the installer:
|
||
|
||
| Distro | Cloud image | What it stresses |
|
||
|---|---|---|
|
||
| **Ubuntu** (noble) | `cloud-images.ubuntu.com` | The common case; `apt`'s `pipx`, externally-managed Python (PEP 668) → install.sh's pipx path. |
|
||
| **Fedora** | Fedora Cloud Base Generic | `dnf` packaging, a different default Python, BSD-style checksum file. |
|
||
| **Arch** | `geo.mirror.pkgbuild.com/images/latest` | Rolling / newest Python; `python-pipx`. |
|
||
| **Alpine** | Alpine "cloud" (cloudinit) image | musl libc + BusyBox `sh` — the harshest POSIX-`sh` host for a `#!/bin/sh` installer. |
|
||
| **NixOS** | `channels.nixos.org` OpenStack image | No FHS `~/.local` on PATH by default; `nix-env` user-profile prereqs; pipx laying a self-contained venv on a non-FHS host. |
|
||
|
||
### Validation run (2026-07-27)
|
||
|
||
First full run on the delphi KVM host, QEMU 11.0.2:
|
||
|
||
| Distro | `test` (bare) | `test-ready` (prepared) |
|
||
|---|---|---|
|
||
| Ubuntu 24.04 | ✅ declines at git gate | ✅ installs, doctor python+config green |
|
||
| Fedora 44 | ✅ declines at git gate | ✅ installs, doctor python+config green |
|
||
| Arch (latest) | ✅ declines at git gate | ✅ installs, doctor python+config green |
|
||
| Alpine 3.21 | ⚠️ blocked (boot) | ⚠️ blocked (boot) |
|
||
| NixOS 24.11 | ⚠️ blocked (no image) | ⚠️ blocked (no image) |
|
||
|
||
On all three working distros the bare `test` sees `install.sh` exit 1 at the
|
||
git-for-git-specs gate (cloud images ship python3 but not git) — a sound
|
||
decline — and `test-ready` installs cleanly with `doctor` reporting a usable
|
||
python and config (backends all not-ready, as expected in a plain VM).
|
||
|
||
**Two known gaps, both in image provisioning rather than the installer:**
|
||
|
||
- **Alpine** boots to a `localhost login:` prompt but its cloud image does not
|
||
apply our NoCloud (`cloud-localds`) seed the way Ubuntu/Fedora/Arch do, so
|
||
the SSH key is never injected and `wait_for_ssh` times out. Needs an
|
||
Alpine-specific seed path (e.g. presenting the seed as a CD-ROM, or Alpine's
|
||
`tiny-cloud` expectations) before it can run.
|
||
- **NixOS** publishes no downloadable cloud qcow2 on `channels.nixos.org` or
|
||
the release bucket (its cloud images are Hydra-built AMIs). The table URL
|
||
404s; a working NixOS cell needs a locally built image (`nixos-generators`,
|
||
baking the test key), which is a follow-up.
|
||
|
||
The run also fixed real defects it surfaced: the pinned Fedora 41 was EOL/404
|
||
(bumped to 44), and the liveness probe used `bot-bottle --version` — which the
|
||
CLI does not implement, so every successful install was misreported as failed
|
||
until it was switched to `bot-bottle --help`.
|
||
|
||
In `test-ready` the harness installs `python3 + git + pipx` first on each
|
||
distro (install.sh installs none of them), so all five drive the recommended
|
||
pipx path. `test` then removes that scaffolding and lets each distro's bare
|
||
image collide with install.sh's guards — on most cloud images python3 is
|
||
present (cloud-init needs it) but git and pipx are not, so install.sh is
|
||
expected to decline at the git-for-git-specs gate or the PEP-668 pip check with
|
||
an actionable message. Both are legitimate, and the two variants together cover
|
||
the whole first half of the installer as well as the happy path.
|
||
|
||
## What a clean install touches (the footprint that decides "wipeable")
|
||
|
||
| Artifact | Location | In the guest's `$HOME`? | Survives VM teardown? |
|
||
|---|---|---|---|
|
||
| Config / state / db | `~/.bot-bottle/{agents,bottles,contrib,…}` ([`install.sh`](../../install.sh)) | ✅ | ❌ overlay deleted |
|
||
| pipx venv + shim | `~/.local/pipx/venvs/bot-bottle`, shim in `~/.local/bin` | ✅ | ❌ overlay deleted |
|
||
| pip `--user` fallback | `~/.local/lib` + `~/.local/bin` | ✅ | ❌ overlay deleted |
|
||
| **Distro prerequisites** (python/git/pipx, `test-ready` only) | system paths via `apt`/`dnf`/`pacman`/`apk`/`nix-env` | ❌ | ❌ **overlay deleted** |
|
||
|
||
Unlike the macOS throwaway user (whose Homebrew / Apple-Container / Rosetta
|
||
footprint *survives*), **every row here dies with the overlay** — that is the
|
||
VM's whole advantage. The cached base image is read-only backing and is the
|
||
only thing that persists between runs, on purpose.
|
||
|
||
## Design notes baked into the harness
|
||
|
||
- **User-mode networking** (`-netdev user,hostfwd=tcp:127.0.0.1:PORT-:22`):
|
||
no root, no bridge, no host network state touched. Only SSH is forwarded.
|
||
- **cloud-init seed ISO** (`cloud-localds`) injects an ephemeral SSH keypair
|
||
and a passwordless-sudo login. The keypair is generated per run and deleted
|
||
on teardown; the guest can't be logged into after it's gone.
|
||
- **Copy-on-write overlay**: the cached base is never mutated, so a corrupt or
|
||
interrupted run can't poison the cache; downloads land at `*.partial` and
|
||
are renamed only after checksum verification.
|
||
- **Checksums**: verified against each vendor's published sums file at
|
||
download time (GNU `hash file`, bare-hash, and Fedora's BSD
|
||
`SHA256 (file) = hash` formats are all handled). The NixOS channel image
|
||
ships no stable sums file, so that one needs `BB_TEST_SKIP_VERIFY=1`.
|
||
- **`test-all`** runs every distro × both variants (`test` and `test-ready`),
|
||
each cell in its own subshell on its own forwarded port, so one cell's
|
||
failure (or teardown trap) can't abort the matrix; it prints a per-cell
|
||
PASS/FAIL summary.
|
||
|
||
## Not wired into PR CI
|
||
|
||
Like the macOS harness, the runtime is host-specific (needs `/dev/kvm`,
|
||
`qemu`, and `cloud-localds`) and is not exercised by the Linux pull-request
|
||
runner. It is validated statically (`bash -n`, `shellcheck`) and run by hand
|
||
on a KVM-capable host. The cloud-image URLs in the `DISTRO` table are the one
|
||
place to bump when a distro cuts a newer build.
|