From 3e7440de698dc4f1ea3c92ca4d0b6393465477fe Mon Sep 17 00:00:00 2001 From: didericis Date: Mon, 27 Jul 2026 11:33:48 -0400 Subject: [PATCH] feat: get Alpine and NixOS cells to green MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Completes the matrix: all five distros now pass both `test` and `test-ready`, validated end-to-end on the delphi KVM host (QEMU 11.0.2). Alpine — the generic_ cloud image silently ignores a NoCloud seed, so cloud-init never ran. Fixes, all confirmed by booting: - switch to the nocloud_ image variant (built for a local seed), - start sshd from a runcmd (OpenRC doesn't auto-start it after key injection), - give the account a throwaway password — Alpine's non-PAM sshd refuses pubkey auth for a cloud-init-locked ('!') account, unlike the UsePAM=yes distros, - install sudo via cloud-init packages: (the minimal image has none, so the test-ready `sudo apk add` prereq failed). Also add instance-id to the NoCloud meta-data, which the nocloud image requires. NixOS — publishes no downloadable cloud qcow2 (Hydra builds AMIs), so the harness now BUILDS one with nixos-generators (new scripts/linux-install-test-nixos.nix: cloud-init for the key, flakes enabled, deliberately no python/git/pipx). ensure_base_image branches to build_nixos_image for the nixos distro. The test-ready prereq uses `nix profile install` pinned to nixpkgs/nixos-24.11, because the guest's default unstable registry builds pipx from source and its test suite currently fails to build. Validation: 10/10 cells pass, clean teardown, no leaked VMs or run dirs. Co-Authored-By: Claude Opus 4.8 --- .../testing-clean-install-on-linux.md | 74 ++++++++------- scripts/linux-install-test-nixos.nix | 24 +++++ scripts/linux-install-test.sh | 89 ++++++++++++++++--- 3 files changed, 144 insertions(+), 43 deletions(-) create mode 100644 scripts/linux-install-test-nixos.nix diff --git a/docs/research/testing-clean-install-on-linux.md b/docs/research/testing-clean-install-on-linux.md index c46fc541..7ad15cde 100644 --- a/docs/research/testing-clean-install-on-linux.md +++ b/docs/research/testing-clean-install-on-linux.md @@ -81,42 +81,50 @@ Each distro exercises a different corner of the installer: | **Ubuntu** (noble) | `cloud-images.ubuntu.com` | The common case; `apt`'s `pipx`, externally-managed Python (PEP 668) → install.sh's pipx path. | | **Fedora** | Fedora Cloud Base Generic | `dnf` packaging, a different default Python, BSD-style checksum file. | | **Arch** | `geo.mirror.pkgbuild.com/images/latest` | Rolling / newest Python; `python-pipx`. | -| **Alpine** | Alpine "cloud" (cloudinit) image | musl libc + BusyBox `sh` — the harshest POSIX-`sh` host for a `#!/bin/sh` installer. | -| **NixOS** | `channels.nixos.org` OpenStack image | No FHS `~/.local` on PATH by default; `nix-env` user-profile prereqs; pipx laying a self-contained venv on a non-FHS host. | +| **Alpine** | Alpine `nocloud_` (cloudinit) image | musl libc + BusyBox `sh` — the harshest POSIX-`sh` host for a `#!/bin/sh` installer. | +| **NixOS** | locally built with `nixos-generators` | No FHS `~/.local` on PATH by default; `nix profile install` prereqs; pipx laying a self-contained venv on a non-FHS host. | -### Validation run (2026-07-27) +### Validation run (2026-07-27) — full green -First full run on the delphi KVM host, QEMU 11.0.2: +Full matrix on the delphi KVM host, QEMU 11.0.2, both variants × all five +distros passing: | Distro | `test` (bare) | `test-ready` (prepared) | |---|---|---| | Ubuntu 24.04 | ✅ declines at git gate | ✅ installs, doctor python+config green | -| Fedora 44 | ✅ declines at git gate | ✅ installs, doctor python+config green | -| Arch (latest) | ✅ declines at git gate | ✅ installs, doctor python+config green | -| Alpine 3.21 | ⚠️ blocked (boot) | ⚠️ blocked (boot) | -| NixOS 24.11 | ⚠️ blocked (no image) | ⚠️ blocked (no image) | +| Fedora 44 | ✅ declines at git gate | ✅ installs | +| Arch (latest) | ✅ declines at git gate | ✅ installs | +| Alpine 3.21 | ✅ declines at git gate | ✅ installs | +| NixOS 24.11 | ✅ declines (no python3) | ✅ installs | -On all three working distros the bare `test` sees `install.sh` exit 1 at the -git-for-git-specs gate (cloud images ship python3 but not git) — a sound -decline — and `test-ready` installs cleanly with `doctor` reporting a usable -python and config (backends all not-ready, as expected in a plain VM). +The bare `test` sees `install.sh` decline soundly — exit 1 at the +git-for-git-specs gate on the Debian/Fedora/Arch/Alpine images (they ship +python3 but not git), and at the python3 gate on NixOS (no python3 on PATH) — +and `test-ready` installs cleanly with `doctor` reporting a usable python and +config (backends all not-ready, as expected in a plain VM). -**Two known gaps, both in image provisioning rather than the installer:** +Getting to green surfaced and fixed a series of real defects: -- **Alpine** boots to a `localhost login:` prompt but its cloud image does not - apply our NoCloud (`cloud-localds`) seed the way Ubuntu/Fedora/Arch do, so - the SSH key is never injected and `wait_for_ssh` times out. Needs an - Alpine-specific seed path (e.g. presenting the seed as a CD-ROM, or Alpine's - `tiny-cloud` expectations) before it can run. -- **NixOS** publishes no downloadable cloud qcow2 on `channels.nixos.org` or - the release bucket (its cloud images are Hydra-built AMIs). The table URL - 404s; a working NixOS cell needs a locally built image (`nixos-generators`, - baking the test key), which is a follow-up. - -The run also fixed real defects it surfaced: the pinned Fedora 41 was EOL/404 -(bumped to 44), and the liveness probe used `bot-bottle --version` — which the -CLI does not implement, so every successful install was misreported as failed -until it was switched to `bot-bottle --help`. +- **Fedora 41 was EOL/404** → bumped to 44. +- The liveness probe used `bot-bottle --version`, which the CLI does not + implement (unknown args die non-zero), so every *successful* install was + misreported as failed → switched to `bot-bottle --help`. +- **Alpine** needed three fixes: the `generic_` image ignores a NoCloud seed + (switched to the `nocloud_` variant); OpenRC does not auto-start sshd after + cloud-init injects the key (start it via `runcmd`); and Alpine's non-PAM + sshd refuses pubkey auth for a cloud-init-*locked* account (give it a + throwaway password). It also has no `sudo` by default (install it via + cloud-init `packages:`). +- **NixOS** publishes no downloadable cloud qcow2 (its cloud images are + Hydra-built AMIs), so the harness builds one with `nixos-generators` + ([`linux-install-test-nixos.nix`](../../scripts/linux-install-test-nixos.nix)): + cloud-init for the key, flakes enabled, deliberately no python/git/pipx. The + `test-ready` prereq install pins `nixpkgs/nixos-24.11` because the guest's + default unstable registry builds pipx from source (and its test suite + currently fails to build). +- Two harness-hygiene bugs also fixed: `cmd_down` left `serial.log` behind + (orphaned run dirs), and the teardown trap was armed after `cmd_up`, leaking + a VM when `wait_for_ssh` timed out. In `test-ready` the harness installs `python3 + git + pipx` first on each distro (install.sh installs none of them), so all five drive the recommended @@ -134,7 +142,7 @@ the whole first half of the installer as well as the happy path. | Config / state / db | `~/.bot-bottle/{agents,bottles,contrib,…}` ([`install.sh`](../../install.sh)) | ✅ | ❌ overlay deleted | | pipx venv + shim | `~/.local/pipx/venvs/bot-bottle`, shim in `~/.local/bin` | ✅ | ❌ overlay deleted | | pip `--user` fallback | `~/.local/lib` + `~/.local/bin` | ✅ | ❌ overlay deleted | -| **Distro prerequisites** (python/git/pipx, `test-ready` only) | system paths via `apt`/`dnf`/`pacman`/`apk`/`nix-env` | ❌ | ❌ **overlay deleted** | +| **Distro prerequisites** (python/git/pipx, `test-ready` only) | system paths via `apt`/`dnf`/`pacman`/`apk`/`nix profile` | ❌ | ❌ **overlay deleted** | Unlike the macOS throwaway user (whose Homebrew / Apple-Container / Rosetta footprint *survives*), **every row here dies with the overlay** — that is the @@ -153,8 +161,14 @@ only thing that persists between runs, on purpose. are renamed only after checksum verification. - **Checksums**: verified against each vendor's published sums file at download time (GNU `hash file`, bare-hash, and Fedora's BSD - `SHA256 (file) = hash` formats are all handled). The NixOS channel image - ships no stable sums file, so that one needs `BB_TEST_SKIP_VERIFY=1`. + `SHA256 (file) = hash` formats are all handled). Alpine ships `.sha512` only + (this verifier is sha256) so it is skipped; NixOS is built locally, not + downloaded, so there is nothing to verify. +- **NixOS is built, not downloaded**: `ensure_base_image` runs + `nixos-generate -f qcow` against + [`linux-install-test-nixos.nix`](../../scripts/linux-install-test-nixos.nix) + once and caches the result; the per-run seed/overlay flow is otherwise + identical to the downloaded distros. - **`test-all`** runs every distro × both variants (`test` and `test-ready`), each cell in its own subshell on its own forwarded port, so one cell's failure (or teardown trap) can't abort the matrix; it prints a per-cell diff --git a/scripts/linux-install-test-nixos.nix b/scripts/linux-install-test-nixos.nix new file mode 100644 index 00000000..804783c9 --- /dev/null +++ b/scripts/linux-install-test-nixos.nix @@ -0,0 +1,24 @@ +# NixOS image for scripts/linux-install-test.sh. +# +# NixOS publishes no downloadable cloud qcow2 (its cloud images are Hydra-built +# AMIs), so the harness BUILDS this one with nixos-generators (-f qcow). It is +# deliberately minimal — no python3/git/pipx — so the bare `test` variant is +# genuinely under-provisioned and exercises install.sh's guards; `test-ready` +# provisions them with `nix profile install` (hence flakes below). +{ lib, ... }: +{ + # Consume the same NoCloud seed the other distros use: cloud-init injects the + # per-run ephemeral SSH key for root. Leave networking to NixOS's default + # dhcpcd (QEMU user-mode NAT) — enabling cloud-init's networkd here conflicts + # with dhcpcd and can drop the guest's network. + services.cloud-init.enable = true; + services.cloud-init.network.enable = false; + + services.openssh.enable = true; + services.openssh.settings.PermitRootLogin = lib.mkForce "prohibit-password"; + + # Flakes so the `test-ready` prereq step can `nix profile install nixpkgs#...`. + nix.settings.experimental-features = [ "nix-command" "flakes" ]; + + system.stateVersion = "24.11"; +} diff --git a/scripts/linux-install-test.sh b/scripts/linux-install-test.sh index 02b9babd..9ba06440 100755 --- a/scripts/linux-install-test.sh +++ b/scripts/linux-install-test.sh @@ -74,7 +74,8 @@ # # Notes: # * Needs /dev/kvm, qemu-system-x86_64, and cloud-localds (cloud-image-utils -# / cloud-utils). On the NixOS host: nix shell nixpkgs#qemu nixpkgs#cloud-utils +# / cloud-utils); the nixos distro additionally needs nixos-generate. On the +# NixOS host: nix shell nixpkgs#qemu nixpkgs#cloud-utils nixpkgs#nixos-generators # * Networking is user-mode (`-netdev user,hostfwd`) so the harness needs no # root, no bridge, and touches no host network state. Only SSH is forwarded. # * The cloud-image URLs in the DISTRO table are the one place to bump when a @@ -132,18 +133,26 @@ SUM_URL[arch]="https://geo.mirror.pkgbuild.com/images/latest/Arch-Linux-x86_64-c SSH_USER[arch]="arch" PREREQ[arch]="sudo pacman -Sy --noconfirm python git python-pipx" -IMAGE_URL[alpine]="https://dl-cdn.alpinelinux.org/alpine/v3.21/releases/cloud/generic_alpine-3.21.7-x86_64-bios-cloudinit-r0.qcow2" -SUM_URL[alpine]="" # Alpine cloud images ship .sha512 only; this verifier is sha256. Set BB_TEST_SKIP_VERIFY handling applies. +# Use the *nocloud_* Alpine variant, not generic_: the generic image probes +# network datasources and ignores the local NoCloud seed, so cloud-init never +# runs and the SSH key is never injected. (Only published at .0 patch levels.) +IMAGE_URL[alpine]="https://dl-cdn.alpinelinux.org/alpine/v3.21/releases/cloud/nocloud_alpine-3.21.0-x86_64-bios-cloudinit-r0.qcow2" +SUM_URL[alpine]="" # Alpine cloud images ship .sha512 only; this verifier is sha256. SSH_USER[alpine]="alpine" PREREQ[alpine]="sudo apk add --no-cache python3 git pipx" -# NixOS is externally-managed in its own way (no FHS ~/.local on PATH by -# default, python/pipx not preinstalled). nix-env installs into the user -# profile — no sudo — and pipx then lays down a self-contained venv. -IMAGE_URL[nixos]="https://channels.nixos.org/nixos-24.11/latest-nixos-openstack-x86_64-linux.qcow2" -SUM_URL[nixos]="" # channel image ships no stable sums file; set BB_TEST_SKIP_VERIFY=1 +# NixOS publishes no downloadable cloud qcow2 (its cloud images are Hydra-built +# AMIs), so the harness BUILDS one with nixos-generators — see build_nixos_image +# and linux-install-test-nixos.nix. It's externally-managed in its own way (no +# FHS ~/.local on PATH by default); `nix profile install` provisions the +# prerequisites into root's profile (the image enables flakes for this). +IMAGE_URL[nixos]="nix:build" # sentinel: ensure_base_image builds instead of downloading +SUM_URL[nixos]="" SSH_USER[nixos]="root" -PREREQ[nixos]="nix-env -iA nixos.python3 nixos.git nixos.pipx" +# Pin to a stable release: the guest's default `nixpkgs` registry is unstable, +# where pipx isn't in the binary cache and builds from source (its test suite +# currently fails to build). nixos-24.11 has these cached as substitutes. +PREREQ[nixos]="nix profile install nixpkgs/nixos-24.11#python3 nixpkgs/nixos-24.11#git nixpkgs/nixos-24.11#pipx" ALL_DISTROS=(ubuntu fedora arch alpine nixos) @@ -242,12 +251,36 @@ wait_for_ssh() { # --- image cache ----------------------------------------------------- _base_image() { - # One cached file per distro, keyed by the image's basename so a URL bump - # to a newer build lands as a new cache entry rather than a stale hit. + # One cached file per distro. NixOS is built (not downloaded), so it has a + # fixed cache name; the rest are keyed by the image's basename so a URL bump + # lands as a new cache entry rather than a stale hit. + if [ "$DISTRO" = nixos ]; then + echo "$CACHE_DIR/nixos-built.qcow2" + return + fi local url="${IMAGE_URL[$DISTRO]}" echo "$CACHE_DIR/$DISTRO-$(basename "$url")" } +# NixOS has no upstream cloud qcow2; build one with nixos-generators and copy it +# out of the (immutable, GC-able) store into the cache. +build_nixos_image() { + local out; out="$(_base_image)" + [ -f "$out" ] && { echo "== nixos base image cached: $out =="; return 0; } + command -v nixos-generate >/dev/null 2>&1 || { + echo "error: 'nixos-generate' is required to build the NixOS image" >&2 + echo " run inside: nix shell nixpkgs#nixos-generators nixpkgs#qemu nixpkgs#cloud-utils" >&2 + return 1 + } + echo "== building NixOS cloud image with nixos-generators (first run is slow) ==" + local link="$CACHE_DIR/nixos-result" + nixos-generate -f qcow --system x86_64-linux \ + -c "$_SCRIPT_DIR/linux-install-test-nixos.nix" -o "$link" + cp -L "$link"/*.qcow2 "$out" + rm -f "$link" + echo " built + cached: $out" +} + verify_checksum() { local file="$1" sums_url="${SUM_URL[$DISTRO]}" base base="$(basename "${IMAGE_URL[$DISTRO]}")" @@ -277,6 +310,10 @@ verify_checksum() { ensure_base_image() { mkdir -p "$CACHE_DIR" + if [ "$DISTRO" = nixos ]; then + build_nixos_image + return + fi local base; base="$(_base_image)" if [ -f "$base" ]; then echo "== base image cached: $base ==" @@ -306,6 +343,30 @@ make_seed() { #cloud-config ssh_authorized_keys: - $pub +EOF + elif [ "$DISTRO" = alpine ]; then + # Alpine needs three things the systemd distros don't: cloud-init locks + # the account (lock_passwd), but Alpine's non-PAM sshd then refuses + # pubkey auth for a locked account — so give it a throwaway password; + # and OpenRC does not auto-start sshd after the key is injected, so + # start it via runcmd. + cat > "$user_data" </dev/null || true" ] EOF else cat > "$user_data" < "$RUN_DIR/meta-data" + # NoCloud wants a meta-data with an instance-id, or cloud-init may not treat + # the seed as a new instance (the Alpine nocloud image is strict about this). + printf 'instance-id: bbtest-%s\nlocal-hostname: bbtest-%s\n' "$DISTRO" "$DISTRO" \ + > "$RUN_DIR/meta-data" cloud-localds "$(_seed)" "$user_data" "$RUN_DIR/meta-data" }