feat: get Alpine and NixOS cells to green

Completes the matrix: all five distros now pass both `test` and `test-ready`,
validated end-to-end on the delphi KVM host (QEMU 11.0.2).

Alpine — the generic_ cloud image silently ignores a NoCloud seed, so cloud-init
never ran. Fixes, all confirmed by booting:
- switch to the nocloud_ image variant (built for a local seed),
- start sshd from a runcmd (OpenRC doesn't auto-start it after key injection),
- give the account a throwaway password — Alpine's non-PAM sshd refuses pubkey
  auth for a cloud-init-locked ('!') account, unlike the UsePAM=yes distros,
- install sudo via cloud-init packages: (the minimal image has none, so the
  test-ready `sudo apk add` prereq failed).
Also add instance-id to the NoCloud meta-data, which the nocloud image requires.

NixOS — publishes no downloadable cloud qcow2 (Hydra builds AMIs), so the harness
now BUILDS one with nixos-generators (new scripts/linux-install-test-nixos.nix:
cloud-init for the key, flakes enabled, deliberately no python/git/pipx).
ensure_base_image branches to build_nixos_image for the nixos distro. The
test-ready prereq uses `nix profile install` pinned to nixpkgs/nixos-24.11,
because the guest's default unstable registry builds pipx from source and its
test suite currently fails to build.

Validation: 10/10 cells pass, clean teardown, no leaked VMs or run dirs.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

docs/research/testing-clean-install-on-linux.md

@@ -81,42 +81,50 @@ Each distro exercises a different corner of the installer:
 | **Ubuntu** (noble) | `cloud-images.ubuntu.com` | The common case; `apt`'s `pipx`, externally-managed Python (PEP 668) → install.sh's pipx path. |
 | **Fedora** | Fedora Cloud Base Generic | `dnf` packaging, a different default Python, BSD-style checksum file. |
 | **Arch** | `geo.mirror.pkgbuild.com/images/latest` | Rolling / newest Python; `python-pipx`. |
-| **Alpine** | Alpine "cloud" (cloudinit) image | musl libc + BusyBox `sh` — the harshest POSIX-`sh` host for a `#!/bin/sh` installer. |
-| **NixOS** | `channels.nixos.org` OpenStack image | No FHS `~/.local` on PATH by default; `nix-env` user-profile prereqs; pipx laying a self-contained venv on a non-FHS host. |
+| **Alpine** | Alpine `nocloud_` (cloudinit) image | musl libc + BusyBox `sh` — the harshest POSIX-`sh` host for a `#!/bin/sh` installer. |
+| **NixOS** | locally built with `nixos-generators` | No FHS `~/.local` on PATH by default; `nix profile install` prereqs; pipx laying a self-contained venv on a non-FHS host. |
 
-### Validation run (2026-07-27)
+### Validation run (2026-07-27) — full green
 
-First full run on the delphi KVM host, QEMU 11.0.2:
+Full matrix on the delphi KVM host, QEMU 11.0.2, both variants × all five
+distros passing:
 
 | Distro | `test` (bare) | `test-ready` (prepared) |
 |---|---|---|
 | Ubuntu 24.04 | ✅ declines at git gate | ✅ installs, doctor python+config green |
-| Fedora 44 | ✅ declines at git gate | ✅ installs, doctor python+config green |
-| Arch (latest) | ✅ declines at git gate | ✅ installs, doctor python+config green |
-| Alpine 3.21 | ⚠️ blocked (boot) | ⚠️ blocked (boot) |
-| NixOS 24.11 | ⚠️ blocked (no image) | ⚠️ blocked (no image) |
+| Fedora 44 | ✅ declines at git gate | ✅ installs |
+| Arch (latest) | ✅ declines at git gate | ✅ installs |
+| Alpine 3.21 | ✅ declines at git gate | ✅ installs |
+| NixOS 24.11 | ✅ declines (no python3) | ✅ installs |
 
-On all three working distros the bare `test` sees `install.sh` exit 1 at the
-git-for-git-specs gate (cloud images ship python3 but not git) — a sound
-decline — and `test-ready` installs cleanly with `doctor` reporting a usable
-python and config (backends all not-ready, as expected in a plain VM).
+The bare `test` sees `install.sh` decline soundly — exit 1 at the
+git-for-git-specs gate on the Debian/Fedora/Arch/Alpine images (they ship
+python3 but not git), and at the python3 gate on NixOS (no python3 on PATH) —
+and `test-ready` installs cleanly with `doctor` reporting a usable python and
+config (backends all not-ready, as expected in a plain VM).
 
-**Two known gaps, both in image provisioning rather than the installer:**
+Getting to green surfaced and fixed a series of real defects:
 
-- **Alpine** boots to a `localhost login:` prompt but its cloud image does not
-  apply our NoCloud (`cloud-localds`) seed the way Ubuntu/Fedora/Arch do, so
-  the SSH key is never injected and `wait_for_ssh` times out. Needs an
-  Alpine-specific seed path (e.g. presenting the seed as a CD-ROM, or Alpine's
-  `tiny-cloud` expectations) before it can run.
-- **NixOS** publishes no downloadable cloud qcow2 on `channels.nixos.org` or
-  the release bucket (its cloud images are Hydra-built AMIs). The table URL
-  404s; a working NixOS cell needs a locally built image (`nixos-generators`,
-  baking the test key), which is a follow-up.
-
-The run also fixed real defects it surfaced: the pinned Fedora 41 was EOL/404
-(bumped to 44), and the liveness probe used `bot-bottle --version` — which the
-CLI does not implement, so every successful install was misreported as failed
-until it was switched to `bot-bottle --help`.
+- **Fedora 41 was EOL/404** → bumped to 44.
+- The liveness probe used `bot-bottle --version`, which the CLI does not
+  implement (unknown args die non-zero), so every *successful* install was
+  misreported as failed → switched to `bot-bottle --help`.
+- **Alpine** needed three fixes: the `generic_` image ignores a NoCloud seed
+  (switched to the `nocloud_` variant); OpenRC does not auto-start sshd after
+  cloud-init injects the key (start it via `runcmd`); and Alpine's non-PAM
+  sshd refuses pubkey auth for a cloud-init-*locked* account (give it a
+  throwaway password). It also has no `sudo` by default (install it via
+  cloud-init `packages:`).
+- **NixOS** publishes no downloadable cloud qcow2 (its cloud images are
+  Hydra-built AMIs), so the harness builds one with `nixos-generators`
+  ([`linux-install-test-nixos.nix`](../../scripts/linux-install-test-nixos.nix)):
+  cloud-init for the key, flakes enabled, deliberately no python/git/pipx. The
+  `test-ready` prereq install pins `nixpkgs/nixos-24.11` because the guest's
+  default unstable registry builds pipx from source (and its test suite
+  currently fails to build).
+- Two harness-hygiene bugs also fixed: `cmd_down` left `serial.log` behind
+  (orphaned run dirs), and the teardown trap was armed after `cmd_up`, leaking
+  a VM when `wait_for_ssh` timed out.
 
 In `test-ready` the harness installs `python3 + git + pipx` first on each
 distro (install.sh installs none of them), so all five drive the recommended
@@ -134,7 +142,7 @@ the whole first half of the installer as well as the happy path.
 | Config / state / db | `~/.bot-bottle/{agents,bottles,contrib,…}` ([`install.sh`](../../install.sh)) | ✅ | ❌ overlay deleted |
 | pipx venv + shim | `~/.local/pipx/venvs/bot-bottle`, shim in `~/.local/bin` | ✅ | ❌ overlay deleted |
 | pip `--user` fallback | `~/.local/lib` + `~/.local/bin` | ✅ | ❌ overlay deleted |
-| **Distro prerequisites** (python/git/pipx, `test-ready` only) | system paths via `apt`/`dnf`/`pacman`/`apk`/`nix-env` | ❌ | ❌ **overlay deleted** |
+| **Distro prerequisites** (python/git/pipx, `test-ready` only) | system paths via `apt`/`dnf`/`pacman`/`apk`/`nix profile` | ❌ | ❌ **overlay deleted** |
 
 Unlike the macOS throwaway user (whose Homebrew / Apple-Container / Rosetta
 footprint *survives*), **every row here dies with the overlay** — that is the
@@ -153,8 +161,14 @@ only thing that persists between runs, on purpose.
   are renamed only after checksum verification.
 - **Checksums**: verified against each vendor's published sums file at
   download time (GNU `hash  file`, bare-hash, and Fedora's BSD
-  `SHA256 (file) = hash` formats are all handled). The NixOS channel image
-  ships no stable sums file, so that one needs `BB_TEST_SKIP_VERIFY=1`.
+  `SHA256 (file) = hash` formats are all handled). Alpine ships `.sha512` only
+  (this verifier is sha256) so it is skipped; NixOS is built locally, not
+  downloaded, so there is nothing to verify.
+- **NixOS is built, not downloaded**: `ensure_base_image` runs
+  `nixos-generate -f qcow` against
+  [`linux-install-test-nixos.nix`](../../scripts/linux-install-test-nixos.nix)
+  once and caches the result; the per-run seed/overlay flow is otherwise
+  identical to the downloaded distros.
 - **`test-all`** runs every distro × both variants (`test` and `test-ready`),
   each cell in its own subshell on its own forwarded port, so one cell's
   failure (or teardown trap) can't abort the matrix; it prints a per-cell

scripts/linux-install-test-nixos.nix

@@ -0,0 +1,24 @@
+# NixOS image for scripts/linux-install-test.sh.
+#
+# NixOS publishes no downloadable cloud qcow2 (its cloud images are Hydra-built
+# AMIs), so the harness BUILDS this one with nixos-generators (-f qcow). It is
+# deliberately minimal — no python3/git/pipx — so the bare `test` variant is
+# genuinely under-provisioned and exercises install.sh's guards; `test-ready`
+# provisions them with `nix profile install` (hence flakes below).
+{ lib, ... }:
+{
+  # Consume the same NoCloud seed the other distros use: cloud-init injects the
+  # per-run ephemeral SSH key for root. Leave networking to NixOS's default
+  # dhcpcd (QEMU user-mode NAT) — enabling cloud-init's networkd here conflicts
+  # with dhcpcd and can drop the guest's network.
+  services.cloud-init.enable = true;
+  services.cloud-init.network.enable = false;
+
+  services.openssh.enable = true;
+  services.openssh.settings.PermitRootLogin = lib.mkForce "prohibit-password";
+
+  # Flakes so the `test-ready` prereq step can `nix profile install nixpkgs#...`.
+  nix.settings.experimental-features = [ "nix-command" "flakes" ];
+
+  system.stateVersion = "24.11";
+}

scripts/linux-install-test.sh

@@ -74,7 +74,8 @@
 #
 # Notes:
 #   * Needs /dev/kvm, qemu-system-x86_64, and cloud-localds (cloud-image-utils
-#     / cloud-utils). On the NixOS host:  nix shell nixpkgs#qemu nixpkgs#cloud-utils
+#     / cloud-utils); the nixos distro additionally needs nixos-generate. On the
+#     NixOS host:  nix shell nixpkgs#qemu nixpkgs#cloud-utils nixpkgs#nixos-generators
 #   * Networking is user-mode (`-netdev user,hostfwd`) so the harness needs no
 #     root, no bridge, and touches no host network state. Only SSH is forwarded.
 #   * The cloud-image URLs in the DISTRO table are the one place to bump when a
@@ -132,18 +133,26 @@ SUM_URL[arch]="https://geo.mirror.pkgbuild.com/images/latest/Arch-Linux-x86_64-c
 SSH_USER[arch]="arch"
 PREREQ[arch]="sudo pacman -Sy --noconfirm python git python-pipx"
 
-IMAGE_URL[alpine]="https://dl-cdn.alpinelinux.org/alpine/v3.21/releases/cloud/generic_alpine-3.21.7-x86_64-bios-cloudinit-r0.qcow2"
-SUM_URL[alpine]=""   # Alpine cloud images ship .sha512 only; this verifier is sha256. Set BB_TEST_SKIP_VERIFY handling applies.
+# Use the *nocloud_* Alpine variant, not generic_: the generic image probes
+# network datasources and ignores the local NoCloud seed, so cloud-init never
+# runs and the SSH key is never injected. (Only published at .0 patch levels.)
+IMAGE_URL[alpine]="https://dl-cdn.alpinelinux.org/alpine/v3.21/releases/cloud/nocloud_alpine-3.21.0-x86_64-bios-cloudinit-r0.qcow2"
+SUM_URL[alpine]=""   # Alpine cloud images ship .sha512 only; this verifier is sha256.
 SSH_USER[alpine]="alpine"
 PREREQ[alpine]="sudo apk add --no-cache python3 git pipx"
 
-# NixOS is externally-managed in its own way (no FHS ~/.local on PATH by
-# default, python/pipx not preinstalled). nix-env installs into the user
-# profile — no sudo — and pipx then lays down a self-contained venv.
-IMAGE_URL[nixos]="https://channels.nixos.org/nixos-24.11/latest-nixos-openstack-x86_64-linux.qcow2"
-SUM_URL[nixos]=""   # channel image ships no stable sums file; set BB_TEST_SKIP_VERIFY=1
+# NixOS publishes no downloadable cloud qcow2 (its cloud images are Hydra-built
+# AMIs), so the harness BUILDS one with nixos-generators — see build_nixos_image
+# and linux-install-test-nixos.nix. It's externally-managed in its own way (no
+# FHS ~/.local on PATH by default); `nix profile install` provisions the
+# prerequisites into root's profile (the image enables flakes for this).
+IMAGE_URL[nixos]="nix:build"   # sentinel: ensure_base_image builds instead of downloading
+SUM_URL[nixos]=""
 SSH_USER[nixos]="root"
-PREREQ[nixos]="nix-env -iA nixos.python3 nixos.git nixos.pipx"
+# Pin to a stable release: the guest's default `nixpkgs` registry is unstable,
+# where pipx isn't in the binary cache and builds from source (its test suite
+# currently fails to build). nixos-24.11 has these cached as substitutes.
+PREREQ[nixos]="nix profile install nixpkgs/nixos-24.11#python3 nixpkgs/nixos-24.11#git nixpkgs/nixos-24.11#pipx"
 
 ALL_DISTROS=(ubuntu fedora arch alpine nixos)
 
@@ -242,12 +251,36 @@ wait_for_ssh() {
 
 # --- image cache -----------------------------------------------------
 _base_image() {
-    # One cached file per distro, keyed by the image's basename so a URL bump
-    # to a newer build lands as a new cache entry rather than a stale hit.
+    # One cached file per distro. NixOS is built (not downloaded), so it has a
+    # fixed cache name; the rest are keyed by the image's basename so a URL bump
+    # lands as a new cache entry rather than a stale hit.
+    if [ "$DISTRO" = nixos ]; then
+        echo "$CACHE_DIR/nixos-built.qcow2"
+        return
+    fi
     local url="${IMAGE_URL[$DISTRO]}"
     echo "$CACHE_DIR/$DISTRO-$(basename "$url")"
 }
 
+# NixOS has no upstream cloud qcow2; build one with nixos-generators and copy it
+# out of the (immutable, GC-able) store into the cache.
+build_nixos_image() {
+    local out; out="$(_base_image)"
+    [ -f "$out" ] && { echo "== nixos base image cached: $out =="; return 0; }
+    command -v nixos-generate >/dev/null 2>&1 || {
+        echo "error: 'nixos-generate' is required to build the NixOS image" >&2
+        echo "  run inside:  nix shell nixpkgs#nixos-generators nixpkgs#qemu nixpkgs#cloud-utils" >&2
+        return 1
+    }
+    echo "== building NixOS cloud image with nixos-generators (first run is slow) =="
+    local link="$CACHE_DIR/nixos-result"
+    nixos-generate -f qcow --system x86_64-linux \
+        -c "$_SCRIPT_DIR/linux-install-test-nixos.nix" -o "$link"
+    cp -L "$link"/*.qcow2 "$out"
+    rm -f "$link"
+    echo "   built + cached: $out"
+}
+
 verify_checksum() {
     local file="$1" sums_url="${SUM_URL[$DISTRO]}" base
     base="$(basename "${IMAGE_URL[$DISTRO]}")"
@@ -277,6 +310,10 @@ verify_checksum() {
 
 ensure_base_image() {
     mkdir -p "$CACHE_DIR"
+    if [ "$DISTRO" = nixos ]; then
+        build_nixos_image
+        return
+    fi
     local base; base="$(_base_image)"
     if [ -f "$base" ]; then
         echo "== base image cached: $base =="
@@ -306,6 +343,30 @@ make_seed() {
 #cloud-config
 ssh_authorized_keys:
   - $pub
+EOF
+    elif [ "$DISTRO" = alpine ]; then
+        # Alpine needs three things the systemd distros don't: cloud-init locks
+        # the account (lock_passwd), but Alpine's non-PAM sshd then refuses
+        # pubkey auth for a locked account — so give it a throwaway password;
+        # and OpenRC does not auto-start sshd after the key is injected, so
+        # start it via runcmd.
+        cat > "$user_data" <<EOF
+#cloud-config
+users:
+  - name: $user
+    sudo: ALL=(ALL) NOPASSWD:ALL
+    shell: /bin/sh
+    lock_passwd: false
+    ssh_authorized_keys:
+      - $pub
+chpasswd:
+  expire: false
+  list: |
+    $user:bbtest
+packages:
+  - sudo
+runcmd:
+  - [ sh, -c, "rc-service sshd start 2>/dev/null || true" ]
 EOF
     else
         cat > "$user_data" <<EOF
@@ -319,8 +380,10 @@ users:
       - $pub
 EOF
     fi
-    # meta-data can be empty but must exist for NoCloud.
-    : > "$RUN_DIR/meta-data"
+    # NoCloud wants a meta-data with an instance-id, or cloud-init may not treat
+    # the seed as a new instance (the Alpine nocloud image is strict about this).
+    printf 'instance-id: bbtest-%s\nlocal-hostname: bbtest-%s\n' "$DISTRO" "$DISTRO" \
+        > "$RUN_DIR/meta-data"
     cloud-localds "$(_seed)" "$user_data" "$RUN_DIR/meta-data"
 }