From fdec887beb02f536d8cea67240fa4cf9db910a53 Mon Sep 17 00:00:00 2001 From: claude Date: Mon, 27 Jul 2026 03:48:39 +0000 Subject: [PATCH] feat: add macOS clean-install test harness Add scripts/macos-install-test.sh, a throwaway-user harness for exercising install.sh the way a brand-new user would on macOS, plus the research note that motivates the approach. The harness has up/run/status/down/deep-reset subcommands. Because install.sh writes only to the user home (pipx venv, ~/.bot-bottle, a PATH line) and never installs the backend, deleting the account is a complete, deterministic reset of the install surface. A disposable macOS VM can't stand in on M1/M2: the Apple `container` backend needs Virtualization.framework, and running it inside a guest VM requires nested virtualization (M3+ only), so a throwaway user is the only way to reach the real host backend from a clean $HOME. Co-Authored-By: Claude Opus 4.8 --- .../testing-clean-install-on-macos.md | 229 ++++++++++++++++++ scripts/macos-install-test.sh | 195 +++++++++++++++ 2 files changed, 424 insertions(+) create mode 100644 docs/research/testing-clean-install-on-macos.md create mode 100755 scripts/macos-install-test.sh diff --git a/docs/research/testing-clean-install-on-macos.md b/docs/research/testing-clean-install-on-macos.md new file mode 100644 index 00000000..7c3e76b5 --- /dev/null +++ b/docs/research/testing-clean-install-on-macos.md @@ -0,0 +1,229 @@ +# Testing a clean bot-bottle install on macOS + +How do you exercise `install.sh` (and, ideally, a first `bot-bottle start`) +the way a brand-new user would — on a pristine macOS environment you can +throw away afterward — *without* permanently polluting your daily-driver +Mac? The user's framing: is there a VM or boundary that avoids creating a +separate account, or is spinning up and tearing down a throwaway macOS +user on the CLI easy enough to just do that? + +## Summary + +There is no lightweight, in-place macOS sandbox that hands you a clean home +directory and wipeable system state without *either* a VM or a separate +user account. `sandbox-exec` (Seatbelt) is deprecated and confines a +process, not an environment; App Sandbox is for shipping apps, not for +provisioning a fresh dev host. So the real choice is exactly the two the +user named: **a disposable macOS VM** or **a throwaway user account** — +and which one is right turns on a detail specific to *this* project. + +bot-bottle's default macOS backend is Apple's `container`, which runs each +container in its own lightweight VM via `Virtualization.framework` +([`README.md:27`](../README.md), [`apple-container-backend.md`](apple-container-backend.md)). +That means a full end-to-end test — install *and* `bot-bottle start` — +needs virtualization to work wherever bot-bottle runs. Inside a macOS guest +VM that requires **nested virtualization, which Apple gates to M3 or newer +chips on macOS 15+**. On M1/M2 you cannot run the Apple Container backend +(or Docker Desktop, same reason) inside a macOS VM at all. + +The recommendation splits on what you're testing and what silicon you have: + +- **Install-script correctness only** (does `curl | sh` → pipx → config dir + → `doctor`'s Python/config checks pass?): a **disposable Tart VM** is the + cleanest boundary and works on any Apple Silicon Mac. `doctor` will report + the backend as not-ready inside the VM on M1/M2, which is fine — you're + testing the installer, not the runtime. +- **Full runtime** (actually launch a bottle) on **M3/M4**: a **disposable + Tart VM from a golden base image, cloned per run** is the gold standard — + a genuine kernel/state boundary that wipes to nothing. +- **Full runtime** on **M1/M2**, or when you'd rather not fight nested virt: + a **throwaway admin user via `sysadminctl`** is the pragmatic pick. It + tests the real backend because the backend runs on the host hypervisor — + but it is a *hygiene* boundary, not a security one, and it does **not** + clean the system-level footprint (see below). + +Prefer the VM. Reach for the throwaway user only when nested virt is off the +table and you accept an imperfect wipe. + +## Why "a boundary without a separate user" doesn't really exist on macOS + +macOS has no namespace/overlay story like Linux `unshare` + tmpfs. The +options that sound like in-place sandboxes don't fit: + +| Mechanism | Why it doesn't give you a clean, wipeable env | +|---|---| +| `sandbox-exec` / Seatbelt | Officially deprecated; confines *one process's* syscalls against a profile. It cannot present a fresh `$HOME` or a pristine `/usr/local`, and it won't let the Apple Container system service work. | +| App Sandbox | Entitlement-based confinement for signed `.app` bundles, not a provisioning tool for a CLI dev environment. | +| A second `$HOME` via `HOME=/tmp/foo` | Redirects only what honors `$HOME`. `install.sh` mostly does (it writes `~/.bot-bottle` and pipx/pip `--user` paths), but the Apple `container` install lands in `/usr/local` + a **system service**, and Homebrew lands in `/opt/homebrew` — all outside any `$HOME` you set. You'd get a false sense of "clean." | +| APFS snapshot rollback (`tmutil localsnapshot`) | You can't roll the live boot volume back to a local snapshot without booting to Recovery; it's not a per-run userspace undo. | + +So the honest answer to "is there some boundary that avoids a separate +user?": yes — a **VM** — and it's the *stronger* boundary anyway. The only +lighter-weight option is the separate user, with the caveats below. + +## What a clean install actually touches (the footprint that decides "wipeable") + +Grounding the teardown story in what `install.sh` and the backend create: + +| Artifact | Location | In `$HOME`? | Survives user deletion? | +|---|---|---|---| +| Config / state / db | `~/.bot-bottle/{agents,bottles,contrib,state,db}` ([`install.sh:80-83`](../install.sh), [`bot_bottle/paths.py:59`](../bot_bottle/paths.py)) | ✅ | ❌ removed with home | +| pipx venv + shim | `~/.local/pipx/venvs/bot-bottle`, shim in `~/.local/bin` ([`install.sh:87-89`](../install.sh)) | ✅ | ❌ removed with home | +| pip `--user` fallback | `~/Library/Python//{lib,bin}` ([`install.sh:100-104`](../install.sh)) | ✅ | ❌ removed with home | +| PATH / token exports | shell profile (`~/.zprofile`, etc.); `BOT_BOTTLE_CLAUDE_OAUTH_TOKEN` ([`README.md:74`](../README.md)) | ✅ | ❌ removed with home | +| **Apple `container` install** | `/usr/local/...` + notarized `.pkg` receipts | ❌ | ✅ **stays** | +| **Apple `container` system service** | launchd system service (`container system start`) | ❌ | ✅ **stays** | +| **Homebrew** (if used for `container`/python) | `/opt/homebrew` | ❌ | ✅ **stays** | +| Rosetta 2 (needed for image builds) | system | ❌ | ✅ **stays** | + +The three bold rows are the crux: **deleting the throwaway user does not +uninstall the Apple Container runtime, its system service, Homebrew, or +Rosetta.** A VM, by contrast, wipes 100% of the above by definition — +that's its entire advantage for this task. + +## Option A — Disposable Tart VM (recommended) + +[Tart](https://tart.run) is a CLI-first macOS/Linux VM manager built on +`Virtualization.framework`, purpose-built for exactly this "does it work on +a clean macOS, without my settings/permissions/data" workflow. Keep one +pristine *golden* image, clone a throwaway per run, delete it after. + +```sh +brew install cirruslabs/cli/tart + +# One-time: build a golden base (either a prebuilt image or a vanilla IPSW). +tart clone ghcr.io/cirruslabs/macos-tahoe-base:latest golden # ~25 GB pull +# — or a truly vanilla install you click through once — +# tart create golden --from-ipsw latest --disk-size 60 + +# Per test run: clone → boot → test → destroy. +tart clone golden test-run +tart run test-run & +ssh admin@"$(tart ip test-run)" +# inside the guest: +# curl -fsSL https://gitea.dideric.is/didericis/bot-bottle/raw/branch/main/install.sh | sh +# bot-bottle doctor +tart stop test-run +tart delete test-run # back to pristine; golden is untouched +``` + +Cloning is cheap (sparse files), so the golden image is your reset button — +every `tart clone` is a fresh macOS. This is the closest thing to a Linux +`docker run --rm` for a whole Mac. + +**The nested-virt caveat (read before relying on it for runtime tests).** +The Apple Container backend inside the guest needs +`Virtualization.framework` to work *inside* the VM. Apple enables nested +virtualization only on **M3 or newer**, on **macOS 15 (Sequoia) or later**; +M2 and earlier are excluded by Apple, confirmed by Apple DTS. Consequences: + +- **M3/M4 host:** full runtime works in the guest. `bot-bottle doctor` + reports the backend ready and `start` can launch a bottle. Gold standard. +- **M1/M2 host:** the guest can install bot-bottle and pass the Python / + config-dir checks, but `doctor`'s backend check will fail and you cannot + launch a bottle in the VM. Still perfectly good for testing *the + installer*; not for the runtime. +- **M4-specific:** a known bug blocks pre-Ventura guests on M4; use a + current macOS guest (which you want anyway, since Apple `container` + targets macOS 26 Tahoe). + +UTM is the GUI equivalent on the same framework (and was first to expose +nested virt) if you'd rather click; Tart wins for a scriptable +spin-up/tear-down loop. + +## Option B — Throwaway user via `sysadminctl` (pragmatic fallback) + +Creating and deleting a user from the CLI is genuinely a two-liner, and it +tests the **real** backend on any Apple Silicon Mac because the backend runs +on the host hypervisor — no nested virt needed. + +```sh +# Create a self-contained admin user (admin needed for the container service). +sudo sysadminctl -addUser bbtest -fullName "bot-bottle test" \ + -password 'throwaway' -admin + +# Log into that account (fast-user-switch or the login window), then run the +# installer as bbtest exactly as a new user would. When done: + +sudo sysadminctl -deleteUser bbtest -secure # -secure erases the home dir +``` + +Honest accounting of what this does and doesn't buy you: + +- **Boundary strength:** it's a *hygiene / fresh-`$HOME`* boundary, **not a + security boundary.** Same kernel, same admin group; an admin test user can + touch system state. If the point is "clean environment," fine. If the point + is "contain something untrusted," this is the wrong tool — use a VM. +- **Wipe completeness:** `-secure` erases the home dir (so `~/.bot-bottle`, + the pipx venv, and profile exports go away), but as the footprint table + shows, the **Apple Container runtime, its launchd system service, + Homebrew, and Rosetta persist.** For a truly repeatable "did a *system with + nothing installed* work?" test, that residue defeats the purpose — the + second run isn't clean. +- **Operational gotchas:** don't pass real passwords on the command line (they + land in `ps` and history — this is a throwaway credential, so it's + tolerable here). Deletion must run as root from a normally-booted, admin- + logged-in session; the Terminal needs **Full Disk Access** or you'll hit + error `-14120` and a half-deleted account. Prefer letting the system place + the home dir (don't pass `-home`), or deletion can orphan it. + +Use this when you're on M1/M2, you specifically want to exercise the live +backend, and you can tolerate the system-level runtime staying installed +between runs (or you uninstall Apple `container` / brew by hand to reset). + +## Honorable mentions + +- **External bootable macOS volume.** A fresh macOS on an external SSD (or a + separate APFS volume) is bare-metal disposable: no nested-virt limit, real + backend works, and you `diskutil` the volume away to reset. Cost is reboot + friction per run — good for an occasional thorough pass, poor for a tight + loop. +- **Rented / cloud Mac.** AWS EC2 Mac (dedicated Mac minis), Scaleway Apple + silicon, or MacStadium give a genuinely throwaway host you release when + done. Overkill for local iteration, but this is essentially what the + project's own advisory `integration-macos` CI job needs — a self-hosted + Apple Silicon runner with the `container` CLI, Python ≥ 3.11, and coverage + on the launchd service's PATH ([`README.md:78`](../README.md)). If you end + up standing up a cloud Mac for install testing, it doubles as that runner. + +## Recommendation + +Default to a **disposable Tart VM** — it's the only option that wipes the +*entire* footprint (including the Apple Container system service that a user +deletion leaves behind), it's a real boundary, and the spin-up/tear-down +loop is a two-command `tart clone` / `tart delete`. Confirm your chip first: +on **M3/M4** it tests install *and* runtime end-to-end; on **M1/M2** it still +cleanly tests `install.sh` + `doctor`'s Python/config path, and you fall back +to a **throwaway `sysadminctl` admin user** for live-backend testing — +accepting that it's a hygiene boundary and that you'll manually uninstall the +Apple Container runtime / Homebrew between runs to get back to truly clean. + +There is no third, lighter-weight "in-place boundary without a user" that +actually delivers a clean, wipeable macOS — the VM *is* that answer, and it's +the better one. + +## Harness + +The throwaway-user loop is scripted in +[`scripts/macos-install-test.sh`](../../scripts/macos-install-test.sh): +`up` creates the account, `run` pipes *this checkout's* `install.sh` into it +headlessly (so a PR is verifiable before it lands) and lets the installer run +`doctor`, `down` deletes the account and its home (the full reset), and +`deep-reset` additionally uninstalls the host `container` runtime. It leans on +the footprint analysis above — the reset is just user deletion because +everything `install.sh` writes is user-home-local. + +## Sources + +- [Apple Containers on macOS: technical comparison with Docker — The New Stack](https://thenewstack.io/apple-containers-on-macos-a-technical-comparison-with-docker/) +- [How to Set Up Apple Containerization on macOS 26 — Stéphane Paquet](https://spaquet.medium.com/how-to-set-up-apple-containerization-on-macos-26-f870cc8c26cd) +- [Install Apple Container CLI (macOS 15/26) — 4sysops](https://4sysops.com/archives/install-apple-container-cli-running-containers-natively-on-macos-15-sequoia-and-macos-26-tahoe/) +- [Nested virtualization on Apple Silicon (M3+, macOS 15) — UTM issue #6700](https://github.com/utmapp/UTM/issues/6700) +- [macOS 15 Sequoia nested virtualization for M3+ — Parallels Forums](https://forum.parallels.com/threads/macos-15-sequoia-nested-virtualization-for-m3-macs.364397/) +- [M2 nested virtualization restriction (Apple DTS) — Apple Developer Forums](https://developer.apple.com/forums/thread/756723) +- [M4 can't virtualize older macOS — Yahoo/Tech](https://tech.yahoo.com/computing/articles/m4-mac-computers-cant-virtualize-175122301.html) +- [Tart — macOS/Linux VMs on Apple Silicon (Cirrus Labs)](https://tart.run/quick-start/) +- [Tart GitHub](https://github.com/cirruslabs/tart) +- [macOS VMs in a single command — frr.dev](https://www.frr.dev/posts/tart-macos-vms-from-terminal/) +- [sysadminctl reference — SS64](https://ss64.com/mac/sysadminctl.html) +- [User management from the macOS command line — macnotes](https://macnotes.wordpress.com/2019/03/28/user-management-create-remove-change-password-secure-token-from-macos-command-line/) diff --git a/scripts/macos-install-test.sh b/scripts/macos-install-test.sh new file mode 100755 index 00000000..37c318e9 --- /dev/null +++ b/scripts/macos-install-test.sh @@ -0,0 +1,195 @@ +#!/usr/bin/env bash +# Clean-install test harness for the macOS (Apple `container`) path. +# +# Exercises install.sh the way a brand-new user would, inside a throwaway +# macOS account you create and delete from the CLI. install.sh's entire +# footprint is user-home-local — the pipx venv under ~/.local (or the pip +# --user tree under ~/Library/Python), the ~/.bot-bottle config dir, and a +# PATH line in the login shell. It never installs the backend itself (see +# the header of install.sh), so deleting the user is a complete, +# deterministic reset of everything the installer touched. The Apple +# `container` runtime is a HOST prerequisite installed once and kept; +# `deep-reset` is the rare escape hatch that also removes it. +# +# Why a throwaway user and not a disposable VM: bot-bottle's default macOS +# backend is Apple `container`, which runs each container in its own +# Virtualization.framework microVM. Running that backend inside a macOS +# guest VM needs nested virtualization, which Apple gates to M3+ silicon. +# On M1/M2 a separate user account is the only way to get a clean $HOME +# while still reaching the real host backend. Full rationale in +# docs/research/testing-clean-install-on-macos.md. +# +# Usage: +# sudo ./scripts/macos-install-test.sh up # create the throwaway user +# sudo ./scripts/macos-install-test.sh run # run install.sh (+doctor) as it +# ./scripts/macos-install-test.sh status # user present? backend ready? +# sudo ./scripts/macos-install-test.sh down # delete user + home (the reset) +# sudo ./scripts/macos-install-test.sh deep-reset # ALSO uninstall host `container` +# +# A full clean cycle is: sudo ... up && sudo ... run && sudo ... down +# +# Config via env: +# BB_TEST_USER account short name (default: bbtest) +# BB_TEST_FULLNAME account full name (default: "bot-bottle install test") +# BB_TEST_ADMIN 1=admin (reach container svc), 0=standard (default: 1) +# BB_TEST_INSTALL_URL curl this install.sh instead of piping the local checkout +# BOT_BOTTLE_INSTALL_SPEC passed through to install.sh (pip / git spec) +# +# Notes: +# * Run from a normally-booted admin session. Grant Terminal *Full Disk +# Access* (System Settings -> Privacy & Security) or `down` half-fails +# with error -14120 and leaves an orphaned account. +# * `sysadminctl` always exits 0 even on failure, so `up`/`down` verify +# the result with `dscl` and fail loudly on a mismatch. +# * The account is created without a password: `run` drives it headlessly +# via `sudo -u`, which never needs the target's password. The account +# cannot GUI-login, which this harness does not require. +# * `run` covers the installer + `bot-bottle doctor`. Actually launching a +# bottle from the throwaway user may need a full launchd user session +# (`launchctl asuser`); on M1/M2 the backend can't run under nested virt +# anyway, so this harness stops at install + doctor. + +set -euo pipefail + +USER_NAME="${BB_TEST_USER:-bbtest}" +FULL_NAME="${BB_TEST_FULLNAME:-bot-bottle install test}" +ADMIN="${BB_TEST_ADMIN:-1}" + +_SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +_REPO_ROOT="$(cd "$_SCRIPT_DIR/.." && pwd)" + +# --- guards ---------------------------------------------------------- +require_macos() { + [ "$(uname -s)" = "Darwin" ] \ + || { echo "error: this harness is macOS-only (uname is $(uname -s))" >&2; exit 1; } +} + +require_root() { + if [ "$(id -u)" -ne 0 ]; then + echo "error: '$1' needs root; re-run under sudo" >&2 + exit 1 + fi +} + +user_exists() { dscl . -read "/Users/$USER_NAME" >/dev/null 2>&1; } + +# Run a shell snippet as the throwaway user in a fresh login shell. +run_as_user() { sudo -u "$USER_NAME" -i sh -c "$1"; } + +# --- commands -------------------------------------------------------- +cmd_up() { + require_macos + require_root up + if user_exists; then + echo "$USER_NAME already exists; nothing to do (run 'down' first to reset)" + return 0 + fi + local admin_flag=() + [ "$ADMIN" = "1" ] && admin_flag=(-admin) + # No -password: the account is only ever driven headlessly via `sudo -u`, + # which doesn't need one. sysadminctl warns about FileVault here; that's + # irrelevant to a headless test account. + sysadminctl -addUser "$USER_NAME" -fullName "$FULL_NAME" "${admin_flag[@]}" || true + # sysadminctl exits 0 regardless of outcome, so confirm the account landed. + user_exists || { echo "error: failed to create $USER_NAME" >&2; exit 1; } + echo "created $USER_NAME (admin=$ADMIN). Install into it with: sudo $0 run" +} + +cmd_run() { + require_macos + require_root run + user_exists || { echo "error: $USER_NAME does not exist; run 'sudo $0 up' first" >&2; exit 1; } + + local spec_env="" + [ -n "${BOT_BOTTLE_INSTALL_SPEC:-}" ] \ + && spec_env="BOT_BOTTLE_INSTALL_SPEC='$BOT_BOTTLE_INSTALL_SPEC' " + + echo "== installing bot-bottle as $USER_NAME ==" + if [ -n "${BB_TEST_INSTALL_URL:-}" ]; then + run_as_user "curl -fsSL '$BB_TEST_INSTALL_URL' | ${spec_env}sh" + else + # Test THIS checkout's install.sh, not the published one, so a PR is + # verifiable before it lands. Stage it world-readable in /tmp because + # the throwaway user can't read the tester's (mode 700) home. + local staged + staged="$(mktemp /tmp/bb-install.XXXXXX.sh)" + cp "$_REPO_ROOT/install.sh" "$staged" + chmod 644 "$staged" + run_as_user "${spec_env}sh '$staged'" || { rm -f "$staged"; exit 1; } + rm -f "$staged" + fi + echo "== install.sh runs 'doctor' itself; re-check anytime with: $0 status ==" +} + +cmd_status() { + require_macos + if user_exists; then + echo "user: $USER_NAME present" + if [ "$(id -u)" -eq 0 ]; then + echo "doctor (as $USER_NAME):" + run_as_user 'command -v bot-bottle >/dev/null 2>&1 \ + && bot-bottle doctor \ + || echo " bot-bottle not installed for this user yet"' || true + else + echo " (re-run under sudo to run 'bot-bottle doctor' as $USER_NAME)" + fi + else + echo "user: $USER_NAME absent" + fi + if command -v container >/dev/null 2>&1; then + echo "backend: apple 'container' present ($(container --version 2>/dev/null | head -1))" + else + echo "backend: apple 'container' NOT on PATH (host prerequisite; install once)" + fi +} + +cmd_down() { + require_macos + require_root down + if ! user_exists; then + echo "$USER_NAME not present; nothing to remove" + return 0 + fi + # A plain -deleteUser removes the home dir, which is the whole reset. + # -secure is a no-op on modern macOS (secure erase of the home folder + # was removed in Sierra), so it buys nothing here. + sysadminctl -deleteUser "$USER_NAME" || true + if user_exists; then + echo "error: $USER_NAME still present after delete." >&2 + echo " - grant Terminal Full Disk Access (System Settings > Privacy & Security), or" >&2 + echo " - it may hold the last Secure Token (won't happen while another admin exists)" >&2 + exit 1 + fi + echo "removed $USER_NAME and its home — install surface is clean." +} + +cmd_deep_reset() { + require_macos + require_root deep-reset + # Remove the user first (idempotent), then the HOST-level container + # runtime that a user deletion leaves behind under /usr/local + launchd. + cmd_down || true + if command -v container >/dev/null 2>&1; then + # The service can run in more than one launchd context (the invoking + # user's and root's), so stop both, best-effort. + [ -n "${SUDO_USER:-}" ] && sudo -u "$SUDO_USER" container system stop 2>/dev/null || true + container system stop 2>/dev/null || true + if [ -x /usr/local/bin/uninstall-container.sh ]; then + /usr/local/bin/uninstall-container.sh -d || true + echo "uninstalled the host Apple 'container' runtime" + else + echo "note: /usr/local/bin/uninstall-container.sh not found; runtime left as-is" >&2 + fi + else + echo "no 'container' runtime on PATH; nothing further to remove" + fi +} + +case "${1:-}" in + up) cmd_up ;; + run) cmd_run ;; + status) cmd_status ;; + down) cmd_down ;; + deep-reset) cmd_deep_reset ;; + *) echo "usage: $0 {up|run|status|down|deep-reset}" >&2 ; exit 2 ;; +esac