didericis/bot-bottle
1
0
Fork 0
Code Issues 34 Pull Requests 11 Actions Packages Releases Wiki Activity

feat: add macOS clean-install test harness
prd-number-check / require-numbered-prds (pull_request) Successful in 14s
Details
tracker-policy-pr / check-pr (pull_request) Successful in 10s
Details

Browse Source 
Add scripts/macos-install-test.sh, a throwaway-user harness for exercising
install.sh the way a brand-new user would on macOS, plus the research note
that motivates the approach.

The harness has up/run/status/down/deep-reset subcommands. Because install.sh
writes only to the user home (pipx venv, ~/.bot-bottle, a PATH line) and never
installs the backend, deleting the account is a complete, deterministic reset
of the install surface. A disposable macOS VM can't stand in on M1/M2: the
Apple `container` backend needs Virtualization.framework, and running it inside
a guest VM requires nested virtualization (M3+ only), so a throwaway user is
the only way to reach the real host backend from a clean $HOME.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
didericis (claude)
2026-07-27 03:48:39 +00:00
parent a6e1aebda1
commit fdec887beb
2 changed files with 424 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/research/testing-clean-install-on-macos.md
+229
View File
@@ -0,0 +1,229 @@
# Testing a clean bot-bottle install on macOS
How do you exercise `install.sh` (and, ideally, a first `bot-bottle start`)
the way a brand-new user would — on a pristine macOS environment you can
throw away afterward — *without* permanently polluting your daily-driver
Mac? The user's framing: is there a VM or boundary that avoids creating a
separate account, or is spinning up and tearing down a throwaway macOS
user on the CLI easy enough to just do that?
## Summary
There is no lightweight, in-place macOS sandbox that hands you a clean home
directory and wipeable system state without *either* a VM or a separate
user account. `sandbox-exec` (Seatbelt) is deprecated and confines a
process, not an environment; App Sandbox is for shipping apps, not for
provisioning a fresh dev host. So the real choice is exactly the two the
user named: **a disposable macOS VM** or **a throwaway user account** —
and which one is right turns on a detail specific to *this* project.
bot-bottle's default macOS backend is Apple's `container`, which runs each
container in its own lightweight VM via `Virtualization.framework`
([`README.md:27`](../README.md), [`apple-container-backend.md`](apple-container-backend.md)).
That means a full end-to-end test — install *and* `bot-bottle start` —
needs virtualization to work wherever bot-bottle runs. Inside a macOS guest
VM that requires **nested virtualization, which Apple gates to M3 or newer
chips on macOS 15+**. On M1/M2 you cannot run the Apple Container backend
(or Docker Desktop, same reason) inside a macOS VM at all.
The recommendation splits on what you're testing and what silicon you have:
- **Install-script correctness only** (does `curl | sh` → pipx → config dir
→ `doctor`'s Python/config checks pass?): a **disposable Tart VM** is the
cleanest boundary and works on any Apple Silicon Mac. `doctor` will report
the backend as not-ready inside the VM on M1/M2, which is fine — you're
testing the installer, not the runtime.
- **Full runtime** (actually launch a bottle) on **M3/M4**: a **disposable
Tart VM from a golden base image, cloned per run** is the gold standard —
a genuine kernel/state boundary that wipes to nothing.
- **Full runtime** on **M1/M2**, or when you'd rather not fight nested virt:
a **throwaway admin user via `sysadminctl`** is the pragmatic pick. It
tests the real backend because the backend runs on the host hypervisor —
but it is a *hygiene* boundary, not a security one, and it does **not**
clean the system-level footprint (see below).
Prefer the VM. Reach for the throwaway user only when nested virt is off the
table and you accept an imperfect wipe.
## Why "a boundary without a separate user" doesn't really exist on macOS
macOS has no namespace/overlay story like Linux `unshare` + tmpfs. The
options that sound like in-place sandboxes don't fit:
| Mechanism | Why it doesn't give you a clean, wipeable env |
|---|---|
| `sandbox-exec` / Seatbelt | Officially deprecated; confines *one process's* syscalls against a profile. It cannot present a fresh `$HOME` or a pristine `/usr/local`, and it won't let the Apple Container system service work. |
| App Sandbox | Entitlement-based confinement for signed `.app` bundles, not a provisioning tool for a CLI dev environment. |
| A second `$HOME` via `HOME=/tmp/foo` | Redirects only what honors `$HOME`. `install.sh` mostly does (it writes `~/.bot-bottle` and pipx/pip `--user` paths), but the Apple `container` install lands in `/usr/local` + a **system service**, and Homebrew lands in `/opt/homebrew` — all outside any `$HOME` you set. You'd get a false sense of "clean." |
| APFS snapshot rollback (`tmutil localsnapshot`) | You can't roll the live boot volume back to a local snapshot without booting to Recovery; it's not a per-run userspace undo. |
So the honest answer to "is there some boundary that avoids a separate
user?": yes — a **VM** — and it's the *stronger* boundary anyway. The only
lighter-weight option is the separate user, with the caveats below.
## What a clean install actually touches (the footprint that decides "wipeable")
Grounding the teardown story in what `install.sh` and the backend create:
| Artifact | Location | In `$HOME`? | Survives user deletion? |
|---|---|---|---|
| Config / state / db | `~/.bot-bottle/{agents,bottles,contrib,state,db}` ([`install.sh:80-83`](../install.sh), [`bot_bottle/paths.py:59`](../bot_bottle/paths.py)) | ✅ | ❌ removed with home |
| pipx venv + shim | `~/.local/pipx/venvs/bot-bottle`, shim in `~/.local/bin` ([`install.sh:87-89`](../install.sh)) | ✅ | ❌ removed with home |
| pip `--user` fallback | `~/Library/Python/<X.Y>/{lib,bin}` ([`install.sh:100-104`](../install.sh)) | ✅ | ❌ removed with home |
| PATH / token exports | shell profile (`~/.zprofile`, etc.); `BOT_BOTTLE_CLAUDE_OAUTH_TOKEN` ([`README.md:74`](../README.md)) | ✅ | ❌ removed with home |
| **Apple `container` install** | `/usr/local/...` + notarized `.pkg` receipts | ❌ | ✅ **stays** |
| **Apple `container` system service** | launchd system service (`container system start`) | ❌ | ✅ **stays** |
| **Homebrew** (if used for `container`/python) | `/opt/homebrew` | ❌ | ✅ **stays** |
| Rosetta 2 (needed for image builds) | system | ❌ | ✅ **stays** |
The three bold rows are the crux: **deleting the throwaway user does not
uninstall the Apple Container runtime, its system service, Homebrew, or
Rosetta.** A VM, by contrast, wipes 100% of the above by definition —
that's its entire advantage for this task.
## Option A — Disposable Tart VM (recommended)
[Tart](https://tart.run) is a CLI-first macOS/Linux VM manager built on
`Virtualization.framework`, purpose-built for exactly this "does it work on
a clean macOS, without my settings/permissions/data" workflow. Keep one
pristine *golden* image, clone a throwaway per run, delete it after.
```sh
brew install cirruslabs/cli/tart
# One-time: build a golden base (either a prebuilt image or a vanilla IPSW).
tart clone ghcr.io/cirruslabs/macos-tahoe-base:latest golden # ~25 GB pull
# — or a truly vanilla install you click through once —
# tart create golden --from-ipsw latest --disk-size 60
# Per test run: clone → boot → test → destroy.
tart clone golden test-run
tart run test-run &
ssh admin@"$(tart ip test-run)"
# inside the guest:
# curl -fsSL https://gitea.dideric.is/didericis/bot-bottle/raw/branch/main/install.sh | sh
# bot-bottle doctor
tart stop test-run
tart delete test-run # back to pristine; golden is untouched
```
Cloning is cheap (sparse files), so the golden image is your reset button —
every `tart clone` is a fresh macOS. This is the closest thing to a Linux
`docker run --rm` for a whole Mac.
**The nested-virt caveat (read before relying on it for runtime tests).**
The Apple Container backend inside the guest needs
`Virtualization.framework` to work *inside* the VM. Apple enables nested
virtualization only on **M3 or newer**, on **macOS 15 (Sequoia) or later**;
M2 and earlier are excluded by Apple, confirmed by Apple DTS. Consequences:
- **M3/M4 host:** full runtime works in the guest. `bot-bottle doctor`
reports the backend ready and `start` can launch a bottle. Gold standard.
- **M1/M2 host:** the guest can install bot-bottle and pass the Python /
config-dir checks, but `doctor`'s backend check will fail and you cannot
launch a bottle in the VM. Still perfectly good for testing *the
installer*; not for the runtime.
- **M4-specific:** a known bug blocks pre-Ventura guests on M4; use a
current macOS guest (which you want anyway, since Apple `container`
targets macOS 26 Tahoe).
UTM is the GUI equivalent on the same framework (and was first to expose
nested virt) if you'd rather click; Tart wins for a scriptable
spin-up/tear-down loop.
## Option B — Throwaway user via `sysadminctl` (pragmatic fallback)
Creating and deleting a user from the CLI is genuinely a two-liner, and it
tests the **real** backend on any Apple Silicon Mac because the backend runs
on the host hypervisor — no nested virt needed.
```sh
# Create a self-contained admin user (admin needed for the container service).
sudo sysadminctl -addUser bbtest -fullName "bot-bottle test" \
-password 'throwaway' -admin
# Log into that account (fast-user-switch or the login window), then run the
# installer as bbtest exactly as a new user would. When done:
sudo sysadminctl -deleteUser bbtest -secure # -secure erases the home dir
```
Honest accounting of what this does and doesn't buy you:
- **Boundary strength:** it's a *hygiene / fresh-`$HOME`* boundary, **not a
security boundary.** Same kernel, same admin group; an admin test user can
touch system state. If the point is "clean environment," fine. If the point
is "contain something untrusted," this is the wrong tool — use a VM.
- **Wipe completeness:** `-secure` erases the home dir (so `~/.bot-bottle`,
the pipx venv, and profile exports go away), but as the footprint table
shows, the **Apple Container runtime, its launchd system service,
Homebrew, and Rosetta persist.** For a truly repeatable "did a *system with
nothing installed* work?" test, that residue defeats the purpose — the
second run isn't clean.
- **Operational gotchas:** don't pass real passwords on the command line (they
land in `ps` and history — this is a throwaway credential, so it's
tolerable here). Deletion must run as root from a normally-booted, admin-
logged-in session; the Terminal needs **Full Disk Access** or you'll hit
error `-14120` and a half-deleted account. Prefer letting the system place
the home dir (don't pass `-home`), or deletion can orphan it.
Use this when you're on M1/M2, you specifically want to exercise the live
backend, and you can tolerate the system-level runtime staying installed
between runs (or you uninstall Apple `container` / brew by hand to reset).
## Honorable mentions
- **External bootable macOS volume.** A fresh macOS on an external SSD (or a
separate APFS volume) is bare-metal disposable: no nested-virt limit, real
backend works, and you `diskutil` the volume away to reset. Cost is reboot
friction per run — good for an occasional thorough pass, poor for a tight
loop.
- **Rented / cloud Mac.** AWS EC2 Mac (dedicated Mac minis), Scaleway Apple
silicon, or MacStadium give a genuinely throwaway host you release when
done. Overkill for local iteration, but this is essentially what the
project's own advisory `integration-macos` CI job needs — a self-hosted
Apple Silicon runner with the `container` CLI, Python ≥ 3.11, and coverage
on the launchd service's PATH ([`README.md:78`](../README.md)). If you end
up standing up a cloud Mac for install testing, it doubles as that runner.
## Recommendation
Default to a **disposable Tart VM** — it's the only option that wipes the
*entire* footprint (including the Apple Container system service that a user
deletion leaves behind), it's a real boundary, and the spin-up/tear-down
loop is a two-command `tart clone` / `tart delete`. Confirm your chip first:
on **M3/M4** it tests install *and* runtime end-to-end; on **M1/M2** it still
cleanly tests `install.sh` + `doctor`'s Python/config path, and you fall back
to a **throwaway `sysadminctl` admin user** for live-backend testing —
accepting that it's a hygiene boundary and that you'll manually uninstall the
Apple Container runtime / Homebrew between runs to get back to truly clean.
There is no third, lighter-weight "in-place boundary without a user" that
actually delivers a clean, wipeable macOS — the VM *is* that answer, and it's
the better one.
## Harness
The throwaway-user loop is scripted in
[`scripts/macos-install-test.sh`](../../scripts/macos-install-test.sh):
`up` creates the account, `run` pipes *this checkout's* `install.sh` into it
headlessly (so a PR is verifiable before it lands) and lets the installer run
`doctor`, `down` deletes the account and its home (the full reset), and
`deep-reset` additionally uninstalls the host `container` runtime. It leans on
the footprint analysis above — the reset is just user deletion because
everything `install.sh` writes is user-home-local.
## Sources
- [Apple Containers on macOS: technical comparison with Docker — The New Stack](https://thenewstack.io/apple-containers-on-macos-a-technical-comparison-with-docker/)
- [How to Set Up Apple Containerization on macOS 26 — Stéphane Paquet](https://spaquet.medium.com/how-to-set-up-apple-containerization-on-macos-26-f870cc8c26cd)
- [Install Apple Container CLI (macOS 15/26) — 4sysops](https://4sysops.com/archives/install-apple-container-cli-running-containers-natively-on-macos-15-sequoia-and-macos-26-tahoe/)
- [Nested virtualization on Apple Silicon (M3+, macOS 15) — UTM issue #6700](https://github.com/utmapp/UTM/issues/6700)
- [macOS 15 Sequoia nested virtualization for M3+ — Parallels Forums](https://forum.parallels.com/threads/macos-15-sequoia-nested-virtualization-for-m3-macs.364397/)
- [M2 nested virtualization restriction (Apple DTS) — Apple Developer Forums](https://developer.apple.com/forums/thread/756723)
- [M4 can't virtualize older macOS — Yahoo/Tech](https://tech.yahoo.com/computing/articles/m4-mac-computers-cant-virtualize-175122301.html)
- [Tart — macOS/Linux VMs on Apple Silicon (Cirrus Labs)](https://tart.run/quick-start/)
- [Tart GitHub](https://github.com/cirruslabs/tart)
- [macOS VMs in a single command — frr.dev](https://www.frr.dev/posts/tart-macos-vms-from-terminal/)
- [sysadminctl reference — SS64](https://ss64.com/mac/sysadminctl.html)
- [User management from the macOS command line — macnotes](https://macnotes.wordpress.com/2019/03/28/user-management-create-remove-change-password-secure-token-from-macos-command-line/)
scripts/macos-install-test.sh
Executable
+195
View File
@@ -0,0 +1,195 @@
#!/usr/bin/env bash
# Clean-install test harness for the macOS (Apple `container`) path.
#
# Exercises install.sh the way a brand-new user would, inside a throwaway
# macOS account you create and delete from the CLI. install.sh's entire
# footprint is user-home-local — the pipx venv under ~/.local (or the pip
# --user tree under ~/Library/Python), the ~/.bot-bottle config dir, and a
# PATH line in the login shell. It never installs the backend itself (see
# the header of install.sh), so deleting the user is a complete,
# deterministic reset of everything the installer touched. The Apple
# `container` runtime is a HOST prerequisite installed once and kept;
# `deep-reset` is the rare escape hatch that also removes it.
#
# Why a throwaway user and not a disposable VM: bot-bottle's default macOS
# backend is Apple `container`, which runs each container in its own
# Virtualization.framework microVM. Running that backend inside a macOS
# guest VM needs nested virtualization, which Apple gates to M3+ silicon.
# On M1/M2 a separate user account is the only way to get a clean $HOME
# while still reaching the real host backend. Full rationale in
# docs/research/testing-clean-install-on-macos.md.
#
# Usage:
# sudo ./scripts/macos-install-test.sh up # create the throwaway user
# sudo ./scripts/macos-install-test.sh run # run install.sh (+doctor) as it
# ./scripts/macos-install-test.sh status # user present? backend ready?
# sudo ./scripts/macos-install-test.sh down # delete user + home (the reset)
# sudo ./scripts/macos-install-test.sh deep-reset # ALSO uninstall host `container`
#
# A full clean cycle is: sudo ... up && sudo ... run && sudo ... down
#
# Config via env:
# BB_TEST_USER account short name (default: bbtest)
# BB_TEST_FULLNAME account full name (default: "bot-bottle install test")
# BB_TEST_ADMIN 1=admin (reach container svc), 0=standard (default: 1)
# BB_TEST_INSTALL_URL curl this install.sh instead of piping the local checkout
# BOT_BOTTLE_INSTALL_SPEC passed through to install.sh (pip / git spec)
#
# Notes:
# * Run from a normally-booted admin session. Grant Terminal *Full Disk
# Access* (System Settings -> Privacy & Security) or `down` half-fails
# with error -14120 and leaves an orphaned account.
# * `sysadminctl` always exits 0 even on failure, so `up`/`down` verify
# the result with `dscl` and fail loudly on a mismatch.
# * The account is created without a password: `run` drives it headlessly
# via `sudo -u`, which never needs the target's password. The account
# cannot GUI-login, which this harness does not require.
# * `run` covers the installer + `bot-bottle doctor`. Actually launching a
# bottle from the throwaway user may need a full launchd user session
# (`launchctl asuser`); on M1/M2 the backend can't run under nested virt
# anyway, so this harness stops at install + doctor.
set -euo pipefail
USER_NAME="${BB_TEST_USER:-bbtest}"
FULL_NAME="${BB_TEST_FULLNAME:-bot-bottle install test}"
ADMIN="${BB_TEST_ADMIN:-1}"
_SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
_REPO_ROOT="$(cd "$_SCRIPT_DIR/.." && pwd)"
# --- guards ----------------------------------------------------------
require_macos() {
[ "$(uname -s)" = "Darwin" ] \
|| { echo "error: this harness is macOS-only (uname is $(uname -s))" >&2; exit 1; }
}
require_root() {
if [ "$(id -u)" -ne 0 ]; then
echo "error: '$1' needs root; re-run under sudo" >&2
exit 1
fi
}
user_exists() { dscl . -read "/Users/$USER_NAME" >/dev/null 2>&1; }
# Run a shell snippet as the throwaway user in a fresh login shell.
run_as_user() { sudo -u "$USER_NAME" -i sh -c "$1"; }
# --- commands --------------------------------------------------------
cmd_up() {
require_macos
require_root up
if user_exists; then
echo "$USER_NAME already exists; nothing to do (run 'down' first to reset)"
return 0
fi
local admin_flag=()
[ "$ADMIN" = "1" ] && admin_flag=(-admin)
# No -password: the account is only ever driven headlessly via `sudo -u`,
# which doesn't need one. sysadminctl warns about FileVault here; that's
# irrelevant to a headless test account.
sysadminctl -addUser "$USER_NAME" -fullName "$FULL_NAME" "${admin_flag[@]}" || true
# sysadminctl exits 0 regardless of outcome, so confirm the account landed.
user_exists || { echo "error: failed to create $USER_NAME" >&2; exit 1; }
echo "created $USER_NAME (admin=$ADMIN). Install into it with: sudo $0 run"
}
cmd_run() {
require_macos
require_root run
user_exists || { echo "error: $USER_NAME does not exist; run 'sudo $0 up' first" >&2; exit 1; }
local spec_env=""
[ -n "${BOT_BOTTLE_INSTALL_SPEC:-}" ] \
&& spec_env="BOT_BOTTLE_INSTALL_SPEC='$BOT_BOTTLE_INSTALL_SPEC' "
echo "== installing bot-bottle as $USER_NAME =="
if [ -n "${BB_TEST_INSTALL_URL:-}" ]; then
run_as_user "curl -fsSL '$BB_TEST_INSTALL_URL' | ${spec_env}sh"
else
# Test THIS checkout's install.sh, not the published one, so a PR is
# verifiable before it lands. Stage it world-readable in /tmp because
# the throwaway user can't read the tester's (mode 700) home.
local staged
staged="$(mktemp /tmp/bb-install.XXXXXX.sh)"
cp "$_REPO_ROOT/install.sh" "$staged"
chmod 644 "$staged"
run_as_user "${spec_env}sh '$staged'" || { rm -f "$staged"; exit 1; }
rm -f "$staged"
fi
echo "== install.sh runs 'doctor' itself; re-check anytime with: $0 status =="
}
cmd_status() {
require_macos
if user_exists; then
echo "user: $USER_NAME present"
if [ "$(id -u)" -eq 0 ]; then
echo "doctor (as $USER_NAME):"
run_as_user 'command -v bot-bottle >/dev/null 2>&1 \
&& bot-bottle doctor \
|| echo " bot-bottle not installed for this user yet"' || true
else
echo " (re-run under sudo to run 'bot-bottle doctor' as $USER_NAME)"
fi
else
echo "user: $USER_NAME absent"
fi
if command -v container >/dev/null 2>&1; then
echo "backend: apple 'container' present ($(container --version 2>/dev/null | head -1))"
else
echo "backend: apple 'container' NOT on PATH (host prerequisite; install once)"
fi
}
cmd_down() {
require_macos
require_root down
if ! user_exists; then
echo "$USER_NAME not present; nothing to remove"
return 0
fi
# A plain -deleteUser removes the home dir, which is the whole reset.
# -secure is a no-op on modern macOS (secure erase of the home folder
# was removed in Sierra), so it buys nothing here.
sysadminctl -deleteUser "$USER_NAME" || true
if user_exists; then
echo "error: $USER_NAME still present after delete." >&2
echo " - grant Terminal Full Disk Access (System Settings > Privacy & Security), or" >&2
echo " - it may hold the last Secure Token (won't happen while another admin exists)" >&2
exit 1
fi
echo "removed $USER_NAME and its home — install surface is clean."
}
cmd_deep_reset() {
require_macos
require_root deep-reset
# Remove the user first (idempotent), then the HOST-level container
# runtime that a user deletion leaves behind under /usr/local + launchd.
cmd_down || true
if command -v container >/dev/null 2>&1; then
# The service can run in more than one launchd context (the invoking
# user's and root's), so stop both, best-effort.
[ -n "${SUDO_USER:-}" ] && sudo -u "$SUDO_USER" container system stop 2>/dev/null || true
container system stop 2>/dev/null || true
if [ -x /usr/local/bin/uninstall-container.sh ]; then
/usr/local/bin/uninstall-container.sh -d || true
echo "uninstalled the host Apple 'container' runtime"
else
echo "note: /usr/local/bin/uninstall-container.sh not found; runtime left as-is" >&2
fi
else
echo "no 'container' runtime on PATH; nothing further to remove"
fi
}
case "${1:-}" in
up) cmd_up ;;
run) cmd_run ;;
status) cmd_status ;;
down) cmd_down ;;
deep-reset) cmd_deep_reset ;;
*) echo "usage: $0 {up|run|status|down|deep-reset}" >&2 ; exit 2 ;;
esac