bot-bottle/docs/research/testing-clean-install-on-macos.md

Testing a clean bot-bottle install on macOS

How do you exercise install.sh (and, ideally, a first bot-bottle start) the way a brand-new user would — on a pristine macOS environment you can throw away afterward — without permanently polluting your daily-driver Mac? The user's framing: is there a VM or boundary that avoids creating a separate account, or is spinning up and tearing down a throwaway macOS user on the CLI easy enough to just do that?

Summary

There is no lightweight, in-place macOS sandbox that hands you a clean home directory and wipeable system state without either a VM or a separate user account. sandbox-exec (Seatbelt) is deprecated and confines a process, not an environment; App Sandbox is for shipping apps, not for provisioning a fresh dev host. So the real choice is exactly the two the user named: a disposable macOS VM or a throwaway user account — and which one is right turns on a detail specific to this project.

bot-bottle's default macOS backend is Apple's container, which runs each container in its own lightweight VM via Virtualization.framework (README.md:27, apple-container-backend.md). That means a full end-to-end test — install and bot-bottle start — needs virtualization to work wherever bot-bottle runs. Inside a macOS guest VM that requires nested virtualization, which Apple gates to M3 or newer chips on macOS 15+. On M1/M2 you cannot run the Apple Container backend (or Docker Desktop, same reason) inside a macOS VM at all.

The recommendation splits on what you're testing and what silicon you have:

• Install-script correctness only (does curl | sh → pipx → config dir → doctor's Python/config checks pass?): a disposable Tart VM is the cleanest boundary and works on any Apple Silicon Mac. doctor will report the backend as not-ready inside the VM on M1/M2, which is fine — you're testing the installer, not the runtime.
• Full runtime (actually launch a bottle) on M3/M4: a disposable Tart VM from a golden base image, cloned per run is the gold standard — a genuine kernel/state boundary that wipes to nothing.
• Full runtime on M1/M2, or when you'd rather not fight nested virt: a throwaway admin user via sysadminctl is the pragmatic pick. It tests the real backend because the backend runs on the host hypervisor — but it is a hygiene boundary, not a security one, and it does not clean the system-level footprint (see below).

Prefer the VM. Reach for the throwaway user only when nested virt is off the table and you accept an imperfect wipe.

Why "a boundary without a separate user" doesn't really exist on macOS

macOS has no namespace/overlay story like Linux unshare + tmpfs. The options that sound like in-place sandboxes don't fit:

Mechanism	Why it doesn't give you a clean, wipeable env
sandbox-exec / Seatbelt	Officially deprecated; confines one process's syscalls against a profile. It cannot present a fresh $HOME or a pristine /usr/local, and it won't let the Apple Container system service work.
App Sandbox	Entitlement-based confinement for signed .app bundles, not a provisioning tool for a CLI dev environment.
A second $HOME via HOME=/tmp/foo	Redirects only what honors $HOME. install.sh mostly does (it writes ~/.bot-bottle and pipx/pip --user paths), but the Apple container install lands in /usr/local + a system service, and Homebrew lands in /opt/homebrew — all outside any $HOME you set. You'd get a false sense of "clean."
APFS snapshot rollback (tmutil localsnapshot)	You can't roll the live boot volume back to a local snapshot without booting to Recovery; it's not a per-run userspace undo.

So the honest answer to "is there some boundary that avoids a separate user?": yes — a VM — and it's the stronger boundary anyway. The only lighter-weight option is the separate user, with the caveats below.

What a clean install actually touches (the footprint that decides "wipeable")

Grounding the teardown story in what install.sh and the backend create:

Artifact	Location	In $HOME?	Survives user deletion?
Config / state / db	~/.bot-bottle/{agents,bottles,contrib,state,db} (install.sh:80-83, bot_bottle/paths.py:59)	✅	❌ removed with home
pipx venv + shim	~/.local/pipx/venvs/bot-bottle, shim in ~/.local/bin (install.sh:87-89)	✅	❌ removed with home
private venv fallback (no pipx)	~/.bot-bottle/venv + symlink in ~/.local/bin (install.sh)	✅	❌ removed with home
PATH / token exports	shell profile (~/.zprofile, etc.); BOT_BOTTLE_CLAUDE_OAUTH_TOKEN (README.md:74)	✅	❌ removed with home
Apple container install	/usr/local/... + notarized .pkg receipts	❌	✅ stays
Apple container service state	per-user: ~/Library/Application Support/com.apple.container/ (container system start)	✅	❌ removed with home
Homebrew (if used for container/python)	/opt/homebrew	❌	✅ stays
Rosetta 2 (needed for image builds)	system	❌	✅ stays

The bold rows are the crux: deleting the throwaway user does not uninstall the Apple Container runtime, Homebrew, or Rosetta. A VM, by contrast, wipes 100% of the above by definition — that's its entire advantage for this task.

The service row is the exception, and a live harness run corrected it: the Apple container service is per-user, not a host-wide launchd service. container system status reports an appRoot under ~/Library/Application Support/com.apple.container/, and a freshly created account sees container system service: NOT running even while the creating admin's is running. That cuts both ways — the state is genuinely removed with the home, so the reset is more complete than this table first claimed, but it also means no brand-new user can run a bottle until they run container system start once. doctor correctly fails until they do, which is why test judges backend readiness separately from install correctness.

Option A — Disposable Tart VM (recommended)

Tart is a CLI-first macOS/Linux VM manager built on Virtualization.framework, purpose-built for exactly this "does it work on a clean macOS, without my settings/permissions/data" workflow. Keep one pristine golden image, clone a throwaway per run, delete it after.

brew install cirruslabs/cli/tart

# One-time: build a golden base (either a prebuilt image or a vanilla IPSW).
tart clone ghcr.io/cirruslabs/macos-tahoe-base:latest golden   # ~25 GB pull
# — or a truly vanilla install you click through once —
# tart create golden --from-ipsw latest --disk-size 60

# Per test run: clone → boot → test → destroy.
tart clone golden test-run
tart run test-run &
ssh admin@"$(tart ip test-run)"
#   inside the guest:
#   curl -fsSL https://gitea.dideric.is/didericis/bot-bottle/raw/branch/main/install.sh | sh
#   bot-bottle doctor
tart stop test-run
tart delete test-run          # back to pristine; golden is untouched

Cloning is cheap (sparse files), so the golden image is your reset button — every tart clone is a fresh macOS. This is the closest thing to a Linux docker run --rm for a whole Mac.

The nested-virt caveat (read before relying on it for runtime tests). The Apple Container backend inside the guest needs Virtualization.framework to work inside the VM. Apple enables nested virtualization only on M3 or newer, on macOS 15 (Sequoia) or later; M2 and earlier are excluded by Apple, confirmed by Apple DTS. Consequences:

• M3/M4 host: full runtime works in the guest. bot-bottle doctor reports the backend ready and start can launch a bottle. Gold standard.
• M1/M2 host: the guest can install bot-bottle and pass the Python / config-dir checks, but doctor's backend check will fail and you cannot launch a bottle in the VM. Still perfectly good for testing the installer; not for the runtime.
• M4-specific: a known bug blocks pre-Ventura guests on M4; use a current macOS guest (which you want anyway, since Apple container targets macOS 26 Tahoe).

UTM is the GUI equivalent on the same framework (and was first to expose nested virt) if you'd rather click; Tart wins for a scriptable spin-up/tear-down loop.

Option B — Throwaway user via sysadminctl (pragmatic fallback)

Creating and deleting a user from the CLI is genuinely a two-liner, and it tests the real backend on any Apple Silicon Mac because the backend runs on the host hypervisor — no nested virt needed.

# Create a self-contained admin user (admin needed for the container service).
sudo sysadminctl -addUser bbtest -fullName "bot-bottle test" \
    -password 'throwaway' -admin

# Log into that account (fast-user-switch or the login window), then run the
# installer as bbtest exactly as a new user would. When done:

sudo sysadminctl -deleteUser bbtest -secure   # -secure erases the home dir

Honest accounting of what this does and doesn't buy you:

• Boundary strength: it's a hygiene / fresh-$HOME boundary, not a security boundary. Same kernel, same admin group; an admin test user can touch system state. If the point is "clean environment," fine. If the point is "contain something untrusted," this is the wrong tool — use a VM.
• Wipe completeness: -secure erases the home dir (so ~/.bot-bottle, the pipx venv, and profile exports go away), but as the footprint table shows, the Apple Container runtime, its launchd system service, Homebrew, and Rosetta persist. For a truly repeatable "did a system with nothing installed work?" test, that residue defeats the purpose — the second run isn't clean.
• Operational gotchas: don't pass real passwords on the command line (they land in ps and history — this is a throwaway credential, so it's tolerable here). Deletion must run as root from a normally-booted, admin- logged-in session; the Terminal needs Full Disk Access or you'll hit error -14120 and a half-deleted account. Prefer letting the system place the home dir (don't pass -home), or deletion can orphan it.

Use this when you're on M1/M2, you specifically want to exercise the live backend, and you can tolerate the system-level runtime staying installed between runs (or you uninstall Apple container / brew by hand to reset).

Honorable mentions

• External bootable macOS volume. A fresh macOS on an external SSD (or a separate APFS volume) is bare-metal disposable: no nested-virt limit, real backend works, and you diskutil the volume away to reset. Cost is reboot friction per run — good for an occasional thorough pass, poor for a tight loop.
• Rented / cloud Mac. AWS EC2 Mac (dedicated Mac minis), Scaleway Apple silicon, or MacStadium give a genuinely throwaway host you release when done. Overkill for local iteration, but this is essentially what the project's own advisory integration-macos CI job needs — a self-hosted Apple Silicon runner with the container CLI, Python ≥ 3.11, and coverage on the launchd service's PATH (README.md:78). If you end up standing up a cloud Mac for install testing, it doubles as that runner.

Recommendation

Default to a disposable Tart VM — it's the only option that wipes the entire footprint (including the Apple Container system service that a user deletion leaves behind), it's a real boundary, and the spin-up/tear-down loop is a two-command tart clone / tart delete. Confirm your chip first: on M3/M4 it tests install and runtime end-to-end; on M1/M2 it still cleanly tests install.sh + doctor's Python/config path, and you fall back to a throwaway sysadminctl admin user for live-backend testing — accepting that it's a hygiene boundary and that you'll manually uninstall the Apple Container runtime / Homebrew between runs to get back to truly clean.

There is no third, lighter-weight "in-place boundary without a user" that actually delivers a clean, wipeable macOS — the VM is that answer, and it's the better one.

Harness

The throwaway-user loop is scripted in scripts/macos-install-test.sh: up creates the account, run pipes this checkout's install.sh into it headlessly (so a PR is verifiable before it lands) and lets the installer run doctor, down deletes the account and its home (the full reset), and deep-reset additionally uninstalls the host container runtime. It leans on the footprint analysis above — the reset is just user deletion because everything install.sh writes is user-home-local.

There are two one-shot cycles, because "does the installer work" and "can a new user actually run a bottle" are different questions:

sudo ./scripts/macos-install-test.sh test        # up → run → status → down
sudo ./scripts/macos-install-test.sh test-ready  # ... + prereqs before status

test models a macOS system without the prerequisites set up for this user — which is the default state of every new account, since the container service is per-user. It asserts the install is sound and reports backend readiness without failing on it, because install.sh provides no backend and so cannot regress one.

test-ready models a system with them, then demands doctor go fully green, backend included. Both variants pass as of this writing.

The per-user prerequisite is two steps, not one

Running it revealed that "set up the backend for this account" is more than starting a service:

1. container system start — the service is per-user. The run confirms it directly: the throwaway account's appRoot is /Users/bbtest/Library/Application Support/com.apple.container/, its own, and starting it left the admin's service untouched.
2. A guest kernel, which also lives in that per-user app root. A fresh account has none, so container system start prompts to download one — and only prompts, since the flags default to asking. Headless callers must pass --enable-kernel-install or the command dies on failed to read user input.

Neither step is done by bot-bottle backend setup --backend=macos-container, which only checks and then tells you to run container system start yourself. So a new account's real path to a working backend is: container system start --enable-kernel-install.

The harness must also enter the user's launchd domain via launchctl asuser <uid> to do any of this. container system start registers com.apple.container.apiserver as a per-user launchd agent and talks to it over XPC; from plain sudo -u the caller is still in root's bootstrap namespace, the lookup crosses domains, and the apiserver answers invalidState: "unauthorized request" even though the agent started fine.

It refuses to start against an existing account (a reused home is not a clean install), and it tears the account down from an EXIT/INT trap armed the moment the account exists, so a failed or Ctrl-C'd run still leaves the machine clean. Its verdict is deliberately stricter than the installer's own: note that install.sh exits 0 when it finishes but doctor reports unmet prerequisites, so "the installer succeeded" is not the assertion — test fails if the install fails, if bot-bottle never reached the new user's PATH, or if doctor is unhappy. BB_TEST_KEEP=1 skips the teardown to poke at a failure.

What a fresh account actually inherits

Expect the first honest run on a developer Mac to fail at the Python gate, and expect that to be correct. A new account's PATH is just /etc/paths (/usr/local/bin:/System/Cryptexes/App/usr/bin:/usr/bin:/bin:/usr/sbin:/sbin), which notably does not include /opt/homebrew/bin. Homebrew's shellenv line lives in the installing user's ~/.zprofile and is not inherited, so a throwaway user resolves python3 to /usr/bin/python3 — the Command Line Tools stub, still 3.9.6 on macOS 26 — and install.sh correctly dies on its 3.11+ requirement. Your own shell resolving python3 to a 3.14 Homebrew build says nothing about what a new user sees; that gap is exactly what this harness exists to expose.

Sources

• Apple Containers on macOS: technical comparison with Docker — The New Stack
• How to Set Up Apple Containerization on macOS 26 — Stéphane Paquet
• Install Apple Container CLI (macOS 15/26) — 4sysops
• Nested virtualization on Apple Silicon (M3+, macOS 15) — UTM issue #6700
• macOS 15 Sequoia nested virtualization for M3+ — Parallels Forums
• M2 nested virtualization restriction (Apple DTS) — Apple Developer Forums
• M4 can't virtualize older macOS — Yahoo/Tech
• Tart — macOS/Linux VMs on Apple Silicon (Cirrus Labs)
• Tart GitHub
• macOS VMs in a single command — frr.dev
• sysadminctl reference — SS64
• User management from the macOS command line — macnotes