bot-bottle/docs/research/testing-clean-install-on-macos.md

# Testing a clean bot-bottle install on macOS

How do you exercise `install.sh` (and, ideally, a first `bot-bottle start`)
the way a brand-new user would — on a pristine macOS environment you can
throw away afterward — *without* permanently polluting your daily-driver
Mac? The user's framing: is there a VM or boundary that avoids creating a
separate account, or is spinning up and tearing down a throwaway macOS
user on the CLI easy enough to just do that?

## Summary

There is no lightweight, in-place macOS sandbox that hands you a clean home
directory and wipeable system state without *either* a VM or a separate
user account. `sandbox-exec` (Seatbelt) is deprecated and confines a
process, not an environment; App Sandbox is for shipping apps, not for
provisioning a fresh dev host. So the real choice is exactly the two the
user named: **a disposable macOS VM** or **a throwaway user account** —
and which one is right turns on a detail specific to *this* project.

bot-bottle's default macOS backend is Apple's `container`, which runs each
container in its own lightweight VM via `Virtualization.framework`
([`README.md:27`](../README.md), [`apple-container-backend.md`](apple-container-backend.md)).
That means a full end-to-end test — install *and* `bot-bottle start` —
needs virtualization to work wherever bot-bottle runs. Inside a macOS guest
VM that requires **nested virtualization, which Apple gates to M3 or newer
chips on macOS 15+**. On M1/M2 you cannot run the Apple Container backend
(or Docker Desktop, same reason) inside a macOS VM at all.

The recommendation splits on what you're testing and what silicon you have:

- **Install-script correctness only** (does `curl | sh` → pipx → config dir
  → `doctor`'s Python/config checks pass?): a **disposable Tart VM** is the
  cleanest boundary and works on any Apple Silicon Mac. `doctor` will report
  the backend as not-ready inside the VM on M1/M2, which is fine — you're
  testing the installer, not the runtime.
- **Full runtime** (actually launch a bottle) on **M3/M4**: a **disposable
  Tart VM from a golden base image, cloned per run** is the gold standard —
  a genuine kernel/state boundary that wipes to nothing.
- **Full runtime** on **M1/M2**, or when you'd rather not fight nested virt:
  a **throwaway admin user via `sysadminctl`** is the pragmatic pick. It
  tests the real backend because the backend runs on the host hypervisor —
  but it is a *hygiene* boundary, not a security one, and it does **not**
  clean the system-level footprint (see below).

Prefer the VM. Reach for the throwaway user only when nested virt is off the
table and you accept an imperfect wipe.

## Why "a boundary without a separate user" doesn't really exist on macOS

macOS has no namespace/overlay story like Linux `unshare` + tmpfs. The
options that sound like in-place sandboxes don't fit:

| Mechanism | Why it doesn't give you a clean, wipeable env |
|---|---|
| `sandbox-exec` / Seatbelt | Officially deprecated; confines *one process's* syscalls against a profile. It cannot present a fresh `$HOME` or a pristine `/usr/local`, and it won't let the Apple Container system service work. |
| App Sandbox | Entitlement-based confinement for signed `.app` bundles, not a provisioning tool for a CLI dev environment. |
| A second `$HOME` via `HOME=/tmp/foo` | Redirects only what honors `$HOME`. `install.sh` mostly does (it writes `~/.bot-bottle` and pipx/pip `--user` paths), but the Apple `container` install lands in `/usr/local` + a **system service**, and Homebrew lands in `/opt/homebrew` — all outside any `$HOME` you set. You'd get a false sense of "clean." |
| APFS snapshot rollback (`tmutil localsnapshot`) | You can't roll the live boot volume back to a local snapshot without booting to Recovery; it's not a per-run userspace undo. |

So the honest answer to "is there some boundary that avoids a separate
user?": yes — a **VM** — and it's the *stronger* boundary anyway. The only
lighter-weight option is the separate user, with the caveats below.

## What a clean install actually touches (the footprint that decides "wipeable")

Grounding the teardown story in what `install.sh` and the backend create:

| Artifact | Location | In `$HOME`? | Survives user deletion? |
|---|---|---|---|
| Config / state / db | `~/.bot-bottle/{agents,bottles,contrib,state,db}` ([`install.sh:80-83`](../install.sh), [`bot_bottle/paths.py:59`](../bot_bottle/paths.py)) | ✅ | ❌ removed with home |
| pipx venv + shim | `~/.local/pipx/venvs/bot-bottle`, shim in `~/.local/bin` ([`install.sh:87-89`](../install.sh)) | ✅ | ❌ removed with home |
| private venv fallback (no pipx) | `~/.bot-bottle/venv` + symlink in `~/.local/bin` ([`install.sh`](../install.sh)) | ✅ | ❌ removed with home |
| PATH / token exports | shell profile (`~/.zprofile`, etc.); `BOT_BOTTLE_CLAUDE_OAUTH_TOKEN` ([`README.md:74`](../README.md)) | ✅ | ❌ removed with home |
| **Apple `container` install** | `/usr/local/...` + notarized `.pkg` receipts | ❌ | ✅ **stays** |
| Apple `container` **service state** | per-user: `~/Library/Application Support/com.apple.container/` (`container system start`) | ✅ | ❌ removed with home |
| **Homebrew** (if used for `container`/python) | `/opt/homebrew` | ❌ | ✅ **stays** |
| Rosetta 2 (needed for image builds) | system | ❌ | ✅ **stays** |

The bold rows are the crux: **deleting the throwaway user does not uninstall
the Apple Container runtime, Homebrew, or Rosetta.** A VM, by contrast, wipes
100% of the above by definition — that's its entire advantage for this task.

The service row is the exception, and a live harness run corrected it: the
Apple `container` service is **per-user**, not a host-wide launchd service.
`container system status` reports an `appRoot` under
`~/Library/Application Support/com.apple.container/`, and a freshly created
account sees `container system service: NOT running` even while the creating
admin's is running. That cuts both ways — the state is genuinely removed with
the home, so the reset is *more* complete than this table first claimed, but
it also means **no brand-new user can run a bottle until they run
`container system start` once**. `doctor` correctly fails until they do, which
is why `test` judges backend readiness separately from install correctness.

## Option A — Disposable Tart VM (recommended)

[Tart](https://tart.run) is a CLI-first macOS/Linux VM manager built on
`Virtualization.framework`, purpose-built for exactly this "does it work on
a clean macOS, without my settings/permissions/data" workflow. Keep one
pristine *golden* image, clone a throwaway per run, delete it after.

```sh
brew install cirruslabs/cli/tart

# One-time: build a golden base (either a prebuilt image or a vanilla IPSW).
tart clone ghcr.io/cirruslabs/macos-tahoe-base:latest golden   # ~25 GB pull
# — or a truly vanilla install you click through once —
# tart create golden --from-ipsw latest --disk-size 60

# Per test run: clone → boot → test → destroy.
tart clone golden test-run
tart run test-run &
ssh admin@"$(tart ip test-run)"
#   inside the guest:
#   curl -fsSL https://gitea.dideric.is/didericis/bot-bottle/raw/branch/main/install.sh | sh
#   bot-bottle doctor
tart stop test-run
tart delete test-run          # back to pristine; golden is untouched
```

Cloning is cheap (sparse files), so the golden image is your reset button —
every `tart clone` is a fresh macOS. This is the closest thing to a Linux
`docker run --rm` for a whole Mac.

**The nested-virt caveat (read before relying on it for runtime tests).**
The Apple Container backend inside the guest needs
`Virtualization.framework` to work *inside* the VM. Apple enables nested
virtualization only on **M3 or newer**, on **macOS 15 (Sequoia) or later**;
M2 and earlier are excluded by Apple, confirmed by Apple DTS. Consequences:

- **M3/M4 host:** full runtime works in the guest. `bot-bottle doctor`
  reports the backend ready and `start` can launch a bottle. Gold standard.
- **M1/M2 host:** the guest can install bot-bottle and pass the Python /
  config-dir checks, but `doctor`'s backend check will fail and you cannot
  launch a bottle in the VM. Still perfectly good for testing *the
  installer*; not for the runtime.
- **M4-specific:** a known bug blocks pre-Ventura guests on M4; use a
  current macOS guest (which you want anyway, since Apple `container`
  targets macOS 26 Tahoe).

UTM is the GUI equivalent on the same framework (and was first to expose
nested virt) if you'd rather click; Tart wins for a scriptable
spin-up/tear-down loop.

## Option B — Throwaway user via `sysadminctl` (pragmatic fallback)

Creating and deleting a user from the CLI is genuinely a two-liner, and it
tests the **real** backend on any Apple Silicon Mac because the backend runs
on the host hypervisor — no nested virt needed.

```sh
# Create a self-contained admin user (admin needed for the container service).
sudo sysadminctl -addUser bbtest -fullName "bot-bottle test" \
    -password 'throwaway' -admin

# Log into that account (fast-user-switch or the login window), then run the
# installer as bbtest exactly as a new user would. When done:

sudo sysadminctl -deleteUser bbtest -secure   # -secure erases the home dir
```

Honest accounting of what this does and doesn't buy you:

- **Boundary strength:** it's a *hygiene / fresh-`$HOME`* boundary, **not a
  security boundary.** Same kernel, same admin group; an admin test user can
  touch system state. If the point is "clean environment," fine. If the point
  is "contain something untrusted," this is the wrong tool — use a VM.
- **Wipe completeness:** `-secure` erases the home dir (so `~/.bot-bottle`,
  the pipx venv, and profile exports go away), but as the footprint table
  shows, the **Apple Container runtime, its launchd system service,
  Homebrew, and Rosetta persist.** For a truly repeatable "did a *system with
  nothing installed* work?" test, that residue defeats the purpose — the
  second run isn't clean.
- **Operational gotchas:** don't pass real passwords on the command line (they
  land in `ps` and history — this is a throwaway credential, so it's
  tolerable here). Deletion must run as root from a normally-booted, admin-
  logged-in session; the Terminal needs **Full Disk Access** or you'll hit
  error `-14120` and a half-deleted account. Prefer letting the system place
  the home dir (don't pass `-home`), or deletion can orphan it.

Use this when you're on M1/M2, you specifically want to exercise the live
backend, and you can tolerate the system-level runtime staying installed
between runs (or you uninstall Apple `container` / brew by hand to reset).

## Honorable mentions

- **External bootable macOS volume.** A fresh macOS on an external SSD (or a
  separate APFS volume) is bare-metal disposable: no nested-virt limit, real
  backend works, and you `diskutil` the volume away to reset. Cost is reboot
  friction per run — good for an occasional thorough pass, poor for a tight
  loop.
- **Rented / cloud Mac.** AWS EC2 Mac (dedicated Mac minis), Scaleway Apple
  silicon, or MacStadium give a genuinely throwaway host you release when
  done. Overkill for local iteration, but this is essentially what the
  project's own advisory `integration-macos` CI job needs — a self-hosted
  Apple Silicon runner with the `container` CLI, Python ≥ 3.11, and coverage
  on the launchd service's PATH ([`README.md:78`](../README.md)). If you end
  up standing up a cloud Mac for install testing, it doubles as that runner.

## Recommendation

Default to a **disposable Tart VM** — it's the only option that wipes the
*entire* footprint (including the Apple Container system service that a user
deletion leaves behind), it's a real boundary, and the spin-up/tear-down
loop is a two-command `tart clone` / `tart delete`. Confirm your chip first:
on **M3/M4** it tests install *and* runtime end-to-end; on **M1/M2** it still
cleanly tests `install.sh` + `doctor`'s Python/config path, and you fall back
to a **throwaway `sysadminctl` admin user** for live-backend testing —
accepting that it's a hygiene boundary and that you'll manually uninstall the
Apple Container runtime / Homebrew between runs to get back to truly clean.

There is no third, lighter-weight "in-place boundary without a user" that
actually delivers a clean, wipeable macOS — the VM *is* that answer, and it's
the better one.

## Harness

The throwaway-user loop is scripted in
[`scripts/macos-install-test.sh`](../../scripts/macos-install-test.sh):
`up` creates the account, `run` pipes *this checkout's* `install.sh` into it
headlessly (so a PR is verifiable before it lands) and lets the installer run
`doctor`, `down` deletes the account and its home (the full reset), and
`deep-reset` additionally uninstalls the host `container` runtime. It leans on
the footprint analysis above — the reset is just user deletion because
everything `install.sh` writes is user-home-local.

There are two one-shot cycles, because "does the installer work" and "can a new
user actually run a bottle" are different questions:

```sh
sudo ./scripts/macos-install-test.sh test        # up → run → status → down
sudo ./scripts/macos-install-test.sh test-ready  # ... + prereqs before status
```

`test` models a macOS system **without** the prerequisites set up for this user
— which is the default state of every new account, since the `container`
service is per-user. It asserts the install is sound and *reports* backend
readiness without failing on it, because install.sh provides no backend and so
cannot regress one.

`test-ready` models a system **with** them, then demands doctor go fully green,
backend included. Both variants pass as of this writing.

### The per-user prerequisite is two steps, not one

Running it revealed that "set up the backend for this account" is more than
starting a service:

1. **`container system start`** — the service is per-user. The run confirms it
   directly: the throwaway account's `appRoot` is
   `/Users/bbtest/Library/Application Support/com.apple.container/`, its own,
   and starting it left the admin's service untouched.
2. **A guest kernel**, which also lives in that per-user app root. A fresh
   account has none, so `container system start` prompts to download one —
   and *only* prompts, since the flags default to asking. Headless callers
   must pass `--enable-kernel-install` or the command dies on
   `failed to read user input`.

Neither step is done by `bot-bottle backend setup --backend=macos-container`,
which only checks and then tells you to run `container system start` yourself.
So a new account's real path to a working backend is:
`container system start --enable-kernel-install`.

The harness must also enter the user's launchd domain via
`launchctl asuser <uid>` to do any of this. `container system start` registers
`com.apple.container.apiserver` as a per-user launchd agent and talks to it
over XPC; from plain `sudo -u` the caller is still in root's bootstrap
namespace, the lookup crosses domains, and the apiserver answers
`invalidState: "unauthorized request"` even though the agent started fine.

It refuses to start against an existing account (a reused home is not a clean
install), and it tears the account down from an `EXIT`/`INT` trap armed the
moment the account exists, so a failed or Ctrl-C'd run still leaves the machine
clean. Its verdict is deliberately stricter than the installer's own: note that
`install.sh` exits **0** when it finishes but `doctor` reports unmet
prerequisites, so "the installer succeeded" is not the assertion — `test` fails
if the install fails, if `bot-bottle` never reached the new user's `PATH`, or if
`doctor` is unhappy. `BB_TEST_KEEP=1` skips the teardown to poke at a failure.

### What a fresh account actually inherits

Expect the first honest run on a developer Mac to fail at the *Python* gate,
and expect that to be correct. A new account's `PATH` is just `/etc/paths`
(`/usr/local/bin:/System/Cryptexes/App/usr/bin:/usr/bin:/bin:/usr/sbin:/sbin`),
which notably does **not** include `/opt/homebrew/bin`. Homebrew's `shellenv`
line lives in the *installing* user's `~/.zprofile` and is not inherited, so a
throwaway user resolves `python3` to `/usr/bin/python3` — the Command Line
Tools stub, still **3.9.6** on macOS 26 — and `install.sh` correctly dies on its
`3.11+` requirement. Your own shell resolving `python3` to a 3.14 Homebrew
build says nothing about what a new user sees; that gap is exactly what this
harness exists to expose.

## Sources

- [Apple Containers on macOS: technical comparison with Docker — The New Stack](https://thenewstack.io/apple-containers-on-macos-a-technical-comparison-with-docker/)
- [How to Set Up Apple Containerization on macOS 26 — Stéphane Paquet](https://spaquet.medium.com/how-to-set-up-apple-containerization-on-macos-26-f870cc8c26cd)
- [Install Apple Container CLI (macOS 15/26) — 4sysops](https://4sysops.com/archives/install-apple-container-cli-running-containers-natively-on-macos-15-sequoia-and-macos-26-tahoe/)
- [Nested virtualization on Apple Silicon (M3+, macOS 15) — UTM issue #6700](https://github.com/utmapp/UTM/issues/6700)
- [macOS 15 Sequoia nested virtualization for M3+ — Parallels Forums](https://forum.parallels.com/threads/macos-15-sequoia-nested-virtualization-for-m3-macs.364397/)
- [M2 nested virtualization restriction (Apple DTS) — Apple Developer Forums](https://developer.apple.com/forums/thread/756723)
- [M4 can't virtualize older macOS — Yahoo/Tech](https://tech.yahoo.com/computing/articles/m4-mac-computers-cant-virtualize-175122301.html)
- [Tart — macOS/Linux VMs on Apple Silicon (Cirrus Labs)](https://tart.run/quick-start/)
- [Tart GitHub](https://github.com/cirruslabs/tart)
- [macOS VMs in a single command — frr.dev](https://www.frr.dev/posts/tart-macos-vms-from-terminal/)
- [sysadminctl reference — SS64](https://ss64.com/mac/sysadminctl.html)
- [User management from the macOS command line — macnotes](https://macnotes.wordpress.com/2019/03/28/user-management-create-remove-change-password-secure-token-from-macos-command-line/)