didericis
90d6104e17
prd-number-check / require-numbered-prds (pull_request) Successful in 5s
test / image-input-builds (pull_request) Successful in 38s
test / unit (pull_request) Successful in 44s
tracker-policy-pr / check-pr (pull_request) Successful in 4s
lint / lint (push) Successful in 53s
test / integration-docker (pull_request) Successful in 58s
test / coverage (pull_request) Successful in 18s
The harness's second run hit the wall the first one predicted: a fresh account has no pipx, so install.sh fell to `pip install --user`, and every Python a Mac offers — Homebrew and python.org alike — is externally managed, so PEP 668 blocked it. That fallback was never a fallback on macOS; it was a dead end that printed instructions. Replace it with a venv at ~/.bot-bottle/venv (BOT_BOTTLE_VENV to move it), with the console script symlinked into ~/.local/bin. PEP 668 does not apply inside a venv, and venv is stdlib, so unlike pipx there is nothing to bootstrap first. pipx stays the preferred path when present, so anyone already managing their Python apps that way is unaffected — and the post-install PATH check now asks pipx for PIPX_BIN_DIR instead of assuming ~/.local/bin. Keeping the venv under ~/.bot-bottle rather than ~/.local/share means the whole footprint stays in one directory, which is what lets the throwaway-account teardown remain a complete reset. This removes the PEP 668 pre-flight and the sysconfig user-scheme lookup, both of which existed only to serve the --user path. Their tests go with them: * `detects_externally_managed_python` asserted the check that is now moot; replaced by one asserting pipx is still preferred when present. * `checks_pip_usable_before_fallback` pinned a pip probe that no longer runs; replaced by one asserting the venv's own pip does the install, since using the base interpreter's would install outside the venv. * `resolves_user_scripts_dir_not_hardcoded` and `macos_user_scheme_is_not_dot_local_bin` guarded the ~/Library/Python scripts-dir lookup. Nothing installs there now. The surviving "don't hardcode" concern is pipx's bin dir, which has its own test. Five tests are added for the new path: the venv fallback exists, no --user path survives, the venv is under the config dir, venv creation failure names python3-venv (Debian ships it separately), and the entry point is exposed outside the venv. Verified end to end in a sandbox HOME with a fresh-account PATH and no pipx: venv built, package installed, symlink created, `doctor` reached and green (python 3.14.5, macos-container ready), exit 0. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WEfZZhakx13bxTfXcZCoS5
259 lines
15 KiB
Markdown
259 lines
15 KiB
Markdown
# Testing a clean bot-bottle install on macOS
|
|
|
|
How do you exercise `install.sh` (and, ideally, a first `bot-bottle start`)
|
|
the way a brand-new user would — on a pristine macOS environment you can
|
|
throw away afterward — *without* permanently polluting your daily-driver
|
|
Mac? The user's framing: is there a VM or boundary that avoids creating a
|
|
separate account, or is spinning up and tearing down a throwaway macOS
|
|
user on the CLI easy enough to just do that?
|
|
|
|
## Summary
|
|
|
|
There is no lightweight, in-place macOS sandbox that hands you a clean home
|
|
directory and wipeable system state without *either* a VM or a separate
|
|
user account. `sandbox-exec` (Seatbelt) is deprecated and confines a
|
|
process, not an environment; App Sandbox is for shipping apps, not for
|
|
provisioning a fresh dev host. So the real choice is exactly the two the
|
|
user named: **a disposable macOS VM** or **a throwaway user account** —
|
|
and which one is right turns on a detail specific to *this* project.
|
|
|
|
bot-bottle's default macOS backend is Apple's `container`, which runs each
|
|
container in its own lightweight VM via `Virtualization.framework`
|
|
([`README.md:27`](../README.md), [`apple-container-backend.md`](apple-container-backend.md)).
|
|
That means a full end-to-end test — install *and* `bot-bottle start` —
|
|
needs virtualization to work wherever bot-bottle runs. Inside a macOS guest
|
|
VM that requires **nested virtualization, which Apple gates to M3 or newer
|
|
chips on macOS 15+**. On M1/M2 you cannot run the Apple Container backend
|
|
(or Docker Desktop, same reason) inside a macOS VM at all.
|
|
|
|
The recommendation splits on what you're testing and what silicon you have:
|
|
|
|
- **Install-script correctness only** (does `curl | sh` → pipx → config dir
|
|
→ `doctor`'s Python/config checks pass?): a **disposable Tart VM** is the
|
|
cleanest boundary and works on any Apple Silicon Mac. `doctor` will report
|
|
the backend as not-ready inside the VM on M1/M2, which is fine — you're
|
|
testing the installer, not the runtime.
|
|
- **Full runtime** (actually launch a bottle) on **M3/M4**: a **disposable
|
|
Tart VM from a golden base image, cloned per run** is the gold standard —
|
|
a genuine kernel/state boundary that wipes to nothing.
|
|
- **Full runtime** on **M1/M2**, or when you'd rather not fight nested virt:
|
|
a **throwaway admin user via `sysadminctl`** is the pragmatic pick. It
|
|
tests the real backend because the backend runs on the host hypervisor —
|
|
but it is a *hygiene* boundary, not a security one, and it does **not**
|
|
clean the system-level footprint (see below).
|
|
|
|
Prefer the VM. Reach for the throwaway user only when nested virt is off the
|
|
table and you accept an imperfect wipe.
|
|
|
|
## Why "a boundary without a separate user" doesn't really exist on macOS
|
|
|
|
macOS has no namespace/overlay story like Linux `unshare` + tmpfs. The
|
|
options that sound like in-place sandboxes don't fit:
|
|
|
|
| Mechanism | Why it doesn't give you a clean, wipeable env |
|
|
|---|---|
|
|
| `sandbox-exec` / Seatbelt | Officially deprecated; confines *one process's* syscalls against a profile. It cannot present a fresh `$HOME` or a pristine `/usr/local`, and it won't let the Apple Container system service work. |
|
|
| App Sandbox | Entitlement-based confinement for signed `.app` bundles, not a provisioning tool for a CLI dev environment. |
|
|
| A second `$HOME` via `HOME=/tmp/foo` | Redirects only what honors `$HOME`. `install.sh` mostly does (it writes `~/.bot-bottle` and pipx/pip `--user` paths), but the Apple `container` install lands in `/usr/local` + a **system service**, and Homebrew lands in `/opt/homebrew` — all outside any `$HOME` you set. You'd get a false sense of "clean." |
|
|
| APFS snapshot rollback (`tmutil localsnapshot`) | You can't roll the live boot volume back to a local snapshot without booting to Recovery; it's not a per-run userspace undo. |
|
|
|
|
So the honest answer to "is there some boundary that avoids a separate
|
|
user?": yes — a **VM** — and it's the *stronger* boundary anyway. The only
|
|
lighter-weight option is the separate user, with the caveats below.
|
|
|
|
## What a clean install actually touches (the footprint that decides "wipeable")
|
|
|
|
Grounding the teardown story in what `install.sh` and the backend create:
|
|
|
|
| Artifact | Location | In `$HOME`? | Survives user deletion? |
|
|
|---|---|---|---|
|
|
| Config / state / db | `~/.bot-bottle/{agents,bottles,contrib,state,db}` ([`install.sh:80-83`](../install.sh), [`bot_bottle/paths.py:59`](../bot_bottle/paths.py)) | ✅ | ❌ removed with home |
|
|
| pipx venv + shim | `~/.local/pipx/venvs/bot-bottle`, shim in `~/.local/bin` ([`install.sh:87-89`](../install.sh)) | ✅ | ❌ removed with home |
|
|
| private venv fallback (no pipx) | `~/.bot-bottle/venv` + symlink in `~/.local/bin` ([`install.sh`](../install.sh)) | ✅ | ❌ removed with home |
|
|
| PATH / token exports | shell profile (`~/.zprofile`, etc.); `BOT_BOTTLE_CLAUDE_OAUTH_TOKEN` ([`README.md:74`](../README.md)) | ✅ | ❌ removed with home |
|
|
| **Apple `container` install** | `/usr/local/...` + notarized `.pkg` receipts | ❌ | ✅ **stays** |
|
|
| **Apple `container` system service** | launchd system service (`container system start`) | ❌ | ✅ **stays** |
|
|
| **Homebrew** (if used for `container`/python) | `/opt/homebrew` | ❌ | ✅ **stays** |
|
|
| Rosetta 2 (needed for image builds) | system | ❌ | ✅ **stays** |
|
|
|
|
The three bold rows are the crux: **deleting the throwaway user does not
|
|
uninstall the Apple Container runtime, its system service, Homebrew, or
|
|
Rosetta.** A VM, by contrast, wipes 100% of the above by definition —
|
|
that's its entire advantage for this task.
|
|
|
|
## Option A — Disposable Tart VM (recommended)
|
|
|
|
[Tart](https://tart.run) is a CLI-first macOS/Linux VM manager built on
|
|
`Virtualization.framework`, purpose-built for exactly this "does it work on
|
|
a clean macOS, without my settings/permissions/data" workflow. Keep one
|
|
pristine *golden* image, clone a throwaway per run, delete it after.
|
|
|
|
```sh
|
|
brew install cirruslabs/cli/tart
|
|
|
|
# One-time: build a golden base (either a prebuilt image or a vanilla IPSW).
|
|
tart clone ghcr.io/cirruslabs/macos-tahoe-base:latest golden # ~25 GB pull
|
|
# — or a truly vanilla install you click through once —
|
|
# tart create golden --from-ipsw latest --disk-size 60
|
|
|
|
# Per test run: clone → boot → test → destroy.
|
|
tart clone golden test-run
|
|
tart run test-run &
|
|
ssh admin@"$(tart ip test-run)"
|
|
# inside the guest:
|
|
# curl -fsSL https://gitea.dideric.is/didericis/bot-bottle/raw/branch/main/install.sh | sh
|
|
# bot-bottle doctor
|
|
tart stop test-run
|
|
tart delete test-run # back to pristine; golden is untouched
|
|
```
|
|
|
|
Cloning is cheap (sparse files), so the golden image is your reset button —
|
|
every `tart clone` is a fresh macOS. This is the closest thing to a Linux
|
|
`docker run --rm` for a whole Mac.
|
|
|
|
**The nested-virt caveat (read before relying on it for runtime tests).**
|
|
The Apple Container backend inside the guest needs
|
|
`Virtualization.framework` to work *inside* the VM. Apple enables nested
|
|
virtualization only on **M3 or newer**, on **macOS 15 (Sequoia) or later**;
|
|
M2 and earlier are excluded by Apple, confirmed by Apple DTS. Consequences:
|
|
|
|
- **M3/M4 host:** full runtime works in the guest. `bot-bottle doctor`
|
|
reports the backend ready and `start` can launch a bottle. Gold standard.
|
|
- **M1/M2 host:** the guest can install bot-bottle and pass the Python /
|
|
config-dir checks, but `doctor`'s backend check will fail and you cannot
|
|
launch a bottle in the VM. Still perfectly good for testing *the
|
|
installer*; not for the runtime.
|
|
- **M4-specific:** a known bug blocks pre-Ventura guests on M4; use a
|
|
current macOS guest (which you want anyway, since Apple `container`
|
|
targets macOS 26 Tahoe).
|
|
|
|
UTM is the GUI equivalent on the same framework (and was first to expose
|
|
nested virt) if you'd rather click; Tart wins for a scriptable
|
|
spin-up/tear-down loop.
|
|
|
|
## Option B — Throwaway user via `sysadminctl` (pragmatic fallback)
|
|
|
|
Creating and deleting a user from the CLI is genuinely a two-liner, and it
|
|
tests the **real** backend on any Apple Silicon Mac because the backend runs
|
|
on the host hypervisor — no nested virt needed.
|
|
|
|
```sh
|
|
# Create a self-contained admin user (admin needed for the container service).
|
|
sudo sysadminctl -addUser bbtest -fullName "bot-bottle test" \
|
|
-password 'throwaway' -admin
|
|
|
|
# Log into that account (fast-user-switch or the login window), then run the
|
|
# installer as bbtest exactly as a new user would. When done:
|
|
|
|
sudo sysadminctl -deleteUser bbtest -secure # -secure erases the home dir
|
|
```
|
|
|
|
Honest accounting of what this does and doesn't buy you:
|
|
|
|
- **Boundary strength:** it's a *hygiene / fresh-`$HOME`* boundary, **not a
|
|
security boundary.** Same kernel, same admin group; an admin test user can
|
|
touch system state. If the point is "clean environment," fine. If the point
|
|
is "contain something untrusted," this is the wrong tool — use a VM.
|
|
- **Wipe completeness:** `-secure` erases the home dir (so `~/.bot-bottle`,
|
|
the pipx venv, and profile exports go away), but as the footprint table
|
|
shows, the **Apple Container runtime, its launchd system service,
|
|
Homebrew, and Rosetta persist.** For a truly repeatable "did a *system with
|
|
nothing installed* work?" test, that residue defeats the purpose — the
|
|
second run isn't clean.
|
|
- **Operational gotchas:** don't pass real passwords on the command line (they
|
|
land in `ps` and history — this is a throwaway credential, so it's
|
|
tolerable here). Deletion must run as root from a normally-booted, admin-
|
|
logged-in session; the Terminal needs **Full Disk Access** or you'll hit
|
|
error `-14120` and a half-deleted account. Prefer letting the system place
|
|
the home dir (don't pass `-home`), or deletion can orphan it.
|
|
|
|
Use this when you're on M1/M2, you specifically want to exercise the live
|
|
backend, and you can tolerate the system-level runtime staying installed
|
|
between runs (or you uninstall Apple `container` / brew by hand to reset).
|
|
|
|
## Honorable mentions
|
|
|
|
- **External bootable macOS volume.** A fresh macOS on an external SSD (or a
|
|
separate APFS volume) is bare-metal disposable: no nested-virt limit, real
|
|
backend works, and you `diskutil` the volume away to reset. Cost is reboot
|
|
friction per run — good for an occasional thorough pass, poor for a tight
|
|
loop.
|
|
- **Rented / cloud Mac.** AWS EC2 Mac (dedicated Mac minis), Scaleway Apple
|
|
silicon, or MacStadium give a genuinely throwaway host you release when
|
|
done. Overkill for local iteration, but this is essentially what the
|
|
project's own advisory `integration-macos` CI job needs — a self-hosted
|
|
Apple Silicon runner with the `container` CLI, Python ≥ 3.11, and coverage
|
|
on the launchd service's PATH ([`README.md:78`](../README.md)). If you end
|
|
up standing up a cloud Mac for install testing, it doubles as that runner.
|
|
|
|
## Recommendation
|
|
|
|
Default to a **disposable Tart VM** — it's the only option that wipes the
|
|
*entire* footprint (including the Apple Container system service that a user
|
|
deletion leaves behind), it's a real boundary, and the spin-up/tear-down
|
|
loop is a two-command `tart clone` / `tart delete`. Confirm your chip first:
|
|
on **M3/M4** it tests install *and* runtime end-to-end; on **M1/M2** it still
|
|
cleanly tests `install.sh` + `doctor`'s Python/config path, and you fall back
|
|
to a **throwaway `sysadminctl` admin user** for live-backend testing —
|
|
accepting that it's a hygiene boundary and that you'll manually uninstall the
|
|
Apple Container runtime / Homebrew between runs to get back to truly clean.
|
|
|
|
There is no third, lighter-weight "in-place boundary without a user" that
|
|
actually delivers a clean, wipeable macOS — the VM *is* that answer, and it's
|
|
the better one.
|
|
|
|
## Harness
|
|
|
|
The throwaway-user loop is scripted in
|
|
[`scripts/macos-install-test.sh`](../../scripts/macos-install-test.sh):
|
|
`up` creates the account, `run` pipes *this checkout's* `install.sh` into it
|
|
headlessly (so a PR is verifiable before it lands) and lets the installer run
|
|
`doctor`, `down` deletes the account and its home (the full reset), and
|
|
`deep-reset` additionally uninstalls the host `container` runtime. It leans on
|
|
the footprint analysis above — the reset is just user deletion because
|
|
everything `install.sh` writes is user-home-local.
|
|
|
|
`test` chains `up → run → status → down` into the one-shot cycle you normally
|
|
want:
|
|
|
|
```sh
|
|
sudo ./scripts/macos-install-test.sh test
|
|
```
|
|
|
|
It refuses to start against an existing account (a reused home is not a clean
|
|
install), and it tears the account down from an `EXIT`/`INT` trap armed the
|
|
moment the account exists, so a failed or Ctrl-C'd run still leaves the machine
|
|
clean. Its verdict is deliberately stricter than the installer's own: note that
|
|
`install.sh` exits **0** when it finishes but `doctor` reports unmet
|
|
prerequisites, so "the installer succeeded" is not the assertion — `test` fails
|
|
if the install fails, if `bot-bottle` never reached the new user's `PATH`, or if
|
|
`doctor` is unhappy. `BB_TEST_KEEP=1` skips the teardown to poke at a failure.
|
|
|
|
### What a fresh account actually inherits
|
|
|
|
Expect the first honest run on a developer Mac to fail at the *Python* gate,
|
|
and expect that to be correct. A new account's `PATH` is just `/etc/paths`
|
|
(`/usr/local/bin:/System/Cryptexes/App/usr/bin:/usr/bin:/bin:/usr/sbin:/sbin`),
|
|
which notably does **not** include `/opt/homebrew/bin`. Homebrew's `shellenv`
|
|
line lives in the *installing* user's `~/.zprofile` and is not inherited, so a
|
|
throwaway user resolves `python3` to `/usr/bin/python3` — the Command Line
|
|
Tools stub, still **3.9.6** on macOS 26 — and `install.sh` correctly dies on its
|
|
`3.11+` requirement. Your own shell resolving `python3` to a 3.14 Homebrew
|
|
build says nothing about what a new user sees; that gap is exactly what this
|
|
harness exists to expose.
|
|
|
|
## Sources
|
|
|
|
- [Apple Containers on macOS: technical comparison with Docker — The New Stack](https://thenewstack.io/apple-containers-on-macos-a-technical-comparison-with-docker/)
|
|
- [How to Set Up Apple Containerization on macOS 26 — Stéphane Paquet](https://spaquet.medium.com/how-to-set-up-apple-containerization-on-macos-26-f870cc8c26cd)
|
|
- [Install Apple Container CLI (macOS 15/26) — 4sysops](https://4sysops.com/archives/install-apple-container-cli-running-containers-natively-on-macos-15-sequoia-and-macos-26-tahoe/)
|
|
- [Nested virtualization on Apple Silicon (M3+, macOS 15) — UTM issue #6700](https://github.com/utmapp/UTM/issues/6700)
|
|
- [macOS 15 Sequoia nested virtualization for M3+ — Parallels Forums](https://forum.parallels.com/threads/macos-15-sequoia-nested-virtualization-for-m3-macs.364397/)
|
|
- [M2 nested virtualization restriction (Apple DTS) — Apple Developer Forums](https://developer.apple.com/forums/thread/756723)
|
|
- [M4 can't virtualize older macOS — Yahoo/Tech](https://tech.yahoo.com/computing/articles/m4-mac-computers-cant-virtualize-175122301.html)
|
|
- [Tart — macOS/Linux VMs on Apple Silicon (Cirrus Labs)](https://tart.run/quick-start/)
|
|
- [Tart GitHub](https://github.com/cirruslabs/tart)
|
|
- [macOS VMs in a single command — frr.dev](https://www.frr.dev/posts/tart-macos-vms-from-terminal/)
|
|
- [sysadminctl reference — SS64](https://ss64.com/mac/sysadminctl.html)
|
|
- [User management from the macOS command line — macnotes](https://macnotes.wordpress.com/2019/03/28/user-management-create-remove-change-password-secure-token-from-macos-command-line/)
|