fix(install): install into a private venv when pipx is absent

The harness's second run hit the wall the first one predicted: a fresh account
has no pipx, so install.sh fell to `pip install --user`, and every Python a Mac
offers — Homebrew and python.org alike — is externally managed, so PEP 668
blocked it. That fallback was never a fallback on macOS; it was a dead end that
printed instructions.

Replace it with a venv at ~/.bot-bottle/venv (BOT_BOTTLE_VENV to move it),
with the console script symlinked into ~/.local/bin. PEP 668 does not apply
inside a venv, and venv is stdlib, so unlike pipx there is nothing to bootstrap
first. pipx stays the preferred path when present, so anyone already managing
their Python apps that way is unaffected — and the post-install PATH check now
asks pipx for PIPX_BIN_DIR instead of assuming ~/.local/bin.

Keeping the venv under ~/.bot-bottle rather than ~/.local/share means the whole
footprint stays in one directory, which is what lets the throwaway-account
teardown remain a complete reset.

This removes the PEP 668 pre-flight and the sysconfig user-scheme lookup, both
of which existed only to serve the --user path. Their tests go with them:

* `detects_externally_managed_python` asserted the check that is now moot;
  replaced by one asserting pipx is still preferred when present.
* `checks_pip_usable_before_fallback` pinned a pip probe that no longer runs;
  replaced by one asserting the venv's own pip does the install, since using
  the base interpreter's would install outside the venv.
* `resolves_user_scripts_dir_not_hardcoded` and
  `macos_user_scheme_is_not_dot_local_bin` guarded the ~/Library/Python
  scripts-dir lookup. Nothing installs there now. The surviving "don't
  hardcode" concern is pipx's bin dir, which has its own test.

Five tests are added for the new path: the venv fallback exists, no --user path
survives, the venv is under the config dir, venv creation failure names
python3-venv (Debian ships it separately), and the entry point is exposed
outside the venv.

Verified end to end in a sandbox HOME with a fresh-account PATH and no pipx:
venv built, package installed, symlink created, `doctor` reached and green
(python 3.14.5, macos-container ready), exit 0.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WEfZZhakx13bxTfXcZCoS5

README.md

@@ -81,7 +81,7 @@ The installer is a bootstrapper: it finds a suitable Python, installs bot-bottle
 
 **Python ≥ 3.11**, and this is the one that trips people up on macOS: the `python3` Apple ships at `/usr/bin/python3` is **3.9.6**, which is too old. Bare `python3` resolves to that stub far more often than people expect. `path_helper` builds a login shell's `PATH` from `/etc/paths` and then appends `/etc/paths.d/*`, and `/usr/bin` sits in the former — so even when `/opt/homebrew/bin` *is* on the `PATH` (via `/etc/paths.d/homebrew`), it comes after `/usr/bin` and loses. Prepending a newer Python is something your shell profile does, and a fresh account, a launchd job, or a CI runner has no such profile. So the installer looks past bare `python3` before giving up: it tries `python3`, then the versioned `python3.11`–`python3.14` names, then `/opt/homebrew/bin`, `/usr/local/bin`, `~/.local/bin`, and python.org framework builds — and tells you which one it picked when it isn't the obvious one. Point it somewhere specific with `BOT_BOTTLE_PYTHON=/path/to/python3`.
 
-**`pipx`** is strongly preferred over the `pip --user` fallback. Homebrew and Debian/Ubuntu interpreters are externally managed (PEP 668), which blocks `pip install --user` outright; the installer detects this and tells you to install `pipx` rather than failing deep inside pip.
+**No `pipx` required.** If `pipx` is present the installer uses it and stays out of the way. If it isn't, bot-bottle installs into a private venv at `~/.bot-bottle/venv` (override with `BOT_BOTTLE_VENV`) and symlinks the entry point into `~/.local/bin`. There is deliberately no `pip install --user` path: Homebrew, python.org and Debian/Ubuntu interpreters are all externally managed (PEP 668), which blocks `--user` outright — so on a Mac it is never the fallback it appears to be. A venv is exempt from PEP 668, and `venv` is stdlib, so unlike `pipx` there is nothing to bootstrap first.
 
 **`git`**, because the default install spec is a `git+` URL. Set `BOT_BOTTLE_INSTALL_SPEC` to a wheel path or index name to avoid it.
 

docs/research/testing-clean-install-on-macos.md

@@ -69,7 +69,7 @@ Grounding the teardown story in what `install.sh` and the backend create:
 |---|---|---|---|
 | Config / state / db | `~/.bot-bottle/{agents,bottles,contrib,state,db}` ([`install.sh:80-83`](../install.sh), [`bot_bottle/paths.py:59`](../bot_bottle/paths.py)) | ✅ | ❌ removed with home |
 | pipx venv + shim | `~/.local/pipx/venvs/bot-bottle`, shim in `~/.local/bin` ([`install.sh:87-89`](../install.sh)) | ✅ | ❌ removed with home |
-| pip `--user` fallback | `~/Library/Python/<X.Y>/{lib,bin}` ([`install.sh:100-104`](../install.sh)) | ✅ | ❌ removed with home |
+| private venv fallback (no pipx) | `~/.bot-bottle/venv` + symlink in `~/.local/bin` ([`install.sh`](../install.sh)) | ✅ | ❌ removed with home |
 | PATH / token exports | shell profile (`~/.zprofile`, etc.); `BOT_BOTTLE_CLAUDE_OAUTH_TOKEN` ([`README.md:74`](../README.md)) | ✅ | ❌ removed with home |
 | **Apple `container` install** | `/usr/local/...` + notarized `.pkg` receipts | ❌ | ✅ **stays** |
 | **Apple `container` system service** | launchd system service (`container system start`) | ❌ | ✅ **stays** |

install.sh

@@ -9,7 +9,7 @@
 #   uv tool install bot-bottle
 #
 # This script is a thin bootstrapper: it finds a Python 3.11+ interpreter,
-# installs the package with pipx (falling back to pip --user), creates the
+# installs the package with pipx (falling back to a private venv), creates the
 # config dir, and runs `bot-bottle doctor`. It is idempotent (safe to re-run)
 # and never uses sudo. It does NOT install Docker or a VM backend for you —
 # `doctor` reports what's missing after install.
@@ -17,9 +17,11 @@
 # Env:
 #   BOT_BOTTLE_PYTHON        interpreter to install with (skips the search)
 #   BOT_BOTTLE_INSTALL_SPEC  pip/git spec to install instead of the default
+#   BOT_BOTTLE_VENV          where the non-pipx install lives (~/.bot-bottle/venv)
 set -eu
 
 PACKAGE_SPEC="${BOT_BOTTLE_INSTALL_SPEC:-git+https://gitea.dideric.is/didericis/bot-bottle.git}"
+VENV="${BOT_BOTTLE_VENV:-${HOME}/.bot-bottle/venv}"
 MIN_PYTHON_MAJOR=3
 MIN_PYTHON_MINOR=11
 
@@ -135,30 +137,6 @@ case "${PACKAGE_SPEC}" in
         ;;
 esac
 
-# The pip fallback needs a usable pip. Externally-managed interpreters
-# (PEP 668, common on Debian/Ubuntu/Homebrew) reject `pip install --user`;
-# pipx sidesteps that, so recommend it when pip can't be used.
-if ! command -v pipx >/dev/null 2>&1; then
-    "${PYTHON}" -m pip --version >/dev/null 2>&1 || die \
-        "neither pipx nor a usable '${PYTHON} -m pip' was found. Install pipx "\
-"(recommended): '${PYTHON} -m pip install --user pipx' or your OS package manager."
-    if "${PYTHON}" - <<'PY'
-import os
-import sys
-import sysconfig
-
-# PEP 668: an EXTERNALLY-MANAGED marker in the stdlib dir means pip refuses
-# to install into this interpreter without --break-system-packages.
-marker = os.path.join(sysconfig.get_path("stdlib"), "EXTERNALLY-MANAGED")
-raise SystemExit(0 if os.path.exists(marker) else 1)
-PY
-    then
-        die "${PYTHON} is externally managed (PEP 668), so 'pip install --user' is "\
-"blocked. Install pipx and re-run: '${PYTHON} -m pip install --user --break-system-packages pipx', "\
-"then 'pipx ensurepath'."
-    fi
-fi
-
 # --- config directories ------------------------------------------------------
 
 mkdir -p \
@@ -168,35 +146,43 @@ mkdir -p \
 
 # --- install -----------------------------------------------------------------
 
+BIN_DIR="${HOME}/.local/bin"
+
 if command -v pipx >/dev/null 2>&1; then
     # --python pins the venv to the interpreter we vetted. Without it pipx uses
     # whichever Python it was itself installed with, which is not necessarily
     # the one that passed the version check above.
     say "installing with pipx (python: ${PYTHON})"
     pipx install --python "${PYTHON}" --force "${PACKAGE_SPEC}"
+    # Ask pipx where it puts entry points rather than assuming ~/.local/bin.
+    pipx_bin="$(pipx environment --value PIPX_BIN_DIR 2>/dev/null || true)"
+    [ -n "${pipx_bin}" ] && BIN_DIR="${pipx_bin}"
 else
-    say "pipx not found; installing with '${PYTHON} -m pip install --user'"
-    "${PYTHON}" -m pip install --user --upgrade "${PACKAGE_SPEC}"
+    # No `pip install --user` fallback: PEP 668 makes it unusable on nearly
+    # every interpreter a Mac offers (Homebrew and python.org are both
+    # externally managed), and on Debian/Ubuntu too. A private venv sidesteps
+    # that entirely — PEP 668 does not apply inside a venv — and `venv` is
+    # stdlib, so unlike pipx there is nothing to bootstrap first.
+    say "pipx not found; installing into a managed venv at ${VENV}"
+    "${PYTHON}" -m venv --clear "${VENV}" || die \
+        "could not create a virtualenv at ${VENV} using ${PYTHON}. On Debian/Ubuntu "\
+"the venv module ships separately: 'sudo apt install python3-venv'."
+    "${VENV}/bin/python" -m pip install --upgrade "${PACKAGE_SPEC}"
+    # Expose the entry point outside the venv, the way pipx would.
+    mkdir -p "${BIN_DIR}"
+    ln -sf "${VENV}/bin/bot-bottle" "${BIN_DIR}/bot-bottle"
 fi
 
 # --- locate the entry point --------------------------------------------------
 
-# The pip --user scripts directory is platform-specific: ~/.local/bin on Linux,
-# but ~/Library/Python/<X.Y>/bin on a python.org macOS interpreter. Ask the
-# interpreter for its own user-scheme scripts dir instead of hardcoding.
-USER_SCRIPTS="$("${PYTHON}" - <<'PY'
-import sysconfig
-print(sysconfig.get_path("scripts", sysconfig.get_preferred_scheme("user")))
-PY
-)"
-
 if command -v bot-bottle >/dev/null 2>&1; then
     BOT_BOTTLE_BIN="bot-bottle"
-elif [ -n "${USER_SCRIPTS}" ] && [ -x "${USER_SCRIPTS}/bot-bottle" ]; then
-    BOT_BOTTLE_BIN="${USER_SCRIPTS}/bot-bottle"
-    say "note: add ${USER_SCRIPTS} to your PATH to run 'bot-bottle' directly"
+elif [ -x "${BIN_DIR}/bot-bottle" ]; then
+    BOT_BOTTLE_BIN="${BIN_DIR}/bot-bottle"
+    say "note: add ${BIN_DIR} to your PATH to run 'bot-bottle' directly:"
+    say "    echo 'export PATH=\"${BIN_DIR}:\$PATH\"' >> ~/.zprofile"
 else
-    die "bot-bottle was installed but is not on PATH; add ${USER_SCRIPTS:-your user scripts dir} to PATH and re-run"
+    die "bot-bottle was installed but no entry point turned up in ${BIN_DIR}"
 fi
 
 # --- verify ------------------------------------------------------------------

scripts/macos-install-test.sh

@@ -3,9 +3,10 @@
 #
 # Exercises install.sh the way a brand-new user would, inside a throwaway
 # macOS account you create and delete from the CLI. install.sh's entire
-# footprint is user-home-local — the pipx venv under ~/.local (or the pip
-# --user tree under ~/Library/Python) and the ~/.bot-bottle config dir. It
-# writes no shell-profile PATH line, and never installs the backend (see
+# footprint is user-home-local — the pipx venv under ~/.local, or the private
+# venv at ~/.bot-bottle/venv plus a ~/.local/bin symlink, and the ~/.bot-bottle
+# config dir. It writes no shell-profile PATH line, and never installs the
+# backend (see
 # the header of install.sh), so deleting the user is a complete,
 # deterministic reset of everything the installer touched. The Apple
 # `container` runtime is a HOST prerequisite installed once and kept;

tests/unit/test_install_script.py

@@ -10,7 +10,6 @@ from __future__ import annotations
 
 import os
 import re
-import sysconfig
 import unittest
 from pathlib import Path
 
@@ -57,9 +56,37 @@ class TestInstallScript(unittest.TestCase):
         self.assertIn(".bot-bottle/agents", self.text)
         self.assertIn(".bot-bottle/bottles", self.text)
 
-    def test_installs_via_pipx_with_pip_fallback(self):
+    def test_installs_via_pipx_with_venv_fallback(self):
         self.assertIn("pipx install", self.text)
-        self.assertIn("pip install --user", self.text)
+        self.assertIn("-m venv", self.text)
+
+    def test_no_pip_user_fallback(self):
+        # `pip install --user` is not a fallback, it's a dead end: PEP 668
+        # blocks it on Homebrew, python.org and Debian/Ubuntu interpreters,
+        # which is every Python a Mac realistically offers. A private venv is
+        # exempt from PEP 668 and needs no bootstrap, since venv is stdlib.
+        # code_only, because the comment explaining the absence says the words.
+        code = code_only(self.text)
+        self.assertNotIn("pip install --user", code)
+        self.assertNotIn("--break-system-packages", code)
+
+    def test_venv_lives_under_the_config_dir(self):
+        # Keeps the whole install footprint inside ~/.bot-bottle (plus the
+        # entry-point symlink), which is what makes deleting a throwaway
+        # account a complete reset in scripts/macos-install-test.sh.
+        self.assertIn(".bot-bottle/venv", self.text)
+        self.assertIn("BOT_BOTTLE_VENV", self.text)
+
+    def test_venv_failure_is_actionable(self):
+        # Debian/Ubuntu ship venv separately; failing there must say so rather
+        # than dumping ensurepip's error.
+        self.assertIn("python3-venv", self.text)
+
+    def test_entry_point_is_exposed_outside_the_venv(self):
+        # A venv's bin dir is never on PATH, so the console script has to be
+        # linked somewhere conventional or `bot-bottle` is unreachable.
+        self.assertIn(".local/bin", self.text)
+        self.assertIn("ln -sf", self.text)
 
     def test_runs_doctor_after_install(self):
         self.assertIn("doctor", self.text)
@@ -74,29 +101,20 @@ class TestInstallScript(unittest.TestCase):
         self.assertIn("command -v git", self.text)
         self.assertIn("git+*|*.git", self.text)
 
-    def test_checks_pip_usable_before_fallback(self):
-        # Against the discovered interpreter, which is not necessarily the
-        # `python3` on PATH — hence no literal command name here.
-        self.assertIn("-m pip --version", self.text)
+    def test_installs_into_the_venv_with_its_own_pip(self):
+        # The venv's pip, not the base interpreter's — the base one may not
+        # exist, and using it would install outside the venv.
+        self.assertIn("${VENV}/bin/python\" -m pip install", self.text)
 
-    def test_detects_externally_managed_python(self):
-        # PEP 668: 'pip install --user' is blocked on externally-managed
-        # interpreters; the script must detect this and point at pipx.
-        self.assertIn("EXTERNALLY-MANAGED", self.text)
-        self.assertIn("pipx", self.text)
+    def test_pipx_is_preferred_when_present(self):
+        # The venv is a fallback, not a takeover: someone who already manages
+        # their Python apps with pipx keeps doing so.
+        self.assertIn("command -v pipx", self.text)
 
-    def test_resolves_user_scripts_dir_not_hardcoded(self):
-        # The pip --user scripts dir differs by platform; the script must ask
-        # the interpreter (sysconfig + the preferred *user* scheme) rather than
-        # hardcoding Linux's ~/.local/bin (which is wrong on macOS python.org).
-        self.assertIn("get_preferred_scheme", self.text)
-        self.assertIn("sysconfig", self.text)
-        # ~/.local/bin appears legitimately elsewhere — it's one of the places
-        # the interpreter search looks. What must never happen is USER_SCRIPTS
-        # itself being hardcoded to it.
-        for line in code_only(self.text).splitlines():
-            if "USER_SCRIPTS" in line:
-                self.assertNotIn(".local/bin", line)
+    def test_asks_pipx_where_its_bin_dir_is(self):
+        # PIPX_BIN_DIR is configurable, so the post-install "is it on PATH?"
+        # check must ask rather than assume ~/.local/bin.
+        self.assertIn("PIPX_BIN_DIR", self.text)
 
     def test_searches_beyond_path_for_an_interpreter(self):
         # `python3` on PATH is the *oldest* interpreter on a stock Mac: a fresh
@@ -121,20 +139,5 @@ class TestInstallScript(unittest.TestCase):
         self.assertIn("brew install python@", self.text)
         self.assertIn("BOT_BOTTLE_PYTHON=/path/to/python3", self.text)
 
-    def test_macos_user_scheme_is_not_dot_local_bin(self):
-        # The case the fix exists for: a python.org macOS interpreter uses the
-        # osx_framework_user scheme, whose scripts land under
-        # ~/Library/Python/<X.Y>/bin — NOT ~/.local/bin. Drive the same
-        # sysconfig lookup install.sh uses, with a mac-like userbase, to prove
-        # it resolves a non-~/.local/bin directory.
-        self.assertIn("osx_framework_user", sysconfig.get_scheme_names())
-        scripts = sysconfig.get_path(
-            "scripts", "osx_framework_user",
-            vars={"userbase": "/Users/dev/Library/Python/3.11"},
-        )
-        self.assertEqual("/Users/dev/Library/Python/3.11/bin", scripts)
-        self.assertNotIn("/.local/bin", scripts)
-
-
 if __name__ == "__main__":
     unittest.main()