feat(harness): add test-ready, the with-prerequisites variant
prd-number-check / require-numbered-prds (pull_request) Successful in 11s
tracker-policy-pr / check-pr (pull_request) Successful in 21s
test / image-input-builds (pull_request) Successful in 48s
test / integration-docker (pull_request) Successful in 1m4s
test / unit (pull_request) Successful in 2m52s
test / coverage (pull_request) Failing after 20s

"Does the installer work" and "can a new user actually run a bottle" are
different questions, and collapsing them is what made the previous run
ambiguous. Split them into two cycles over the same throwaway account:

  test        a macOS system WITHOUT the prerequisites set up for this user,
              which is the default state of every new account since the Apple
              `container` service is per-user. Asserts the install is sound;
              reports backend readiness without failing on it, because
              install.sh provides no backend and cannot regress one.

  test-ready  the same system WITH them. Runs `container system start` for the
              throwaway user, then demands doctor go fully green, backend
              included — the end-to-end claim.

The new `prereqs` step runs `container system start` rather than
`bot-bottle backend setup --backend=macos-container`, because that subcommand
does not start anything: it checks, then tells you to run `container system
start` yourself (backend/macos_container/setup.py, "no privileged host setup
required"). The harness runs what the product actually asks for.

test-ready sets BB_TEST_REQUIRE_BACKEND, so it demands exactly what `test`
merely reports. Both share one cycle function; the step count and the PASS
claim are the only differences beyond the extra step, so neither variant can
drift from the other's teardown or dirty-account guarantees.

Rig covers the distinguishing case directly: identical host state where `test`
passes on install soundness and `test-ready` starts the service and reaches a
green backend, plus the service-start failure and an install failure under
test-ready. 22 scenarios, all passing.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WEfZZhakx13bxTfXcZCoS5
This commit is contained in:
2026-07-27 09:51:35 -04:00
parent 556217ae7b
commit 71c0f8ed88
2 changed files with 112 additions and 23 deletions
@@ -223,13 +223,25 @@ headlessly (so a PR is verifiable before it lands) and lets the installer run
the footprint analysis above — the reset is just user deletion because
everything `install.sh` writes is user-home-local.
`test` chains `up → run → status → down` into the one-shot cycle you normally
want:
There are two one-shot cycles, because "does the installer work" and "can a new
user actually run a bottle" are different questions:
```sh
sudo ./scripts/macos-install-test.sh test
sudo ./scripts/macos-install-test.sh test # up → run → status → down
sudo ./scripts/macos-install-test.sh test-ready # ... + prereqs before status
```
`test` models a macOS system **without** the prerequisites set up for this user
— which is the default state of every new account, since the `container`
service is per-user. It asserts the install is sound and *reports* backend
readiness without failing on it, because install.sh provides no backend and so
cannot regress one.
`test-ready` models a system **with** them: it runs `container system start`
for the throwaway user, then demands doctor go fully green, backend included.
(`bot-bottle backend setup --backend=macos-container` does not start the
service — it only checks and tells you to run that command yourself.)
It refuses to start against an existing account (a reused home is not a clean
install), and it tears the account down from an `EXIT`/`INT` trap armed the
moment the account exists, so a failed or Ctrl-C'd run still leaves the machine