feat(harness): add test-ready, the with-prerequisites variant
prd-number-check / require-numbered-prds (pull_request) Successful in 11s
tracker-policy-pr / check-pr (pull_request) Successful in 21s
test / image-input-builds (pull_request) Successful in 48s
test / integration-docker (pull_request) Successful in 1m4s
test / unit (pull_request) Successful in 2m52s
test / coverage (pull_request) Failing after 20s
prd-number-check / require-numbered-prds (pull_request) Successful in 11s
tracker-policy-pr / check-pr (pull_request) Successful in 21s
test / image-input-builds (pull_request) Successful in 48s
test / integration-docker (pull_request) Successful in 1m4s
test / unit (pull_request) Successful in 2m52s
test / coverage (pull_request) Failing after 20s
"Does the installer work" and "can a new user actually run a bottle" are
different questions, and collapsing them is what made the previous run
ambiguous. Split them into two cycles over the same throwaway account:
test a macOS system WITHOUT the prerequisites set up for this user,
which is the default state of every new account since the Apple
`container` service is per-user. Asserts the install is sound;
reports backend readiness without failing on it, because
install.sh provides no backend and cannot regress one.
test-ready the same system WITH them. Runs `container system start` for the
throwaway user, then demands doctor go fully green, backend
included — the end-to-end claim.
The new `prereqs` step runs `container system start` rather than
`bot-bottle backend setup --backend=macos-container`, because that subcommand
does not start anything: it checks, then tells you to run `container system
start` yourself (backend/macos_container/setup.py, "no privileged host setup
required"). The harness runs what the product actually asks for.
test-ready sets BB_TEST_REQUIRE_BACKEND, so it demands exactly what `test`
merely reports. Both share one cycle function; the step count and the PASS
claim are the only differences beyond the extra step, so neither variant can
drift from the other's teardown or dirty-account guarantees.
Rig covers the distinguishing case directly: identical host state where `test`
passes on install soundness and `test-ready` starts the service and reaches a
green backend, plus the service-start failure and an install failure under
test-ready. 22 scenarios, all passing.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WEfZZhakx13bxTfXcZCoS5
This commit is contained in:
@@ -223,13 +223,25 @@ headlessly (so a PR is verifiable before it lands) and lets the installer run
|
||||
the footprint analysis above — the reset is just user deletion because
|
||||
everything `install.sh` writes is user-home-local.
|
||||
|
||||
`test` chains `up → run → status → down` into the one-shot cycle you normally
|
||||
want:
|
||||
There are two one-shot cycles, because "does the installer work" and "can a new
|
||||
user actually run a bottle" are different questions:
|
||||
|
||||
```sh
|
||||
sudo ./scripts/macos-install-test.sh test
|
||||
sudo ./scripts/macos-install-test.sh test # up → run → status → down
|
||||
sudo ./scripts/macos-install-test.sh test-ready # ... + prereqs before status
|
||||
```
|
||||
|
||||
`test` models a macOS system **without** the prerequisites set up for this user
|
||||
— which is the default state of every new account, since the `container`
|
||||
service is per-user. It asserts the install is sound and *reports* backend
|
||||
readiness without failing on it, because install.sh provides no backend and so
|
||||
cannot regress one.
|
||||
|
||||
`test-ready` models a system **with** them: it runs `container system start`
|
||||
for the throwaway user, then demands doctor go fully green, backend included.
|
||||
(`bot-bottle backend setup --backend=macos-container` does not start the
|
||||
service — it only checks and tells you to run that command yourself.)
|
||||
|
||||
It refuses to start against an existing account (a reused home is not a clean
|
||||
install), and it tears the account down from an `EXIT`/`INT` trap armed the
|
||||
moment the account exists, so a failed or Ctrl-C'd run still leaves the machine
|
||||
|
||||
Reference in New Issue
Block a user