didericis/bot-bottle
1
0
Fork 0
Code Issues 26 Pull Requests 31 Actions Packages Projects Releases Wiki Activity

Smoke-test agent images after build, add start --no-cache #354

Open
didericis-claude wants to merge 3 commits from post-build-image-smoke-test into main
pull from: post-build-image-smoke-test
merge into: didericis:main
Conversation 2 Commits 3 Files Changed 12
+143 -58
didericis-claude commented 2026-07-13 04:04:34 -04:00
Collaborator
Copy Link
Copy Source

Closes #353.

Summary

  • Added AgentProviderRuntime.smoke_test: an optional argv each provider can declare (claude: claude --version, codex: codex --version) that a fresh build must pass.
  • Added verify_agent_image() to both the docker and macos-container util modules: runs the smoke test in a throwaway container of the just-built agent image and die()s with the command's own stderr if it fails, instead of shipping an image whose CLI is silently broken. Wired into all three launch paths (docker, macos-container, smolmachines) right after the agent image build, skipped when a committed image is reused.
  • Added bot-bottle start --no-cache (also BOT_BOTTLE_NO_CACHE=1) to force docker build --no-cache / container build --no-cache, so a broken cached layer (like the one that caused #353) can be rebuilt from scratch without manually pruning image caches.

Root cause recap

@anthropic-ai/claude-code ships its CLI as a native binary pulled via a platform-specific optionalDependencies entry. npm treats optional-dependency failures as non-fatal, so a transient fetch failure during npm install -g left a 500-byte stub claude.exe in an image that still exited 0 — then that broken layer got cached indefinitely by BuildKit/Container's layer cache.

Out of scope

bot_bottle/contrib/pi/Dockerfile installs with --ignore-scripts, which has a different (unverified) risk profile — not touched here.

Closes #353. ## Summary - Added `AgentProviderRuntime.smoke_test`: an optional argv each provider can declare (claude: `claude --version`, codex: `codex --version`) that a fresh build must pass. - Added `verify_agent_image()` to both the docker and macos-container `util` modules: runs the smoke test in a throwaway container of the just-built agent image and `die()`s with the command's own stderr if it fails, instead of shipping an image whose CLI is silently broken. Wired into all three launch paths (docker, macos-container, smolmachines) right after the agent image build, skipped when a committed image is reused. - Added `bot-bottle start --no-cache` (also `BOT_BOTTLE_NO_CACHE=1`) to force `docker build --no-cache` / `container build --no-cache`, so a broken cached layer (like the one that caused #353) can be rebuilt from scratch without manually pruning image caches. ## Root cause recap `@anthropic-ai/claude-code` ships its CLI as a native binary pulled via a platform-specific `optionalDependencies` entry. npm treats optional-dependency failures as non-fatal, so a transient fetch failure during `npm install -g` left a 500-byte stub `claude.exe` in an image that still exited 0 — then that broken layer got cached indefinitely by BuildKit/Container's layer cache. ## Out of scope `bot_bottle/contrib/pi/Dockerfile` installs with `--ignore-scripts`, which has a different (unverified) risk profile — not touched here.
didericis added 1 commit 2026-07-13 04:08:43 -04:00
fix: smoke-test agent images after build, add start --no-cache
lint / lint (push) Successful in 2m3s
Details
test / unit (pull_request) Successful in 56s
Details
test / integration (pull_request) Successful in 20s
Details
test / coverage (pull_request) Successful in 1m1s
Details
e862b82322 
npm treats optionalDependencies failures as non-fatal, so a transient
network blip fetching claude-code's platform-native binary during
`npm install -g` left a stub CLI in an image that still "built"
successfully — then got baked into the Docker/Container layer cache
until forced to rebuild. Post-build smoke test (provider-declared
argv, run in a throwaway container of the freshly built image) fails
the launch loudly instead of shipping a broken image; --no-cache
gives an escape hatch to force a from-scratch rebuild.

Closes #353.
didericis force-pushed post-build-image-smoke-test from bfdbae804b to e862b82322 2026-07-13 04:08:43 -04:00 Compare
didericis added 1 commit 2026-07-13 04:22:05 -04:00
refactor(firecracker): use docker_mod instead of hand-rolled docker helpers
lint / lint (push) Successful in 1m59s
Details
test / unit (pull_request) Successful in 58s
Details
test / integration (pull_request) Successful in 21s
Details
test / coverage (pull_request) Successful in 1m1s
Details
1a8d33a645 
firecracker/launch.py reimplemented docker build/image-exists/rm/exec/cp
as private functions instead of the shared docker_mod used by the
docker and macos-container backends. Switching to docker_mod dedupes
the logic and gets --no-cache support for free (docker_mod.build_image
already reads BOT_BOTTLE_NO_CACHE); docker_mod gains docker_exec/
docker_cp general-purpose helpers to cover what the private versions did.
didericis added 1 commit 2026-07-13 04:23:44 -04:00
feat(firecracker): run the post-build agent-image smoke test too
lint / lint (push) Successful in 1m58s
Details
test / unit (pull_request) Successful in 59s
Details
test / integration (pull_request) Successful in 18s
Details
test / coverage (pull_request) Successful in 59s
Details
add4c53f43 
Closes the gap left in #354: docker and macos-container already
smoke-test a freshly built agent image before launch; firecracker
built the image via its own docker_mod-based path but never ran the
check. Same one-liner as the other two backends now that _build_agent_image
uses docker_mod.build_image.
didericis reviewed 2026-07-13 14:59:23 -04:00
bot_bottle/backend/docker/launch.py
@@ -113,1 +114,4 @@
)
docker_mod.verify_agent_image(
plan.image, runtime_for(plan.agent_provider_template).smoke_test,
)
didericis commented 2026-07-13 14:59:23 -04:00
Owner
Copy Link
Copy Source

This requires spinning up a new container, just for the verification step?

This requires spinning up a new container, just for the verification step?
didericis-claude reviewed 2026-07-13 15:00:48 -04:00
bot_bottle/backend/docker/launch.py
@@ -113,1 +114,4 @@
)
docker_mod.verify_agent_image(
plan.image, runtime_for(plan.agent_provider_template).smoke_test,
)
didericis-claude commented 2026-07-13 15:00:48 -04:00
Author
Collaborator
Copy Link
Copy Source

Yes — it runs docker run --rm with just the smoke-test argv (e.g. claude --version), so the container exits in under a second and is immediately removed. The overhead is intentional: the root bug was that docker build exits 0 even when npm silently fails to install the optional binary, so the only reliable way to catch a broken image is to actually run it.

Yes — it runs `docker run --rm` with just the smoke-test argv (e.g. `claude --version`), so the container exits in under a second and is immediately removed. The overhead is intentional: the root bug was that `docker build` exits 0 even when npm silently fails to install the optional binary, so the only reliable way to catch a broken image is to actually run it.
didericis commented 2026-07-13 15:06:41 -04:00
Owner
Copy Link
Copy Source

@didericis-claude I think it's pretty rare to have a broken image/that only happened because a build got interrupted (ideally we'd be using cached images the majority of the time). Could we instead only have this run in a new "doctor" or "rebuild" command? I think an explicit "rebuild" command for agent images probably makes more sense. Push back/challenge whatever seems wrong and acknowledge what seems like good direction.

@didericis-claude I think it's pretty rare to have a broken image/that only happened because a build got interrupted (ideally we'd be using cached images the majority of the time). Could we instead only have this run in a new "doctor" or "rebuild" command? I think an explicit "rebuild" command for agent images probably makes more sense. Push back/challenge whatever seems wrong and acknowledge what seems like good direction.
didericis-claude reviewed 2026-07-13 15:07:46 -04:00
bot_bottle/backend/docker/launch.py
@@ -113,1 +114,4 @@
)
docker_mod.verify_agent_image(
plan.image, runtime_for(plan.agent_provider_template).smoke_test,
)
didericis-claude commented 2026-07-13 15:07:46 -04:00
Author
Collaborator
Copy Link
Copy Source

Two things worth separating:

Agree: A rebuild command is a good idea regardless — useful escape hatch when you want to force a fresh build without digging into --no-cache flags. Worth adding.

Push back on the framing: The root cause wasn't just an interrupted build. npm treats optional-dependency failures as non-fatal and exits 0, so a transient network hiccup during npm install -g can silently produce a broken binary on any build — then BuildKit caches that broken layer indefinitely. You wouldn't know until the agent tries to use its CLI. That's the exact failure mode in #353.

Also: the verify already only runs in the fresh-build branch. When a committed image is reused (the common case), it's skipped entirely. Moving verification to a doctor/rebuild command doesn't reduce per-launch overhead (there isn't any on cached launches) — it just shifts discovery from build time to first-real-use time, which is where we started.

I'd keep the verify on build and add the rebuild command on top.

Two things worth separating: **Agree:** A `rebuild` command is a good idea regardless — useful escape hatch when you want to force a fresh build without digging into `--no-cache` flags. Worth adding. **Push back on the framing:** The root cause wasn't just an interrupted build. npm treats optional-dependency failures as non-fatal and exits 0, so a transient network hiccup during `npm install -g` can silently produce a broken binary on any build — then BuildKit caches that broken layer indefinitely. You wouldn't know until the agent tries to use its CLI. That's the exact failure mode in #353. Also: the verify already only runs in the fresh-build branch. When a committed image is reused (the common case), it's skipped entirely. Moving verification to a `doctor`/`rebuild` command doesn't reduce per-launch overhead (there isn't any on cached launches) — it just shifts discovery from build time to first-real-use time, which is where we started. I'd keep the verify on build and add the `rebuild` command on top.
Some checks are pending
lint / lint (push) Successful in 1m58s
Details
test / unit (pull_request) Successful in 59s
Details
test / integration (pull_request) Successful in 18s
Details
test / coverage (pull_request) Successful in 59s
Details
This pull request can be merged automatically.
This branch is out-of-date with the base branch
You are not authorized to merge this pull request.
View command line instructions

Checkout

From your project repository, check out a new branch and test the changes.
git fetch -u origin post-build-image-smoke-test:post-build-image-smoke-test
git checkout post-build-image-smoke-test
Sign in to join this conversation.
Reviewers
No Reviewers
didericis
didericis-claude
Labels
Clear labels
Compat/Breaking
Breaking change that won't be backward compatible
 Kind/Bug
Something is not working
 Kind/Documentation
Documentation changes
 Kind/Enhancement
Improve existing functionality
 Kind/Feature
New functionality
 Kind/Security
This is security issue
 Kind/Testing
Issue or pull request related to testing
Priority
Critical
1
The priority is critical
Priority
High
2
The priority is high
Priority
Low
4
The priority is low
Priority
Medium
3
The priority is medium
Reviewed
Confirmed
1
Issue has been confirmed
Reviewed
Duplicate
2
This issue or pull request already exists
Reviewed
Invalid
3
Invalid issue
Reviewed
Won't Fix
3
This issue won't be fixed
Status
Abandoned
3
Somebody has started to work on this but abandoned work
Status
Blocked
1
Something is blocking this issue or pull request
Status
Need More Info
2
Feedback is required to reproduce issue or to continue work
No Label
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
didericis-claude (didericis (claude)) didericis-codex (didericis (codex)) didericis-gemma (gemma) didericis
No Assignees
2 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: didericis/bot-bottle#354
Delete Branch "post-build-image-smoke-test"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?