didericis/bot-bottle
1
0
Fork 0
Code Issues 25 Pull Requests 9 Actions Packages Projects Releases Wiki Activity

feat(firecracker): consolidated per-host gateway launch (PRD 0070, part of #348) #354

Open
didericis-claude wants to merge 6 commits from firecracker-consolidated-launch into main
pull from: firecracker-consolidated-launch
merge into: didericis:main
Conversation 3 Commits 6 Files Changed 19
+716 -42
didericis-claude commented 2026-07-13 04:04:34 -04:00
Collaborator
Copy Link
Copy Source

Part of #348 (PRD 0069 — Firecracker-native, Docker-free backend). Closes #353.

Consolidated Firecracker launch (PRD 0070)

Re-enables the Firecracker backend on the shared per-host gateway instead of the per-bottle companion container that the de-sidecar work removed. This is the source-IP-keyed persistent data plane that stage (1) of #348 calls for: one gateway per host, shared by every microVM, rather than one docker run per bottle.

  • firecracker/consolidated_launch.py (new) mirrors docker/consolidated_launch.py but wired for Firecracker's TAP topology: it ensures a Firecracker-flavoured orchestrator + gateway are up, registers the bottle by its guest IP (the attribution key), provisions the bottle's git-gate state into the running gateway, and returns the shared gateway CA for the provisioner to install in the rootfs.
  • firecracker/launch.py is rebuilt around that sequence. The per-bottle sidecar bundle is gone; VMs reach the shared gateway at their slot's host-side TAP IP, where Docker's PREROUTING DNAT + the nft table's ct status dnat accept forward rule route the traffic. Attribution is unspoofable by construction (the /31 point-to-point TAP + the bot_bottle_fc nft table guarantee only the expected VM can source-IP its guest address).
  • orchestrator/lifecycle.py and orchestrator/gateway.py grow small, additive knobs so both backends can run on one host without collision: OrchestratorService takes orchestrator_name/orchestrator_label and exposes a _gateway() override hook; DockerGateway takes host_port_bindings so the Firecracker gateway can publish its ports on the host for VMs to reach over their TAP link. The Firecracker orchestrator uses bot-bottle-fc-* container names and the heavy gateway data-plane image (_gateway_image, kept distinct from the lean control-plane image after the #384 split).

The TAP slot allocation, rootfs build, and VM boot are unchanged; only the data-plane wiring moved from per-bottle sidecar to shared gateway.

Post-build agent-image smoke test (Closes #353)

npm treats optionalDependencies failures as non-fatal, so a transient network blip during npm install -g can leave a stub CLI in an image that still docker build-exits 0 — then gets baked into the Docker layer cache indefinitely (this is exactly the broken claude-code native-binary image in #353). A provider-declared smoke test, run in a throwaway container of the freshly built image, fails the launch loudly at build time instead of at first agent use.

  • docker/util.py gains docker_exec, docker_cp, and verify_agent_image helpers plus --no-cache support in build_image, all built on the shared run_docker primitive.
  • docker/launch.py, firecracker/launch.py, and macos_container/util.py run verify_agent_image after every fresh agent-image build.
  • cli/start.py adds --no-cache to force a from-scratch agent-image rebuild (the escape hatch when a broken layer is already cached).
Part of #348 (PRD 0069 — Firecracker-native, Docker-free backend). Closes #353. ## Consolidated Firecracker launch (PRD 0070) Re-enables the Firecracker backend on the **shared per-host gateway** instead of the per-bottle companion container that the de-sidecar work removed. This is the source-IP-keyed persistent data plane that stage (1) of #348 calls for: one gateway per host, shared by every microVM, rather than one `docker run` per bottle. - `firecracker/consolidated_launch.py` (new) mirrors `docker/consolidated_launch.py` but wired for Firecracker's TAP topology: it ensures a Firecracker-flavoured orchestrator + gateway are up, registers the bottle by its **guest IP** (the attribution key), provisions the bottle's git-gate state into the running gateway, and returns the shared gateway CA for the provisioner to install in the rootfs. - `firecracker/launch.py` is rebuilt around that sequence. The per-bottle sidecar bundle is gone; VMs reach the shared gateway at their slot's host-side TAP IP, where Docker's PREROUTING DNAT + the nft table's `ct status dnat accept` forward rule route the traffic. Attribution is unspoofable by construction (the /31 point-to-point TAP + the `bot_bottle_fc` nft table guarantee only the expected VM can source-IP its guest address). - `orchestrator/lifecycle.py` and `orchestrator/gateway.py` grow small, additive knobs so both backends can run on one host without collision: `OrchestratorService` takes `orchestrator_name`/`orchestrator_label` and exposes a `_gateway()` override hook; `DockerGateway` takes `host_port_bindings` so the Firecracker gateway can publish its ports on the host for VMs to reach over their TAP link. The Firecracker orchestrator uses `bot-bottle-fc-*` container names and the heavy gateway data-plane image (`_gateway_image`, kept distinct from the lean control-plane image after the #384 split). The TAP slot allocation, rootfs build, and VM boot are unchanged; only the data-plane wiring moved from per-bottle sidecar to shared gateway. ## Post-build agent-image smoke test (Closes #353) npm treats `optionalDependencies` failures as non-fatal, so a transient network blip during `npm install -g` can leave a stub CLI in an image that still `docker build`-exits 0 — then gets baked into the Docker layer cache indefinitely (this is exactly the broken `claude-code` native-binary image in #353). A provider-declared smoke test, run in a throwaway container of the freshly built image, fails the launch loudly at build time instead of at first agent use. - `docker/util.py` gains `docker_exec`, `docker_cp`, and `verify_agent_image` helpers plus `--no-cache` support in `build_image`, all built on the shared `run_docker` primitive. - `docker/launch.py`, `firecracker/launch.py`, and `macos_container/util.py` run `verify_agent_image` after every fresh agent-image build. - `cli/start.py` adds `--no-cache` to force a from-scratch agent-image rebuild (the escape hatch when a broken layer is already cached).
didericis force-pushed firecracker-consolidated-launch from bfdbae804b to e862b82322 2026-07-13 04:08:43 -04:00 Compare
didericis reviewed 2026-07-13 14:59:23 -04:00
bot_bottle/backend/docker/launch.py Outdated
@@ -113,1 +114,4 @@
)
docker_mod.verify_agent_image(
plan.image, runtime_for(plan.agent_provider_template).smoke_test,
)
didericis commented 2026-07-13 14:59:23 -04:00
Owner
Copy Link
Copy Source

This requires spinning up a new container, just for the verification step?

This requires spinning up a new container, just for the verification step?
didericis marked this conversation as resolved
didericis-claude reviewed 2026-07-13 15:00:48 -04:00
bot_bottle/backend/docker/launch.py Outdated
@@ -113,1 +114,4 @@
)
docker_mod.verify_agent_image(
plan.image, runtime_for(plan.agent_provider_template).smoke_test,
)
didericis-claude commented 2026-07-13 15:00:48 -04:00
Author
Collaborator
Copy Link
Copy Source

Yes — it runs docker run --rm with just the smoke-test argv (e.g. claude --version), so the container exits in under a second and is immediately removed. The overhead is intentional: the root bug was that docker build exits 0 even when npm silently fails to install the optional binary, so the only reliable way to catch a broken image is to actually run it.

Yes — it runs `docker run --rm` with just the smoke-test argv (e.g. `claude --version`), so the container exits in under a second and is immediately removed. The overhead is intentional: the root bug was that `docker build` exits 0 even when npm silently fails to install the optional binary, so the only reliable way to catch a broken image is to actually run it.
didericis commented 2026-07-13 15:06:41 -04:00
Owner
Copy Link
Copy Source

@didericis-claude I think it's pretty rare to have a broken image/that only happened because a build got interrupted (ideally we'd be using cached images the majority of the time). Could we instead only have this run in a new "doctor" or "rebuild" command? I think an explicit "rebuild" command for agent images probably makes more sense. Push back/challenge whatever seems wrong and acknowledge what seems like good direction.

@didericis-claude I think it's pretty rare to have a broken image/that only happened because a build got interrupted (ideally we'd be using cached images the majority of the time). Could we instead only have this run in a new "doctor" or "rebuild" command? I think an explicit "rebuild" command for agent images probably makes more sense. Push back/challenge whatever seems wrong and acknowledge what seems like good direction.
didericis-claude reviewed 2026-07-13 15:07:46 -04:00
bot_bottle/backend/docker/launch.py Outdated
@@ -113,1 +114,4 @@
)
docker_mod.verify_agent_image(
plan.image, runtime_for(plan.agent_provider_template).smoke_test,
)
didericis-claude commented 2026-07-13 15:07:46 -04:00
Author
Collaborator
Copy Link
Copy Source

Two things worth separating:

Agree: A rebuild command is a good idea regardless — useful escape hatch when you want to force a fresh build without digging into --no-cache flags. Worth adding.

Push back on the framing: The root cause wasn't just an interrupted build. npm treats optional-dependency failures as non-fatal and exits 0, so a transient network hiccup during npm install -g can silently produce a broken binary on any build — then BuildKit caches that broken layer indefinitely. You wouldn't know until the agent tries to use its CLI. That's the exact failure mode in #353.

Also: the verify already only runs in the fresh-build branch. When a committed image is reused (the common case), it's skipped entirely. Moving verification to a doctor/rebuild command doesn't reduce per-launch overhead (there isn't any on cached launches) — it just shifts discovery from build time to first-real-use time, which is where we started.

I'd keep the verify on build and add the rebuild command on top.

Two things worth separating: **Agree:** A `rebuild` command is a good idea regardless — useful escape hatch when you want to force a fresh build without digging into `--no-cache` flags. Worth adding. **Push back on the framing:** The root cause wasn't just an interrupted build. npm treats optional-dependency failures as non-fatal and exits 0, so a transient network hiccup during `npm install -g` can silently produce a broken binary on any build — then BuildKit caches that broken layer indefinitely. You wouldn't know until the agent tries to use its CLI. That's the exact failure mode in #353. Also: the verify already only runs in the fresh-build branch. When a committed image is reused (the common case), it's skipped entirely. Moving verification to a `doctor`/`rebuild` command doesn't reduce per-launch overhead (there isn't any on cached launches) — it just shifts discovery from build time to first-real-use time, which is where we started. I'd keep the verify on build and add the `rebuild` command on top.
didericis marked this conversation as resolved
didericis force-pushed firecracker-consolidated-launch from add4c53f43 to f47957d847 2026-07-14 03:57:07 -04:00 Compare
didericis-claude changed title from Smoke-test agent images after build, add start --no-cache to feat(firecracker): consolidated orchestrator launch (PRD 0070) 2026-07-14 03:57:25 -04:00
didericis force-pushed firecracker-consolidated-launch from f47957d847 to 06a920f08d 2026-07-14 04:38:46 -04:00 Compare
didericis force-pushed firecracker-consolidated-launch from 06a920f08d to 79616a6460 2026-07-14 05:09:08 -04:00 Compare
didericis added 4 commits 2026-07-14 17:31:57 -04:00
fix: smoke-test agent images after build, add start --no-cache 1d3e9e859c 
npm treats optionalDependencies failures as non-fatal, so a transient
network blip fetching claude-code's platform-native binary during
`npm install -g` left a stub CLI in an image that still "built"
successfully — then got baked into the Docker/Container layer cache
until forced to rebuild. Post-build smoke test (provider-declared
argv, run in a throwaway container of the freshly built image) fails
the launch loudly instead of shipping a broken image; --no-cache
gives an escape hatch to force a from-scratch rebuild.

Closes #353.
refactor(firecracker): use docker_mod instead of hand-rolled docker helpers 3c426f45f5 
firecracker/launch.py reimplemented docker build/image-exists/rm/exec/cp
as private functions instead of the shared docker_mod used by the
docker and macos-container backends. Switching to docker_mod dedupes
the logic and gets --no-cache support for free (docker_mod.build_image
already reads BOT_BOTTLE_NO_CACHE); docker_mod gains docker_exec/
docker_cp general-purpose helpers to cover what the private versions did.
feat(firecracker): implement consolidated orchestrator launch (PRD 0070) 64ca39ddd4 
Replace the per-bottle Docker sidecar bundle with the shared per-host
orchestrator + gateway, mirroring what the Docker backend already has.

- Add `bot_bottle/backend/firecracker/consolidated_launch.py`:
  `_FirecrackerOrchestratorService` (subclasses `OrchestratorService`,
  overrides `_gateway()` to return a `DockerGateway` with host port
  bindings so Firecracker VMs can reach it via their TAP link);
  `launch_consolidated()` registers the bottle by guest IP (attribution
  key), provisions git-gate into the shared gateway, and returns the
  shared CA + orchestrator URL for teardown; `teardown_consolidated()`
  deregisters and cleans up.

- Rewrite `bot_bottle/backend/firecracker/launch.py`: removes the
  per-bottle sidecar bundle (`_start_sidecar_bundle`, `_stage_git_gate`,
  etc.) and `_mint_certs`; wires `launch_consolidated()` instead. The VM
  still sends to `host_tap_ip:PORT` — Docker's PREROUTING DNAT + the nft
  `ct status dnat accept` rule in the forward chain route the traffic to
  the shared gateway container.

- Extend `DockerGateway` with `host_port_bindings` so the Firecracker
  gateway publishes its ports on the host (`0.0.0.0:PORT`).

- Parameterise `OrchestratorService` with `orchestrator_name` /
  `orchestrator_label` so Docker and Firecracker orchestrators can
  coexist on the same host (`bot-bottle-orchestrator` vs
  `bot-bottle-fc-orchestrator`).

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
fix(docker): use run_docker in docker_exec, docker_cp, verify_agent_image
lint / lint (push) Successful in 2m12s
Details
test / unit (pull_request) Successful in 1m7s
Details
test / integration (pull_request) Successful in 24s
Details
test / coverage (pull_request) Successful in 1m16s
Details
0a1f949937 
The rebase onto lazy-backend-imports converts existing helpers (image_exists,
container_exists, etc.) to run_docker; the three new functions added in this
branch still called subprocess.run directly. Switch them over for consistency.
didericis force-pushed firecracker-consolidated-launch from 69cbf82e6d to 0a1f949937 2026-07-14 17:31:57 -04:00 Compare
didericis-claude changed title from feat(firecracker): consolidated orchestrator launch (PRD 0070) to feat(firecracker): consolidated per-host gateway launch (PRD 0070, part of #348) 2026-07-14 17:32:04 -04:00
didericis added 1 commit 2026-07-14 18:03:42 -04:00
feat(orchestrator): add buildah for in-VM agent-image builds (PRD 0069 Stage 3)
test / unit (pull_request) Successful in 1m16s
Details
test / integration (pull_request) Successful in 26s
Details
test / coverage (pull_request) Successful in 1m23s
Details
bab44dbe97 
The Firecracker backend's remaining host-Docker dependency is building
users' agent Dockerfiles. Move that build *into the orchestrator VM*:
give the orchestrator image buildah (rootless, daemonless) so it builds
agent images itself, and the host needs no Docker daemon and no
root-equivalent `docker` group. An untrusted Dockerfile then builds
inside the confined orchestrator VM rather than on the host — strictly
more isolated than host `docker build`.

- buildah + crun (OCI runtime) + netavark/aardvark-dns (network backend
  for FROM pulls and RUN egress), installed explicitly since
  --no-install-recommends strips buildah's helper deps.
- vfs storage + chroot isolation (STORAGE_DRIVER/BUILDAH_ISOLATION) so
  buildah needs neither fuse-overlayfs / an overlay kernel module nor
  configured subuid maps — it works unconditionally as root in a minimal
  microVM rootfs. (Slower than overlay; a build-cache pass is deferred.)

Bootstrap: the orchestrator rootfs is still produced from this image via
host docker export; a later step pulls a pre-built image instead.

Verified: the orchestrator image boots as a Firecracker VM and buildah
runs inside it as real VM root (reaches the base-image pull; build-time
egress wiring is the next step).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WBMWTEtQdJ4W5UrWuLHCck
didericis added 1 commit 2026-07-14 18:13:45 -04:00
feat(firecracker): NAT'd egress link for the orchestrator/builder VM
lint / lint (push) Successful in 2m22s
Details
test / unit (pull_request) Successful in 1m20s
Details
test / integration (pull_request) Successful in 29s
Details
test / coverage (pull_request) Successful in 1m27s
Details
95981ea9d3 
The orchestrator/gateway VM is trusted infra, not an isolated agent: it
builds agent images in-VM (buildah must FROM-pull + apt/npm) and, in the
Stage B cutover, forwards agent egress upstream. Give it a dedicated TAP
(`bborch0`) on a /31 at the top of the IP_BASE /16 (clear of the bbfc*
agent pool at the bottom), NAT'd out the host uplink — while agent VMs
keep their fail-closed, gateway-only isolation table.

- netpool.defaults.env / netpool.py: new BOT_BOTTLE_FC_ORCH_IFACE +
  `orch_slot()` (index -1 sentinel; host x.y.255.0 / guest x.y.255.1).
- scripts/firecracker-netpool.sh: create + address the orchestrator TAP;
  `bot_bottle_fc_nat` table masquerades its /31 out the uplink and
  accepts its forward path. Because bootstrap still runs Docker (whose
  FORWARD policy is DROP), a best-effort, guarded, idempotent DOCKER-USER
  ACCEPT is added too (skipped once Docker is gone). down/status updated.
- nix/firecracker-netpool.nix: mirror the option, pass it via the unit
  Environment= (the store-copied script can't read the defaults file),
  and add iptables to the unit path for the DOCKER-USER step.

Agent isolation is unchanged: the new rules only ever accept/masquerade
the orchestrator link and never drop, so they can't weaken the bbfc*
drops. Applied by re-running `sudo ./scripts/firecracker-netpool.sh up`.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WBMWTEtQdJ4W5UrWuLHCck
Some checks are pending
lint / lint (push) Successful in 2m22s
Details
test / unit (pull_request) Successful in 1m20s
Details
test / integration (pull_request) Successful in 29s
Details
test / coverage (pull_request) Successful in 1m27s
Details
This pull request can be merged automatically.
This branch is out-of-date with the base branch
You are not authorized to merge this pull request.
View command line instructions

Checkout

From your project repository, check out a new branch and test the changes.
git fetch -u origin firecracker-consolidated-launch:firecracker-consolidated-launch
git checkout firecracker-consolidated-launch
Sign in to join this conversation.
Reviewers
No Reviewers
didericis
didericis-claude
Labels
Clear labels
Compat/Breaking
Breaking change that won't be backward compatible
 Kind/Bug
Something is not working
 Kind/Documentation
Documentation changes
 Kind/Enhancement
Improve existing functionality
 Kind/Feature
New functionality
 Kind/Security
This is security issue
 Kind/Testing
Issue or pull request related to testing
Priority
Critical
1
The priority is critical
Priority
High
2
The priority is high
Priority
Low
4
The priority is low
Priority
Medium
3
The priority is medium
Reviewed
Confirmed
1
Issue has been confirmed
Reviewed
Duplicate
2
This issue or pull request already exists
Reviewed
Invalid
3
Invalid issue
Reviewed
Won't Fix
3
This issue won't be fixed
Status
Abandoned
3
Somebody has started to work on this but abandoned work
Status
Blocked
1
Something is blocking this issue or pull request
Status
Need More Info
2
Feedback is required to reproduce issue or to continue work
No Label
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
didericis-claude (didericis (claude)) didericis-codex (didericis (codex)) didericis-gemma (gemma) didericis
No Assignees
2 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: didericis/bot-bottle#354
Delete Branch "firecracker-consolidated-launch"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?