didericis/bot-bottle
1
0
Fork 0
Code Issues 23 Pull Requests 7 Actions Packages Projects Releases Wiki Activity

feat(firecracker): Linux microVM backend to replace smolmachines #343

Open
didericis-claude wants to merge 13 commits from firecracker-backend into main
pull from: firecracker-backend
merge into: didericis:main
Conversation 0 Commits 13 Files Changed 103
+3725 -6170

13 Commits

Author SHA1 Message Date
didericis 049103ac99 docs: mark smolmachines-specific PRDs + research as superseded
test / unit (pull_request) Successful in 50s
Details
test / integration (pull_request) Successful in 17s
Details
test / coverage (pull_request) Failing after 1m4s
Details 
Add a "Superseded (2026-07-11)" banner and flip Status: Active →
Superseded on the 8 smolmachines-specific docs (7 PRDs + the VM-backend
research). Bodies are left intact as a historical record; the banner
points at the removal commit and the landscape doc so a reader isn't
misled into thinking the backend still exists.

Incidental mentions in other PRDs (git-gate, cred-proxy, print-parity,
etc.) are left as-is.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WBMWTEtQdJ4W5UrWuLHCck
 2026-07-11 19:09:43 -04:00
didericis f71558aff6 fix(firecracker): run the agent from /home/node, not the /root SSH cwd
lint / lint (push) Failing after 1m55s
Details
test / unit (pull_request) Successful in 53s
Details
test / integration (pull_request) Successful in 19s
Details
test / coverage (pull_request) Failing after 1m3s
Details 
The control SSH logs in as root, so the agent (dropped to node via
runuser) inherited cwd=/root. That was harmless while the rootless
rootfs left /root node-owned, but the dropbear fix (chown /root →
root:root 0700) made /root unreadable to node — so the agent's cwd was
inaccessible, breaking Node's process.cwd(), Claude Code's shell-snapshot
machinery, and `/doctor` (which reported /root/.claude EACCES and failed
every Bash command).

Run the agent from its workdir (default /home/node) via `env --chdir`.
Not a `sh -c 'cd … && exec "$@"'` wrapper: ssh space-joins everything
after the host into one string for the guest shell, so the quoted script
+ $@ get re-split and mangled (it exec'd the $0 placeholder → exit 127).
`env --chdir=DIR …` is all simple words, so it survives the join.

Verified end-to-end: a real headless claude bottle now runs its Bash tool
and exits 0 (was EACCES/127 before).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WBMWTEtQdJ4W5UrWuLHCck
 2026-07-11 18:57:33 -04:00
didericis a80356af75 feat(firecracker): portable systemd-unit install for the network pool
lint / lint (push) Failing after 2m5s
Details
test / unit (pull_request) Successful in 1m5s
Details
test / integration (pull_request) Successful in 22s
Details
test / coverage (pull_request) Failing after 1m5s
Details 
Make the network pool a persistent, distro-uniform resource instead of a
non-persistent per-distro shell command. systemd is the common
denominator across Debian/Ubuntu/Fedora/RHEL/Arch/… (and NixOS), so
`backend setup` on any systemd host now installs one bot-bottle-owned
oneshot unit (bot-bottle-firecracker-netpool.service): params pinned via
Environment= (so it doesn't depend on $SUDO_USER at boot), ExecStart/Stop
delegating to the bundled bring-up script (single source of logic).

- render_systemd_unit() in netpool.py (derived from the same constants
  as the shell script + nix module).
- backend setup: install + enable the unit directly when run as root,
  else print a self-contained copy-paste block. NixOS keeps its
  declarative module (which produces the same-named unit); non-systemd
  hosts fall back to the raw imperative script.
- backend teardown: symmetric — disable + remove the unit.
- backend status: report the unit's active/inactive state so you can
  tell a persistent install from an imperative one.

Net: one install path (`backend setup`), persistent, identical across
distros; per-distro variance shrinks to installing nft + iproute2.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WBMWTEtQdJ4W5UrWuLHCck
 2026-07-11 18:33:42 -04:00
didericis 3d7c508dc4 fix(firecracker): make the launch path work end-to-end
lint / lint (push) Failing after 2m2s
Details
test / unit (pull_request) Successful in 56s
Details
test / integration (pull_request) Successful in 24s
Details
test / coverage (pull_request) Failing after 1m11s
Details 
First real end-to-end launch (there was no firecracker integration test)
surfaced three bugs in the control/provision path, all now verified fixed
by tests/integration/test_firecracker_launch:

1. Guest SSH rejected the correct key. The rootless rootfs build
   (`docker export | tar` as a non-root user) can't preserve uid 0, so
   every path — including /root — is owned by the build uid (node in
   guest). dropbear refuses root's authorized_keys when /root isn't
   root-owned, so auth fell back to password → denied. bb-init now
   `chown -R 0:0 /root`.

2. The SSH client (newer OpenSSH) didn't reliably present the -i key
   against the operator's ~/.ssh config; add `IdentitiesOnly=yes` to
   ssh_base_argv so only the per-bottle key is offered.

3. cp_in double-wrapped the remote command as `sh -c <remote>`, but ssh
   space-joins everything after the host into one string for the guest
   shell, collapsing `sh -c mkdir -p X && …` to `mkdir` with no operand
   ("missing operand"). Pass the command as a single arg and let the
   guest login shell run it (stdin still carries the tar).

Also stop discarding dropbear's stderr (`-E` now reaches the host-side
console.log) so future guest-auth issues are debuggable.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WBMWTEtQdJ4W5UrWuLHCck
 2026-07-11 18:16:57 -04:00
didericis f42a0dc7fe fix(firecracker): status() defers unverifiable nft, like the preflight 
`status()` hard-failed when it couldn't confirm the nft table, but listing
nftables usually needs root — so an unprivileged `backend status` reported
"not ready" even with the pool fully up, making it useless as a launch
gate (and skipping the firecracker integration test on a set-up host).

Base readiness on what the launch preflight actually hard-requires: the
TAP pool present (unprivileged, authoritative) + no range overlap. Report
the nft table state (present / unverified / not-confirmable-unprivileged)
but don't let it flip readiness — the post-boot isolation probe is the
authoritative isolation check, same as the preflight's deferral.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WBMWTEtQdJ4W5UrWuLHCck
 2026-07-11 17:28:17 -04:00
didericis 480269b116 refactor(firecracker): non-invasive NixOS module (no firewall switch) 
The module used `networking.nftables.tables.*` (which forces
`networking.nftables.enable = true`, flipping the host firewall backend)
and `systemd.network.enable` (handing interfaces to systemd-networkd) —
disruptive on an iptables + Docker daily driver.

Rewrite it to a single systemd oneshot that brings the pool up: idempotent
`ip tuntap` for the TAPs + `nft -f` for the independent `inet bot_bottle_fc`
table (its own hooks at priority -10), with ExecStop teardown. Same as the
imperative script, but declarative. It touches neither the firewall backend
nor networkd, so it coexists with iptables/Docker/ufw/firewalld. Verified
through the NixOS module system (service present, firewall untouched).

Drop the now-redundant `render_nixos_module()` paste generator (the module
is a real importable file) and point `backend setup` at importing it (flake
output or the file path), noting it's non-invasive.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WBMWTEtQdJ4W5UrWuLHCck
 2026-07-11 17:25:50 -04:00
didericis 949b001464 feat(firecracker): show binary/KVM prerequisites in backend setup
lint / lint (push) Failing after 1m58s
Details
test / unit (pull_request) Successful in 57s
Details
test / integration (pull_request) Successful in 19s
Details
test / coverage (pull_request) Failing after 1m6s
Details 
`backend setup --backend=firecracker` only covered the network pool. Lead
with the other prerequisites: the firecracker binary (with a release
install pointer when it's not on PATH, plus a NixOS note), a /dev/kvm
check, and a reminder about the cached guest kernel + static dropbear and
mke2fs. The network pool is now step 2.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WBMWTEtQdJ4W5UrWuLHCck
 2026-07-11 17:03:04 -04:00
didericis 656966c2c4 feat(backend): add teardown() to the contract; backend teardown
lint / lint (push) Failing after 1m56s
Details
test / unit (pull_request) Successful in 52s
Details
test / integration (pull_request) Successful in 17s
Details
test / coverage (pull_request) Failing after 1m2s
Details 
Add an abstract teardown() classmethod to BottleBackend — the inverse of
setup(), surfaced as `./cli.py backend teardown [--backend=NAME]`
(uninstall). Symmetric with setup: it prints the privileged commands /
declarative config change to remove the host prerequisites.

- firecracker: NixOS-aware — disable the flake module (or drop the
  import) and rebuild, or `firecracker-netpool.sh down` imperatively.
- docker / macos-container: nothing to undo (no privileged host state);
  print a short note.

Not called by the launch path or the test suite. Extends test_cli_backend
for the new dispatch.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WBMWTEtQdJ4W5UrWuLHCck
 2026-07-11 16:56:17 -04:00
didericis 15f0e0b507 test(integration): unblock sandbox-escape; add firecracker launch smoke 
Sandbox-escape couldn't run: its git-gate fixture had no host_key, so the
preflight ssh-keyscanned the deliberately-unreachable upstream and die()d
in setUpClass. Preset a throwaway host_key so the keyscan is skipped (the
key is never used — the push is rejected by gitleaks first).

That unblocked a second, pre-existing issue: the planted secrets weren't
caught by gitleaks (the AWS example key is allowlisted; the others hit
entropy/keyword gates in the keyword-free URL the attack embeds). Reshape
the fixtures — three structural, high-entropy shapes gitleaks matches
without a keyword (github / slack / gitlab) for the git-push attack, and
a separate alphanumeric secret for the DNS attack (a gitleaks-matchable
token carries separators that aren't valid DNS labels, so the two uses
can't share one secret). All five sandbox-escape tests now pass.

Add test_firecracker_launch: a launch smoke (exec + proxy env) gated on
`FirecrackerBottleBackend.status() == 0` (the `backend status` result),
skipping with setup instructions when the TAP pool / nft table aren't
provisioned.

The git-gate-only-matches-gitleaks-patterns asymmetry the reshape exposed
is tracked separately (#346).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WBMWTEtQdJ4W5UrWuLHCck
 2026-07-11 16:55:59 -04:00
didericis dd2e83b8a9 refactor(backend): generic setup/status; drop firecracker-only CLI
lint / lint (push) Failing after 2m1s
Details
test / unit (pull_request) Successful in 1m3s
Details
test / integration (pull_request) Successful in 23s
Details
test / coverage (pull_request) Failing after 1m11s
Details 
Add abstract `setup()`/`status()` classmethods to BottleBackend (same
per-host, no-instance shape as is_available) so host provisioning is
part of the backend contract, not a per-backend command. Replace the
`./cli.py firecracker {setup,status}` command with a generic
`./cli.py backend {setup,status} [--backend=NAME]` that resolves a
backend (flag / $BOT_BOTTLE_BACKEND / host default) and dispatches —
swapping backends is just a different --backend.

Implementations:
- firecracker: moved out of cli/ into backend/firecracker/setup.py
  (network pool module/script + range-overlap check), unchanged output.
- docker: new backend/docker/setup.py — reports docker on PATH, daemon
  reachability, and gVisor runsc; setup notes no privileged pool is
  needed. Minimal placeholder; richer version tracked in #345.
- macos-container: new setup.py — container CLI + system-service checks.

Also retarget the launch-preflight / isolation-probe pointers to the new
command. New test_cli_backend covers dispatch + docker setup/status.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WBMWTEtQdJ4W5UrWuLHCck
 2026-07-11 15:18:40 -04:00
didericis ce3fad9320 feat(firecracker): move pool off CGNAT, add overlap guard + flake module 
The default TAP-pool base was 100.64.0.0/10 (RFC-6598 CGNAT) — chosen to
dodge RFC-1918, but that's exactly the range Tailscale assigns node
addresses from, so on a Tailscale host it's the worst pick. Move the
default to 10.243.0.0/16, an obscure RFC-1918 block that steers clear of
docker/libvirt/k8s/LAN and Tailscale.

No default is collision-proof, so add netpool.overlapping_routes(): it
parses `ip -json route show table all` and flags any route intersecting
the pool range (excluding our own bbfc* TAPs and the default route). The
launch preflight warns on overlap; `backend status` reports it.

Distribute the NixOS host setup as a flake module instead of a
copy-pasted blob: nix/firecracker-netpool.nix computes the taps / nft
table from typed options (poolSize, ipBase, ifacePrefix, owner) with a
/31-alignment assertion, and flake.nix exposes it as
nixosModules.firecracker-netpool. Defaults mirror the backend constants;
writeEnvFile emits the matching BOT_BOTTLE_FC_* so the host pool and the
launcher can't drift.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WBMWTEtQdJ4W5UrWuLHCck
 2026-07-11 15:18:21 -04:00
didericis c07ebca867 feat(backend): remove smolmachines; firecracker is the Linux default
lint / lint (push) Successful in 1m54s
Details
test / unit (pull_request) Successful in 58s
Details
test / integration (pull_request) Successful in 18s
Details
test / coverage (pull_request) Failing after 1m3s
Details 
Delete the smolmachines backend (the whole bot_bottle/backend/smolmachines
package and its tests). It had fatal Linux issues (TSI networking under
sustained use, exec-channel contention, no SIGWINCH) and is superseded by
the Firecracker backend (issue #342).

Backend selection now:
- default is macos-container on macOS, firecracker on KVM-capable Linux
  hosts, and docker as the last resort (was smolmachines).
- firecracker is selected on a KVM host even when the `firecracker`
  binary isn't installed, so start routes through its preflight and
  prints an install pointer (same UX as require_container), instead of
  silently falling back. Split is_host_capable() (Linux + KVM) out of
  is_available() (adds the binary check) to drive this.

Retarget the cross-backend tests (parity, print-parity, prepare,
workspace, freezer, selection) from smolmachines to firecracker rather
than dropping the coverage. Remove docker.util.image_id/save, which only
smolmachines used. Update README/AGENTS/example bottles and stale
comments; historical docs/prds are left as a point-in-time record.

BREAKING: BOT_BOTTLE_BACKEND=smolmachines now errors as unknown.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WBMWTEtQdJ4W5UrWuLHCck
 2026-07-11 13:24:47 -04:00
didericis c276f7b0b1 feat(firecracker): add Linux microVM backend to replace smolmachines
lint / lint (push) Successful in 2m4s
Details
test / unit (pull_request) Successful in 59s
Details
test / integration (pull_request) Successful in 18s
Details
test / coverage (pull_request) Failing after 1m7s
Details 
Adds a Firecracker-based backend for Linux, providing mature KVM-based
microVM isolation to replace smolmachines/libkrun (issue #342, closes
the dead-end tracked in #332).

Architecture:
- Guest control over SSH (dropbear injected into the rootfs) on a
  point-to-point TAP link. `ssh -t` forwards SIGWINCH natively, so no
  resize bridge is needed.
- Networking: a one-time, root-provisioned pool of user-owned TAP
  devices (no shared bridge → no docker0/virbr0/cni0 collisions) plus a
  dedicated `table inet bot_bottle_fc` nftables table (independent of
  Docker/ufw/firewalld rules). `./cli.py firecracker setup` prints the
  host-appropriate config (NixOS module or sudo script).
- Rootfs: `docker export` → ext4 via `mke2fs -d` (rootless, no mount),
  cached by image digest; per-bottle SSH pubkey + IP passed via the
  kernel cmdline.
- Sidecar: reuses the Docker bundle, published on the slot's host TAP IP.
- Fail-closed isolation: TAP pool verified at preflight; the egress
  boundary is proven empirically post-boot (before the agent runs) by a
  canary probe — the VM must fail to reach the host directly, or launch
  is refused.

Linux hosts with Firecracker + KVM now default to this backend;
macOS stays on macos-container.

Not yet validated end-to-end on live hardware (requires the one-time
network pool). Unit tests + pyright pass.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01G8p32HJgPoS1hLPWubbftM
 2026-07-11 10:32:55 -04:00