Make the network pool a persistent, distro-uniform resource instead of a
non-persistent per-distro shell command. systemd is the common
denominator across Debian/Ubuntu/Fedora/RHEL/Arch/… (and NixOS), so
`backend setup` on any systemd host now installs one bot-bottle-owned
oneshot unit (bot-bottle-firecracker-netpool.service): params pinned via
Environment= (so it doesn't depend on $SUDO_USER at boot), ExecStart/Stop
delegating to the bundled bring-up script (single source of logic).
- render_systemd_unit() in netpool.py (derived from the same constants
as the shell script + nix module).
- backend setup: install + enable the unit directly when run as root,
else print a self-contained copy-paste block. NixOS keeps its
declarative module (which produces the same-named unit); non-systemd
hosts fall back to the raw imperative script.
- backend teardown: symmetric — disable + remove the unit.
- backend status: report the unit's active/inactive state so you can
tell a persistent install from an imperative one.
Net: one install path (`backend setup`), persistent, identical across
distros; per-distro variance shrinks to installing nft + iproute2.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WBMWTEtQdJ4W5UrWuLHCck
First real end-to-end launch (there was no firecracker integration test)
surfaced three bugs in the control/provision path, all now verified fixed
by tests/integration/test_firecracker_launch:
1. Guest SSH rejected the correct key. The rootless rootfs build
(`docker export | tar` as a non-root user) can't preserve uid 0, so
every path — including /root — is owned by the build uid (node in
guest). dropbear refuses root's authorized_keys when /root isn't
root-owned, so auth fell back to password → denied. bb-init now
`chown -R 0:0 /root`.
2. The SSH client (newer OpenSSH) didn't reliably present the -i key
against the operator's ~/.ssh config; add `IdentitiesOnly=yes` to
ssh_base_argv so only the per-bottle key is offered.
3. cp_in double-wrapped the remote command as `sh -c <remote>`, but ssh
space-joins everything after the host into one string for the guest
shell, collapsing `sh -c mkdir -p X && …` to `mkdir` with no operand
("missing operand"). Pass the command as a single arg and let the
guest login shell run it (stdin still carries the tar).
Also stop discarding dropbear's stderr (`-E` now reaches the host-side
console.log) so future guest-auth issues are debuggable.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WBMWTEtQdJ4W5UrWuLHCck
`status()` hard-failed when it couldn't confirm the nft table, but listing
nftables usually needs root — so an unprivileged `backend status` reported
"not ready" even with the pool fully up, making it useless as a launch
gate (and skipping the firecracker integration test on a set-up host).
Base readiness on what the launch preflight actually hard-requires: the
TAP pool present (unprivileged, authoritative) + no range overlap. Report
the nft table state (present / unverified / not-confirmable-unprivileged)
but don't let it flip readiness — the post-boot isolation probe is the
authoritative isolation check, same as the preflight's deferral.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WBMWTEtQdJ4W5UrWuLHCck
The module used `networking.nftables.tables.*` (which forces
`networking.nftables.enable = true`, flipping the host firewall backend)
and `systemd.network.enable` (handing interfaces to systemd-networkd) —
disruptive on an iptables + Docker daily driver.
Rewrite it to a single systemd oneshot that brings the pool up: idempotent
`ip tuntap` for the TAPs + `nft -f` for the independent `inet bot_bottle_fc`
table (its own hooks at priority -10), with ExecStop teardown. Same as the
imperative script, but declarative. It touches neither the firewall backend
nor networkd, so it coexists with iptables/Docker/ufw/firewalld. Verified
through the NixOS module system (service present, firewall untouched).
Drop the now-redundant `render_nixos_module()` paste generator (the module
is a real importable file) and point `backend setup` at importing it (flake
output or the file path), noting it's non-invasive.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WBMWTEtQdJ4W5UrWuLHCck
`backend setup --backend=firecracker` only covered the network pool. Lead
with the other prerequisites: the firecracker binary (with a release
install pointer when it's not on PATH, plus a NixOS note), a /dev/kvm
check, and a reminder about the cached guest kernel + static dropbear and
mke2fs. The network pool is now step 2.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WBMWTEtQdJ4W5UrWuLHCck
Add an abstract teardown() classmethod to BottleBackend — the inverse of
setup(), surfaced as `./cli.py backend teardown [--backend=NAME]`
(uninstall). Symmetric with setup: it prints the privileged commands /
declarative config change to remove the host prerequisites.
- firecracker: NixOS-aware — disable the flake module (or drop the
import) and rebuild, or `firecracker-netpool.sh down` imperatively.
- docker / macos-container: nothing to undo (no privileged host state);
print a short note.
Not called by the launch path or the test suite. Extends test_cli_backend
for the new dispatch.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WBMWTEtQdJ4W5UrWuLHCck
Add abstract `setup()`/`status()` classmethods to BottleBackend (same
per-host, no-instance shape as is_available) so host provisioning is
part of the backend contract, not a per-backend command. Replace the
`./cli.py firecracker {setup,status}` command with a generic
`./cli.py backend {setup,status} [--backend=NAME]` that resolves a
backend (flag / $BOT_BOTTLE_BACKEND / host default) and dispatches —
swapping backends is just a different --backend.
Implementations:
- firecracker: moved out of cli/ into backend/firecracker/setup.py
(network pool module/script + range-overlap check), unchanged output.
- docker: new backend/docker/setup.py — reports docker on PATH, daemon
reachability, and gVisor runsc; setup notes no privileged pool is
needed. Minimal placeholder; richer version tracked in #345.
- macos-container: new setup.py — container CLI + system-service checks.
Also retarget the launch-preflight / isolation-probe pointers to the new
command. New test_cli_backend covers dispatch + docker setup/status.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WBMWTEtQdJ4W5UrWuLHCck
The default TAP-pool base was 100.64.0.0/10 (RFC-6598 CGNAT) — chosen to
dodge RFC-1918, but that's exactly the range Tailscale assigns node
addresses from, so on a Tailscale host it's the worst pick. Move the
default to 10.243.0.0/16, an obscure RFC-1918 block that steers clear of
docker/libvirt/k8s/LAN and Tailscale.
No default is collision-proof, so add netpool.overlapping_routes(): it
parses `ip -json route show table all` and flags any route intersecting
the pool range (excluding our own bbfc* TAPs and the default route). The
launch preflight warns on overlap; `backend status` reports it.
Distribute the NixOS host setup as a flake module instead of a
copy-pasted blob: nix/firecracker-netpool.nix computes the taps / nft
table from typed options (poolSize, ipBase, ifacePrefix, owner) with a
/31-alignment assertion, and flake.nix exposes it as
nixosModules.firecracker-netpool. Defaults mirror the backend constants;
writeEnvFile emits the matching BOT_BOTTLE_FC_* so the host pool and the
launcher can't drift.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WBMWTEtQdJ4W5UrWuLHCck
Delete the smolmachines backend (the whole bot_bottle/backend/smolmachines
package and its tests). It had fatal Linux issues (TSI networking under
sustained use, exec-channel contention, no SIGWINCH) and is superseded by
the Firecracker backend (issue #342).
Backend selection now:
- default is macos-container on macOS, firecracker on KVM-capable Linux
hosts, and docker as the last resort (was smolmachines).
- firecracker is selected on a KVM host even when the `firecracker`
binary isn't installed, so start routes through its preflight and
prints an install pointer (same UX as require_container), instead of
silently falling back. Split is_host_capable() (Linux + KVM) out of
is_available() (adds the binary check) to drive this.
Retarget the cross-backend tests (parity, print-parity, prepare,
workspace, freezer, selection) from smolmachines to firecracker rather
than dropping the coverage. Remove docker.util.image_id/save, which only
smolmachines used. Update README/AGENTS/example bottles and stale
comments; historical docs/prds are left as a point-in-time record.
BREAKING: BOT_BOTTLE_BACKEND=smolmachines now errors as unknown.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WBMWTEtQdJ4W5UrWuLHCck
Adds a Firecracker-based backend for Linux, providing mature KVM-based
microVM isolation to replace smolmachines/libkrun (issue #342, closes
the dead-end tracked in #332).
Architecture:
- Guest control over SSH (dropbear injected into the rootfs) on a
point-to-point TAP link. `ssh -t` forwards SIGWINCH natively, so no
resize bridge is needed.
- Networking: a one-time, root-provisioned pool of user-owned TAP
devices (no shared bridge → no docker0/virbr0/cni0 collisions) plus a
dedicated `table inet bot_bottle_fc` nftables table (independent of
Docker/ufw/firewalld rules). `./cli.py firecracker setup` prints the
host-appropriate config (NixOS module or sudo script).
- Rootfs: `docker export` → ext4 via `mke2fs -d` (rootless, no mount),
cached by image digest; per-bottle SSH pubkey + IP passed via the
kernel cmdline.
- Sidecar: reuses the Docker bundle, published on the slot's host TAP IP.
- Fail-closed isolation: TAP pool verified at preflight; the egress
boundary is proven empirically post-boot (before the agent runs) by a
canary probe — the VM must fail to reach the host directly, or launch
is refused.
Linux hosts with Firecracker + KVM now default to this backend;
macOS stays on macos-container.
Not yet validated end-to-end on live hardware (requires the one-time
network pool). Unit tests + pyright pass.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01G8p32HJgPoS1hLPWubbftM
When a git-gate repo entry has no host_key configured, the prepare step
now either errors (headless) or fetches the key via ssh-keyscan, shows
it to the operator for confirmation, and optionally persists it to the
bottle config .md file on disk.
Closes#333
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
On Linux the guest kernel's LOCAL routing table routes all
127.0.0.0/8 to the guest's own loopback interface (priority 0,
checked before any main-table route), so TSI never sees
connections to the per-bottle loopback alias — the fix_guest_
loopback_routing approach confirmed this at the kernel level.
Use the per-bottle docker bridge gateway (192.168.N.1) instead.
It is not a loopback address, so the guest routes it via eth0
and TSI intercepts it normally. The TSI allowlist remains a
/32 that is distinct from the container IP (192.168.N.2), so
direct bypass to egress:9099 is still blocked by TSI.
Changes:
- Add _proxy_host() helper: returns bundle_gateway on Linux,
loopback alias on macOS
- Thread proxy_host through _start_bundle, _discover_urls,
_launch_vm, and _bundle_launch_spec (publish_host_ip)
- Remove _fix_guest_loopback_routing (no longer needed)
- Relax proxy-URL assertion in the integration test to accept
any http://IP:port (with a comment explaining the difference)
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Port the smolmachines backend so BOT_BOTTLE_BACKEND=smolmachines
works on Linux (KVM), not just macOS:
- Preflight gates /dev/kvm presence + accessibility on Linux with
actionable remediation (kvm module, kvm group).
- smolvm state-DB path is platform-derived (XDG on Linux).
- force_allowlist runs on both platforms and is fail-closed: it
verifies the persisted TSI allowlist and dies rather than booting
a VM whose egress confinement it can't confirm. Previously it
no-oped on Linux, failing OPEN.
- allocate() does per-bottle 127.0.0.<N> scoping on Linux too (no
ifconfig needed — all of 127/8 is already loopback); only
ensure_pool's lo0 aliasing stays macOS-only.
- README documents Linux + NixOS host setup.
Linux/KVM integration (the sandbox-escape acceptance gate) is
pending verification on a NixOS host; unit tests cover the new
platform branches.
Issue: #283
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NkwFXLFff9PYPy4wgVBJp9
Apple's `container run --mount type=bind` only accepts directory
sources — a file source fails with "path '<file>' is not a
directory". Move the sqlite db into its own ~/.bot-bottle/db/
subdirectory so it can be bind-mounted the same way the CA/routes
mounts already are, without exposing the rest of ~/.bot-bottle.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
- `bottle:` in agent frontmatter is now optional; agents without it
are portable and require bottles to be selected at launch.
- Adds `filter_multiselect` to `tui.py`: multi-select picker with
ordered selection list, Space/Enter to toggle, Ctrl-D to confirm.
- `ManifestIndex` gains `all_bottle_names` and `load_for_agent` accepts
`bottle_names: tuple[str, ...]` to merge bottles in order at runtime.
- `merge_bottles_runtime` in `manifest_extends.py` applies the same
field-merge rules as `extends:` to pre-resolved bottle objects.
- `BottleSpec` gains `bottle_names`; `_validate` and `write_launch_metadata`
thread it through so `resume` replays the same bottle configuration.
- `cmd_start` shows the bottle multiselect after agent selection,
pre-populated from the agent's `bottle:` field when present.
- Existing agents with `bottle:` declared continue to work unchanged.
`container build` resolves -f relative to the current working directory,
not the build context, so builds failed from any cwd other than the repo
root. Anchor a relative Dockerfile to the context before passing it.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
EgressPlan gains a `canary: str` field (default "") populated in Egress.prepare()
using secrets.token_urlsafe(32). Each launched bottle:
- sidecar receives EGRESS_TOKEN_CANARY=<value> (literal env entry, scanned by
existing known-secrets detector without any detector code changes)
- agent receives BOT_BOTTLE_CANARY=<value> (visible fake secret that signals
exfiltration with zero false positives if it appears in outbound traffic)
Docker compose and macos-container backends updated; smolmachines shares docker
compose and so picks this up automatically. Unit tests cover canary uniqueness,
detection via scan_known_secrets, and EgressPlan backward-compat default.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
On resume from a committed snapshot, smolvm's pack process remaps all
file uids to the host uid (501 on macOS). Files in /tmp that were
created during the session (e.g. /tmp/claude-1000 owned by node=uid
1000) get remapped to 501. Claude Code then refuses to use the temp
directory because it's owned by a different uid.
Two-part fix:
- Exclude ./tmp and ./var/tmp from the tar in _exec_tar_to_file.
Both directories are ephemeral; a resumed VM should start with clean
temp directories identical to a fresh VM.
- Add mkdir -p /tmp /var/tmp to _init_vm before chown/chmod, so the
directories are created if the committed snapshot omitted them.
Replace the Popen/stdout=PIPE approach with a write-then-copy
strategy that avoids binary-stdout piping through the smolvm exec
channel entirely:
1. Probe connectivity with `machine_exec(machine, ["true"])` first.
If this fails while an interactive session is running, the error
now says "concurrent exec not available" instead of the opaque
"<no stderr>".
2. Run `tar --create --gzip --file=/var/tmp/.bot-bottle-commit.tar.gz`
inside the VM via machine_exec (same mechanism used during
provisioning). tar writes to a file in the VM, not stdout, so
smolvm never has to transmit binary data over the exec channel.
3. Copy the compressed archive to the host with machine_cp.
4. Dockerfile switches to ADD rootfs.tar.gz / — Docker decompresses
gzip tarballs automatically.
smolvm machine exec requires stdout to be a pipe, not a regular
file descriptor. Passing stdout=file caused smolvm to return
non-zero with no stderr (the error was silently swallowed or went
to the regular-file fd instead of reaching us).
Switch _snapshot_running_vm to a new _exec_tar_to_file helper that
uses Popen with stdout=PIPE and streams the tar to disk via
shutil.copyfileobj. A background thread drains stderr concurrently
to prevent deadlock when the stderr pipe buffer fills while we are
writing stdout data.
The terminal-decoration wrapper script is invoked with sh -lc, which
sources login-shell init files (/etc/profile, ~/.profile) rather than
interactive-shell files (~/.zshrc). smolvm is typically installed via
homebrew whose PATH setup lands in ~/.zprofile or ~/.zshrc — not picked
up by sh -l — so pty_resize.py's Popen(["smolvm", ...]) raises
FileNotFoundError, pty_resize exits non-zero, and the trailing reset-
printf makes sh exit 0. The caller sees "session ended (exit 0)"
immediately with no agent output.
Use sh -c instead. The calling process (./cli.py) inherits the user's
interactive shell PATH where smolvm is present, confirmed by the
provision steps (machine_exec) succeeding before exec_agent is reached.
smolvm pack create --from-vm requires the VM to be stopped, and stopping
a smolmachines VM terminates any running interactive session.
Instead, mirror the macos-container approach: exec into the running VM as
root and stream the root filesystem via tar (smolvm machine exec -- tar),
build a Docker image from the archive, push to an ephemeral local registry,
and run smolvm pack create --image to produce the .smolmachine artifact.
The VM stays running throughout the commit.
Remove the stop-confirm prompt and machine_is_running check that were
added in the previous commit — neither is needed when we no longer stop.
smolvm pack create --from-vm requires the VM to be stopped. Add
machine_is_running() to smolvm.py (via machine ls --json state field),
and add the same confirm-stop flow to SmolmachinesFreezer that was
originally designed for macos-container: if running, prompt the user,
stop the VM, then pack. Already-stopped VMs are packed directly.
Apple Container removes containers when they stop, making the
stop-then-export flow impossible regardless of the --rm flag.
Replace `container export` (requires stopped container) with
`container exec --user root <name> tar --create ... --file=- --directory=/ .`
streamed to a temp file, then build the committed image from that archive
as before. The bottle stays running after commit, which is better UX.
Drop the stop-confirm prompt from MacosContainerFreezer since we no longer
need to stop the container at all.
container stop was removing the container immediately (due to --rm)
before container export could run. The force_remove_container teardown
callback on the ExitStack already handles cleanup on normal exit, so
--rm was redundant. Without it, the stopped container stays available
for container export to snapshot.
Freezer._freeze only ever used bottle.name, which is always
f"bot-bottle-{agent.slug}". Remove the Bottle parameter from
commit() and _freeze(), derive the container name from agent.slug
directly in each subclass, and delete the _NamedBottle stub that
existed solely to paper over this.
Adds a Freezer ABC (backend/freeze.py) that encapsulates the
stop-commit-mark-preserved flow for all backends, following the same
pattern as BottleBackend. Each backend gets its own Freezer subclass:
DockerFreezer — docker commit
MacosContainerFreezer — container export + image rebuild; prompts
to stop if the container is running
SmolmachinesFreezer — smolvm pack create --from-vm
The base class owns write_committed_image, mark_preserved, and the
resume hint. Subclasses implement _freeze() and optionally override
_export_hint() for migration instructions.
Freezer.commit(agent, bottle) is the primary entry point for use
within a live launch context. Freezer.commit_slug(slug) is a
convenience wrapper for cmd_commit, which no longer branches on
backend names itself.
get_freezer(backend_name) is the factory, analogous to
get_bottle_backend(). CommitCancelled is raised by MacosContainerFreezer
when the user declines the stop prompt; cmd_commit catches it and
returns 0.
`container export` requires the container to be stopped first. When a
running bottle is detected, prompt the user to confirm, stop the
container, then commit. Adds `container_is_running` and
`stop_container` helpers to the macos-container util.
Addresses #240 (comment)
Adds `./cli.py commit [<slug>]` which runs `docker commit` on the
active agent container and stores the resulting image tag in per-bottle
state. The next `./cli.py resume <slug>` automatically boots from the
committed snapshot instead of rebuilding from the Dockerfile, preserving
all in-container state across restarts and migrations.
- bottle_state: add write_committed_image / read_committed_image helpers
- docker/util: add commit_container wrapper around `docker commit`
- docker/launch: check for a committed image before the Dockerfile build
step; fall back to normal build if the image is absent from the daemon
- cli/commit: new command with interactive slug picker; errors clearly on
non-Docker backends
- 50 new unit tests covering all paths
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Replace module-level apply_routes_change wrappers with a public
applicator singleton in each backend. Callers now work with the
EgressApplicator instance directly (applicator.apply_routes_change)
rather than through a function shim.
Pulls the duplicated apply_routes_change / validate_routes_content /
_routes_path logic into EgressApplicator (ABC) in backend/egress_apply.py.
DockerEgressApplicator and MacOSContainerEgressApplicator override the
single abstract _signal_bundle_reload method with their respective kill
commands. Module-level shims preserve the existing public API.
- Set http_proxy/https_proxy (lowercase) alongside uppercase variants in smolmachines guest env for tools that only check lowercase
- Replace dataclasses.asdict with route_to_yaml_dict in /allowlist introspection so returned routes use YAML-schema-compatible keys
- Expand routes_yaml tool description in supervise_server to document all accepted route keys, making the round-trip from list-egress-routes to propose/apply explicit
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Mirrors the fix already applied to the macos-container backend in
eb3e64e: bind-mount the parent egress directory instead of the
routes file itself, so the live routes update is visible inside the
running sidecar bundle when the host overwrites the file.
didericis
didericis-claude
didericis-codex