On a Firecracker gateway cold boot, reconcile every live agent VM against
the fresh gateway: push the new mitmproxy CA into each agent's trust store,
re-provision git-gate repos/creds from the persisted upstreams snapshot,
and restore egress tokens. Per-bottle failures are logged and skipped so
one unreachable VM does not block the rest.
New modules / changes:
- backend/firecracker/reconcile.py: attach_bottled_agents_to_gateway,
_push_ca, _reprovision_git_gate, _guest_ip_from_config
- git_gate/provision.py: write upstreams.json after key provisioning so
the bring-up reconcile can reconstruct the upstream table without the
manifest
- backend/firecracker/infra.py: call attach_bottled_agents_to_gateway in
the cold-boot branch of ensure_running()
- backend/base.py: no-op default on BottleBackend
- backend/firecracker/consolidated_launch.py: remove superseded
_reprovision_running_bottles / _guest_ip_from_config
- orchestrator: OrchestratorCore.update_agent_secret + POST
/bottles/<id>/secret + client.update_agent_secret (single-secret
in-place update, the reusable primitive from the 2026-07-26 hotfix)
- tests: 15 new tests in test_firecracker_reconcile.py; updated
test_firecracker_infra.py cold-boot cases; stale FC reprovision tests
removed from test_backend_secret_reprovision.py
Closes#516.
The firecracker setup message told users to run
sudo bot-bottle backend setup --backend=firecracker
which fails for exactly the users who followed the documented install. sudo
replaces PATH with sudoers' secure_path — /usr/local/sbin:/usr/local/bin:
/usr/sbin:/usr/bin:/sbin:/bin on Debian and Ubuntu — which deliberately
excludes user-writable directories. Both supported install paths land in one:
pipx uses ~/.local/bin, and install.sh's venv fallback symlinks there. So the
hint works for anyone who installed system-wide and breaks with "command not
found" for everyone else, which is how it survives a read-through.
Add bot_bottle/invocation.py: self_path() resolves the running entry point to
an absolute path, and sudo_command() builds the copy-pasteable form. The one
sudo recommendation in the tree now uses it. Non-sudo hints keep the bare
`bot-bottle`, which is correct — the user reached them by running it.
self_path() falls back to the bare name when argv[0] cannot be resolved (a
`python -m` style invocation), because a slightly wrong hint beats a traceback
raised while reporting some unrelated problem.
Tested behaviourally rather than by scanning source: the first version of the
test grepped the module and failed on the comment explaining why the bare form
is wrong. It now drives _setup_systemd() as non-root and asserts what is
actually printed. Verified the guard bites by restoring the bare form and
watching it fail.
Not verified end to end: this message only prints on the systemd path, so it
is unreachable on macOS, where the rest of this work was tested.
`doctor` told users to run `./cli.py backend setup`. cli.py is a four-line
wrapper at the repo root that calls bot_bottle.cli:main, and it does not ship —
[tool.setuptools.packages.find] includes only bot_bottle*, so anyone who
installed rather than cloned has no such file. Confirmed against a real
install: the user's tree contains exactly one executable, `bot-bottle`, and no
cli.py anywhere, while doctor recommended `./cli.py` three times. Invisible in
development, where ./cli.py works fine from a checkout, which is why only a
clean-install test surfaced it.
The two entry points are the same code, so the fix is to name the one that
always exists. 141 replacements across 45 files: the runtime messages that
caused this, plus README, docs, PRDs, research notes and test prose, so nothing
teaches the invocation a user cannot run.
scripts/demo.sh is deliberately untouched — it *executes* ./cli.py from a
checkout, where that is the correct and available path.
Verified end to end: a sandbox install now reports
"Run: bot-bottle backend setup --backend=firecracker", and bot-bottle is on
that user's PATH. Unit suite unchanged against the pre-existing baseline.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WEfZZhakx13bxTfXcZCoS5
Addresses the review on PR #481.
Self-contained wheel (review point 1): the gateway/infra/orchestrator
images build from a context that must hold bot_bottle/, pyproject.toml,
and the root-level Dockerfiles. Modules previously located these by
walking __file__ to the repo root, so an installed wheel (package in
site-packages, no repo root) passed `doctor` but failed `start`.
- Add bot_bottle/resources.py: build_root() returns the repo root in a
checkout (unchanged) or a staged copy from the wheel's bundled
_resources/ otherwise; dockerfile()/nix_netpool_module()/
netpool_script() derive from it.
- setup.py bundles the root Dockerfiles, nix module, netpool script, and
pyproject.toml into bot_bottle/_resources/ at build; MANIFEST.in ships
them in the sdist.
- Route every _REPO_ROOT/_REPO_DIR call site (docker/macos launch, macos
infra, firecracker infra_vm/infra_artifact/setup, orchestrator
lifecycle/gateway) through resources. Checkout behavior is unchanged.
install.sh prerequisites (review point 2): check for git when installing
a git+ spec, and — before the pip fallback — that pip is usable and the
interpreter isn't externally managed (PEP 668), pointing at pipx.
Tests: test_resources covers checkout + staged-wheel layouts;
test_wheel_install builds the wheel, installs it into an isolated venv,
and asserts `doctor` runs and build_root() yields a valid context.
Running `start` end-to-end still needs a Docker/KVM host (CI).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Two reviewer findings addressed:
1. _kvm_accessible() now opens /dev/kvm with O_RDWR|O_CLOEXEC instead
of read-only. VM creation requires write access; a read-only
descriptor can satisfy KVM_GET_API_VERSION but fails at boot time.
2. status() now checks every hard prerequisite that require_firecracker()
checks at launch: guest kernel image, static dropbear binary, and
mke2fs. Previously a host with a configured TAP pool but missing
artifacts could pass status() and then fail during launch.
Regression coverage: TestFirecrackerKvmCheck gains test_kvm_open_rdwr_fails;
TestFirecrackerArtifactCheck covers each missing artifact; existing
TestFirecrackerStatus fixtures updated to stub the new checks.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Adds `_firecracker_binary_ok()` (runs `firecracker --version`) and
`_kvm_accessible()` (opens /dev/kvm, issues KVM_GET_API_VERSION ioctl)
and gates `status()` on both, so the launch preflight reports missing
binary or inaccessible KVM before the caller ever attempts a boot.
Regression tests in TestFirecrackerBinaryCheck, TestFirecrackerKvmCheck,
and TestFirecrackerStatusRuntime cover all branches. Updated the
pre-existing TestFirecrackerStatus fixture to also stub the two new
helpers so it still exercises only the tap-pool/nft logic it was
designed for.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Make the network pool a persistent, distro-uniform resource instead of a
non-persistent per-distro shell command. systemd is the common
denominator across Debian/Ubuntu/Fedora/RHEL/Arch/… (and NixOS), so
`backend setup` on any systemd host now installs one bot-bottle-owned
oneshot unit (bot-bottle-firecracker-netpool.service): params pinned via
Environment= (so it doesn't depend on $SUDO_USER at boot), ExecStart/Stop
delegating to the bundled bring-up script (single source of logic).
- render_systemd_unit() in netpool.py (derived from the same constants
as the shell script + nix module).
- backend setup: install + enable the unit directly when run as root,
else print a self-contained copy-paste block. NixOS keeps its
declarative module (which produces the same-named unit); non-systemd
hosts fall back to the raw imperative script.
- backend teardown: symmetric — disable + remove the unit.
- backend status: report the unit's active/inactive state so you can
tell a persistent install from an imperative one.
Net: one install path (`backend setup`), persistent, identical across
distros; per-distro variance shrinks to installing nft + iproute2.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WBMWTEtQdJ4W5UrWuLHCck
`status()` hard-failed when it couldn't confirm the nft table, but listing
nftables usually needs root — so an unprivileged `backend status` reported
"not ready" even with the pool fully up, making it useless as a launch
gate (and skipping the firecracker integration test on a set-up host).
Base readiness on what the launch preflight actually hard-requires: the
TAP pool present (unprivileged, authoritative) + no range overlap. Report
the nft table state (present / unverified / not-confirmable-unprivileged)
but don't let it flip readiness — the post-boot isolation probe is the
authoritative isolation check, same as the preflight's deferral.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WBMWTEtQdJ4W5UrWuLHCck
The module used `networking.nftables.tables.*` (which forces
`networking.nftables.enable = true`, flipping the host firewall backend)
and `systemd.network.enable` (handing interfaces to systemd-networkd) —
disruptive on an iptables + Docker daily driver.
Rewrite it to a single systemd oneshot that brings the pool up: idempotent
`ip tuntap` for the TAPs + `nft -f` for the independent `inet bot_bottle_fc`
table (its own hooks at priority -10), with ExecStop teardown. Same as the
imperative script, but declarative. It touches neither the firewall backend
nor networkd, so it coexists with iptables/Docker/ufw/firewalld. Verified
through the NixOS module system (service present, firewall untouched).
Drop the now-redundant `render_nixos_module()` paste generator (the module
is a real importable file) and point `backend setup` at importing it (flake
output or the file path), noting it's non-invasive.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WBMWTEtQdJ4W5UrWuLHCck
`backend setup --backend=firecracker` only covered the network pool. Lead
with the other prerequisites: the firecracker binary (with a release
install pointer when it's not on PATH, plus a NixOS note), a /dev/kvm
check, and a reminder about the cached guest kernel + static dropbear and
mke2fs. The network pool is now step 2.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WBMWTEtQdJ4W5UrWuLHCck
Add an abstract teardown() classmethod to BottleBackend — the inverse of
setup(), surfaced as `./cli.py backend teardown [--backend=NAME]`
(uninstall). Symmetric with setup: it prints the privileged commands /
declarative config change to remove the host prerequisites.
- firecracker: NixOS-aware — disable the flake module (or drop the
import) and rebuild, or `firecracker-netpool.sh down` imperatively.
- docker / macos-container: nothing to undo (no privileged host state);
print a short note.
Not called by the launch path or the test suite. Extends test_cli_backend
for the new dispatch.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WBMWTEtQdJ4W5UrWuLHCck
Add abstract `setup()`/`status()` classmethods to BottleBackend (same
per-host, no-instance shape as is_available) so host provisioning is
part of the backend contract, not a per-backend command. Replace the
`./cli.py firecracker {setup,status}` command with a generic
`./cli.py backend {setup,status} [--backend=NAME]` that resolves a
backend (flag / $BOT_BOTTLE_BACKEND / host default) and dispatches —
swapping backends is just a different --backend.
Implementations:
- firecracker: moved out of cli/ into backend/firecracker/setup.py
(network pool module/script + range-overlap check), unchanged output.
- docker: new backend/docker/setup.py — reports docker on PATH, daemon
reachability, and gVisor runsc; setup notes no privileged pool is
needed. Minimal placeholder; richer version tracked in #345.
- macos-container: new setup.py — container CLI + system-service checks.
Also retarget the launch-preflight / isolation-probe pointers to the new
command. New test_cli_backend covers dispatch + docker setup/status.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WBMWTEtQdJ4W5UrWuLHCck
didericis-claude
didericis