On a Firecracker gateway cold boot, reconcile every live agent VM against
the fresh gateway: push the new mitmproxy CA into each agent's trust store,
re-provision git-gate repos/creds from the persisted upstreams snapshot,
and restore egress tokens. Per-bottle failures are logged and skipped so
one unreachable VM does not block the rest.
New modules / changes:
- backend/firecracker/reconcile.py: attach_bottled_agents_to_gateway,
_push_ca, _reprovision_git_gate, _guest_ip_from_config
- git_gate/provision.py: write upstreams.json after key provisioning so
the bring-up reconcile can reconstruct the upstream table without the
manifest
- backend/firecracker/infra.py: call attach_bottled_agents_to_gateway in
the cold-boot branch of ensure_running()
- backend/base.py: no-op default on BottleBackend
- backend/firecracker/consolidated_launch.py: remove superseded
_reprovision_running_bottles / _guest_ip_from_config
- orchestrator: OrchestratorCore.update_agent_secret + POST
/bottles/<id>/secret + client.update_agent_secret (single-secret
in-place update, the reusable primitive from the 2026-07-26 hotfix)
- tests: 15 new tests in test_firecracker_reconcile.py; updated
test_firecracker_infra.py cold-boot cases; stale FC reprovision tests
removed from test_backend_secret_reprovision.py
Closes#516.
Agent images build with buildah inside the infra VM, and
`_build_in_infra` shipped only the Dockerfile into an empty `{ctx}/ctx`
context — on the documented assumption that agent Dockerfiles COPY
nothing from the context. The "pin & verify image inputs" work broke
that: the codex Dockerfile COPYs `codex-package_SHA256SUMS` and the
claude/pi Dockerfiles COPY their npm `package.json`/`package-lock.json`,
so any fresh agent image build fails with `COPY ...: no such file or
directory` (existing agents only survive on a rootfs cached from before
the change).
Parse each Dockerfile's context COPY sources, tar those committed files
from the build root into the VM-side `{ctx}/ctx`, and fold their content
into the rootfs cache digest so a repinned input rebuilds instead of
reusing stale bytes. `COPY --from=<stage>` and absolute/traversing
sources are excluded.
Closes#538.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The firecracker setup message told users to run
sudo bot-bottle backend setup --backend=firecracker
which fails for exactly the users who followed the documented install. sudo
replaces PATH with sudoers' secure_path — /usr/local/sbin:/usr/local/bin:
/usr/sbin:/usr/bin:/sbin:/bin on Debian and Ubuntu — which deliberately
excludes user-writable directories. Both supported install paths land in one:
pipx uses ~/.local/bin, and install.sh's venv fallback symlinks there. So the
hint works for anyone who installed system-wide and breaks with "command not
found" for everyone else, which is how it survives a read-through.
Add bot_bottle/invocation.py: self_path() resolves the running entry point to
an absolute path, and sudo_command() builds the copy-pasteable form. The one
sudo recommendation in the tree now uses it. Non-sudo hints keep the bare
`bot-bottle`, which is correct — the user reached them by running it.
self_path() falls back to the bare name when argv[0] cannot be resolved (a
`python -m` style invocation), because a slightly wrong hint beats a traceback
raised while reporting some unrelated problem.
Tested behaviourally rather than by scanning source: the first version of the
test grepped the module and failed on the comment explaining why the bare form
is wrong. It now drives _setup_systemd() as non-root and asserts what is
actually printed. Verified the guard bites by restoring the bare form and
watching it fail.
Not verified end to end: this message only prints on the systemd path, so it
is unreachable on macOS, where the rest of this work was tested.
`container image tag` only accepts image-name[:tag] as its source and
rejects a bare 64-hex image ID ("cannot specify 64 byte hex string as
reference"), so pinning the agent image for the nested-containers layer
died before the build could start. Tag from the ref instead; the
existing post-tag inspection is what keeps the handoff fail-closed if
the ref moves between the two commands.
The unit test mocked subprocess and asserted the ID-as-source call
shape, so it never saw the CLI's rejection. It now pins the ref as the
source, and covers the bare-hex ID that `container image inspect`
actually reports plus the mid-flight tag move.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
`doctor` told users to run `./cli.py backend setup`. cli.py is a four-line
wrapper at the repo root that calls bot_bottle.cli:main, and it does not ship —
[tool.setuptools.packages.find] includes only bot_bottle*, so anyone who
installed rather than cloned has no such file. Confirmed against a real
install: the user's tree contains exactly one executable, `bot-bottle`, and no
cli.py anywhere, while doctor recommended `./cli.py` three times. Invisible in
development, where ./cli.py works fine from a checkout, which is why only a
clean-install test surfaced it.
The two entry points are the same code, so the fix is to name the one that
always exists. 141 replacements across 45 files: the runtime messages that
caused this, plus README, docs, PRDs, research notes and test prose, so nothing
teaches the invocation a user cannot run.
scripts/demo.sh is deliberately untouched — it *executes* ./cli.py from a
checkout, where that is the correct and available path.
Verified end to end: a sandbox install now reports
"Run: bot-bottle backend setup --backend=firecracker", and bot-bottle is on
that user's PATH. Unit suite unchanged against the pre-existing baseline.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WEfZZhakx13bxTfXcZCoS5
The install itself now works end to end on a fresh macOS account — venv built,
package installed from git, entry point linked — and `doctor` then died with an
unhandled traceback:
PermissionError: [Errno 13] Permission denied: 'ip'
netpool's probe helpers caught only FileNotFoundError. That is not the only way
a probe binary can be unavailable: when a name on PATH exists but this user
cannot execute it, exec fails with EACCES, and CPython reports that in
preference to the ENOENT from the other PATH entries. So `except
FileNotFoundError` misses it and the crash propagates all the way out of
`doctor`. Reproduced directly: a mode-000 file named `ip` on PATH yields
exactly the error above.
_run_ok's docstring already stated the intent — treat an unavailable binary as
failure rather than crashing — so this widens the catch to OSError to match
what it says. Any OSError means the probe could not run, which for a
fail-closed check is indistinguishable from "not present". The same narrow
catch is fixed in overlapping_routes and in the two docker probes
(compose ls, docker ps), which are the same shape and equally reachable.
Deliberately not touched: the FileNotFoundError catches around file I/O in
bottle_state and orchestrator/service, where the narrow exception is correct.
The harness also required `bot-bottle` on PATH before running doctor, which
could never be true: install.sh prints the PATH line rather than editing a
shell profile, by design, so on a fresh account the entry point is installed
and working but not on PATH. It now looks where the installer actually puts
it (~/.local/bin, then the venv) and notes when it's running by absolute path.
That was the harness failing a run for a reason the installer intends.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WEfZZhakx13bxTfXcZCoS5
Address review: the self-heal classified every non-absence `network inspect`
failure as a poisoned network and force-removed the shared gateway. But
inspect also fails on transient daemon/API errors, permission failures,
timeouts, or a bad context — destroying a healthy gateway on that guess would
tear the network out from under every live bottle.
Now the destructive path runs only for the known poison signature (docker's
`ParseAddr` error from the malformed `::1/64` IPv6 gateway). The absent case
(`No such network`) still just creates; any other inspect failure raises a
clear GatewayError without mutating shared state.
Adds a regression test asserting a generic inspect error issues neither
`docker rm --force` nor `docker network rm` and surfaces the error.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
PR #515 stopped bot-bottle from *creating* a gateway network with a
malformed IPv6 subnet (--ipv6=false), but it can't recover a network that
is *already* poisoned. On a daemon that default-enables IPv6, the fixed-name
`bot-bottle-gateway` network gets an `fdd0::/64` subnet whose `::1/64`
gateway trips docker's own netip.ParseAddr, so `docker network inspect`/`ls`
exit non-zero — poisoning every command that reads networks.
Such a network can survive on a shared runner from a pre-fix or concurrent
launch. `_ensure_network` never healed it: its migrate/recreate branch only
ran when `network inspect` *succeeded*, but a poisoned network makes inspect
*fail*, so the code fell through to `network create`, which no-ops on
"already exists" — leaving the poison in place. The next subnet read then
failed with ConsolidatedLaunchError (test_multitenant_isolation), and a bare
`docker network ls` failed too (test_orphan_cleanup).
Fix, two parts:
- gateway.py: when `network inspect` fails for a reason other than "no such
network", treat the network as poisoned and force-remove + recreate it
IPv4-only. Absent-vs-poisoned is distinguished by the inspect stderr.
- test.yml: add an integration-docker preflight that drops the leftover
gateway network (and its container) before the suite, so direct `network
ls`/`inspect` calls in tests are clean even on a daemon where the in-code
heal can't run because inspect itself is what's broken.
Adds unit coverage for the poisoned-heal and the absent-create split.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
On a docker daemon that default-enables IPv6 (default-address-pools),
creating the gateway network with only `--subnet` lets the daemon also
attach an fdd0::/64 IPv6 subnet. Its gateway is stored as `::1/64`,
which trips docker's own netip.ParseAddr in `network inspect`/`ls`:
ParseAddr("fdd0:0:0:6::1/64"): unexpected character, want colon
That poisons every `_network_cidr`/`network ls` read and fails the
docker integration suite intermittently (whichever run the runner's
IPv6 pool index lands on a broken network). bot-bottle attribution
pins IPv4 source IPs and has no IPv6 support, so pass `--ipv6=false`
explicitly at network create to keep the gateway network IPv4-only
regardless of the daemon default.
Note: an already-poisoned runner still needs a one-time
`docker network rm bot-bottle-gateway` (and possibly a daemon
restart) to clear the malformed network.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
didericis-claude
didericis-codex
didericis
Gitea Actions