FROM-scratch images (commit_container output) and registry images built
for reproducibility often omit the created field entirely. Both backends
were calling die() in that case, crashing the cached-image quickstart
before any container started.
image_created_at now returns datetime | None — None when the field is
absent or unparseable, die() only on real inspect failures (non-zero
exit, malformed JSON). stale_checks in both backends skips the staleness
check when None is returned.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
pyright strict reportUnusedImport flagged BottleImages in
test_docker_launch_committed_image.py and test_macos_container_launch.py;
neither file references the type by name (they only use the returned
value's attributes).
`from __future__ import annotations` already defers all annotation
evaluation, so quoting `str | Path`, `BottleImages` inside the same
module was redundant and tripped pyright strict mode.
Addresses review comments 3098, 3099, 3100 on PR #336:
- Add BottleImages(agent, sidecar) dataclass to backend/__init__.py.
Docker/macOS backends use str image refs; smolmachines uses Path
artifacts. Replaces the singular `image` variable from the canonical
pattern in comment 3100.
- Replace _image_stale_checks/skip_stale with public prelaunch_checks().
CLI now calls backend.prelaunch_checks(plan) before backend.launch(plan);
if StaleImageError is raised and the operator confirms, launch proceeds
without re-checking. Removes the while-True/skip_stale retry loop.
- Add abstract _build_or_load_images(plan) -> BottleImages to
BottleBackend. launch() calls it then passes images to _launch_impl.
Each backend implements both methods.
- Fix comment 3098 (macos-container): _build_images is removed.
build_or_load_images() has separate fresh/cached code paths — the
cached path never calls a build helper.
- Update _start_bundle (smolmachines) to accept sidecar_artifact: Path
directly. Sidecar artifact resolution moves to _sidecar_from_path(),
called by build_or_load_images alongside _agent_from_path().
- image_cache: StaleImageError exception + check_stale/check_stale_path (raise instead of warn)
- BottleBackend.launch: template method (skip_stale flag) that calls _image_stale_checks then _launch_impl
- Each backend: _image_stale_checks delegates to a stale_checks() function in its launch module; _launch_impl replaces launch override
- macos_container: adds image_created_at to util, cached-image support in _build_images, stale_checks
- cli/start.py: catches StaleImageError, prompts interactively, retries with skip_stale=True; headless mode dies on it
_auto_select_backend gains a prompt parameter (default True). When
prompt=False the docker-fallback [i/d/q] menu is skipped and the call
dies immediately with an actionable message ("set
BOT_BOTTLE_BACKEND=docker or install a VM backend"), preventing hangs
in CI, webhook dispatch, and orchestrator launches.
prepare_with_preflight passes prompt=not spec.headless so the headless
start path can never block waiting for TTY input it cannot receive.
Remove the private _read_tty_line duplicate from backend/__init__.py and
the local definition from cli/_common.py. Both now import from the
shared bot_bottle.util module.
- Remove hardcoded --backend=macos-container flag reference in
firecracker/util.py require_firecracker() error message
- Remove --backend flag from cli.py start; backend selection now
driven exclusively by BOT_BOTTLE_BACKEND env var or auto-selection
- Skip unavailable backends in cli.py cleanup (fixes crash on Linux
when macos-container.prepare_cleanup calls require_container())
- Add two-tier auto-selection: VM backend first (macos-container on
macOS, firecracker on Linux+KVM); fall back to docker with a
security warning and interactive i/d/q prompt; exit if docker
also unavailable and print VM install instructions
Closes#344
The macOS infra container's init script still spawned the supervisor as
`python3 /app/gateway_init.py`. 5ad3449 moved bot_bottle from flat files
under /app into a pip-installed package, so that file no longer exists:
the supervisor never started, mitmdump never generated its CA, and the
launch failed downstream with a misleading "gateway CA not available"
error pointing at TLS rather than at the supervisor.
This is the same defect 127ba49 fixed for the firecracker backend. It
survived on macOS because CI has no Apple Container runner — the KVM
runner added in c193b04 covers firecracker, and ubuntu-latest covers
docker, but nothing exercises macos_container.
test_macos_infra asserted the substring "gateway_init.py", which the
broken path satisfied; it now asserts the module form, matching the
assertion 127ba49 already made in test_firecracker_infra_vm.
Verified end to end on the macos-container backend: image builds, CA is
generated, agent registers, and a headless bottle runs a prompt.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Review follow-ups for the infra candidate-artifact flow:
1. publish-infra would fail on the first merge to main. The rootfs version now
hashes the dropbear bytes, and build-infra sets BOT_BOTTLE_FC_DROPBEAR to the
staged dropbear — but publish-infra (ubuntu-latest) set none, so _publish_
bundle re-derived the version with a "<missing>" dropbear and rejected the
candidate as "does not match checkout". Download the same firecracker-inputs
dropbear and export BOT_BOTTLE_FC_DROPBEAR in publish-infra (and add
stage-firecracker-inputs to its needs, since it now consumes that artifact).
2. Guard stage-firecracker-inputs with the same fork-PR check as the other
KVM-runner jobs, so a fork PR can't spin the privileged runner (it only
copies a static binary, but keep the posture consistent; it gates the whole
Firecracker chain via needs).
3. ensure_artifact_gz / _publish_bundle read version.txt before checking it
exists — a bundle missing it raised a raw FileNotFoundError instead of the
intended "bundle is incomplete" die. Guard the read (kept before the gz/sha
completeness check so a wrong-version bundle still reports the version
mismatch, not "incomplete").
4. test_infra_artifact.setUp didn't isolate BOT_BOTTLE_INFRA_ARTIFACT_DIR, so
an ambient candidate dir (the coverage job exports one) would send the
registry-pull tests down the local-bundle path. Pin it off in setUp; the
candidate-path cases set it explicitly.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01A9qa3xoavjQScufDfZaXKR
Dockerfile.gateway COPYs pyproject.toml into /src and runs pip install
/src, so it is a real input to the baked rootfs. A dependency-only change
previously reused stale artifact versions, potentially booting a rootfs
whose installed packages differed from the current checkout.
Also adds _fake_repo fixture support and a regression test so this input
can't silently drop out of the hash again.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
build_agent_rootfs_dir cached the built rootfs by Dockerfile content alone,
but util.inject_guest_boot then writes util._GUEST_INIT into it. So a fix to
the init — making /tmp world-writable (1777) so the agent can create scratch
dirs / git worktrees there — did NOT bust the cache: the KVM runner kept
reusing a stale agent-<dockerfilehash> rootfs built with the old init, and the
sandbox-escape README-push test kept failing at `git init /tmp/...` with
"Permission denied".
Key the cache on Dockerfile content AND the injected init (_rootfs_digest), so
an init change rebuilds. Self-busting: the new key yields a fresh cache dir, so
no manual cache clear on the runner.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01A9qa3xoavjQScufDfZaXKR
The infra VM is a per-host singleton that outlives short-lived launchers, so
`ensure_running` adopts it when its control plane is healthy. But it adopted
ANY healthy VM regardless of the code that built it — so after an infra-code
change the old VM kept being adopted and the new code never booted. The only
way to dislodge it was an out-of-band `kill`, which then raced whatever
launched next. On CI this meant every infra change needed a manual VM kill.
Make adoption version-aware:
* boot records the infra-artifact version it booted from in a `booted-version`
marker beside the singleton; `stop` clears it.
* `ensure_running` adopts only when the marker matches the current version
(`_adoptable`); a missing/mismatched marker falls through to stop + reboot.
So a stale VM is replaced automatically on the next launch — no manual kill,
and it's concurrency-safe (the reboot happens under the singleton flock).
Also harden teardown: the PID file drifts after crashes / out-of-band kills,
so `stop` now also reaps any orphaned firecracker still bound to the infra
config path (scoped to that path, so interactive-pool VMs are untouched) —
otherwise a survivor holds the orchestrator TAP and the fresh boot dies with
"tap … Resource busy".
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01A9qa3xoavjQScufDfZaXKR
The rootless agent rootfs build can land /tmp as 0755/root-owned, so the
agent (uid 1000 node) can't create scratch dirs there. The sandbox-escape
suite's README-push test does `cd /tmp && git init sandbox-escape-repo` and
died with "cannot mkdir sandbox-escape-repo: Permission denied" — before the
git-gate gitleaks hook could run — so the test read it as a missing hook.
On docker the agent inherits node:22-slim's 1777 /tmp, which is why only the
Firecracker path was affected (and only now that the suite runs end-to-end).
Set /tmp to 1777 in the guest PID-1 init, so every agent VM boots with a
usable /tmp regardless of rootfs perm drift.
Also update the infra-init unit test to assert the gateway launches via the
`bot_bottle.gateway_init` module (matching the prior fix), not a file path.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01A9qa3xoavjQScufDfZaXKR
The infra VM's init boots the orchestrator control plane AND the gateway data
plane (egress/git-http/supervise). Since 5ad3449 moved the daemons into the
installed `bot_bottle` package, there is no `/app/gateway_init.py` file — the
gateway image's entrypoint is `python3 -m bot_bottle.gateway_init`. But the
firecracker init still spawned the old file path, so the guest logged:
python3: can't open file '/app/gateway_init.py': No such file or directory
The control plane came up (it already used `python3 -m bot_bottle.orchestrator`)
but the gateway never started, so mitmproxy never generated its CA and launch
died with "gateway CA not available after 30s". Mirror the orchestrator line:
run the gateway as `python3 -m bot_bottle.gateway_init` (bot_bottle resolves
from the /app CWD, same as the control plane; egress-entrypoint.sh /
egress_addon.py are present from the gateway base image).
Third build/runtime bug from 5ad3449's package refactor that the Firecracker
integration suite never exercised (the artifact publish was broken, so the
job 404'd before boot). Changes the init, so the infra artifact version moves;
the matching rootfs has been rebuilt and published.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01A9qa3xoavjQScufDfZaXKR
On a self-hosted KVM runner the process has a real controlling terminal
so name_color_modal successfully opens /dev/tty and enters a curses
loop waiting for keyboard input, hanging the test indefinitely.
Docker containers (ubuntu-latest runners) don't have a real /dev/tty,
causing an OSError that triggers the existing fallback — this is why
the hang was invisible in ubuntu-latest CI.
Also add timeout=5 to _daemon_reachable() to match the same defensive
fix already applied to docker_available() in tests/_docker.py.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Rename _BACKENDS → _backends: pyright treats uppercase module-level
names as constants and flags the reassignment in _get_backends() as
reportConstantRedefinition; lowercase avoids this.
- Add TYPE_CHECKING guard importing CommitCancelled/Freezer/get_freezer
from .freeze: pyright cannot see module-level __getattr__ bindings, so
reportUnsupportedDunderAll fired for those three __all__ entries; the
guard makes them visible to the type checker without running at import
time.
- Update test_backend_selection.py to patch _backends (lowercase).
`undefined-all-variable` fires on CommitCancelled / Freezer / get_freezer
in __all__ because pylint can't see module-level __getattr__ bindings;
`global-statement` fires on the _BACKENDS singleton setter. Both are
intentional patterns — add inline disables rather than suppress globally.
Importing backend.docker.util previously triggered eager loading of all
three backend packages (~76 modules) because backend/__init__.py imported
DockerBottleBackend, FirecrackerBottleBackend, and MacosContainerBottleBackend
at module scope. This made the module prohibitively expensive to import
from the orchestrator layer and elsewhere.
The three backend imports are now deferred into _get_backends(), which
loads all three on first call and caches the result in the module-level
_BACKENDS variable (initially None). Module-level __getattr__ exposes
backend classes and freeze symbols lazily for existing import/patch sites.
backend/docker/util.py raw subprocess.run(["docker", ...]) calls are
replaced with the shared run_docker primitive from docker_cmd, eliminating
the duplication between the backend and orchestrator implementations.
_silent_run() is removed; image_exists() is inlined directly onto
run_docker. The commit_container test is updated to patch run_docker
instead of subprocess.run.
Cloning/fetching from the git-gate on the Apple-container backend failed with
"empty reply from server" (curl exit 52). Root cause: the git-http handler
crashed on every upload-pack with
PermissionError: [Errno 13] Permission denied: '/etc/git-gate/access-hook'
The access-hook is exec'd directly, so it needs the x bit. prepare() stages it
0o700 and trusted the gateway copy to carry that mode. `docker cp` does; the
Apple `container cp` (AppleGatewayTransport) does not, landing the hook 0o644 →
EACCES. The unhandled exception killed the handler thread, closing the socket
with no HTTP response — which the client sees as the opaque empty reply.
- provision_git_gate now `chmod +x`es the access-hook on the gateway side after
the copy, so it's executable under every transport (docker/apple/firecracker).
- git-http handler wraps the access-hook subprocess.run: an OSError /
SubprocessError (un-execable, timed out) now fails closed with a 503 instead
of crashing the thread into an empty reply — a gate that can't run its hook
should deny, visibly.
- Updates the now-misleading "docker cp preserves source mode" comment in
git_gate.prepare().
Regression tests: provisioning applies +x to the access-hook; the handler
returns 503 (not an empty reply) when the hook can't be exec'd.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The control plane had no caller authentication — dispatch() routed every
request straight through, and the only guards anywhere were the
(source_ip, identity_token) checks inside /resolve and /attribute, which
authenticate the *bottle* a request is about, not the *caller*. On the
consolidated backends the agent shares a network with the control-plane port
(macOS: the infra container runs 8099 on the agent's --internal network;
docker: the orchestrator container sits on the shared gateway network), so a
prompt-injected agent could GET /bottles (enumerate siblings + policies),
PUT /bottles/<id>/policy (rewrite its own egress allowlist to allow-all),
POST /resolve (read the injected upstream credentials it's never meant to
see), and POST /supervise/respond (self-approve its own proposals). Every
README guarantee collapsed to the agent's discretion.
Fix: a per-host control-plane secret required on every route but GET /health,
compared with hmac.compare_digest. It is held only by the trusted callers and
never handed to an agent:
- minted + persisted 0600 at <root>/control-plane-token (paths.host_control_plane_token);
- injected as $BOT_BOTTLE_CONTROL_PLANE_TOKEN into the orchestrator + gateway
containers via bare `--env NAME` (value inherited from the launch process,
so it never lands on argv or in `container/docker inspect`);
- presented by the gateway's PolicyResolver (reads the env) on /resolve, and by
the host CLI's OrchestratorClient (reads the host file) on every call.
The agent container is never given the env var or the host file, so from a
bottle every /bottles*, /resolve, /attribute, and /supervise/* call now
returns 401 — closing the enumeration, allowlist-rewrite, credential-lift, and
self-approval. The existing (source_ip, identity_token) checks stay as
defense-in-depth.
Enforced when configured: macOS + docker inject the secret (→ enforced). With
no secret set the server runs open and warns loudly at startup — a
fail-visible fallback for the unit suite and for Firecracker, whose
port-scoped nft already blocks agents from 8099 (wiring the secret into its
infra-VM init is a clean fast-follow, left out here to avoid churning the
prebuilt-artifact hash).
Verified end-to-end on real Apple Container: infra comes up healthy, the host
CLI (with the secret) lists bottles while an unauthenticated GET /bottles gets
401, all five issue-#400 attacks from inside the agent get 401, and egress
policy still works (200 allowed / 403 denied) — proving the gateway
authenticates to /resolve with the secret. 1829 unit tests pass, pyright
clean, pylint 9.91.
Refs #400.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Adopts the firecracker infra-VM pattern for macOS: the orchestrator control
plane and the gateway data plane now run in a SINGLE Apple container instead of
two. Apple Containers are lightweight VMs with separate kernels, so the prior
two-container design had both guests writing one bot-bottle.db over virtiofs,
where fcntl locks are not coherent across kernels — concurrent writes (the
orchestrator's registry vs the gateway supervise daemon's queue) could corrupt
it. One container = one kernel = coherent locking.
The DB moves onto a container-only Apple volume (bot-bottle-mac-db), never
bind-mounted from the host, so no host process opens the live file either. The
host CLI already reaches registry + supervise state over the control-plane HTTP
surface (cli/supervise.py uses OrchestratorClient), exactly as firecracker's
VM-only DB requires.
Two simplifications fall out of the single container:
- No DNS dance: the control plane and gateway daemons reach each other over
127.0.0.1, so the orchestrator-before-gateway ordering (a workaround for
Apple having no container DNS) is gone, along with the moved-IP recreate
logic it needed.
- Net -243 lines.
Mechanics: the infra container runs from the gateway image with the
control-plane source bind-mounted read-only (like the docker orchestrator, so a
code change needs no rebuild) and a small sh -c init that starts both processes
(mirrors firecracker's _infra_init). Also implements the macOS backend's
ensure_orchestrator() and adds it to discover_orchestrator_url, so operator
tools (supervise) can bring up / find the control plane on demand — previously
the macOS backend died with "no orchestrator control plane".
Verified end-to-end on real Apple Container 1.0.0: the single infra container
comes up healthy (one address for control plane + gateway), both processes run,
the DB is written on the container-only volume, host-side supervise works over
HTTP, and a registered agent gets 200 for an allowed host / 403 for a denied
one. 1824 unit tests pass with `container` absent (CI parity), pyright clean,
pylint 9.89.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Addresses findings from a high-effort review of the PRD 0070 macOS backend.
Correctness:
- Stamp identity_token onto MacosContainerBottlePlan after registration. git's
gitconfig extraHeader and the supervise MCP --header read
getattr(plan,"identity_token","") at provision time, and both reach the
gateway on NO_PROXY (bypassing the egress proxy that carries the token). The
plan never carried it, so /resolve fail-closed and every git fetch/push and
supervise call from a macOS bottle would have been denied. Registration
precedes provision(), so — unlike the run-time env — the plan can carry it.
- Self-heal the orchestrator: recreate when it is not (source-current AND
answering /health), not on the source-hash label alone. A container running
current code but with a wedged HTTP server was left alone and polled to
death, failing every launch until manual deletion.
- image_digest and container_image_digest now read the same descriptor.digest
field; dropped image_digest's id/tag fallback that could yield a value the
container side can't produce — a permanent mismatch would have recreated the
shared gateway on every launch (severing every live bottle's egress, since
the replacement gets a new DHCP address).
- Poll for the agent's and gateway's DHCP address instead of a fatal read
right after `container run` (there is no --ip; the address can lag start).
Cleanup:
- One _inspect_first + _descriptor_digest behind the four inspect readers.
- Shared bind_mount_spec (util) and host_db_dir (paths) replace per-module
copies; _GIT_HTTP_PORT now imports git_http_backend.DEFAULT_PORT.
- Drop the dead _url cache / url property and the write-only agent_proxy_url.
Deferred (noted on the PR, not fixed here): the gateway image rebuilding on
every launch (needs source-hash-labeled build), SQLite shared across VM
guests, and the sh -lc profile-override edge — each is design-level or
behavior-risk beyond a review fix.
Verified: real Apple Container bring-up is green and idempotent; 1826 unit
tests pass with `container` absent (CI parity), pyright clean, pylint 9.86.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Re-enables the macos-container backend on the shared per-host orchestrator +
gateway, replacing the per-bottle companion container removed in #385. This is
the last backend in PRD 0070's roadmap.
Apple Container 1.0.0 forced three departures from the docker shape, each
verified against the live CLI (findings recorded in the networking spike):
- No `--ip`. The address is DHCP-assigned and knowable only once the container
runs, so the order inverts: gateway up -> run agent -> read its address ->
register. The identity token is minted by registration and therefore cannot
be in the agent's run-time env; it rides the proxy URL applied at
`container exec` time (bare `--env` names keep it off argv).
- No container DNS. The gateway can only be handed the control plane's IP, so
the orchestrator starts first and the gateway is pointed at its address.
- No `network connect`. Networks are fixed at run time, so the shared host-only
network is created up front; per-bottle networks would restart the gateway
on every launch and defeat the consolidation.
The agent runs with `--cap-drop CAP_NET_RAW`: Apple grants NET_RAW by default,
which would let an agent forge a neighbour's source address on the shared
segment. NET_ADMIN is already absent, so this closes the source-address half of
PRD 0070's attribution invariant.
Verified end-to-end on real Apple Container 1.0.0: both images build, the
control plane comes up healthy, the gateway reaches it by IP, and a registered
agent gets 200 for a host in its routes and 403 for one outside them. Bring-up
is idempotent — a second launch does not churn the singletons.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Follow-up to the codex review on #398. os.open's mode arg only applies on
creation, so a committed-rootfs.tar.partial left 0644 by an interrupted run
would be opened/truncated (not re-moded) and stay world-readable for the
whole SSH stream. Unlink any leftover and exclusively recreate it
(O_EXCL|O_NOFOLLOW), then fchmod 0600 immediately so umask can't loosen it.
Test pre-creates a 0644 partial and asserts the fd is 0600 mid-stream.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UoEZHDjv84ChoZbozQERhJ
Address the codex review on #398:
- P1: inject_guest_boot no longer follows a symlink at bb-init/bb-dropbear.
A committed snapshot is guest-controlled and could plant those paths as
symlinks aimed at a host file (e.g. bb-init -> ~/.bashrc); write_text /
copy2 would then overwrite the target as the host user during resume.
Replace any pre-existing entry and create the files with
O_EXCL|O_NOFOLLOW so the write stays inside the staging tree.
- P2: write the snapshot tar owner-only (0600). It can contain the bottle's
private workspace; it was being created world-readable (0644).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UoEZHDjv84ChoZbozQERhJ
The last host-Docker dependency in the Firecracker launch path. Freeze
and resume no longer touch the docker daemon, so the backend needs
firecracker + KVM only — completing #348.
Freeze: stream the guest rootfs over SSH straight into a persistent
committed-rootfs.tar (the resumable/migratable artifact) instead of
round-tripping through `docker build` from a scratch image. Written to
a .partial sibling and atomically renamed so a failed freeze leaves no
truncated artifact.
Resume: extract the snapshot tar into a cached base dir and feed it to
the existing rootless `mke2fs -d` pipeline, replacing the
`docker create` + `docker export | tar` path. Recreate the
proc/sys/dev/run mount points the freezer excludes so the guest init
can mount them.
`util.build_base_rootfs_dir` / `docker_image_id` stay — they still back
the opt-in BOT_BOTTLE_INFRA_BUILD=local dev path and off-host
publish_infra, which are out of scope.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UoEZHDjv84ChoZbozQERhJ
Rename the Gitea generic package from bot-bottle-infra to
bot-bottle-firecracker-infra so it's self-evident in the package list which
backend it serves (and leaves room for other artifacts, e.g. a shipped
kernel). The version slot stays the content hash — "firecracker" belongs in
the package name, not the version. Docker image / VM names are unchanged.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UoEZHDjv84ChoZbozQERhJ
Generic packages have no description field, so publish_infra now uploads a
short about.txt alongside the rootfs on every publish — it's what identifies
the package as the Firecracker backend's infra rootfs on the package page.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UoEZHDjv84ChoZbozQERhJ
Address PR #395 review (two P1s):
- Version hash covered only `bot_bottle/**.py`, but the image `COPY`s the
whole package — non-Python inputs baked in (egress_entrypoint.sh,
netpool.defaults.env) didn't change the version, so a launch host could
boot a stale rootfs whose code differs from its checkout. Hash every
regular file under bot_bottle/ (excluding __pycache__/.pyc). Regression
tests: a shell-script change bumps the version; .pyc/__pycache__ don't.
- publish_infra `_put` read the whole (hundreds-of-MB) gz into memory via
read_bytes(). Stream it from disk with an explicit Content-Length; the
tiny .sha256 stays in-memory. Test asserts the body is the file object,
not bytes.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UoEZHDjv84ChoZbozQERhJ
Drop the fallback to the general-purpose BOT_BOTTLE_CLAUDE_GITEA_TOKEN so
the artifact pull/publish uses a dedicated, package-scoped token that can
be granted (or revoked) independently of the general Gitea token.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UoEZHDjv84ChoZbozQERhJ
Stage 2 of the docker-free Firecracker backend (#348): stop building the
fixed infra image on the launch host. The infra VM's rootfs is host- and
bottle-agnostic (authorized_keys + guest IP ride the kernel cmdline, not the
rootfs), so it's built once off-host and published as a versioned, ready-to-
boot ext4; the launch host downloads + verifies + boots it — no Docker, no
image tooling, just HTTP + gunzip.
- infra_artifact.py: version = content hash of the rootfs inputs (the shipped
bot_bottle package + the three Dockerfiles + the init), so a launch host
pulls the artifact matching its code and a content change can't silently
boot a stale rootfs. Pull + sha256-verify (fail-closed) + gunzip from a
Gitea generic package; base/owner/token configurable, default this Gitea.
- infra_vm.ensure_built/boot default to the pull path; BOT_BOTTLE_INFRA_BUILD=
local keeps the docker build-from-source path for iterating on Dockerfiles.
- publish_infra.py: the off-host half — builds the images with Docker, mke2fs
the rootfs (with buildah slack), gzips, and PUTs it to the generic package.
Rollout note: default=pull means a launch 404s until an artifact is published;
until the Gitea packages endpoint is enabled + an artifact published, use
BOT_BOTTLE_INFRA_BUILD=local. Freeze/migrate's remaining docker use is a
separate PR.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UoEZHDjv84ChoZbozQERhJ
Add BottleBackend.ensure_orchestrator() -> str: the backend-agnostic
entry point that brings up the per-host orchestrator + shared gateway
(idempotent) and returns its host-reachable control-plane URL. Docker
starts the orchestrator + gateway containers (OrchestratorService);
firecracker boots the infra VM; macos-container dies with a pointer
(no orchestrator). Previously bring-up was reachable only through each
backend's consolidated_launch, with no shared handle.
Wire it into `bot-bottle supervise`: supervise is often the first thing
an operator runs, before any bottle has booted the control plane, so
`_resolve_orchestrator_url` now starts the selected backend's
orchestrator on demand when discovery finds nothing, instead of failing
with "launch a bottle first".
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UoEZHDjv84ChoZbozQERhJ
The firecracker supervise MCP daemon 500'd (-32603) on egress-allow/block:
it ran with no BOT_BOTTLE_ROOT/SUPERVISE_DB_PATH, so it targeted a stray,
unmigrated SQLite file and `write_proposal` hit "no such table:
supervise_proposals". Meanwhile the in-VM control plane migrated only the
registry table (orchestrator_bottles) into its own DB, and the host
operator reads a third, disconnected DB — three files, none shared.
Consolidate to one DB per host, owned by the control plane on the
persisted registry volume (/var/lib/bot-bottle/db/bot-bottle.db):
- orchestrator startup now migrates the supervise queue + audit tables
into the same file it migrates the registry into (StoreManager), so the
control-plane DB carries every table.
- the in-VM supervise daemon is pointed at that same file via
SUPERVISE_DB_PATH, so daemon and control plane share one queue.
`list-egress-routes` already worked (no DB); egress-allow now queues +
waits on the single persisted DB instead of erroring. Validated against a
live infra VM: migrate + write_proposal succeeds and the proposal is
queued. The host-operator HTTP bridge (so approvals complete from the
host, unifying docker onto the same path) is the follow-up.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WBMWTEtQdJ4W5UrWuLHCck
The agent-image build ran over ssh with output fully captured, shown
only as a 20-line tail on failure — so a successful (or in-progress)
first build was a long silent wait through the base pull + apt/npm
installs. Stream the `buildah build` step's stdout/stderr straight to
our own (like the docker backend's `docker build`) via a non-capturing
_ssh_streamed helper, so the operator sees `STEP i/n` progress live.
Failure now points at the streamed output above instead of a captured
tail. The smoke test and rootfs export stay captured (short / piped).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WBMWTEtQdJ4W5UrWuLHCck
The interactive agent command is sent to the guest by spreading the
remote argv as separate ssh arguments; ssh space-joins everything after
the host into one line that the guest login shell re-parses. That only
works while every token is a "simple word" — which held for claude
(`--append-system-prompt-file <path>`) but not for codex's
`read_prompt_file` mode, whose positional is a whole sentence:
"Read and follow the instructions in <path>.". The guest shell re-split
it on spaces, so codex received `Read` as the prompt and `and`, `follow`,
… as extra args — failing with `unrecognized subcommand 'and'` the moment
an interactive codex session attached.
Pre-quote each remote token with shlex.quote before ssh joins them (the
same ssh→guest-shell discipline infra_vm/cp_in already use). Simple words
are unchanged, so existing behaviour and the parity/structure tests are
untouched; an arg with spaces now survives as a single argument.
Regression test round-trips the joined remote command back through
shlex.split and asserts codex's prompt comes out as exactly one arg.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WBMWTEtQdJ4W5UrWuLHCck
Codex review: the /31 TAP doesn't make source IP unspoofable, and the
app-layer token was returned by launch but never delivered or enforced, so
a spoofed source could select a victim bottle's policy/tokens. Make the
token mandatory and deliver it on each attributed plane (anti-spoof landed
separately as the network boundary).
Enforcement (control plane):
- `Orchestrator.resolve` now requires a matching (source_ip, identity_token)
pair (constant-time) — no source-IP-only fallback. `/resolve` fail-closes
(403) on a missing/empty/mismatched token.
Delivery, per plane (the token is `token_urlsafe`, safe in a URL):
- egress: proxy credentials (`HTTPS_PROXY=http://bottle:<token>@gw`). The
addon reads `Proxy-Authorization` — from the request (HTTP) or captured at
the CONNECT for HTTPS tunnels (keyed by client conn, cleared on disconnect)
— validates, and strips it (+ the legacy header) before upstream.
- git-http: a URL-scoped `http.<gate>/.extraHeader: x-bot-bottle-identity`
in the agent's git config (only over the http transport).
- supervise: `mcp add --header x-bot-bottle-identity: <token>` (claude +
codex); the server reads the header and passes it to resolve.
Wiring: thread `ctx.identity_token` onto the firecracker + docker plans and
into the agent env/config at launch.
Verified on a KVM host: egress with the correct proxy-cred token returns
200 (HTTP and HTTPS/CONNECT), and no-token / wrong-token return 403; a real
`cli.py start --backend=firecracker` launch provisions git config + the
supervise MCP header and reaches the agent session, all under mandatory
enforcement. Fixed a `claude mcp add` arg-order bug (--header must follow
the positional name/url) found by that launch.
Transparent proxy for tools that ignore proxy env is deferred to a
follow-up (see thread); anti-spoof + host firewall remain the fail-closed
boundary.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WBMWTEtQdJ4W5UrWuLHCck
Codex flagged that parallel `start`s race on shared state: two cold
launches could both stop/build/boot the singleton on the same rootfs/PID,
concurrent builds share the infra VM's buildah store, the cleanup did
`buildah rm -a` (nuking a peer build's container), and the rootfs cache
was populated non-atomically.
- `infra_vm.ensure_running`: a host flock (`singleton.lock`) around the
cold stop/build/boot path, with a double-checked health re-test under
the lock so a second launcher adopts rather than re-boots. The healthy
fast-path stays lock-free.
- `image_builder.build_agent_rootfs_dir`: a host flock (`.build.lock`)
around cache-lookup + build + publish; build into `.building-<digest>`
and publish by atomic `os.rename`, so a partial build never appears as
`agent-<digest>`.
- scoped cleanup: per-build named working containers
(`<tag>-smoke`/`-export`) removed by name instead of `buildah rm -a`;
also cleared before the build to recover from a crashed prior run.
Verified: agent image builds via the locked/atomic path, infra VM stays
healthy, agent boots, `claude --version` = 2.1.172. Full unit suite +
pyright green.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WBMWTEtQdJ4W5UrWuLHCck
Addresses the review finding that buildah lived only in the orchestrator
image, so the persistent infra VM wasn't the builder — a separate throwaway
builder VM contended with it for the orchestrator TAP. Consolidate:
- **Rebase the gateway (and thus infra) on `python:3.12-slim` = Debian
trixie**, pip-installing mitmproxy instead of `FROM mitmproxy/mitmproxy`
(Debian bookworm). trixie ships buildah 1.39, which can build agent
Dockerfiles that use heredocs; bookworm's 1.28 can't (`Unknown
instruction: "{"`). CA path is unchanged (set via `--set confdir=`).
- **buildah lives only in `Dockerfile.infra`** now (removed from the
orchestrator image, which is lean/stdlib-only again).
- **Shared orchestrator content**: `Dockerfile.orchestrator` is the single
definition of the control-plane payload; the infra image `COPY --from`s
it (same trixie base → clean copy, and future deps like iroh are added
once). The docker backend runs the orchestrator image directly.
- **`image_builder` builds inside the infra VM** (which now has buildah)
over SSH — no throwaway builder VM, so the `bborch0` contention is gone.
`ensure_built` builds orchestrator + gateway before infra (FROM gateway,
COPY --from orchestrator).
Verified on a KVM host: images build (buildah 1.39 in infra), the agent
image builds *inside* the infra VM (heredoc Dockerfile and all), the infra
VM stays healthy, the agent boots and `claude --version` = 2.1.172. The
rebased gateway still starts as a docker container and generates its CA
(docker backend unaffected).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WBMWTEtQdJ4W5UrWuLHCck
git-gate now works end-to-end for git-upstream bottles on the infra VM.
Three fixes surfaced by driving a real clone through git-http:
- `SshGatewayTransport.cp_into` preserves the source file mode (docker cp
does). The access-hook is staged 0700 and git-http execs it directly; a
plain `cat >` landed it 0644 -> EACCES. Keys stay 0600.
- the infra init runs `BOT_BOTTLE_GATEWAY_DAEMONS=egress,git-http,supervise`:
the VM backend reaches git over git-http (9420), so the git:// daemon
(git-gate, whose /git-gate-entrypoint.sh the consolidated model doesn't
stage) is left out instead of crash-looping.
- `build_infra_rootfs_dir` folds the init's content-hash into the rootfs
cache key, so an init change actually rebuilds the rootfs (the base image
digest alone wouldn't catch it).
Verified on a KVM host: launch_consolidated provisions a git-upstream
bottle's repo + creds into the gateway VM over SSH; an agent VM clones via
git-http (source-IP attributed) and the access-hook resolves the
provisioned key + known_hosts and attempts the upstream SSH fetch —
failing closed only because the test used a dummy key + fake upstream
("refusing to serve stale data"). With real creds the fetch serves.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WBMWTEtQdJ4W5UrWuLHCck
The infra VM's rootfs is ephemeral (rebuilt each boot), so the bottle
registry DB needs durable storage across restarts. Give the infra VM a
firecracker analogue of a docker volume: a host-side ext4 file attached as
a second virtio-block device (guest /dev/vdb), mounted at the control
plane's DB dir (/var/lib/bot-bottle, where host_db_path lives at
db/bot-bottle.db).
- firecracker_vm.boot/_config take an optional `data_drive` (a non-root,
RW second drive).
- infra_vm creates the volume on first use (`mke2fs` an empty ext4 at
<fc-cache>/infra/registry.ext4) and mounts /dev/vdb in the PID-1 init
before the control plane starts. It's a plain ext4 file, so
`sudo mount -o loop <path>` (VM stopped) inspects bot-bottle.db directly.
Verified on a KVM host: register a bottle, restart the infra VM (fresh
rootfs, same volume) — /dev/vdb re-mounts and the registry `db/` dir + a
marker file survive the restart.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WBMWTEtQdJ4W5UrWuLHCck
consolidated_launch now drives the persistent infra VM instead of the
`_FirecrackerOrchestratorService` Docker containers — the point Docker
leaves the Firecracker launch path.
- launch_consolidated: `infra_vm.ensure_running()` (singleton) for the
control plane + gateway; register the bottle over HTTP at the infra VM's
guest IP; provision git-gate via `SshGatewayTransport` (over SSH into the
gateway VM); fetch the gateway CA via `InfraVm.gateway_ca_pem()`.
- teardown_consolidated deregisters + deprovisions but does NOT stop the
infra VM (persistent per-host singleton shared by every bottle).
- new `infra_vm.SshGatewayTransport` + `gateway_transport()` (built from the
stable key + orchestrator link IP, so teardown needs no live handle).
- drop the DockerGateway/OrchestratorService machinery from the firecracker
consolidated path (still used by the docker backend).
launch.py already calls launch_consolidated, so the whole firecracker
launch path now uses the VM. pyright + unit suite green.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WBMWTEtQdJ4W5UrWuLHCck
git-gate provisioning into the running gateway was hard-wired to docker
exec/cp. Extract a backend-neutral `GatewayTransport` (exec + cp_into) so
the same provisioning logic serves both the docker gateway container and
the firecracker gateway VM (over SSH, added with the launch swap).
- `provision_git_gate` / `deprovision_git_gate` now take a transport
instead of a gateway name; `DockerGatewayTransport` wraps the existing
docker exec/cp behavior. deprovision is best-effort (catches the
transport error) — matching the prior idempotent teardown.
- both consolidated_launch callers pass `DockerGatewayTransport(name)` —
no behavior change; the firecracker swap flips only its own to SSH.
Pure refactor: docker path unchanged, gateway_provision tests updated to
construct the transport, all green.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WBMWTEtQdJ4W5UrWuLHCck
The infra VM must outlive the short-lived `start` launcher and be reused
across launches. Add an idempotent singleton:
- `ensure_running()` adopts the infra VM when its control plane is already
healthy (a prior launcher booted it), else clears any stale VM and boots
a fresh one. Returns a handle usable for CA fetch / git-gate provisioning
whether we booted it or adopted it.
- boot is `detached` (firecracker in its own session via start_new_session)
so it survives the launcher exiting; its PID is recorded so a later
process can `stop()` it. `_kill_pidfile` SIGTERM/SIGKILLs but only if the
PID is still a firecracker process (guards a recycled PID).
- a STABLE SSH key (generated once under the infra cache dir, re-injected
each boot via the cmdline) so any launcher can SSH in to fetch the CA /
provision, not just the one that booted the VM. `InfraVm.vm` is None in
the adopted case; teardown then goes through the PID file.
Verified on a KVM host: first ensure_running boots; a second adopts it
(same PID, no reboot) and can still reach /health and fetch the gateway CA
over SSH; stop() tears it down (control plane then unreachable).
Next: git-gate provisioning into the VM over SSH (today docker exec/cp),
then swap consolidated_launch.py onto the infra VM.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WBMWTEtQdJ4W5UrWuLHCck
The single infra VM now runs BOTH the orchestrator control plane and the
gateway data plane (egress / supervise / git-http), multi-tenant against
the local control plane — the single-VM shape from the Stage B design.
- Dockerfile.infra: the firecracker infra image = the gateway image +
the baked control-plane source (FROM bot-bottle-gateway, COPY
bot_bottle). Reuses the gateway payload rather than copying mitmproxy/
gitleaks into a third image; the docker backend keeps its two separate
images.
- infra_vm: build from source (gateway then infra image), and the PID-1
init now also launches `gateway_init` with
BOT_BOTTLE_ORCHESTRATOR_URL=http://127.0.0.1:8099. Adds `gateway_ca_pem`
(fetch the mitmproxy CA over SSH) and the agent-facing port constants.
- init exports PATH — a bare-init shell resolves its own execs via a
built-in default path, but that isn't in the environment, so
gateway_init's `python3 ...` daemons would otherwise fail to spawn.
Verified on a KVM host: infra VM boots, control plane /health -> 200, and
egress:9099 / supervise:9100 / git-http:9420 all listen and are reachable
from the host over the TAP link; the gateway CA is retrievable. (git-gate
stays down until a bottle provisions its per-bottle entrypoint/creds, same
as a fresh docker gateway — non-fatal, the supervisor keeps the rest up.)
Next: agent->gateway VM-to-VM routing so bbfc* VMs reach these ports at
the infra VM and nowhere else.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WBMWTEtQdJ4W5UrWuLHCck
First slice of Stage B: the orchestrator control plane runs as a
persistent Firecracker infra VM instead of a Docker container. The host
CLI reaches it over HTTP at the orchestrator link's guest IP; agent VMs
will reach its gateway ports (added next) over VM-to-VM routing.
- Dockerfile.orchestrator bakes the stdlib-only control-plane source
(COPY bot_bottle) so the image is self-contained and runs from a built
image with no runtime bind-mount — a guest VM can't bind-mount host
source. (Build-from-source stays the default; a pull-from-registry mode
lands later. The docker backend's dev bind-mount still overlays this.)
- util.build_base_rootfs_dir / inject_guest_boot take a `variant` +
`init_script`, so the same orchestrator image is prepared two ways
without a cache collision: the builder VM keeps the SSH-only agent init;
the infra VM gets a control-plane PID-1 init.
- new firecracker/infra_vm.py: boot the infra VM on the orchestrator link,
run `python -m bot_bottle.orchestrator` as PID 1, and poll /health.
Verified on a KVM host: infra VM boots, control plane answers
`GET /health -> 200 {"status":"ok"}` from the host over the TAP link.
Next: fold gateway_init (egress/git-gate/supervise) into the same VM,
then agent->gateway routing.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WBMWTEtQdJ4W5UrWuLHCck
didericis-claude
didericis-codex
didericis