Adopts the firecracker infra-VM pattern for macOS: the orchestrator control
plane and the gateway data plane now run in a SINGLE Apple container instead of
two. Apple Containers are lightweight VMs with separate kernels, so the prior
two-container design had both guests writing one bot-bottle.db over virtiofs,
where fcntl locks are not coherent across kernels — concurrent writes (the
orchestrator's registry vs the gateway supervise daemon's queue) could corrupt
it. One container = one kernel = coherent locking.
The DB moves onto a container-only Apple volume (bot-bottle-mac-db), never
bind-mounted from the host, so no host process opens the live file either. The
host CLI already reaches registry + supervise state over the control-plane HTTP
surface (cli/supervise.py uses OrchestratorClient), exactly as firecracker's
VM-only DB requires.
Two simplifications fall out of the single container:
- No DNS dance: the control plane and gateway daemons reach each other over
127.0.0.1, so the orchestrator-before-gateway ordering (a workaround for
Apple having no container DNS) is gone, along with the moved-IP recreate
logic it needed.
- Net -243 lines.
Mechanics: the infra container runs from the gateway image with the
control-plane source bind-mounted read-only (like the docker orchestrator, so a
code change needs no rebuild) and a small sh -c init that starts both processes
(mirrors firecracker's _infra_init). Also implements the macOS backend's
ensure_orchestrator() and adds it to discover_orchestrator_url, so operator
tools (supervise) can bring up / find the control plane on demand — previously
the macOS backend died with "no orchestrator control plane".
Verified end-to-end on real Apple Container 1.0.0: the single infra container
comes up healthy (one address for control plane + gateway), both processes run,
the DB is written on the container-only volume, host-side supervise works over
HTTP, and a registered agent gets 200 for an allowed host / 403 for a denied
one. 1824 unit tests pass with `container` absent (CI parity), pyright clean,
pylint 9.89.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Addresses findings from a high-effort review of the PRD 0070 macOS backend.
Correctness:
- Stamp identity_token onto MacosContainerBottlePlan after registration. git's
gitconfig extraHeader and the supervise MCP --header read
getattr(plan,"identity_token","") at provision time, and both reach the
gateway on NO_PROXY (bypassing the egress proxy that carries the token). The
plan never carried it, so /resolve fail-closed and every git fetch/push and
supervise call from a macOS bottle would have been denied. Registration
precedes provision(), so — unlike the run-time env — the plan can carry it.
- Self-heal the orchestrator: recreate when it is not (source-current AND
answering /health), not on the source-hash label alone. A container running
current code but with a wedged HTTP server was left alone and polled to
death, failing every launch until manual deletion.
- image_digest and container_image_digest now read the same descriptor.digest
field; dropped image_digest's id/tag fallback that could yield a value the
container side can't produce — a permanent mismatch would have recreated the
shared gateway on every launch (severing every live bottle's egress, since
the replacement gets a new DHCP address).
- Poll for the agent's and gateway's DHCP address instead of a fatal read
right after `container run` (there is no --ip; the address can lag start).
Cleanup:
- One _inspect_first + _descriptor_digest behind the four inspect readers.
- Shared bind_mount_spec (util) and host_db_dir (paths) replace per-module
copies; _GIT_HTTP_PORT now imports git_http_backend.DEFAULT_PORT.
- Drop the dead _url cache / url property and the write-only agent_proxy_url.
Deferred (noted on the PR, not fixed here): the gateway image rebuilding on
every launch (needs source-hash-labeled build), SQLite shared across VM
guests, and the sh -lc profile-override edge — each is design-level or
behavior-risk beyond a review fix.
Verified: real Apple Container bring-up is green and idempotent; 1826 unit
tests pass with `container` absent (CI parity), pyright clean, pylint 9.86.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
CI's unit + coverage jobs failed with `FileNotFoundError: 'container'`: three
tests reached the real Apple CLI, which exists on a macOS dev host but not on
the Linux runner. They passed locally for that reason alone — and two of them
were quietly creating real Apple networks on the dev host as a side effect.
- `test_enumerate_active_is_empty_while_disabled` asserted the disabled-era
stub and called `enumerate_active()` unmocked. The backend launches bottles
again, so it now covers the real enumeration: slug parsing, exclusion of the
shared gateway/orchestrator singletons, and the CLI-failure path.
- The two orchestrator tests patched `orchestrator_service.container_mod`, but
`_run_orchestrator_container` reaches the CLI through `ensure_networks`,
which is imported from the gateway module and resolves `container_mod` in
*its* namespace. Patch the imported name instead.
Adds a test that the networks exist before the orchestrator runs — the
ordering the escaped call was hiding.
Verified by reproducing the CI environment locally (`PATH` without the
`container` binary): 3 failures before, 1818 passing after.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Re-enables the macos-container backend on the shared per-host orchestrator +
gateway, replacing the per-bottle companion container removed in #385. This is
the last backend in PRD 0070's roadmap.
Apple Container 1.0.0 forced three departures from the docker shape, each
verified against the live CLI (findings recorded in the networking spike):
- No `--ip`. The address is DHCP-assigned and knowable only once the container
runs, so the order inverts: gateway up -> run agent -> read its address ->
register. The identity token is minted by registration and therefore cannot
be in the agent's run-time env; it rides the proxy URL applied at
`container exec` time (bare `--env` names keep it off argv).
- No container DNS. The gateway can only be handed the control plane's IP, so
the orchestrator starts first and the gateway is pointed at its address.
- No `network connect`. Networks are fixed at run time, so the shared host-only
network is created up front; per-bottle networks would restart the gateway
on every launch and defeat the consolidation.
The agent runs with `--cap-drop CAP_NET_RAW`: Apple grants NET_RAW by default,
which would let an agent forge a neighbour's source address on the shared
segment. NET_ADMIN is already absent, so this closes the source-address half of
PRD 0070's attribution invariant.
Verified end-to-end on real Apple Container 1.0.0: both images build, the
control plane comes up healthy, the gateway reaches it by IP, and a registered
agent gets 200 for a host in its routes and 403 for one outside them. Bring-up
is idempotent — a second launch does not churn the singletons.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Follow-up to the codex review on #398. os.open's mode arg only applies on
creation, so a committed-rootfs.tar.partial left 0644 by an interrupted run
would be opened/truncated (not re-moded) and stay world-readable for the
whole SSH stream. Unlink any leftover and exclusively recreate it
(O_EXCL|O_NOFOLLOW), then fchmod 0600 immediately so umask can't loosen it.
Test pre-creates a 0644 partial and asserts the fd is 0600 mid-stream.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UoEZHDjv84ChoZbozQERhJ
Address the codex review on #398:
- P1: inject_guest_boot no longer follows a symlink at bb-init/bb-dropbear.
A committed snapshot is guest-controlled and could plant those paths as
symlinks aimed at a host file (e.g. bb-init -> ~/.bashrc); write_text /
copy2 would then overwrite the target as the host user during resume.
Replace any pre-existing entry and create the files with
O_EXCL|O_NOFOLLOW so the write stays inside the staging tree.
- P2: write the snapshot tar owner-only (0600). It can contain the bottle's
private workspace; it was being created world-readable (0644).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UoEZHDjv84ChoZbozQERhJ
The last host-Docker dependency in the Firecracker launch path. Freeze
and resume no longer touch the docker daemon, so the backend needs
firecracker + KVM only — completing #348.
Freeze: stream the guest rootfs over SSH straight into a persistent
committed-rootfs.tar (the resumable/migratable artifact) instead of
round-tripping through `docker build` from a scratch image. Written to
a .partial sibling and atomically renamed so a failed freeze leaves no
truncated artifact.
Resume: extract the snapshot tar into a cached base dir and feed it to
the existing rootless `mke2fs -d` pipeline, replacing the
`docker create` + `docker export | tar` path. Recreate the
proc/sys/dev/run mount points the freezer excludes so the guest init
can mount them.
`util.build_base_rootfs_dir` / `docker_image_id` stay — they still back
the opt-in BOT_BOTTLE_INFRA_BUILD=local dev path and off-host
publish_infra, which are out of scope.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UoEZHDjv84ChoZbozQERhJ
Rename the Gitea generic package from bot-bottle-infra to
bot-bottle-firecracker-infra so it's self-evident in the package list which
backend it serves (and leaves room for other artifacts, e.g. a shipped
kernel). The version slot stays the content hash — "firecracker" belongs in
the package name, not the version. Docker image / VM names are unchanged.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UoEZHDjv84ChoZbozQERhJ
Generic packages have no description field, so publish_infra now uploads a
short about.txt alongside the rootfs on every publish — it's what identifies
the package as the Firecracker backend's infra rootfs on the package page.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UoEZHDjv84ChoZbozQERhJ
Address PR #395 review (two P1s):
- Version hash covered only `bot_bottle/**.py`, but the image `COPY`s the
whole package — non-Python inputs baked in (egress_entrypoint.sh,
netpool.defaults.env) didn't change the version, so a launch host could
boot a stale rootfs whose code differs from its checkout. Hash every
regular file under bot_bottle/ (excluding __pycache__/.pyc). Regression
tests: a shell-script change bumps the version; .pyc/__pycache__ don't.
- publish_infra `_put` read the whole (hundreds-of-MB) gz into memory via
read_bytes(). Stream it from disk with an explicit Content-Length; the
tiny .sha256 stays in-memory. Test asserts the body is the file object,
not bytes.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UoEZHDjv84ChoZbozQERhJ
Drop the fallback to the general-purpose BOT_BOTTLE_CLAUDE_GITEA_TOKEN so
the artifact pull/publish uses a dedicated, package-scoped token that can
be granted (or revoked) independently of the general Gitea token.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UoEZHDjv84ChoZbozQERhJ
Stage 2 of the docker-free Firecracker backend (#348): stop building the
fixed infra image on the launch host. The infra VM's rootfs is host- and
bottle-agnostic (authorized_keys + guest IP ride the kernel cmdline, not the
rootfs), so it's built once off-host and published as a versioned, ready-to-
boot ext4; the launch host downloads + verifies + boots it — no Docker, no
image tooling, just HTTP + gunzip.
- infra_artifact.py: version = content hash of the rootfs inputs (the shipped
bot_bottle package + the three Dockerfiles + the init), so a launch host
pulls the artifact matching its code and a content change can't silently
boot a stale rootfs. Pull + sha256-verify (fail-closed) + gunzip from a
Gitea generic package; base/owner/token configurable, default this Gitea.
- infra_vm.ensure_built/boot default to the pull path; BOT_BOTTLE_INFRA_BUILD=
local keeps the docker build-from-source path for iterating on Dockerfiles.
- publish_infra.py: the off-host half — builds the images with Docker, mke2fs
the rootfs (with buildah slack), gzips, and PUTs it to the generic package.
Rollout note: default=pull means a launch 404s until an artifact is published;
until the Gitea packages endpoint is enabled + an artifact published, use
BOT_BOTTLE_INFRA_BUILD=local. Freeze/migrate's remaining docker use is a
separate PR.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UoEZHDjv84ChoZbozQERhJ
Adds globalize_slug(slug) to bottle_state alongside bottle_identity.
git_gate_provision now calls globalize_slug(slug) instead of inlining
socket.gethostname(), so the hostname-qualification logic has a single,
named home. Assumes slug is a mint_slug output.
Title format changes from bot-bottle:{host}:{slug}:{name}
to bot-bottle:{host}-{slug}:{name} to match the globalize_slug contract.
Closes#388 (part 1 of 3). Deploy key titles now carry the machine
hostname so keys provisioned on different hosts don't collide with
each other on the forge when a prior bottle was never torn down.
Title format: bot-bottle:<hostname>:<slug>:<repo-name>
`cmd_supervise` establishes the orchestrator client up front (since the
HTTP-bridge move in 27fe03b), so these tests — which mock `curses.wrapper`
to exercise the KeyboardInterrupt / Die / crash-log paths — only reached
those paths when a live orchestrator happened to be reachable. On CI
(none reachable) the up-front connect errored and `cmd_supervise`
returned 1 before curses, failing all four. They passed locally only
because a dev orchestrator was up.
Stub `supervise_cli._client` in the class setUp so the tests isolate the
post-connect behavior they actually cover, independent of environment.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UoEZHDjv84ChoZbozQERhJ
In consolidated mode the gateway's static route table is empty (routes
are resolved per request by source IP), but `list-egress-routes` was
reading that static table via the `_egress.local/allowlist` introspection
endpoint — so it always returned an empty allowlist. Agents, told to call
it before composing an egress proposal, then sent a routes.yaml with only
the newly-needed host; approving it replaced the whole policy and silently
dropped base routes like api.anthropic.com, breaking the bottle's egress.
Answer `list-egress-routes` from the calling bottle's resolved policy
(same (source_ip, identity-token) attribution the proposal path uses,
same JSON shape the single-tenant introspection endpoint returns).
Fail-closed to an empty list on an unreachable orchestrator; falls back
to the introspection endpoint in single-tenant mode (no resolver).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UoEZHDjv84ChoZbozQERhJ
Consolidated proposals are keyed by the orchestrator-assigned bottle_id,
so the supervise TUI was rendering a hex id (e.g. 3601cbe883c2786d) as
the bottle name — and the resume hint printed `./cli.py resume <id>`,
which resume can't take (it wants the human identity/slug).
`supervise_pending` now tags each dict with `bottle_label` — the human
slug resolved from registry metadata, falling back to the id when the
bottle is gone. `QueuedProposal` carries the label; every display site
(list rows, detail view, status lines, resume hint) shows it, while
approve/reject still key on `proposal.bottle_slug` (the id).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UoEZHDjv84ChoZbozQERhJ
In consolidated mode the supervise server attributes each proposal to
the orchestrator-assigned bottle_id and stores that as the proposal's
`bottle_slug` (supervise_server `_attributed_config`). But
`_record_for_slug` only scanned registry metadata for a matching human
slug, so approving an egress proposal 409'd with "bottle <id> is no
longer registered; cannot apply the route change" — the id never
matched a human slug.
Resolve the record by bottle_id first (the consolidated reality),
keeping the metadata-slug scan as a fallback for legacy single-tenant
proposals. The prior tests keyed proposals by the human slug matching
the metadata, exercising only the fallback path — added a test that
mirrors production (proposal keyed by bottle_id, distinct human slug in
metadata).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UoEZHDjv84ChoZbozQERhJ
Add BottleBackend.ensure_orchestrator() -> str: the backend-agnostic
entry point that brings up the per-host orchestrator + shared gateway
(idempotent) and returns its host-reachable control-plane URL. Docker
starts the orchestrator + gateway containers (OrchestratorService);
firecracker boots the infra VM; macos-container dies with a pointer
(no orchestrator). Previously bring-up was reachable only through each
backend's consolidated_launch, with no shared handle.
Wire it into `bot-bottle supervise`: supervise is often the first thing
an operator runs, before any bottle has booted the control plane, so
`_resolve_orchestrator_url` now starts the selected backend's
orchestrator on demand when discovery finds nothing, instead of failing
with "launch a bottle first".
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UoEZHDjv84ChoZbozQERhJ
The docker gateway container ran the supervise daemon but bind-mounted no
DB and set no SUPERVISE_DB_PATH — so the daemon wrote proposals to a
container-local, ephemeral SQLite file, disconnected from the host DB the
orchestrator (and now the operator, over HTTP) uses. Same split-DB bug
firecracker had; Dockerfile.gateway even documents the mount
(`/run/supervise/bot-bottle.db bind-mounted at run time`) that
ensure_running never provided.
Bind-mount the host DB dir into the gateway at /run/supervise and set
SUPERVISE_DB_PATH, so the daemon queues into the same file the
orchestrator container opens (its BOT_BOTTLE_ROOT bind-mount) and the
operator reaches over the control plane. One DB per host, shared by
bind-mounts — docker now on the exact same supervise path as firecracker.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WBMWTEtQdJ4W5UrWuLHCck
The `bot-bottle supervise` operator TUI read and wrote the queue DB
directly and tried a backend-specific live "apply" (which was unwired —
it raised). It now talks only to the orchestrator control plane:
- OrchestratorClient gains supervise_pending() + supervise_respond().
- discover_orchestrator_url() finds the one running per-host control
plane by health-probing the backends' well-known :8099 addresses
(docker publishes on loopback; the firecracker infra VM serves it on
the orchestrator TAP) — no backend branching in the TUI.
- discover_pending/approve/reject call the client; the server does the
apply + response + audit atomically. The dead direct-DB apply/audit
helpers and the docker/macos applicator imports are gone.
- A missing control plane is now a clean one-line error up front, not a
mid-curses crash.
CLI tests move to mocking the client (the DB-write behaviour they used to
assert is now server-side, covered by test_orchestrator_service). Docker's
orchestrator-container DB wiring lands next so its /supervise endpoints hit
the same shared DB.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WBMWTEtQdJ4W5UrWuLHCck
The orchestrator owns the single DB *and* the live policy, so operator
decisions belong there — applied server-side, reached over HTTP. Adds:
GET /supervise/proposals -> pending proposals across bottles
POST /supervise/respond -> apply + record an operator decision
`supervise_respond` is one atomic server-side op on the one DB: approve/
modify on an egress tool rewrites the bottle's policy (so the gateway
serves the new routes on its next /resolve — the live "apply" that was a
documented TODO), then writes the queued Response (unblocking the agent's
MCP call) and an audit entry. reject records the response + audit only.
Fails closed (409) when the proposal is unknown or the bottle was torn
down before the operator acted (an egress apply would have no target).
This is the server half of unifying every backend onto one HTTP path for
supervise; the host TUI (direct-DB today) moves onto this client next.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WBMWTEtQdJ4W5UrWuLHCck
The firecracker supervise MCP daemon 500'd (-32603) on egress-allow/block:
it ran with no BOT_BOTTLE_ROOT/SUPERVISE_DB_PATH, so it targeted a stray,
unmigrated SQLite file and `write_proposal` hit "no such table:
supervise_proposals". Meanwhile the in-VM control plane migrated only the
registry table (orchestrator_bottles) into its own DB, and the host
operator reads a third, disconnected DB — three files, none shared.
Consolidate to one DB per host, owned by the control plane on the
persisted registry volume (/var/lib/bot-bottle/db/bot-bottle.db):
- orchestrator startup now migrates the supervise queue + audit tables
into the same file it migrates the registry into (StoreManager), so the
control-plane DB carries every table.
- the in-VM supervise daemon is pointed at that same file via
SUPERVISE_DB_PATH, so daemon and control plane share one queue.
`list-egress-routes` already worked (no DB); egress-allow now queues +
waits on the single persisted DB instead of erroring. Validated against a
live infra VM: migrate + write_proposal succeeds and the proposal is
queued. The host-operator HTTP bridge (so approvals complete from the
host, unifying docker onto the same path) is the follow-up.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WBMWTEtQdJ4W5UrWuLHCck
The agent-image build ran over ssh with output fully captured, shown
only as a 20-line tail on failure — so a successful (or in-progress)
first build was a long silent wait through the base pull + apt/npm
installs. Stream the `buildah build` step's stdout/stderr straight to
our own (like the docker backend's `docker build`) via a non-capturing
_ssh_streamed helper, so the operator sees `STEP i/n` progress live.
Failure now points at the streamed output above instead of a captured
tail. The smoke test and rootfs export stay captured (short / piped).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WBMWTEtQdJ4W5UrWuLHCck
The interactive agent command is sent to the guest by spreading the
remote argv as separate ssh arguments; ssh space-joins everything after
the host into one line that the guest login shell re-parses. That only
works while every token is a "simple word" — which held for claude
(`--append-system-prompt-file <path>`) but not for codex's
`read_prompt_file` mode, whose positional is a whole sentence:
"Read and follow the instructions in <path>.". The guest shell re-split
it on spaces, so codex received `Read` as the prompt and `and`, `follow`,
… as extra args — failing with `unrecognized subcommand 'and'` the moment
an interactive codex session attached.
Pre-quote each remote token with shlex.quote before ssh joins them (the
same ssh→guest-shell discipline infra_vm/cp_in already use). Simple words
are unchanged, so existing behaviour and the parity/structure tests are
untouched; an arg with spaces now survives as a single argument.
Regression test round-trips the joined remote command back through
shlex.split and asserts codex's prompt comes out as exactly one arg.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WBMWTEtQdJ4W5UrWuLHCck
The Codex CLI has no `codex mcp add --header` flag (verified against
0.144.5 and the codex-rs `AddMcpStreamableHttpArgs` surface: only
`--url`, `--bearer-token-env-var`, `--oauth-*`, `--env`). The old call
therefore exited nonzero on every codex bottle; provisioning only
warned and continued, so supervise was silently unregistered — and
under mandatory (source_ip, token) attribution the suggested manual
recovery (`codex mcp add supervise --url ...`, no token) could not
restore access either.
Write the `[mcp_servers.supervise]` streamable-HTTP entry directly into
`~/.codex/config.toml` instead, delivering the identity token via the
Codex-supported `http_headers` key (the only way to attach a static
request header to an HTTP MCP server). Registration failure is now
FATAL when supervise is enabled, rather than a warning.
Test validates the generated entry against the real Codex config
surface: it must parse as TOML into a streamable-HTTP server carrying
the token as `http_headers["x-bot-bottle-identity"]`, and must use only
keys accepted by `RawMcpServerConfig` (config.toml is
`deny_unknown_fields`). Also covers custom `CODEX_HOME`, the no-token
case, and the now-fatal failure path.
Refs: PR #354 review (codex P1).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WBMWTEtQdJ4W5UrWuLHCck
Codex review: the /31 TAP doesn't make source IP unspoofable, and the
app-layer token was returned by launch but never delivered or enforced, so
a spoofed source could select a victim bottle's policy/tokens. Make the
token mandatory and deliver it on each attributed plane (anti-spoof landed
separately as the network boundary).
Enforcement (control plane):
- `Orchestrator.resolve` now requires a matching (source_ip, identity_token)
pair (constant-time) — no source-IP-only fallback. `/resolve` fail-closes
(403) on a missing/empty/mismatched token.
Delivery, per plane (the token is `token_urlsafe`, safe in a URL):
- egress: proxy credentials (`HTTPS_PROXY=http://bottle:<token>@gw`). The
addon reads `Proxy-Authorization` — from the request (HTTP) or captured at
the CONNECT for HTTPS tunnels (keyed by client conn, cleared on disconnect)
— validates, and strips it (+ the legacy header) before upstream.
- git-http: a URL-scoped `http.<gate>/.extraHeader: x-bot-bottle-identity`
in the agent's git config (only over the http transport).
- supervise: `mcp add --header x-bot-bottle-identity: <token>` (claude +
codex); the server reads the header and passes it to resolve.
Wiring: thread `ctx.identity_token` onto the firecracker + docker plans and
into the agent env/config at launch.
Verified on a KVM host: egress with the correct proxy-cred token returns
200 (HTTP and HTTPS/CONNECT), and no-token / wrong-token return 403; a real
`cli.py start --backend=firecracker` launch provisions git config + the
supervise MCP header and reaches the agent session, all under mandatory
enforcement. Fixed a `claude mcp add` arg-order bug (--header must follow
the positional name/url) found by that launch.
Transparent proxy for tools that ignore proxy env is deferred to a
follow-up (see thread); anti-spoof + host firewall remain the fail-closed
boundary.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WBMWTEtQdJ4W5UrWuLHCck
Codex flagged that parallel `start`s race on shared state: two cold
launches could both stop/build/boot the singleton on the same rootfs/PID,
concurrent builds share the infra VM's buildah store, the cleanup did
`buildah rm -a` (nuking a peer build's container), and the rootfs cache
was populated non-atomically.
- `infra_vm.ensure_running`: a host flock (`singleton.lock`) around the
cold stop/build/boot path, with a double-checked health re-test under
the lock so a second launcher adopts rather than re-boots. The healthy
fast-path stays lock-free.
- `image_builder.build_agent_rootfs_dir`: a host flock (`.build.lock`)
around cache-lookup + build + publish; build into `.building-<digest>`
and publish by atomic `os.rename`, so a partial build never appears as
`agent-<digest>`.
- scoped cleanup: per-build named working containers
(`<tag>-smoke`/`-export`) removed by name instead of `buildah rm -a`;
also cleared before the build to recover from a crashed prior run.
Verified: agent image builds via the locked/atomic path, infra VM stays
healthy, agent boots, `claude --version` = 2.1.172. Full unit suite +
pyright green.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WBMWTEtQdJ4W5UrWuLHCck
The /31 point-to-point TAP does NOT make a guest source address
unspoofable: root in an agent VM can source another bottle's guest IP on
its own bbfc TAP. The isolation table only matched iifname class + port
(DNAT) and never bound iifname to its assigned ip saddr — so a spoofed
source was DNAT'd to the gateway and attributed to the *victim* bottle,
getting the victim's policy/tokens. Source-IP attribution was therefore
not actually sound.
Add one anti-spoof rule per slot in the isolation forward chain, before
the established/DNAT accepts: `iifname bbfcN ip saddr != <guestN> drop`.
Generated in the existing setup loop — no new dependency, ~pool_size
lines. Legit traffic (correct saddr) is unchanged; a spoofed saddr on any
bbfc TAP is dropped before it can be attributed.
Apply with a nixos-rebuild (the systemd unit re-runs this script). Codex
review blocker; the app-layer identity token (defense-in-depth) is wired
separately.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WBMWTEtQdJ4W5UrWuLHCck
The gateway used `FROM zricethezav/gitleaks AS gitleaks-src` purely to
COPY the binary out — a supply-chain surface (a whole third-party image as
a build input, tying us to its cadence). Install gitleaks from its official
release instead, pinned by version + SHA256 and verified. python (already
in the image) does the download, so no curl/wget is added. trixie apt also
ships gitleaks but an older 8.16; the pinned download keeps the verified
8.30.1 (byte-identical to what the image provided).
Verified: gateway + infra images build, gitleaks 8.30.1 runs in both.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WBMWTEtQdJ4W5UrWuLHCck
Addresses the review finding that buildah lived only in the orchestrator
image, so the persistent infra VM wasn't the builder — a separate throwaway
builder VM contended with it for the orchestrator TAP. Consolidate:
- **Rebase the gateway (and thus infra) on `python:3.12-slim` = Debian
trixie**, pip-installing mitmproxy instead of `FROM mitmproxy/mitmproxy`
(Debian bookworm). trixie ships buildah 1.39, which can build agent
Dockerfiles that use heredocs; bookworm's 1.28 can't (`Unknown
instruction: "{"`). CA path is unchanged (set via `--set confdir=`).
- **buildah lives only in `Dockerfile.infra`** now (removed from the
orchestrator image, which is lean/stdlib-only again).
- **Shared orchestrator content**: `Dockerfile.orchestrator` is the single
definition of the control-plane payload; the infra image `COPY --from`s
it (same trixie base → clean copy, and future deps like iroh are added
once). The docker backend runs the orchestrator image directly.
- **`image_builder` builds inside the infra VM** (which now has buildah)
over SSH — no throwaway builder VM, so the `bborch0` contention is gone.
`ensure_built` builds orchestrator + gateway before infra (FROM gateway,
COPY --from orchestrator).
Verified on a KVM host: images build (buildah 1.39 in infra), the agent
image builds *inside* the infra VM (heredoc Dockerfile and all), the infra
VM stays healthy, the agent boots and `claude --version` = 2.1.172. The
rebased gateway still starts as a docker container and generates its CA
(docker backend unaffected).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WBMWTEtQdJ4W5UrWuLHCck
Each nft table (isolation, orchestrator-egress nat, agent->gateway route)
was re-applied as a plain `table {...}` block, which on a bare `up`
re-apply (not the systemd down->up path) would append duplicate rules or
error on the existing base chains. Use the standard delete-first pattern
(create empty, delete, recreate) so `up` lands identical state regardless
of history — the setup reproduces cleanly on a fresh install and on
re-apply, not just via a full down->up cycle.
No functional change to the resulting ruleset; only its idempotency.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WBMWTEtQdJ4W5UrWuLHCck
git-gate now works end-to-end for git-upstream bottles on the infra VM.
Three fixes surfaced by driving a real clone through git-http:
- `SshGatewayTransport.cp_into` preserves the source file mode (docker cp
does). The access-hook is staged 0700 and git-http execs it directly; a
plain `cat >` landed it 0644 -> EACCES. Keys stay 0600.
- the infra init runs `BOT_BOTTLE_GATEWAY_DAEMONS=egress,git-http,supervise`:
the VM backend reaches git over git-http (9420), so the git:// daemon
(git-gate, whose /git-gate-entrypoint.sh the consolidated model doesn't
stage) is left out instead of crash-looping.
- `build_infra_rootfs_dir` folds the init's content-hash into the rootfs
cache key, so an init change actually rebuilds the rootfs (the base image
digest alone wouldn't catch it).
Verified on a KVM host: launch_consolidated provisions a git-upstream
bottle's repo + creds into the gateway VM over SSH; an agent VM clones via
git-http (source-IP attributed) and the access-hook resolves the
provisioned key + known_hosts and attempts the upstream SSH fetch —
failing closed only because the test used a dummy key + fake upstream
("refusing to serve stale data"). With real creds the fetch serves.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WBMWTEtQdJ4W5UrWuLHCck
The infra VM's rootfs is ephemeral (rebuilt each boot), so the bottle
registry DB needs durable storage across restarts. Give the infra VM a
firecracker analogue of a docker volume: a host-side ext4 file attached as
a second virtio-block device (guest /dev/vdb), mounted at the control
plane's DB dir (/var/lib/bot-bottle, where host_db_path lives at
db/bot-bottle.db).
- firecracker_vm.boot/_config take an optional `data_drive` (a non-root,
RW second drive).
- infra_vm creates the volume on first use (`mke2fs` an empty ext4 at
<fc-cache>/infra/registry.ext4) and mounts /dev/vdb in the PID-1 init
before the control plane starts. It's a plain ext4 file, so
`sudo mount -o loop <path>` (VM stopped) inspects bot-bottle.db directly.
Verified on a KVM host: register a bottle, restart the infra VM (fresh
rootfs, same volume) — /dev/vdb re-mounts and the registry `db/` dir + a
marker file survive the restart.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WBMWTEtQdJ4W5UrWuLHCck
consolidated_launch now drives the persistent infra VM instead of the
`_FirecrackerOrchestratorService` Docker containers — the point Docker
leaves the Firecracker launch path.
- launch_consolidated: `infra_vm.ensure_running()` (singleton) for the
control plane + gateway; register the bottle over HTTP at the infra VM's
guest IP; provision git-gate via `SshGatewayTransport` (over SSH into the
gateway VM); fetch the gateway CA via `InfraVm.gateway_ca_pem()`.
- teardown_consolidated deregisters + deprovisions but does NOT stop the
infra VM (persistent per-host singleton shared by every bottle).
- new `infra_vm.SshGatewayTransport` + `gateway_transport()` (built from the
stable key + orchestrator link IP, so teardown needs no live handle).
- drop the DockerGateway/OrchestratorService machinery from the firecracker
consolidated path (still used by the docker backend).
launch.py already calls launch_consolidated, so the whole firecracker
launch path now uses the VM. pyright + unit suite green.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WBMWTEtQdJ4W5UrWuLHCck
git-gate provisioning into the running gateway was hard-wired to docker
exec/cp. Extract a backend-neutral `GatewayTransport` (exec + cp_into) so
the same provisioning logic serves both the docker gateway container and
the firecracker gateway VM (over SSH, added with the launch swap).
- `provision_git_gate` / `deprovision_git_gate` now take a transport
instead of a gateway name; `DockerGatewayTransport` wraps the existing
docker exec/cp behavior. deprovision is best-effort (catches the
transport error) — matching the prior idempotent teardown.
- both consolidated_launch callers pass `DockerGatewayTransport(name)` —
no behavior change; the firecracker swap flips only its own to SSH.
Pure refactor: docker path unchanged, gateway_provision tests updated to
construct the transport, all green.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WBMWTEtQdJ4W5UrWuLHCck
The infra VM must outlive the short-lived `start` launcher and be reused
across launches. Add an idempotent singleton:
- `ensure_running()` adopts the infra VM when its control plane is already
healthy (a prior launcher booted it), else clears any stale VM and boots
a fresh one. Returns a handle usable for CA fetch / git-gate provisioning
whether we booted it or adopted it.
- boot is `detached` (firecracker in its own session via start_new_session)
so it survives the launcher exiting; its PID is recorded so a later
process can `stop()` it. `_kill_pidfile` SIGTERM/SIGKILLs but only if the
PID is still a firecracker process (guards a recycled PID).
- a STABLE SSH key (generated once under the infra cache dir, re-injected
each boot via the cmdline) so any launcher can SSH in to fetch the CA /
provision, not just the one that booted the VM. `InfraVm.vm` is None in
the adopted case; teardown then goes through the PID file.
Verified on a KVM host: first ensure_running boots; a second adopts it
(same PID, no reboot) and can still reach /health and fetch the gateway CA
over SSH; stop() tears it down (control plane then unreachable).
Next: git-gate provisioning into the VM over SSH (today docker exec/cp),
then swap consolidated_launch.py onto the infra VM.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WBMWTEtQdJ4W5UrWuLHCck
Agent VMs must reach the shared gateway that now runs in the infra VM
(egress:9099 / supervise:9100 / git-http:9420 at the orchestrator link's
guest IP). Add a PREROUTING DNAT: agents keep addressing their own
host-side TAP IP on the gateway ports, and the rule redirects that to the
infra VM. The isolation table's existing `ct status dnat accept` forward
rule lets the DNAT'd traffic through; every other agent egress stays
dropped, so a bottle still reaches only the gateway and nothing else.
Source IP is deliberately NOT masqueraded: the gateway attributes each
request to the originating bottle by its guest IP, which the /31 TAP + the
bot_bottle_fc nft table make unspoofable. Keeping the agent addressed at
its own host TAP IP means no per-bottle config change vs the docker-DNAT
path it replaces.
- scripts/firecracker-netpool.sh: `_install_gateway_route` adds
`table ip <table>_gw` (prerouting dstnat -> orch_guest on the gateway
ports); wired into up/down/status. The nix module needs no change — it
runs this script, and the ports are baked in.
Verified on a KVM host: an agent VM's `curl -x http://<its-host-tap>:9099`
reaches mitmproxy in the infra VM and gets a 403 (correct policy denial
for an unregistered bottle) — i.e. the route lands end-to-end. Persist
with a nixos-rebuild; the imperative rule holds until then.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WBMWTEtQdJ4W5UrWuLHCck
The single infra VM now runs BOTH the orchestrator control plane and the
gateway data plane (egress / supervise / git-http), multi-tenant against
the local control plane — the single-VM shape from the Stage B design.
- Dockerfile.infra: the firecracker infra image = the gateway image +
the baked control-plane source (FROM bot-bottle-gateway, COPY
bot_bottle). Reuses the gateway payload rather than copying mitmproxy/
gitleaks into a third image; the docker backend keeps its two separate
images.
- infra_vm: build from source (gateway then infra image), and the PID-1
init now also launches `gateway_init` with
BOT_BOTTLE_ORCHESTRATOR_URL=http://127.0.0.1:8099. Adds `gateway_ca_pem`
(fetch the mitmproxy CA over SSH) and the agent-facing port constants.
- init exports PATH — a bare-init shell resolves its own execs via a
built-in default path, but that isn't in the environment, so
gateway_init's `python3 ...` daemons would otherwise fail to spawn.
Verified on a KVM host: infra VM boots, control plane /health -> 200, and
egress:9099 / supervise:9100 / git-http:9420 all listen and are reachable
from the host over the TAP link; the gateway CA is retrievable. (git-gate
stays down until a bottle provisions its per-bottle entrypoint/creds, same
as a fresh docker gateway — non-fatal, the supervisor keeps the rest up.)
Next: agent->gateway VM-to-VM routing so bbfc* VMs reach these ports at
the infra VM and nowhere else.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WBMWTEtQdJ4W5UrWuLHCck
First slice of Stage B: the orchestrator control plane runs as a
persistent Firecracker infra VM instead of a Docker container. The host
CLI reaches it over HTTP at the orchestrator link's guest IP; agent VMs
will reach its gateway ports (added next) over VM-to-VM routing.
- Dockerfile.orchestrator bakes the stdlib-only control-plane source
(COPY bot_bottle) so the image is self-contained and runs from a built
image with no runtime bind-mount — a guest VM can't bind-mount host
source. (Build-from-source stays the default; a pull-from-registry mode
lands later. The docker backend's dev bind-mount still overlays this.)
- util.build_base_rootfs_dir / inject_guest_boot take a `variant` +
`init_script`, so the same orchestrator image is prepared two ways
without a cache collision: the builder VM keeps the SSH-only agent init;
the infra VM gets a control-plane PID-1 init.
- new firecracker/infra_vm.py: boot the infra VM on the orchestrator link,
run `python -m bot_bottle.orchestrator` as PID 1, and poll /health.
Verified on a KVM host: infra VM boots, control plane answers
`GET /health -> 200 {"status":"ok"}` from the host over the TAP link.
Next: fold gateway_init (egress/git-gate/supervise) into the same VM,
then agent->gateway routing.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WBMWTEtQdJ4W5UrWuLHCck
Replace the host `docker build` + `docker export` behind the Firecracker
agent rootfs with an in-VM buildah build. `image_builder.build_agent_rootfs_dir`
boots a throwaway builder VM (the orchestrator image, which carries
buildah) on the NAT'd orchestrator link, sends the Dockerfile over SSH,
`buildah build`s it, smoke-tests the result with `buildah run` (the
image's own PATH, so it catches an npm silent-failure stub), and streams
the rootfs tar back into the content-addressed cache dir — the same base
dir `util.build_rootfs_ext4` already turns into a bootable ext4 with
`mke2fs -d`. No host Docker daemon, no root-equivalent `docker` group;
an untrusted Dockerfile runs in a confined microVM, not on the host.
- new firecracker/image_builder.py (boot → build → smoke → stream).
- launch.py: `_build_agent_image` → `_build_agent_base`, returning the
base dir from the builder VM. The committed-snapshot (freeze/migrate)
path still exports via host docker until it too is ported.
- util: `_inject_guest_boot` → public `inject_guest_boot` (shared with
the builder). unit tests for the cache decision + smoke-test paths.
Verified on a KVM host end-to-end: builds the real claude Dockerfile
(node:22-slim + npm claude-code) in-VM in ~60s, the produced agent VM
boots and `claude --version` returns 2.1.172; the content cache skips
the rebuild on a repeat launch.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WBMWTEtQdJ4W5UrWuLHCck
The orchestrator/gateway VM is trusted infra, not an isolated agent: it
builds agent images in-VM (buildah must FROM-pull + apt/npm) and, in the
Stage B cutover, forwards agent egress upstream. Give it a dedicated TAP
(`bborch0`) on a /31 at the top of the IP_BASE /16 (clear of the bbfc*
agent pool at the bottom), NAT'd out the host uplink — while agent VMs
keep their fail-closed, gateway-only isolation table.
- netpool.defaults.env / netpool.py: new BOT_BOTTLE_FC_ORCH_IFACE +
`orch_slot()` (index -1 sentinel; host x.y.255.0 / guest x.y.255.1).
- scripts/firecracker-netpool.sh: create + address the orchestrator TAP;
`bot_bottle_fc_nat` table masquerades its /31 out the uplink and
accepts its forward path. Because bootstrap still runs Docker (whose
FORWARD policy is DROP), a best-effort, guarded, idempotent DOCKER-USER
ACCEPT is added too (skipped once Docker is gone). down/status updated.
- nix/firecracker-netpool.nix: mirror the option, pass it via the unit
Environment= (the store-copied script can't read the defaults file),
and add iptables to the unit path for the DOCKER-USER step.
Agent isolation is unchanged: the new rules only ever accept/masquerade
the orchestrator link and never drop, so they can't weaken the bbfc*
drops. Applied by re-running `sudo ./scripts/firecracker-netpool.sh up`.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WBMWTEtQdJ4W5UrWuLHCck
The Firecracker backend's remaining host-Docker dependency is building
users' agent Dockerfiles. Move that build *into the orchestrator VM*:
give the orchestrator image buildah (rootless, daemonless) so it builds
agent images itself, and the host needs no Docker daemon and no
root-equivalent `docker` group. An untrusted Dockerfile then builds
inside the confined orchestrator VM rather than on the host — strictly
more isolated than host `docker build`.
- buildah + crun (OCI runtime) + netavark/aardvark-dns (network backend
for FROM pulls and RUN egress), installed explicitly since
--no-install-recommends strips buildah's helper deps.
- vfs storage + chroot isolation (STORAGE_DRIVER/BUILDAH_ISOLATION) so
buildah needs neither fuse-overlayfs / an overlay kernel module nor
configured subuid maps — it works unconditionally as root in a minimal
microVM rootfs. (Slower than overlay; a build-cache pass is deferred.)
Bootstrap: the orchestrator rootfs is still produced from this image via
host docker export; a later step pulls a pre-built image instead.
Verified: the orchestrator image boots as a Firecracker VM and buildah
runs inside it as real VM root (reaches the base-image pull; build-time
egress wiring is the next step).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WBMWTEtQdJ4W5UrWuLHCck
The rebase onto lazy-backend-imports converts existing helpers (image_exists,
container_exists, etc.) to run_docker; the three new functions added in this
branch still called subprocess.run directly. Switch them over for consistency.
Replace the per-bottle Docker sidecar bundle with the shared per-host
orchestrator + gateway, mirroring what the Docker backend already has.
- Add `bot_bottle/backend/firecracker/consolidated_launch.py`:
`_FirecrackerOrchestratorService` (subclasses `OrchestratorService`,
overrides `_gateway()` to return a `DockerGateway` with host port
bindings so Firecracker VMs can reach it via their TAP link);
`launch_consolidated()` registers the bottle by guest IP (attribution
key), provisions git-gate into the shared gateway, and returns the
shared CA + orchestrator URL for teardown; `teardown_consolidated()`
deregisters and cleans up.
- Rewrite `bot_bottle/backend/firecracker/launch.py`: removes the
per-bottle sidecar bundle (`_start_sidecar_bundle`, `_stage_git_gate`,
etc.) and `_mint_certs`; wires `launch_consolidated()` instead. The VM
still sends to `host_tap_ip:PORT` — Docker's PREROUTING DNAT + the nft
`ct status dnat accept` rule in the forward chain route the traffic to
the shared gateway container.
- Extend `DockerGateway` with `host_port_bindings` so the Firecracker
gateway publishes its ports on the host (`0.0.0.0:PORT`).
- Parameterise `OrchestratorService` with `orchestrator_name` /
`orchestrator_label` so Docker and Firecracker orchestrators can
coexist on the same host (`bot-bottle-orchestrator` vs
`bot-bottle-fc-orchestrator`).
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
firecracker/launch.py reimplemented docker build/image-exists/rm/exec/cp
as private functions instead of the shared docker_mod used by the
docker and macos-container backends. Switching to docker_mod dedupes
the logic and gets --no-cache support for free (docker_mod.build_image
already reads BOT_BOTTLE_NO_CACHE); docker_mod gains docker_exec/
docker_cp general-purpose helpers to cover what the private versions did.
npm treats optionalDependencies failures as non-fatal, so a transient
network blip fetching claude-code's platform-native binary during
`npm install -g` left a stub CLI in an image that still "built"
successfully — then got baked into the Docker/Container layer cache
until forced to rebuild. Post-build smoke test (provider-declared
argv, run in a throwaway container of the freshly built image) fails
the launch loudly instead of shipping a broken image; --no-cache
gives an escape hatch to force a from-scratch rebuild.
Closes#353.
didericis
didericis-claude
Quality Badge Bot