The Firecracker infra VM started the control plane with no signing key, so it
ran open and granted every unauthenticated caller the `cli` role. The nft
boundary only fences off the separate agent VM — the egress / supervise /
git-http daemons run in this same VM and reach the orchestrator over 127.0.0.1,
so a compromised data-plane daemon could still drive the operator routes
(approve its own supervise proposals, rewrite policy, read injected tokens).
Generate the signing key on the persistent volume, hand it only to the
orchestrator process, mint a `gateway` JWT for the data-plane daemons, and
mirror the key back to the host token file so the CLI signs `cli` tokens the VM
verifies — the same per-process scoping the docker / macOS launchers use.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
PRD 0070's rule — only the orchestrator opens bot-bottle.db; the data
plane reaches state through the control-plane RPC — was not in force.
Three data-plane daemons held a direct read-write handle on the shared
SQLite file: the supervise MCP server, the egress DLP addon (the most
attack-exposed process, TLS-bumping hostile traffic), and the git-gate
pre-receive hook. An RCE in any of them could read every bottle's
plaintext identity_token and forge attribution fleet-wide (issue #469).
Add the agent half of the supervise flow to the control plane:
POST /supervise/propose -> queue a proposal, 201 {proposal_id}
POST /supervise/poll -> non-blocking decision poll, 200 {status,...}
Both attribute the caller by (source_ip, identity_token) exactly like
/resolve — never a caller-supplied slug — so a bottle can only ever queue
or read its own proposals even if the data plane is compromised. A decided
poll archives server-side, preserving the archive-after-read contract.
Data plane: the supervise server, egress addon, and git-gate hook now
queue/poll through PolicyResolver.propose_supervise / poll_supervise
instead of opening the DB. supervise_server keeps its ~30s grace window
by polling the RPC; egress keeps its safelist keyed by resolved bottle;
the git-gate hook gets (source_ip, identity_token) from the CGI env.
Packaging: drop the DB bind-mount and SUPERVISE_DB_PATH from the
data-plane containers/VMs (docker gateway + infra, macOS infra,
firecracker infra). The orchestrator remains the sole opener of the one
file via BOT_BOTTLE_ROOT / host_db_path().
Update PRD 0070: the rule is now in force; remove the transitional caveat.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Extract the shared CA cert polling loop into `backend/util.poll_ca_cert`
(firecracker and macos backends were duplicating deadline/sleep/raise logic).
Each caller now wraps a fetch lambda and converts TimeoutError to its own
error type. Also corrects the PRD port publication line from {port}:{port}
to {host_port}:8099.
Collapses the two-container Docker model (gateway + orchestrator) into one
bot-bottle-infra container, matching the macOS and Firecracker backends.
- Dockerfile.infra: now a shared gateway+orchestrator base (COPY bot_bottle
from orchestrator build, no CMD override)
- Dockerfile.infra.fc: new Firecracker-specific layer (buildah/crun/netavark)
- gateway_init: adds orchestrator daemon with _OPT_IN_DAEMONS gating so it
only starts when BOT_BOTTLE_GATEWAY_DAEMONS explicitly includes it
- orchestrator/lifecycle: OrchestratorService manages one infra container;
builds orchestrator (intermediate) then infra; live source bind-mounted at
/bot-bottle-src with PYTHONPATH so the subprocess uses the checkout
- backend/consolidated_util: extracts provision_bottle + teardown_consolidated
shared across all three backends; removes duplication in docker/fc/macos
consolidated_launch modules
- firecracker/infra_vm: builds four images (orchestrator→gateway→infra→infra.fc)
- All unit tests updated and passing (1878 tests)
- PRD status: Draft → Active
The infra VM is a per-host singleton that outlives short-lived launchers, so
`ensure_running` adopts it when its control plane is healthy. But it adopted
ANY healthy VM regardless of the code that built it — so after an infra-code
change the old VM kept being adopted and the new code never booted. The only
way to dislodge it was an out-of-band `kill`, which then raced whatever
launched next. On CI this meant every infra change needed a manual VM kill.
Make adoption version-aware:
* boot records the infra-artifact version it booted from in a `booted-version`
marker beside the singleton; `stop` clears it.
* `ensure_running` adopts only when the marker matches the current version
(`_adoptable`); a missing/mismatched marker falls through to stop + reboot.
So a stale VM is replaced automatically on the next launch — no manual kill,
and it's concurrency-safe (the reboot happens under the singleton flock).
Also harden teardown: the PID file drifts after crashes / out-of-band kills,
so `stop` now also reaps any orphaned firecracker still bound to the infra
config path (scoped to that path, so interactive-pool VMs are untouched) —
otherwise a survivor holds the orchestrator TAP and the fresh boot dies with
"tap … Resource busy".
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01A9qa3xoavjQScufDfZaXKR
The infra VM's init boots the orchestrator control plane AND the gateway data
plane (egress/git-http/supervise). Since 5ad3449 moved the daemons into the
installed `bot_bottle` package, there is no `/app/gateway_init.py` file — the
gateway image's entrypoint is `python3 -m bot_bottle.gateway_init`. But the
firecracker init still spawned the old file path, so the guest logged:
python3: can't open file '/app/gateway_init.py': No such file or directory
The control plane came up (it already used `python3 -m bot_bottle.orchestrator`)
but the gateway never started, so mitmproxy never generated its CA and launch
died with "gateway CA not available after 30s". Mirror the orchestrator line:
run the gateway as `python3 -m bot_bottle.gateway_init` (bot_bottle resolves
from the /app CWD, same as the control plane; egress-entrypoint.sh /
egress_addon.py are present from the gateway base image).
Third build/runtime bug from 5ad3449's package refactor that the Firecracker
integration suite never exercised (the artifact publish was broken, so the
job 404'd before boot). Changes the init, so the infra artifact version moves;
the matching rootfs has been rebuilt and published.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01A9qa3xoavjQScufDfZaXKR
Stage 2 of the docker-free Firecracker backend (#348): stop building the
fixed infra image on the launch host. The infra VM's rootfs is host- and
bottle-agnostic (authorized_keys + guest IP ride the kernel cmdline, not the
rootfs), so it's built once off-host and published as a versioned, ready-to-
boot ext4; the launch host downloads + verifies + boots it — no Docker, no
image tooling, just HTTP + gunzip.
- infra_artifact.py: version = content hash of the rootfs inputs (the shipped
bot_bottle package + the three Dockerfiles + the init), so a launch host
pulls the artifact matching its code and a content change can't silently
boot a stale rootfs. Pull + sha256-verify (fail-closed) + gunzip from a
Gitea generic package; base/owner/token configurable, default this Gitea.
- infra_vm.ensure_built/boot default to the pull path; BOT_BOTTLE_INFRA_BUILD=
local keeps the docker build-from-source path for iterating on Dockerfiles.
- publish_infra.py: the off-host half — builds the images with Docker, mke2fs
the rootfs (with buildah slack), gzips, and PUTs it to the generic package.
Rollout note: default=pull means a launch 404s until an artifact is published;
until the Gitea packages endpoint is enabled + an artifact published, use
BOT_BOTTLE_INFRA_BUILD=local. Freeze/migrate's remaining docker use is a
separate PR.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UoEZHDjv84ChoZbozQERhJ
The firecracker supervise MCP daemon 500'd (-32603) on egress-allow/block:
it ran with no BOT_BOTTLE_ROOT/SUPERVISE_DB_PATH, so it targeted a stray,
unmigrated SQLite file and `write_proposal` hit "no such table:
supervise_proposals". Meanwhile the in-VM control plane migrated only the
registry table (orchestrator_bottles) into its own DB, and the host
operator reads a third, disconnected DB — three files, none shared.
Consolidate to one DB per host, owned by the control plane on the
persisted registry volume (/var/lib/bot-bottle/db/bot-bottle.db):
- orchestrator startup now migrates the supervise queue + audit tables
into the same file it migrates the registry into (StoreManager), so the
control-plane DB carries every table.
- the in-VM supervise daemon is pointed at that same file via
SUPERVISE_DB_PATH, so daemon and control plane share one queue.
`list-egress-routes` already worked (no DB); egress-allow now queues +
waits on the single persisted DB instead of erroring. Validated against a
live infra VM: migrate + write_proposal succeeds and the proposal is
queued. The host-operator HTTP bridge (so approvals complete from the
host, unifying docker onto the same path) is the follow-up.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WBMWTEtQdJ4W5UrWuLHCck
Codex flagged that parallel `start`s race on shared state: two cold
launches could both stop/build/boot the singleton on the same rootfs/PID,
concurrent builds share the infra VM's buildah store, the cleanup did
`buildah rm -a` (nuking a peer build's container), and the rootfs cache
was populated non-atomically.
- `infra_vm.ensure_running`: a host flock (`singleton.lock`) around the
cold stop/build/boot path, with a double-checked health re-test under
the lock so a second launcher adopts rather than re-boots. The healthy
fast-path stays lock-free.
- `image_builder.build_agent_rootfs_dir`: a host flock (`.build.lock`)
around cache-lookup + build + publish; build into `.building-<digest>`
and publish by atomic `os.rename`, so a partial build never appears as
`agent-<digest>`.
- scoped cleanup: per-build named working containers
(`<tag>-smoke`/`-export`) removed by name instead of `buildah rm -a`;
also cleared before the build to recover from a crashed prior run.
Verified: agent image builds via the locked/atomic path, infra VM stays
healthy, agent boots, `claude --version` = 2.1.172. Full unit suite +
pyright green.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WBMWTEtQdJ4W5UrWuLHCck
Addresses the review finding that buildah lived only in the orchestrator
image, so the persistent infra VM wasn't the builder — a separate throwaway
builder VM contended with it for the orchestrator TAP. Consolidate:
- **Rebase the gateway (and thus infra) on `python:3.12-slim` = Debian
trixie**, pip-installing mitmproxy instead of `FROM mitmproxy/mitmproxy`
(Debian bookworm). trixie ships buildah 1.39, which can build agent
Dockerfiles that use heredocs; bookworm's 1.28 can't (`Unknown
instruction: "{"`). CA path is unchanged (set via `--set confdir=`).
- **buildah lives only in `Dockerfile.infra`** now (removed from the
orchestrator image, which is lean/stdlib-only again).
- **Shared orchestrator content**: `Dockerfile.orchestrator` is the single
definition of the control-plane payload; the infra image `COPY --from`s
it (same trixie base → clean copy, and future deps like iroh are added
once). The docker backend runs the orchestrator image directly.
- **`image_builder` builds inside the infra VM** (which now has buildah)
over SSH — no throwaway builder VM, so the `bborch0` contention is gone.
`ensure_built` builds orchestrator + gateway before infra (FROM gateway,
COPY --from orchestrator).
Verified on a KVM host: images build (buildah 1.39 in infra), the agent
image builds *inside* the infra VM (heredoc Dockerfile and all), the infra
VM stays healthy, the agent boots and `claude --version` = 2.1.172. The
rebased gateway still starts as a docker container and generates its CA
(docker backend unaffected).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WBMWTEtQdJ4W5UrWuLHCck
git-gate now works end-to-end for git-upstream bottles on the infra VM.
Three fixes surfaced by driving a real clone through git-http:
- `SshGatewayTransport.cp_into` preserves the source file mode (docker cp
does). The access-hook is staged 0700 and git-http execs it directly; a
plain `cat >` landed it 0644 -> EACCES. Keys stay 0600.
- the infra init runs `BOT_BOTTLE_GATEWAY_DAEMONS=egress,git-http,supervise`:
the VM backend reaches git over git-http (9420), so the git:// daemon
(git-gate, whose /git-gate-entrypoint.sh the consolidated model doesn't
stage) is left out instead of crash-looping.
- `build_infra_rootfs_dir` folds the init's content-hash into the rootfs
cache key, so an init change actually rebuilds the rootfs (the base image
digest alone wouldn't catch it).
Verified on a KVM host: launch_consolidated provisions a git-upstream
bottle's repo + creds into the gateway VM over SSH; an agent VM clones via
git-http (source-IP attributed) and the access-hook resolves the
provisioned key + known_hosts and attempts the upstream SSH fetch —
failing closed only because the test used a dummy key + fake upstream
("refusing to serve stale data"). With real creds the fetch serves.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WBMWTEtQdJ4W5UrWuLHCck
The infra VM's rootfs is ephemeral (rebuilt each boot), so the bottle
registry DB needs durable storage across restarts. Give the infra VM a
firecracker analogue of a docker volume: a host-side ext4 file attached as
a second virtio-block device (guest /dev/vdb), mounted at the control
plane's DB dir (/var/lib/bot-bottle, where host_db_path lives at
db/bot-bottle.db).
- firecracker_vm.boot/_config take an optional `data_drive` (a non-root,
RW second drive).
- infra_vm creates the volume on first use (`mke2fs` an empty ext4 at
<fc-cache>/infra/registry.ext4) and mounts /dev/vdb in the PID-1 init
before the control plane starts. It's a plain ext4 file, so
`sudo mount -o loop <path>` (VM stopped) inspects bot-bottle.db directly.
Verified on a KVM host: register a bottle, restart the infra VM (fresh
rootfs, same volume) — /dev/vdb re-mounts and the registry `db/` dir + a
marker file survive the restart.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WBMWTEtQdJ4W5UrWuLHCck
consolidated_launch now drives the persistent infra VM instead of the
`_FirecrackerOrchestratorService` Docker containers — the point Docker
leaves the Firecracker launch path.
- launch_consolidated: `infra_vm.ensure_running()` (singleton) for the
control plane + gateway; register the bottle over HTTP at the infra VM's
guest IP; provision git-gate via `SshGatewayTransport` (over SSH into the
gateway VM); fetch the gateway CA via `InfraVm.gateway_ca_pem()`.
- teardown_consolidated deregisters + deprovisions but does NOT stop the
infra VM (persistent per-host singleton shared by every bottle).
- new `infra_vm.SshGatewayTransport` + `gateway_transport()` (built from the
stable key + orchestrator link IP, so teardown needs no live handle).
- drop the DockerGateway/OrchestratorService machinery from the firecracker
consolidated path (still used by the docker backend).
launch.py already calls launch_consolidated, so the whole firecracker
launch path now uses the VM. pyright + unit suite green.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WBMWTEtQdJ4W5UrWuLHCck
The infra VM must outlive the short-lived `start` launcher and be reused
across launches. Add an idempotent singleton:
- `ensure_running()` adopts the infra VM when its control plane is already
healthy (a prior launcher booted it), else clears any stale VM and boots
a fresh one. Returns a handle usable for CA fetch / git-gate provisioning
whether we booted it or adopted it.
- boot is `detached` (firecracker in its own session via start_new_session)
so it survives the launcher exiting; its PID is recorded so a later
process can `stop()` it. `_kill_pidfile` SIGTERM/SIGKILLs but only if the
PID is still a firecracker process (guards a recycled PID).
- a STABLE SSH key (generated once under the infra cache dir, re-injected
each boot via the cmdline) so any launcher can SSH in to fetch the CA /
provision, not just the one that booted the VM. `InfraVm.vm` is None in
the adopted case; teardown then goes through the PID file.
Verified on a KVM host: first ensure_running boots; a second adopts it
(same PID, no reboot) and can still reach /health and fetch the gateway CA
over SSH; stop() tears it down (control plane then unreachable).
Next: git-gate provisioning into the VM over SSH (today docker exec/cp),
then swap consolidated_launch.py onto the infra VM.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WBMWTEtQdJ4W5UrWuLHCck
The single infra VM now runs BOTH the orchestrator control plane and the
gateway data plane (egress / supervise / git-http), multi-tenant against
the local control plane — the single-VM shape from the Stage B design.
- Dockerfile.infra: the firecracker infra image = the gateway image +
the baked control-plane source (FROM bot-bottle-gateway, COPY
bot_bottle). Reuses the gateway payload rather than copying mitmproxy/
gitleaks into a third image; the docker backend keeps its two separate
images.
- infra_vm: build from source (gateway then infra image), and the PID-1
init now also launches `gateway_init` with
BOT_BOTTLE_ORCHESTRATOR_URL=http://127.0.0.1:8099. Adds `gateway_ca_pem`
(fetch the mitmproxy CA over SSH) and the agent-facing port constants.
- init exports PATH — a bare-init shell resolves its own execs via a
built-in default path, but that isn't in the environment, so
gateway_init's `python3 ...` daemons would otherwise fail to spawn.
Verified on a KVM host: infra VM boots, control plane /health -> 200, and
egress:9099 / supervise:9100 / git-http:9420 all listen and are reachable
from the host over the TAP link; the gateway CA is retrievable. (git-gate
stays down until a bottle provisions its per-bottle entrypoint/creds, same
as a fresh docker gateway — non-fatal, the supervisor keeps the rest up.)
Next: agent->gateway VM-to-VM routing so bbfc* VMs reach these ports at
the infra VM and nowhere else.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WBMWTEtQdJ4W5UrWuLHCck
First slice of Stage B: the orchestrator control plane runs as a
persistent Firecracker infra VM instead of a Docker container. The host
CLI reaches it over HTTP at the orchestrator link's guest IP; agent VMs
will reach its gateway ports (added next) over VM-to-VM routing.
- Dockerfile.orchestrator bakes the stdlib-only control-plane source
(COPY bot_bottle) so the image is self-contained and runs from a built
image with no runtime bind-mount — a guest VM can't bind-mount host
source. (Build-from-source stays the default; a pull-from-registry mode
lands later. The docker backend's dev bind-mount still overlays this.)
- util.build_base_rootfs_dir / inject_guest_boot take a `variant` +
`init_script`, so the same orchestrator image is prepared two ways
without a cache collision: the builder VM keeps the SSH-only agent init;
the infra VM gets a control-plane PID-1 init.
- new firecracker/infra_vm.py: boot the infra VM on the orchestrator link,
run `python -m bot_bottle.orchestrator` as PID 1, and poll /health.
Verified on a KVM host: infra VM boots, control plane answers
`GET /health -> 200 {"status":"ok"}` from the host over the TAP link.
Next: fold gateway_init (egress/git-gate/supervise) into the same VM,
then agent->gateway routing.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WBMWTEtQdJ4W5UrWuLHCck
didericis-claude
didericis