39 Commits

Author	SHA1	Message	Date
didericis	dfc693e0b6	fix(firecracker): keep the snapshot partial private even if one was left behind ... Follow-up to the codex review on #398. os.open's mode arg only applies on creation, so a committed-rootfs.tar.partial left 0644 by an interrupted run would be opened/truncated (not re-moded) and stay world-readable for the whole SSH stream. Unlink any leftover and exclusively recreate it (O_EXCL|O_NOFOLLOW), then fchmod 0600 immediately so umask can't loosen it. Test pre-creates a 0644 partial and asserts the fd is 0600 mid-stream. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UoEZHDjv84ChoZbozQERhJ	2026-07-17 01:02:19 -04:00
didericis	39d47b8108	fix(firecracker): harden committed-snapshot resume against guest-controlled data ... Address the codex review on #398: - P1: inject_guest_boot no longer follows a symlink at bb-init/bb-dropbear. A committed snapshot is guest-controlled and could plant those paths as symlinks aimed at a host file (e.g. bb-init -> ~/.bashrc); write_text / copy2 would then overwrite the target as the host user during resume. Replace any pre-existing entry and create the files with O_EXCL|O_NOFOLLOW so the write stays inside the staging tree. - P2: write the snapshot tar owner-only (0600). It can contain the bottle's private workspace; it was being created world-readable (0644). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UoEZHDjv84ChoZbozQERhJ	2026-07-17 00:44:06 -04:00
didericis	5c08701983	feat(firecracker): port freeze/migrate off host Docker (PRD 0069 / #397) ... The last host-Docker dependency in the Firecracker launch path. Freeze and resume no longer touch the docker daemon, so the backend needs firecracker + KVM only — completing #348. Freeze: stream the guest rootfs over SSH straight into a persistent committed-rootfs.tar (the resumable/migratable artifact) instead of round-tripping through `docker build` from a scratch image. Written to a .partial sibling and atomically renamed so a failed freeze leaves no truncated artifact. Resume: extract the snapshot tar into a cached base dir and feed it to the existing rootless `mke2fs -d` pipeline, replacing the `docker create` + `docker export | tar` path. Recreate the proc/sys/dev/run mount points the freezer excludes so the guest init can mount them. `util.build_base_rootfs_dir` / `docker_image_id` stay — they still back the opt-in BOT_BOTTLE_INFRA_BUILD=local dev path and off-host publish_infra, which are out of scope. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UoEZHDjv84ChoZbozQERhJ	2026-07-17 00:31:00 -04:00
didericis	e3e195f866	chore(firecracker): name the artifact package bot-bottle-firecracker-infra ... Rename the Gitea generic package from bot-bottle-infra to bot-bottle-firecracker-infra so it's self-evident in the package list which backend it serves (and leaves room for other artifacts, e.g. a shipped kernel). The version slot stays the content hash — "firecracker" belongs in the package name, not the version. Docker image / VM names are unchanged. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UoEZHDjv84ChoZbozQERhJ	2026-07-17 00:04:48 -04:00
didericis	e3d24b7e41	chore(firecracker): ship an about.txt description with the infra artifact ... Generic packages have no description field, so publish_infra now uploads a short about.txt alongside the rootfs on every publish — it's what identifies the package as the Firecracker backend's infra rootfs on the package page. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UoEZHDjv84ChoZbozQERhJ	2026-07-16 23:59:38 -04:00
didericis	f2891a1634	fix(firecracker): hash all baked-in files + stream the artifact upload ... Address PR #395 review (two P1s): - Version hash covered only `bot_bottle/**.py`, but the image `COPY`s the whole package — non-Python inputs baked in (egress_entrypoint.sh, netpool.defaults.env) didn't change the version, so a launch host could boot a stale rootfs whose code differs from its checkout. Hash every regular file under bot_bottle/ (excluding __pycache__/.pyc). Regression tests: a shell-script change bumps the version; .pyc/__pycache__ don't. - publish_infra `_put` read the whole (hundreds-of-MB) gz into memory via read_bytes(). Stream it from disk with an explicit Content-Length; the tiny .sha256 stays in-memory. Test asserts the body is the file object, not bytes. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UoEZHDjv84ChoZbozQERhJ	2026-07-16 23:46:46 -04:00
didericis	8d8a88aeeb	fix(firecracker): read only BOT_BOTTLE_INFRA_ARTIFACT_TOKEN for the artifact ... Drop the fallback to the general-purpose BOT_BOTTLE_CLAUDE_GITEA_TOKEN so the artifact pull/publish uses a dedicated, package-scoped token that can be granted (or revoked) independently of the general Gitea token. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UoEZHDjv84ChoZbozQERhJ	2026-07-16 23:41:53 -04:00
didericis	18f190b7e3	feat(firecracker): pull the infra rootfs as a prebuilt artifact (PRD 0069 Stage 2) ... Stage 2 of the docker-free Firecracker backend (#348): stop building the fixed infra image on the launch host. The infra VM's rootfs is host- and bottle-agnostic (authorized_keys + guest IP ride the kernel cmdline, not the rootfs), so it's built once off-host and published as a versioned, ready-to- boot ext4; the launch host downloads + verifies + boots it — no Docker, no image tooling, just HTTP + gunzip. - infra_artifact.py: version = content hash of the rootfs inputs (the shipped bot_bottle package + the three Dockerfiles + the init), so a launch host pulls the artifact matching its code and a content change can't silently boot a stale rootfs. Pull + sha256-verify (fail-closed) + gunzip from a Gitea generic package; base/owner/token configurable, default this Gitea. - infra_vm.ensure_built/boot default to the pull path; BOT_BOTTLE_INFRA_BUILD= local keeps the docker build-from-source path for iterating on Dockerfiles. - publish_infra.py: the off-host half — builds the images with Docker, mke2fs the rootfs (with buildah slack), gzips, and PUTs it to the generic package. Rollout note: default=pull means a launch 404s until an artifact is published; until the Gitea packages endpoint is enabled + an artifact published, use BOT_BOTTLE_INFRA_BUILD=local. Freeze/migrate's remaining docker use is a separate PR. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UoEZHDjv84ChoZbozQERhJ	2026-07-16 23:41:53 -04:00
didericis	2641ab70fd	feat(supervise): start orchestrator on demand via backend-agnostic bring-up ... Add BottleBackend.ensure_orchestrator() -> str: the backend-agnostic entry point that brings up the per-host orchestrator + shared gateway (idempotent) and returns its host-reachable control-plane URL. Docker starts the orchestrator + gateway containers (OrchestratorService); firecracker boots the infra VM; macos-container dies with a pointer (no orchestrator). Previously bring-up was reachable only through each backend's consolidated_launch, with no shared handle. Wire it into `bot-bottle supervise`: supervise is often the first thing an operator runs, before any bottle has booted the control plane, so `_resolve_orchestrator_url` now starts the selected backend's orchestrator on demand when discovery finds nothing, instead of failing with "launch a bottle first". Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UoEZHDjv84ChoZbozQERhJ	2026-07-16 21:52:11 -04:00
didericis	38bc555dbf	fix(supervise): single migrated DB for firecracker — orchestrator owns supervise tables ... The firecracker supervise MCP daemon 500'd (-32603) on egress-allow/block: it ran with no BOT_BOTTLE_ROOT/SUPERVISE_DB_PATH, so it targeted a stray, unmigrated SQLite file and `write_proposal` hit "no such table: supervise_proposals". Meanwhile the in-VM control plane migrated only the registry table (orchestrator_bottles) into its own DB, and the host operator reads a third, disconnected DB — three files, none shared. Consolidate to one DB per host, owned by the control plane on the persisted registry volume (/var/lib/bot-bottle/db/bot-bottle.db): - orchestrator startup now migrates the supervise queue + audit tables into the same file it migrates the registry into (StoreManager), so the control-plane DB carries every table. - the in-VM supervise daemon is pointed at that same file via SUPERVISE_DB_PATH, so daemon and control plane share one queue. `list-egress-routes` already worked (no DB); egress-allow now queues + waits on the single persisted DB instead of erroring. Validated against a live infra VM: migrate + write_proposal succeeds and the proposal is queued. The host-operator HTTP bridge (so approvals complete from the host, unifying docker onto the same path) is the follow-up. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WBMWTEtQdJ4W5UrWuLHCck	2026-07-16 18:24:45 -04:00
didericis	a208bcde08	feat(firecracker): stream buildah build output live during agent-image build ... The agent-image build ran over ssh with output fully captured, shown only as a 20-line tail on failure — so a successful (or in-progress) first build was a long silent wait through the base pull + apt/npm installs. Stream the `buildah build` step's stdout/stderr straight to our own (like the docker backend's `docker build`) via a non-capturing _ssh_streamed helper, so the operator sees `STEP i/n` progress live. Failure now points at the streamed output above instead of a captured tail. The smoke test and rootfs export stay captured (short / piped). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WBMWTEtQdJ4W5UrWuLHCck	2026-07-16 18:01:59 -04:00
didericis	c0066d2cd2	fix(firecracker): quote guest argv tokens so codex's multi-word prompt survives ssh ... The interactive agent command is sent to the guest by spreading the remote argv as separate ssh arguments; ssh space-joins everything after the host into one line that the guest login shell re-parses. That only works while every token is a "simple word" — which held for claude (`--append-system-prompt-file <path>`) but not for codex's `read_prompt_file` mode, whose positional is a whole sentence: "Read and follow the instructions in <path>.". The guest shell re-split it on spaces, so codex received `Read` as the prompt and `and`, `follow`, … as extra args — failing with `unrecognized subcommand 'and'` the moment an interactive codex session attached. Pre-quote each remote token with shlex.quote before ssh joins them (the same ssh→guest-shell discipline infra_vm/cp_in already use). Simple words are unchanged, so existing behaviour and the parity/structure tests are untouched; an arg with spaces now survives as a single argument. Regression test round-trips the joined remote command back through shlex.split and asserts codex's prompt comes out as exactly one arg. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WBMWTEtQdJ4W5UrWuLHCck	2026-07-16 18:01:59 -04:00
didericis	d4b27ebf1f	feat(gateway): mandatory identity-token attribution on every data plane (PR #354 review) ... Codex review: the /31 TAP doesn't make source IP unspoofable, and the app-layer token was returned by launch but never delivered or enforced, so a spoofed source could select a victim bottle's policy/tokens. Make the token mandatory and deliver it on each attributed plane (anti-spoof landed separately as the network boundary). Enforcement (control plane): - `Orchestrator.resolve` now requires a matching (source_ip, identity_token) pair (constant-time) — no source-IP-only fallback. `/resolve` fail-closes (403) on a missing/empty/mismatched token. Delivery, per plane (the token is `token_urlsafe`, safe in a URL): - egress: proxy credentials (`HTTPS_PROXY=http://bottle:<token>@gw`). The addon reads `Proxy-Authorization` — from the request (HTTP) or captured at the CONNECT for HTTPS tunnels (keyed by client conn, cleared on disconnect) — validates, and strips it (+ the legacy header) before upstream. - git-http: a URL-scoped `http.<gate>/.extraHeader: x-bot-bottle-identity` in the agent's git config (only over the http transport). - supervise: `mcp add --header x-bot-bottle-identity: <token>` (claude + codex); the server reads the header and passes it to resolve. Wiring: thread `ctx.identity_token` onto the firecracker + docker plans and into the agent env/config at launch. Verified on a KVM host: egress with the correct proxy-cred token returns 200 (HTTP and HTTPS/CONNECT), and no-token / wrong-token return 403; a real `cli.py start --backend=firecracker` launch provisions git config + the supervise MCP header and reaches the agent session, all under mandatory enforcement. Fixed a `claude mcp add` arg-order bug (--header must follow the positional name/url) found by that launch. Transparent proxy for tools that ignore proxy env is deferred to a follow-up (see thread); anti-spoof + host firewall remain the fail-closed boundary. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WBMWTEtQdJ4W5UrWuLHCck	2026-07-16 17:02:41 -04:00
didericis	914f01fa8f	fix(firecracker): serialize infra-VM create + agent builds (PR #354 review) ... Codex flagged that parallel `start`s race on shared state: two cold launches could both stop/build/boot the singleton on the same rootfs/PID, concurrent builds share the infra VM's buildah store, the cleanup did `buildah rm -a` (nuking a peer build's container), and the rootfs cache was populated non-atomically. - `infra_vm.ensure_running`: a host flock (`singleton.lock`) around the cold stop/build/boot path, with a double-checked health re-test under the lock so a second launcher adopts rather than re-boots. The healthy fast-path stays lock-free. - `image_builder.build_agent_rootfs_dir`: a host flock (`.build.lock`) around cache-lookup + build + publish; build into `.building-<digest>` and publish by atomic `os.rename`, so a partial build never appears as `agent-<digest>`. - scoped cleanup: per-build named working containers (`<tag>-smoke`/`-export`) removed by name instead of `buildah rm -a`; also cleared before the build to recover from a crashed prior run. Verified: agent image builds via the locked/atomic path, infra VM stays healthy, agent boots, `claude --version` = 2.1.172. Full unit suite + pyright green. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WBMWTEtQdJ4W5UrWuLHCck	2026-07-16 16:15:58 -04:00
didericis	c60e6b7e9f	refactor(firecracker): single infra VM builds too — buildah in one image (PR #354 review) ... Addresses the review finding that buildah lived only in the orchestrator image, so the persistent infra VM wasn't the builder — a separate throwaway builder VM contended with it for the orchestrator TAP. Consolidate: - **Rebase the gateway (and thus infra) on `python:3.12-slim` = Debian trixie**, pip-installing mitmproxy instead of `FROM mitmproxy/mitmproxy` (Debian bookworm). trixie ships buildah 1.39, which can build agent Dockerfiles that use heredocs; bookworm's 1.28 can't (`Unknown instruction: "{"`). CA path is unchanged (set via `--set confdir=`). - **buildah lives only in `Dockerfile.infra`** now (removed from the orchestrator image, which is lean/stdlib-only again). - **Shared orchestrator content**: `Dockerfile.orchestrator` is the single definition of the control-plane payload; the infra image `COPY --from`s it (same trixie base → clean copy, and future deps like iroh are added once). The docker backend runs the orchestrator image directly. - **`image_builder` builds inside the infra VM** (which now has buildah) over SSH — no throwaway builder VM, so the `bborch0` contention is gone. `ensure_built` builds orchestrator + gateway before infra (FROM gateway, COPY --from orchestrator). Verified on a KVM host: images build (buildah 1.39 in infra), the agent image builds *inside* the infra VM (heredoc Dockerfile and all), the infra VM stays healthy, the agent boots and `claude --version` = 2.1.172. The rebased gateway still starts as a docker container and generates its CA (docker backend unaffected). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WBMWTEtQdJ4W5UrWuLHCck	2026-07-16 15:23:36 -04:00
didericis	4873030550	feat(firecracker): git-gate over SSH into the gateway VM (Stage B, 7/n) ... git-gate now works end-to-end for git-upstream bottles on the infra VM. Three fixes surfaced by driving a real clone through git-http: - `SshGatewayTransport.cp_into` preserves the source file mode (docker cp does). The access-hook is staged 0700 and git-http execs it directly; a plain `cat >` landed it 0644 -> EACCES. Keys stay 0600. - the infra init runs `BOT_BOTTLE_GATEWAY_DAEMONS=egress,git-http,supervise`: the VM backend reaches git over git-http (9420), so the git:// daemon (git-gate, whose /git-gate-entrypoint.sh the consolidated model doesn't stage) is left out instead of crash-looping. - `build_infra_rootfs_dir` folds the init's content-hash into the rootfs cache key, so an init change actually rebuilds the rootfs (the base image digest alone wouldn't catch it). Verified on a KVM host: launch_consolidated provisions a git-upstream bottle's repo + creds into the gateway VM over SSH; an agent VM clones via git-http (source-IP attributed) and the access-hook resolves the provisioned key + known_hosts and attempts the upstream SSH fetch — failing closed only because the test used a dummy key + fake upstream ("refusing to serve stale data"). With real creds the fetch serves. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WBMWTEtQdJ4W5UrWuLHCck	2026-07-16 14:36:54 -04:00
didericis	b93b14f5c2	feat(firecracker): persistent registry volume for the infra VM (Stage B, 6/n) ... The infra VM's rootfs is ephemeral (rebuilt each boot), so the bottle registry DB needs durable storage across restarts. Give the infra VM a firecracker analogue of a docker volume: a host-side ext4 file attached as a second virtio-block device (guest /dev/vdb), mounted at the control plane's DB dir (/var/lib/bot-bottle, where host_db_path lives at db/bot-bottle.db). - firecracker_vm.boot/_config take an optional `data_drive` (a non-root, RW second drive). - infra_vm creates the volume on first use (`mke2fs` an empty ext4 at <fc-cache>/infra/registry.ext4) and mounts /dev/vdb in the PID-1 init before the control plane starts. It's a plain ext4 file, so `sudo mount -o loop <path>` (VM stopped) inspects bot-bottle.db directly. Verified on a KVM host: register a bottle, restart the infra VM (fresh rootfs, same volume) — /dev/vdb re-mounts and the registry `db/` dir + a marker file survive the restart. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WBMWTEtQdJ4W5UrWuLHCck	2026-07-16 14:36:54 -04:00
didericis	957eb19368	feat(firecracker): launch through the infra VM, not Docker (Stage B, 5/n) ... consolidated_launch now drives the persistent infra VM instead of the `_FirecrackerOrchestratorService` Docker containers — the point Docker leaves the Firecracker launch path. - launch_consolidated: `infra_vm.ensure_running()` (singleton) for the control plane + gateway; register the bottle over HTTP at the infra VM's guest IP; provision git-gate via `SshGatewayTransport` (over SSH into the gateway VM); fetch the gateway CA via `InfraVm.gateway_ca_pem()`. - teardown_consolidated deregisters + deprovisions but does NOT stop the infra VM (persistent per-host singleton shared by every bottle). - new `infra_vm.SshGatewayTransport` + `gateway_transport()` (built from the stable key + orchestrator link IP, so teardown needs no live handle). - drop the DockerGateway/OrchestratorService machinery from the firecracker consolidated path (still used by the docker backend). launch.py already calls launch_consolidated, so the whole firecracker launch path now uses the VM. pyright + unit suite green. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WBMWTEtQdJ4W5UrWuLHCck	2026-07-16 14:36:54 -04:00
didericis	5dfb9b0d75	refactor(gateway): parameterize git-gate provisioning transport (Stage B prep) ... git-gate provisioning into the running gateway was hard-wired to docker exec/cp. Extract a backend-neutral `GatewayTransport` (exec + cp_into) so the same provisioning logic serves both the docker gateway container and the firecracker gateway VM (over SSH, added with the launch swap). - `provision_git_gate` / `deprovision_git_gate` now take a transport instead of a gateway name; `DockerGatewayTransport` wraps the existing docker exec/cp behavior. deprovision is best-effort (catches the transport error) — matching the prior idempotent teardown. - both consolidated_launch callers pass `DockerGatewayTransport(name)` — no behavior change; the firecracker swap flips only its own to SSH. Pure refactor: docker path unchanged, gateway_provision tests updated to construct the transport, all green. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WBMWTEtQdJ4W5UrWuLHCck	2026-07-16 14:36:54 -04:00
didericis	bb434b14d7	feat(firecracker): persistent infra-VM singleton lifecycle (Stage B, 4/n) ... The infra VM must outlive the short-lived `start` launcher and be reused across launches. Add an idempotent singleton: - `ensure_running()` adopts the infra VM when its control plane is already healthy (a prior launcher booted it), else clears any stale VM and boots a fresh one. Returns a handle usable for CA fetch / git-gate provisioning whether we booted it or adopted it. - boot is `detached` (firecracker in its own session via start_new_session) so it survives the launcher exiting; its PID is recorded so a later process can `stop()` it. `_kill_pidfile` SIGTERM/SIGKILLs but only if the PID is still a firecracker process (guards a recycled PID). - a STABLE SSH key (generated once under the infra cache dir, re-injected each boot via the cmdline) so any launcher can SSH in to fetch the CA / provision, not just the one that booted the VM. `InfraVm.vm` is None in the adopted case; teardown then goes through the PID file. Verified on a KVM host: first ensure_running boots; a second adopts it (same PID, no reboot) and can still reach /health and fetch the gateway CA over SSH; stop() tears it down (control plane then unreachable). Next: git-gate provisioning into the VM over SSH (today docker exec/cp), then swap consolidated_launch.py onto the infra VM. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WBMWTEtQdJ4W5UrWuLHCck	2026-07-16 14:36:54 -04:00
didericis	e1610121c0	feat(firecracker): run the gateway data plane in the infra VM too (Stage B, 2/n) ... The single infra VM now runs BOTH the orchestrator control plane and the gateway data plane (egress / supervise / git-http), multi-tenant against the local control plane — the single-VM shape from the Stage B design. - Dockerfile.infra: the firecracker infra image = the gateway image + the baked control-plane source (FROM bot-bottle-gateway, COPY bot_bottle). Reuses the gateway payload rather than copying mitmproxy/ gitleaks into a third image; the docker backend keeps its two separate images. - infra_vm: build from source (gateway then infra image), and the PID-1 init now also launches `gateway_init` with BOT_BOTTLE_ORCHESTRATOR_URL=http://127.0.0.1:8099. Adds `gateway_ca_pem` (fetch the mitmproxy CA over SSH) and the agent-facing port constants. - init exports PATH — a bare-init shell resolves its own execs via a built-in default path, but that isn't in the environment, so gateway_init's `python3 ...` daemons would otherwise fail to spawn. Verified on a KVM host: infra VM boots, control plane /health -> 200, and egress:9099 / supervise:9100 / git-http:9420 all listen and are reachable from the host over the TAP link; the gateway CA is retrievable. (git-gate stays down until a bottle provisions its per-bottle entrypoint/creds, same as a fresh docker gateway — non-fatal, the supervisor keeps the rest up.) Next: agent->gateway VM-to-VM routing so bbfc* VMs reach these ports at the infra VM and nowhere else. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WBMWTEtQdJ4W5UrWuLHCck	2026-07-16 14:36:54 -04:00
didericis	1614172423	feat(firecracker): run the orchestrator control plane as a VM (Stage B, 1/n) ... First slice of Stage B: the orchestrator control plane runs as a persistent Firecracker infra VM instead of a Docker container. The host CLI reaches it over HTTP at the orchestrator link's guest IP; agent VMs will reach its gateway ports (added next) over VM-to-VM routing. - Dockerfile.orchestrator bakes the stdlib-only control-plane source (COPY bot_bottle) so the image is self-contained and runs from a built image with no runtime bind-mount — a guest VM can't bind-mount host source. (Build-from-source stays the default; a pull-from-registry mode lands later. The docker backend's dev bind-mount still overlays this.) - util.build_base_rootfs_dir / inject_guest_boot take a `variant` + `init_script`, so the same orchestrator image is prepared two ways without a cache collision: the builder VM keeps the SSH-only agent init; the infra VM gets a control-plane PID-1 init. - new firecracker/infra_vm.py: boot the infra VM on the orchestrator link, run `python -m bot_bottle.orchestrator` as PID 1, and poll /health. Verified on a KVM host: infra VM boots, control plane answers `GET /health -> 200 {"status":"ok"}` from the host over the TAP link. Next: fold gateway_init (egress/git-gate/supervise) into the same VM, then agent->gateway routing. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WBMWTEtQdJ4W5UrWuLHCck	2026-07-16 14:36:54 -04:00
didericis	4edd7803e8	feat(firecracker): build agent images in a builder VM, not host docker (Stage 3) ... Replace the host `docker build` + `docker export` behind the Firecracker agent rootfs with an in-VM buildah build. `image_builder.build_agent_rootfs_dir` boots a throwaway builder VM (the orchestrator image, which carries buildah) on the NAT'd orchestrator link, sends the Dockerfile over SSH, `buildah build`s it, smoke-tests the result with `buildah run` (the image's own PATH, so it catches an npm silent-failure stub), and streams the rootfs tar back into the content-addressed cache dir — the same base dir `util.build_rootfs_ext4` already turns into a bootable ext4 with `mke2fs -d`. No host Docker daemon, no root-equivalent `docker` group; an untrusted Dockerfile runs in a confined microVM, not on the host. - new firecracker/image_builder.py (boot → build → smoke → stream). - launch.py: `_build_agent_image` → `_build_agent_base`, returning the base dir from the builder VM. The committed-snapshot (freeze/migrate) path still exports via host docker until it too is ported. - util: `_inject_guest_boot` → public `inject_guest_boot` (shared with the builder). unit tests for the cache decision + smoke-test paths. Verified on a KVM host end-to-end: builds the real claude Dockerfile (node:22-slim + npm claude-code) in-VM in ~60s, the produced agent VM boots and `claude --version` returns 2.1.172; the content cache skips the rebuild on a repeat launch. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WBMWTEtQdJ4W5UrWuLHCck	2026-07-16 14:36:54 -04:00
didericis	81a2f15046	feat(firecracker): NAT'd egress link for the orchestrator/builder VM ... The orchestrator/gateway VM is trusted infra, not an isolated agent: it builds agent images in-VM (buildah must FROM-pull + apt/npm) and, in the Stage B cutover, forwards agent egress upstream. Give it a dedicated TAP (`bborch0`) on a /31 at the top of the IP_BASE /16 (clear of the bbfc* agent pool at the bottom), NAT'd out the host uplink — while agent VMs keep their fail-closed, gateway-only isolation table. - netpool.defaults.env / netpool.py: new BOT_BOTTLE_FC_ORCH_IFACE + `orch_slot()` (index -1 sentinel; host x.y.255.0 / guest x.y.255.1). - scripts/firecracker-netpool.sh: create + address the orchestrator TAP; `bot_bottle_fc_nat` table masquerades its /31 out the uplink and accepts its forward path. Because bootstrap still runs Docker (whose FORWARD policy is DROP), a best-effort, guarded, idempotent DOCKER-USER ACCEPT is added too (skipped once Docker is gone). down/status updated. - nix/firecracker-netpool.nix: mirror the option, pass it via the unit Environment= (the store-copied script can't read the defaults file), and add iptables to the unit path for the DOCKER-USER step. Agent isolation is unchanged: the new rules only ever accept/masquerade the orchestrator link and never drop, so they can't weaken the bbfc* drops. Applied by re-running `sudo ./scripts/firecracker-netpool.sh up`. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WBMWTEtQdJ4W5UrWuLHCck	2026-07-16 14:36:54 -04:00
didericis-claude	9369fb7de5	feat(firecracker): implement consolidated orchestrator launch (PRD 0070) ... Replace the per-bottle Docker sidecar bundle with the shared per-host orchestrator + gateway, mirroring what the Docker backend already has. - Add `bot_bottle/backend/firecracker/consolidated_launch.py`: `_FirecrackerOrchestratorService` (subclasses `OrchestratorService`, overrides `_gateway()` to return a `DockerGateway` with host port bindings so Firecracker VMs can reach it via their TAP link); `launch_consolidated()` registers the bottle by guest IP (attribution key), provisions git-gate into the shared gateway, and returns the shared CA + orchestrator URL for teardown; `teardown_consolidated()` deregisters and cleans up. - Rewrite `bot_bottle/backend/firecracker/launch.py`: removes the per-bottle sidecar bundle (`_start_sidecar_bundle`, `_stage_git_gate`, etc.) and `_mint_certs`; wires `launch_consolidated()` instead. The VM still sends to `host_tap_ip:PORT` — Docker's PREROUTING DNAT + the nft `ct status dnat accept` rule in the forward chain route the traffic to the shared gateway container. - Extend `DockerGateway` with `host_port_bindings` so the Firecracker gateway publishes its ports on the host (`0.0.0.0:PORT`). - Parameterise `OrchestratorService` with `orchestrator_name` / `orchestrator_label` so Docker and Firecracker orchestrators can coexist on the same host (`bot-bottle-orchestrator` vs `bot-bottle-fc-orchestrator`). Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>	2026-07-16 14:36:54 -04:00
didericis	09393b354b	refactor(de-sidecar): purge the "sidecar" name from live code, tests, and current docs ... Completes the de-sidecar cleanup: no live code, test, current doc, script, or nix file mentions or is named "sidecar" any more. Only the dated PRD/research docs keep the term as historical record (agreed on the #385 thread). - Rename `sidecar_init.py`→`gateway_init.py` was done earlier; this pass sweeps the remaining descriptive uses: the egress / git-gate / supervise components are the gateway's *daemons*, the shared container is the *gateway*, the old per-bottle container was the *companion container*. - Rename `tests/integration/test_sidecar_bundle_image.py`→`test_gateway_image.py` and its class; update `docs/ci.md` + `tests/README.md` for the renamed/ removed integration tests. - `SIDECAR_PORTS` shell var in `scripts/firecracker-netpool.sh`→`GATEWAY_PORTS`. Full unit suite green (bar the pre-existing `/bin/sleep`-missing env errors in test_gateway_init); docker integration — gateway singleton, broker, real two-bottle multitenant isolation, and the gateway-image build — all pass. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WBMWTEtQdJ4W5UrWuLHCck	2026-07-14 17:07:56 -04:00
didericis	77948ef56c	refactor(de-sidecar): remove the per-bottle companion-container architecture ... The per-agent companion container (the egress/git-gate/supervise data plane run once per bottle) is the pre-consolidation architecture. Remove it and disable the backends that still depend on it, per the #385 thread. - Delete `backend/docker/sidecar_bundle.py`; docker's live path uses the consolidated shared gateway, not a per-bottle bundle. - Disable the firecracker and macos-container backends: their `launch()` fails closed (they launched a per-bottle companion; firecracker's consolidated relaunch is #354, macos follows). Their `enumerate` return empty and `cleanup` drop the companion-container discovery (firecracker keeps VMM/run-dir cleanup). - Fail-close both backends' `egress_apply` reload (it signalled the per-bottle container); consolidated egress policy resolves per-request against the orchestrator, so gateway-side apply is a follow-up. - Rename `egress_sidecar_env_entries` → `egress_gateway_env_entries`, `SIDECAR_PORTS` → `GATEWAY_PORTS`. - Move the shared DockerBottlePlan fixture to `tests/unit/_docker_bottle_plan.py`; delete tests for the removed launch paths; update cleanup/egress-apply tests. Docker consolidated launch verified end-to-end (multitenant isolation integration test passes). macos/firecracker are intentionally disabled. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WBMWTEtQdJ4W5UrWuLHCck	2026-07-14 17:02:08 -04:00
didericis	2b970d1170	refactor(firecracker): single-source the network-pool defaults ... The pool params (size, IP base, iface prefix, nft table) were triplicated — hardcoded in netpool.py, scripts/firecracker-netpool.sh, and nix/firecracker-netpool.nix — plus the IP math (3x) and the nft ruleset (2x). Nothing enforced agreement; changing the base (ce3fad9, off CGNAT) forced a coordinated three-file edit, and a missed one would silently provision a range the launcher doesn't expect. Collapse to one source of truth: * netpool.defaults.env — a plain KEY=VALUE file (bash-sourceable, systemd EnvironmentFile-compatible, Python- and Nix-parseable) holding the four defaults. A BOT_BOTTLE_FC_* env var still overrides any key. * netpool.py reads it for the Python defaults (missing file = hard error, not confusing empty defaults). * the shell script falls back to it (no literal `:-8` / `10.243.0.0`), and its `up` is now non-destructive/idempotent (only creates a missing TAP), so re-running never cuts a live VM. * the Nix module readFile-parses it for its option defaults and delegates bring-up to the SAME shell script (dropping its duplicate IP math, nft ruleset, and TAP loop) — passing every value as Environment= so the store-detached script never needs the file. Net: defaults 3x -> 1x, nft ruleset 2x -> 1x, TAP loop 2x -> 1x. The one remaining IP-math dup (Python launch-addressing vs bash bring-up) is justified — different runtimes. Tests now guard the invariant (Python reads the shared file; the script/module hold no literals) instead of pinning duplicated strings. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WBMWTEtQdJ4W5UrWuLHCck	2026-07-13 04:27:48 -04:00
didericis	f71558aff6	fix(firecracker): run the agent from /home/node, not the /root SSH cwd ... The control SSH logs in as root, so the agent (dropped to node via runuser) inherited cwd=/root. That was harmless while the rootless rootfs left /root node-owned, but the dropbear fix (chown /root → root:root 0700) made /root unreadable to node — so the agent's cwd was inaccessible, breaking Node's process.cwd(), Claude Code's shell-snapshot machinery, and `/doctor` (which reported /root/.claude EACCES and failed every Bash command). Run the agent from its workdir (default /home/node) via `env --chdir`. Not a `sh -c 'cd … && exec "$@"'` wrapper: ssh space-joins everything after the host into one string for the guest shell, so the quoted script + $@ get re-split and mangled (it exec'd the $0 placeholder → exit 127). `env --chdir=DIR …` is all simple words, so it survives the join. Verified end-to-end: a real headless claude bottle now runs its Bash tool and exits 0 (was EACCES/127 before). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WBMWTEtQdJ4W5UrWuLHCck	2026-07-11 18:57:33 -04:00
didericis	a80356af75	feat(firecracker): portable systemd-unit install for the network pool ... Make the network pool a persistent, distro-uniform resource instead of a non-persistent per-distro shell command. systemd is the common denominator across Debian/Ubuntu/Fedora/RHEL/Arch/… (and NixOS), so `backend setup` on any systemd host now installs one bot-bottle-owned oneshot unit (bot-bottle-firecracker-netpool.service): params pinned via Environment= (so it doesn't depend on $SUDO_USER at boot), ExecStart/Stop delegating to the bundled bring-up script (single source of logic). - render_systemd_unit() in netpool.py (derived from the same constants as the shell script + nix module). - backend setup: install + enable the unit directly when run as root, else print a self-contained copy-paste block. NixOS keeps its declarative module (which produces the same-named unit); non-systemd hosts fall back to the raw imperative script. - backend teardown: symmetric — disable + remove the unit. - backend status: report the unit's active/inactive state so you can tell a persistent install from an imperative one. Net: one install path (`backend setup`), persistent, identical across distros; per-distro variance shrinks to installing nft + iproute2. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WBMWTEtQdJ4W5UrWuLHCck	2026-07-11 18:33:42 -04:00
didericis	3d7c508dc4	fix(firecracker): make the launch path work end-to-end ... First real end-to-end launch (there was no firecracker integration test) surfaced three bugs in the control/provision path, all now verified fixed by tests/integration/test_firecracker_launch: 1. Guest SSH rejected the correct key. The rootless rootfs build (`docker export | tar` as a non-root user) can't preserve uid 0, so every path — including /root — is owned by the build uid (node in guest). dropbear refuses root's authorized_keys when /root isn't root-owned, so auth fell back to password → denied. bb-init now `chown -R 0:0 /root`. 2. The SSH client (newer OpenSSH) didn't reliably present the -i key against the operator's ~/.ssh config; add `IdentitiesOnly=yes` to ssh_base_argv so only the per-bottle key is offered. 3. cp_in double-wrapped the remote command as `sh -c <remote>`, but ssh space-joins everything after the host into one string for the guest shell, collapsing `sh -c mkdir -p X && …` to `mkdir` with no operand ("missing operand"). Pass the command as a single arg and let the guest login shell run it (stdin still carries the tar). Also stop discarding dropbear's stderr (`-E` now reaches the host-side console.log) so future guest-auth issues are debuggable. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WBMWTEtQdJ4W5UrWuLHCck	2026-07-11 18:16:57 -04:00
didericis	f42a0dc7fe	fix(firecracker): status() defers unverifiable nft, like the preflight ... `status()` hard-failed when it couldn't confirm the nft table, but listing nftables usually needs root — so an unprivileged `backend status` reported "not ready" even with the pool fully up, making it useless as a launch gate (and skipping the firecracker integration test on a set-up host). Base readiness on what the launch preflight actually hard-requires: the TAP pool present (unprivileged, authoritative) + no range overlap. Report the nft table state (present / unverified / not-confirmable-unprivileged) but don't let it flip readiness — the post-boot isolation probe is the authoritative isolation check, same as the preflight's deferral. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WBMWTEtQdJ4W5UrWuLHCck	2026-07-11 17:28:17 -04:00
didericis	480269b116	refactor(firecracker): non-invasive NixOS module (no firewall switch) ... The module used `networking.nftables.tables.*` (which forces `networking.nftables.enable = true`, flipping the host firewall backend) and `systemd.network.enable` (handing interfaces to systemd-networkd) — disruptive on an iptables + Docker daily driver. Rewrite it to a single systemd oneshot that brings the pool up: idempotent `ip tuntap` for the TAPs + `nft -f` for the independent `inet bot_bottle_fc` table (its own hooks at priority -10), with ExecStop teardown. Same as the imperative script, but declarative. It touches neither the firewall backend nor networkd, so it coexists with iptables/Docker/ufw/firewalld. Verified through the NixOS module system (service present, firewall untouched). Drop the now-redundant `render_nixos_module()` paste generator (the module is a real importable file) and point `backend setup` at importing it (flake output or the file path), noting it's non-invasive. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WBMWTEtQdJ4W5UrWuLHCck	2026-07-11 17:25:50 -04:00
didericis	949b001464	feat(firecracker): show binary/KVM prerequisites in backend setup ... `backend setup --backend=firecracker` only covered the network pool. Lead with the other prerequisites: the firecracker binary (with a release install pointer when it's not on PATH, plus a NixOS note), a /dev/kvm check, and a reminder about the cached guest kernel + static dropbear and mke2fs. The network pool is now step 2. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WBMWTEtQdJ4W5UrWuLHCck	2026-07-11 17:03:04 -04:00
didericis	656966c2c4	feat(backend): add teardown() to the contract; backend teardown ... Add an abstract teardown() classmethod to BottleBackend — the inverse of setup(), surfaced as `./cli.py backend teardown [--backend=NAME]` (uninstall). Symmetric with setup: it prints the privileged commands / declarative config change to remove the host prerequisites. - firecracker: NixOS-aware — disable the flake module (or drop the import) and rebuild, or `firecracker-netpool.sh down` imperatively. - docker / macos-container: nothing to undo (no privileged host state); print a short note. Not called by the launch path or the test suite. Extends test_cli_backend for the new dispatch. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WBMWTEtQdJ4W5UrWuLHCck	2026-07-11 16:56:17 -04:00
didericis	dd2e83b8a9	refactor(backend): generic setup/status; drop firecracker-only CLI ... Add abstract `setup()`/`status()` classmethods to BottleBackend (same per-host, no-instance shape as is_available) so host provisioning is part of the backend contract, not a per-backend command. Replace the `./cli.py firecracker {setup,status}` command with a generic `./cli.py backend {setup,status} [--backend=NAME]` that resolves a backend (flag / $BOT_BOTTLE_BACKEND / host default) and dispatches — swapping backends is just a different --backend. Implementations: - firecracker: moved out of cli/ into backend/firecracker/setup.py (network pool module/script + range-overlap check), unchanged output. - docker: new backend/docker/setup.py — reports docker on PATH, daemon reachability, and gVisor runsc; setup notes no privileged pool is needed. Minimal placeholder; richer version tracked in #345. - macos-container: new setup.py — container CLI + system-service checks. Also retarget the launch-preflight / isolation-probe pointers to the new command. New test_cli_backend covers dispatch + docker setup/status. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WBMWTEtQdJ4W5UrWuLHCck	2026-07-11 15:18:40 -04:00
didericis	ce3fad9320	feat(firecracker): move pool off CGNAT, add overlap guard + flake module ... The default TAP-pool base was 100.64.0.0/10 (RFC-6598 CGNAT) — chosen to dodge RFC-1918, but that's exactly the range Tailscale assigns node addresses from, so on a Tailscale host it's the worst pick. Move the default to 10.243.0.0/16, an obscure RFC-1918 block that steers clear of docker/libvirt/k8s/LAN and Tailscale. No default is collision-proof, so add netpool.overlapping_routes(): it parses `ip -json route show table all` and flags any route intersecting the pool range (excluding our own bbfc* TAPs and the default route). The launch preflight warns on overlap; `backend status` reports it. Distribute the NixOS host setup as a flake module instead of a copy-pasted blob: nix/firecracker-netpool.nix computes the taps / nft table from typed options (poolSize, ipBase, ifacePrefix, owner) with a /31-alignment assertion, and flake.nix exposes it as nixosModules.firecracker-netpool. Defaults mirror the backend constants; writeEnvFile emits the matching BOT_BOTTLE_FC_* so the host pool and the launcher can't drift. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WBMWTEtQdJ4W5UrWuLHCck	2026-07-11 15:18:21 -04:00
didericis	c07ebca867	feat(backend): remove smolmachines; firecracker is the Linux default ... Delete the smolmachines backend (the whole bot_bottle/backend/smolmachines package and its tests). It had fatal Linux issues (TSI networking under sustained use, exec-channel contention, no SIGWINCH) and is superseded by the Firecracker backend (issue #342). Backend selection now: - default is macos-container on macOS, firecracker on KVM-capable Linux hosts, and docker as the last resort (was smolmachines). - firecracker is selected on a KVM host even when the `firecracker` binary isn't installed, so start routes through its preflight and prints an install pointer (same UX as require_container), instead of silently falling back. Split is_host_capable() (Linux + KVM) out of is_available() (adds the binary check) to drive this. Retarget the cross-backend tests (parity, print-parity, prepare, workspace, freezer, selection) from smolmachines to firecracker rather than dropping the coverage. Remove docker.util.image_id/save, which only smolmachines used. Update README/AGENTS/example bottles and stale comments; historical docs/prds are left as a point-in-time record. BREAKING: BOT_BOTTLE_BACKEND=smolmachines now errors as unknown. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WBMWTEtQdJ4W5UrWuLHCck	2026-07-11 13:24:47 -04:00
didericis	c276f7b0b1	feat(firecracker): add Linux microVM backend to replace smolmachines ... Adds a Firecracker-based backend for Linux, providing mature KVM-based microVM isolation to replace smolmachines/libkrun (issue #342, closes the dead-end tracked in #332). Architecture: - Guest control over SSH (dropbear injected into the rootfs) on a point-to-point TAP link. `ssh -t` forwards SIGWINCH natively, so no resize bridge is needed. - Networking: a one-time, root-provisioned pool of user-owned TAP devices (no shared bridge → no docker0/virbr0/cni0 collisions) plus a dedicated `table inet bot_bottle_fc` nftables table (independent of Docker/ufw/firewalld rules). `./cli.py firecracker setup` prints the host-appropriate config (NixOS module or sudo script). - Rootfs: `docker export` → ext4 via `mke2fs -d` (rootless, no mount), cached by image digest; per-bottle SSH pubkey + IP passed via the kernel cmdline. - Sidecar: reuses the Docker bundle, published on the slot's host TAP IP. - Fail-closed isolation: TAP pool verified at preflight; the egress boundary is proven empirically post-boot (before the agent runs) by a canary probe — the VM must fail to reach the host directly, or launch is refused. Linux hosts with Firecracker + KVM now default to this backend; macOS stays on macos-container. Not yet validated end-to-end on live hardware (requires the one-time network pool). Unit tests + pyright pass. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01G8p32HJgPoS1hLPWubbftM	2026-07-11 10:32:55 -04:00