205 Commits

Author	SHA1	Message	Date
didericis	4a607ad098	refactor(macos): one infra container (control plane + gateway), fixes shared-DB races ... Adopts the firecracker infra-VM pattern for macOS: the orchestrator control plane and the gateway data plane now run in a SINGLE Apple container instead of two. Apple Containers are lightweight VMs with separate kernels, so the prior two-container design had both guests writing one bot-bottle.db over virtiofs, where fcntl locks are not coherent across kernels — concurrent writes (the orchestrator's registry vs the gateway supervise daemon's queue) could corrupt it. One container = one kernel = coherent locking. The DB moves onto a container-only Apple volume (bot-bottle-mac-db), never bind-mounted from the host, so no host process opens the live file either. The host CLI already reaches registry + supervise state over the control-plane HTTP surface (cli/supervise.py uses OrchestratorClient), exactly as firecracker's VM-only DB requires. Two simplifications fall out of the single container: - No DNS dance: the control plane and gateway daemons reach each other over 127.0.0.1, so the orchestrator-before-gateway ordering (a workaround for Apple having no container DNS) is gone, along with the moved-IP recreate logic it needed. - Net -243 lines. Mechanics: the infra container runs from the gateway image with the control-plane source bind-mounted read-only (like the docker orchestrator, so a code change needs no rebuild) and a small sh -c init that starts both processes (mirrors firecracker's _infra_init). Also implements the macOS backend's ensure_orchestrator() and adds it to discover_orchestrator_url, so operator tools (supervise) can bring up / find the control plane on demand — previously the macOS backend died with "no orchestrator control plane". Verified end-to-end on real Apple Container 1.0.0: the single infra container comes up healthy (one address for control plane + gateway), both processes run, the DB is written on the container-only volume, host-side supervise works over HTTP, and a registered agent gets 200 for an allowed host / 403 for a denied one. 1824 unit tests pass with `container` absent (CI parity), pyright clean, pylint 9.89. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>	2026-07-17 04:14:14 -04:00
didericis	e24b62b6b9	fix(macos): review fixes — token on plan, self-heal, symmetric digest, DHCP poll ... Addresses findings from a high-effort review of the PRD 0070 macOS backend. Correctness: - Stamp identity_token onto MacosContainerBottlePlan after registration. git's gitconfig extraHeader and the supervise MCP --header read getattr(plan,"identity_token","") at provision time, and both reach the gateway on NO_PROXY (bypassing the egress proxy that carries the token). The plan never carried it, so /resolve fail-closed and every git fetch/push and supervise call from a macOS bottle would have been denied. Registration precedes provision(), so — unlike the run-time env — the plan can carry it. - Self-heal the orchestrator: recreate when it is not (source-current AND answering /health), not on the source-hash label alone. A container running current code but with a wedged HTTP server was left alone and polled to death, failing every launch until manual deletion. - image_digest and container_image_digest now read the same descriptor.digest field; dropped image_digest's id/tag fallback that could yield a value the container side can't produce — a permanent mismatch would have recreated the shared gateway on every launch (severing every live bottle's egress, since the replacement gets a new DHCP address). - Poll for the agent's and gateway's DHCP address instead of a fatal read right after `container run` (there is no --ip; the address can lag start). Cleanup: - One _inspect_first + _descriptor_digest behind the four inspect readers. - Shared bind_mount_spec (util) and host_db_dir (paths) replace per-module copies; _GIT_HTTP_PORT now imports git_http_backend.DEFAULT_PORT. - Drop the dead _url cache / url property and the write-only agent_proxy_url. Deferred (noted on the PR, not fixed here): the gateway image rebuilding on every launch (needs source-hash-labeled build), SQLite shared across VM guests, and the sh -lc profile-override edge — each is design-level or behavior-risk beyond a review fix. Verified: real Apple Container bring-up is green and idempotent; 1826 unit tests pass with `container` absent (CI parity), pyright clean, pylint 9.86. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>	2026-07-17 03:26:33 -04:00
didericis	c69642e568	feat(macos): consolidated per-host gateway for the Apple backend (PRD 0070) ... Re-enables the macos-container backend on the shared per-host orchestrator + gateway, replacing the per-bottle companion container removed in #385. This is the last backend in PRD 0070's roadmap. Apple Container 1.0.0 forced three departures from the docker shape, each verified against the live CLI (findings recorded in the networking spike): - No `--ip`. The address is DHCP-assigned and knowable only once the container runs, so the order inverts: gateway up -> run agent -> read its address -> register. The identity token is minted by registration and therefore cannot be in the agent's run-time env; it rides the proxy URL applied at `container exec` time (bare `--env` names keep it off argv). - No container DNS. The gateway can only be handed the control plane's IP, so the orchestrator starts first and the gateway is pointed at its address. - No `network connect`. Networks are fixed at run time, so the shared host-only network is created up front; per-bottle networks would restart the gateway on every launch and defeat the consolidation. The agent runs with `--cap-drop CAP_NET_RAW`: Apple grants NET_RAW by default, which would let an agent forge a neighbour's source address on the shared segment. NET_ADMIN is already absent, so this closes the source-address half of PRD 0070's attribution invariant. Verified end-to-end on real Apple Container 1.0.0: both images build, the control plane comes up healthy, the gateway reaches it by IP, and a registered agent gets 200 for a host in its routes and 403 for one outside them. Bring-up is idempotent — a second launch does not churn the singletons. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-07-17 01:27:40 -04:00
didericis	dfc693e0b6	fix(firecracker): keep the snapshot partial private even if one was left behind ... Follow-up to the codex review on #398. os.open's mode arg only applies on creation, so a committed-rootfs.tar.partial left 0644 by an interrupted run would be opened/truncated (not re-moded) and stay world-readable for the whole SSH stream. Unlink any leftover and exclusively recreate it (O_EXCL|O_NOFOLLOW), then fchmod 0600 immediately so umask can't loosen it. Test pre-creates a 0644 partial and asserts the fd is 0600 mid-stream. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UoEZHDjv84ChoZbozQERhJ	2026-07-17 01:02:19 -04:00
didericis	39d47b8108	fix(firecracker): harden committed-snapshot resume against guest-controlled data ... Address the codex review on #398: - P1: inject_guest_boot no longer follows a symlink at bb-init/bb-dropbear. A committed snapshot is guest-controlled and could plant those paths as symlinks aimed at a host file (e.g. bb-init -> ~/.bashrc); write_text / copy2 would then overwrite the target as the host user during resume. Replace any pre-existing entry and create the files with O_EXCL|O_NOFOLLOW so the write stays inside the staging tree. - P2: write the snapshot tar owner-only (0600). It can contain the bottle's private workspace; it was being created world-readable (0644). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UoEZHDjv84ChoZbozQERhJ	2026-07-17 00:44:06 -04:00
didericis	5c08701983	feat(firecracker): port freeze/migrate off host Docker (PRD 0069 / #397) ... The last host-Docker dependency in the Firecracker launch path. Freeze and resume no longer touch the docker daemon, so the backend needs firecracker + KVM only — completing #348. Freeze: stream the guest rootfs over SSH straight into a persistent committed-rootfs.tar (the resumable/migratable artifact) instead of round-tripping through `docker build` from a scratch image. Written to a .partial sibling and atomically renamed so a failed freeze leaves no truncated artifact. Resume: extract the snapshot tar into a cached base dir and feed it to the existing rootless `mke2fs -d` pipeline, replacing the `docker create` + `docker export | tar` path. Recreate the proc/sys/dev/run mount points the freezer excludes so the guest init can mount them. `util.build_base_rootfs_dir` / `docker_image_id` stay — they still back the opt-in BOT_BOTTLE_INFRA_BUILD=local dev path and off-host publish_infra, which are out of scope. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UoEZHDjv84ChoZbozQERhJ	2026-07-17 00:31:00 -04:00
didericis	e3e195f866	chore(firecracker): name the artifact package bot-bottle-firecracker-infra ... Rename the Gitea generic package from bot-bottle-infra to bot-bottle-firecracker-infra so it's self-evident in the package list which backend it serves (and leaves room for other artifacts, e.g. a shipped kernel). The version slot stays the content hash — "firecracker" belongs in the package name, not the version. Docker image / VM names are unchanged. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UoEZHDjv84ChoZbozQERhJ	2026-07-17 00:04:48 -04:00
didericis	e3d24b7e41	chore(firecracker): ship an about.txt description with the infra artifact ... Generic packages have no description field, so publish_infra now uploads a short about.txt alongside the rootfs on every publish — it's what identifies the package as the Firecracker backend's infra rootfs on the package page. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UoEZHDjv84ChoZbozQERhJ	2026-07-16 23:59:38 -04:00
didericis	f2891a1634	fix(firecracker): hash all baked-in files + stream the artifact upload ... Address PR #395 review (two P1s): - Version hash covered only `bot_bottle/**.py`, but the image `COPY`s the whole package — non-Python inputs baked in (egress_entrypoint.sh, netpool.defaults.env) didn't change the version, so a launch host could boot a stale rootfs whose code differs from its checkout. Hash every regular file under bot_bottle/ (excluding __pycache__/.pyc). Regression tests: a shell-script change bumps the version; .pyc/__pycache__ don't. - publish_infra `_put` read the whole (hundreds-of-MB) gz into memory via read_bytes(). Stream it from disk with an explicit Content-Length; the tiny .sha256 stays in-memory. Test asserts the body is the file object, not bytes. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UoEZHDjv84ChoZbozQERhJ	2026-07-16 23:46:46 -04:00
didericis	8d8a88aeeb	fix(firecracker): read only BOT_BOTTLE_INFRA_ARTIFACT_TOKEN for the artifact ... Drop the fallback to the general-purpose BOT_BOTTLE_CLAUDE_GITEA_TOKEN so the artifact pull/publish uses a dedicated, package-scoped token that can be granted (or revoked) independently of the general Gitea token. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UoEZHDjv84ChoZbozQERhJ	2026-07-16 23:41:53 -04:00
didericis	18f190b7e3	feat(firecracker): pull the infra rootfs as a prebuilt artifact (PRD 0069 Stage 2) ... Stage 2 of the docker-free Firecracker backend (#348): stop building the fixed infra image on the launch host. The infra VM's rootfs is host- and bottle-agnostic (authorized_keys + guest IP ride the kernel cmdline, not the rootfs), so it's built once off-host and published as a versioned, ready-to- boot ext4; the launch host downloads + verifies + boots it — no Docker, no image tooling, just HTTP + gunzip. - infra_artifact.py: version = content hash of the rootfs inputs (the shipped bot_bottle package + the three Dockerfiles + the init), so a launch host pulls the artifact matching its code and a content change can't silently boot a stale rootfs. Pull + sha256-verify (fail-closed) + gunzip from a Gitea generic package; base/owner/token configurable, default this Gitea. - infra_vm.ensure_built/boot default to the pull path; BOT_BOTTLE_INFRA_BUILD= local keeps the docker build-from-source path for iterating on Dockerfiles. - publish_infra.py: the off-host half — builds the images with Docker, mke2fs the rootfs (with buildah slack), gzips, and PUTs it to the generic package. Rollout note: default=pull means a launch 404s until an artifact is published; until the Gitea packages endpoint is enabled + an artifact published, use BOT_BOTTLE_INFRA_BUILD=local. Freeze/migrate's remaining docker use is a separate PR. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UoEZHDjv84ChoZbozQERhJ	2026-07-16 23:41:53 -04:00
didericis	2641ab70fd	feat(supervise): start orchestrator on demand via backend-agnostic bring-up ... Add BottleBackend.ensure_orchestrator() -> str: the backend-agnostic entry point that brings up the per-host orchestrator + shared gateway (idempotent) and returns its host-reachable control-plane URL. Docker starts the orchestrator + gateway containers (OrchestratorService); firecracker boots the infra VM; macos-container dies with a pointer (no orchestrator). Previously bring-up was reachable only through each backend's consolidated_launch, with no shared handle. Wire it into `bot-bottle supervise`: supervise is often the first thing an operator runs, before any bottle has booted the control plane, so `_resolve_orchestrator_url` now starts the selected backend's orchestrator on demand when discovery finds nothing, instead of failing with "launch a bottle first". Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UoEZHDjv84ChoZbozQERhJ	2026-07-16 21:52:11 -04:00
didericis	38bc555dbf	fix(supervise): single migrated DB for firecracker — orchestrator owns supervise tables ... The firecracker supervise MCP daemon 500'd (-32603) on egress-allow/block: it ran with no BOT_BOTTLE_ROOT/SUPERVISE_DB_PATH, so it targeted a stray, unmigrated SQLite file and `write_proposal` hit "no such table: supervise_proposals". Meanwhile the in-VM control plane migrated only the registry table (orchestrator_bottles) into its own DB, and the host operator reads a third, disconnected DB — three files, none shared. Consolidate to one DB per host, owned by the control plane on the persisted registry volume (/var/lib/bot-bottle/db/bot-bottle.db): - orchestrator startup now migrates the supervise queue + audit tables into the same file it migrates the registry into (StoreManager), so the control-plane DB carries every table. - the in-VM supervise daemon is pointed at that same file via SUPERVISE_DB_PATH, so daemon and control plane share one queue. `list-egress-routes` already worked (no DB); egress-allow now queues + waits on the single persisted DB instead of erroring. Validated against a live infra VM: migrate + write_proposal succeeds and the proposal is queued. The host-operator HTTP bridge (so approvals complete from the host, unifying docker onto the same path) is the follow-up. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WBMWTEtQdJ4W5UrWuLHCck	2026-07-16 18:24:45 -04:00
didericis	a208bcde08	feat(firecracker): stream buildah build output live during agent-image build ... The agent-image build ran over ssh with output fully captured, shown only as a 20-line tail on failure — so a successful (or in-progress) first build was a long silent wait through the base pull + apt/npm installs. Stream the `buildah build` step's stdout/stderr straight to our own (like the docker backend's `docker build`) via a non-capturing _ssh_streamed helper, so the operator sees `STEP i/n` progress live. Failure now points at the streamed output above instead of a captured tail. The smoke test and rootfs export stay captured (short / piped). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WBMWTEtQdJ4W5UrWuLHCck	2026-07-16 18:01:59 -04:00
didericis	c0066d2cd2	fix(firecracker): quote guest argv tokens so codex's multi-word prompt survives ssh ... The interactive agent command is sent to the guest by spreading the remote argv as separate ssh arguments; ssh space-joins everything after the host into one line that the guest login shell re-parses. That only works while every token is a "simple word" — which held for claude (`--append-system-prompt-file <path>`) but not for codex's `read_prompt_file` mode, whose positional is a whole sentence: "Read and follow the instructions in <path>.". The guest shell re-split it on spaces, so codex received `Read` as the prompt and `and`, `follow`, … as extra args — failing with `unrecognized subcommand 'and'` the moment an interactive codex session attached. Pre-quote each remote token with shlex.quote before ssh joins them (the same ssh→guest-shell discipline infra_vm/cp_in already use). Simple words are unchanged, so existing behaviour and the parity/structure tests are untouched; an arg with spaces now survives as a single argument. Regression test round-trips the joined remote command back through shlex.split and asserts codex's prompt comes out as exactly one arg. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WBMWTEtQdJ4W5UrWuLHCck	2026-07-16 18:01:59 -04:00
didericis	d4b27ebf1f	feat(gateway): mandatory identity-token attribution on every data plane (PR #354 review) ... Codex review: the /31 TAP doesn't make source IP unspoofable, and the app-layer token was returned by launch but never delivered or enforced, so a spoofed source could select a victim bottle's policy/tokens. Make the token mandatory and deliver it on each attributed plane (anti-spoof landed separately as the network boundary). Enforcement (control plane): - `Orchestrator.resolve` now requires a matching (source_ip, identity_token) pair (constant-time) — no source-IP-only fallback. `/resolve` fail-closes (403) on a missing/empty/mismatched token. Delivery, per plane (the token is `token_urlsafe`, safe in a URL): - egress: proxy credentials (`HTTPS_PROXY=http://bottle:<token>@gw`). The addon reads `Proxy-Authorization` — from the request (HTTP) or captured at the CONNECT for HTTPS tunnels (keyed by client conn, cleared on disconnect) — validates, and strips it (+ the legacy header) before upstream. - git-http: a URL-scoped `http.<gate>/.extraHeader: x-bot-bottle-identity` in the agent's git config (only over the http transport). - supervise: `mcp add --header x-bot-bottle-identity: <token>` (claude + codex); the server reads the header and passes it to resolve. Wiring: thread `ctx.identity_token` onto the firecracker + docker plans and into the agent env/config at launch. Verified on a KVM host: egress with the correct proxy-cred token returns 200 (HTTP and HTTPS/CONNECT), and no-token / wrong-token return 403; a real `cli.py start --backend=firecracker` launch provisions git config + the supervise MCP header and reaches the agent session, all under mandatory enforcement. Fixed a `claude mcp add` arg-order bug (--header must follow the positional name/url) found by that launch. Transparent proxy for tools that ignore proxy env is deferred to a follow-up (see thread); anti-spoof + host firewall remain the fail-closed boundary. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WBMWTEtQdJ4W5UrWuLHCck	2026-07-16 17:02:41 -04:00
didericis	914f01fa8f	fix(firecracker): serialize infra-VM create + agent builds (PR #354 review) ... Codex flagged that parallel `start`s race on shared state: two cold launches could both stop/build/boot the singleton on the same rootfs/PID, concurrent builds share the infra VM's buildah store, the cleanup did `buildah rm -a` (nuking a peer build's container), and the rootfs cache was populated non-atomically. - `infra_vm.ensure_running`: a host flock (`singleton.lock`) around the cold stop/build/boot path, with a double-checked health re-test under the lock so a second launcher adopts rather than re-boots. The healthy fast-path stays lock-free. - `image_builder.build_agent_rootfs_dir`: a host flock (`.build.lock`) around cache-lookup + build + publish; build into `.building-<digest>` and publish by atomic `os.rename`, so a partial build never appears as `agent-<digest>`. - scoped cleanup: per-build named working containers (`<tag>-smoke`/`-export`) removed by name instead of `buildah rm -a`; also cleared before the build to recover from a crashed prior run. Verified: agent image builds via the locked/atomic path, infra VM stays healthy, agent boots, `claude --version` = 2.1.172. Full unit suite + pyright green. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WBMWTEtQdJ4W5UrWuLHCck	2026-07-16 16:15:58 -04:00
didericis	c60e6b7e9f	refactor(firecracker): single infra VM builds too — buildah in one image (PR #354 review) ... Addresses the review finding that buildah lived only in the orchestrator image, so the persistent infra VM wasn't the builder — a separate throwaway builder VM contended with it for the orchestrator TAP. Consolidate: - **Rebase the gateway (and thus infra) on `python:3.12-slim` = Debian trixie**, pip-installing mitmproxy instead of `FROM mitmproxy/mitmproxy` (Debian bookworm). trixie ships buildah 1.39, which can build agent Dockerfiles that use heredocs; bookworm's 1.28 can't (`Unknown instruction: "{"`). CA path is unchanged (set via `--set confdir=`). - **buildah lives only in `Dockerfile.infra`** now (removed from the orchestrator image, which is lean/stdlib-only again). - **Shared orchestrator content**: `Dockerfile.orchestrator` is the single definition of the control-plane payload; the infra image `COPY --from`s it (same trixie base → clean copy, and future deps like iroh are added once). The docker backend runs the orchestrator image directly. - **`image_builder` builds inside the infra VM** (which now has buildah) over SSH — no throwaway builder VM, so the `bborch0` contention is gone. `ensure_built` builds orchestrator + gateway before infra (FROM gateway, COPY --from orchestrator). Verified on a KVM host: images build (buildah 1.39 in infra), the agent image builds *inside* the infra VM (heredoc Dockerfile and all), the infra VM stays healthy, the agent boots and `claude --version` = 2.1.172. The rebased gateway still starts as a docker container and generates its CA (docker backend unaffected). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WBMWTEtQdJ4W5UrWuLHCck	2026-07-16 15:23:36 -04:00
didericis	4873030550	feat(firecracker): git-gate over SSH into the gateway VM (Stage B, 7/n) ... git-gate now works end-to-end for git-upstream bottles on the infra VM. Three fixes surfaced by driving a real clone through git-http: - `SshGatewayTransport.cp_into` preserves the source file mode (docker cp does). The access-hook is staged 0700 and git-http execs it directly; a plain `cat >` landed it 0644 -> EACCES. Keys stay 0600. - the infra init runs `BOT_BOTTLE_GATEWAY_DAEMONS=egress,git-http,supervise`: the VM backend reaches git over git-http (9420), so the git:// daemon (git-gate, whose /git-gate-entrypoint.sh the consolidated model doesn't stage) is left out instead of crash-looping. - `build_infra_rootfs_dir` folds the init's content-hash into the rootfs cache key, so an init change actually rebuilds the rootfs (the base image digest alone wouldn't catch it). Verified on a KVM host: launch_consolidated provisions a git-upstream bottle's repo + creds into the gateway VM over SSH; an agent VM clones via git-http (source-IP attributed) and the access-hook resolves the provisioned key + known_hosts and attempts the upstream SSH fetch — failing closed only because the test used a dummy key + fake upstream ("refusing to serve stale data"). With real creds the fetch serves. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WBMWTEtQdJ4W5UrWuLHCck	2026-07-16 14:36:54 -04:00
didericis	b93b14f5c2	feat(firecracker): persistent registry volume for the infra VM (Stage B, 6/n) ... The infra VM's rootfs is ephemeral (rebuilt each boot), so the bottle registry DB needs durable storage across restarts. Give the infra VM a firecracker analogue of a docker volume: a host-side ext4 file attached as a second virtio-block device (guest /dev/vdb), mounted at the control plane's DB dir (/var/lib/bot-bottle, where host_db_path lives at db/bot-bottle.db). - firecracker_vm.boot/_config take an optional `data_drive` (a non-root, RW second drive). - infra_vm creates the volume on first use (`mke2fs` an empty ext4 at <fc-cache>/infra/registry.ext4) and mounts /dev/vdb in the PID-1 init before the control plane starts. It's a plain ext4 file, so `sudo mount -o loop <path>` (VM stopped) inspects bot-bottle.db directly. Verified on a KVM host: register a bottle, restart the infra VM (fresh rootfs, same volume) — /dev/vdb re-mounts and the registry `db/` dir + a marker file survive the restart. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WBMWTEtQdJ4W5UrWuLHCck	2026-07-16 14:36:54 -04:00
didericis	957eb19368	feat(firecracker): launch through the infra VM, not Docker (Stage B, 5/n) ... consolidated_launch now drives the persistent infra VM instead of the `_FirecrackerOrchestratorService` Docker containers — the point Docker leaves the Firecracker launch path. - launch_consolidated: `infra_vm.ensure_running()` (singleton) for the control plane + gateway; register the bottle over HTTP at the infra VM's guest IP; provision git-gate via `SshGatewayTransport` (over SSH into the gateway VM); fetch the gateway CA via `InfraVm.gateway_ca_pem()`. - teardown_consolidated deregisters + deprovisions but does NOT stop the infra VM (persistent per-host singleton shared by every bottle). - new `infra_vm.SshGatewayTransport` + `gateway_transport()` (built from the stable key + orchestrator link IP, so teardown needs no live handle). - drop the DockerGateway/OrchestratorService machinery from the firecracker consolidated path (still used by the docker backend). launch.py already calls launch_consolidated, so the whole firecracker launch path now uses the VM. pyright + unit suite green. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WBMWTEtQdJ4W5UrWuLHCck	2026-07-16 14:36:54 -04:00
didericis	5dfb9b0d75	refactor(gateway): parameterize git-gate provisioning transport (Stage B prep) ... git-gate provisioning into the running gateway was hard-wired to docker exec/cp. Extract a backend-neutral `GatewayTransport` (exec + cp_into) so the same provisioning logic serves both the docker gateway container and the firecracker gateway VM (over SSH, added with the launch swap). - `provision_git_gate` / `deprovision_git_gate` now take a transport instead of a gateway name; `DockerGatewayTransport` wraps the existing docker exec/cp behavior. deprovision is best-effort (catches the transport error) — matching the prior idempotent teardown. - both consolidated_launch callers pass `DockerGatewayTransport(name)` — no behavior change; the firecracker swap flips only its own to SSH. Pure refactor: docker path unchanged, gateway_provision tests updated to construct the transport, all green. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WBMWTEtQdJ4W5UrWuLHCck	2026-07-16 14:36:54 -04:00
didericis	bb434b14d7	feat(firecracker): persistent infra-VM singleton lifecycle (Stage B, 4/n) ... The infra VM must outlive the short-lived `start` launcher and be reused across launches. Add an idempotent singleton: - `ensure_running()` adopts the infra VM when its control plane is already healthy (a prior launcher booted it), else clears any stale VM and boots a fresh one. Returns a handle usable for CA fetch / git-gate provisioning whether we booted it or adopted it. - boot is `detached` (firecracker in its own session via start_new_session) so it survives the launcher exiting; its PID is recorded so a later process can `stop()` it. `_kill_pidfile` SIGTERM/SIGKILLs but only if the PID is still a firecracker process (guards a recycled PID). - a STABLE SSH key (generated once under the infra cache dir, re-injected each boot via the cmdline) so any launcher can SSH in to fetch the CA / provision, not just the one that booted the VM. `InfraVm.vm` is None in the adopted case; teardown then goes through the PID file. Verified on a KVM host: first ensure_running boots; a second adopts it (same PID, no reboot) and can still reach /health and fetch the gateway CA over SSH; stop() tears it down (control plane then unreachable). Next: git-gate provisioning into the VM over SSH (today docker exec/cp), then swap consolidated_launch.py onto the infra VM. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WBMWTEtQdJ4W5UrWuLHCck	2026-07-16 14:36:54 -04:00
didericis	e1610121c0	feat(firecracker): run the gateway data plane in the infra VM too (Stage B, 2/n) ... The single infra VM now runs BOTH the orchestrator control plane and the gateway data plane (egress / supervise / git-http), multi-tenant against the local control plane — the single-VM shape from the Stage B design. - Dockerfile.infra: the firecracker infra image = the gateway image + the baked control-plane source (FROM bot-bottle-gateway, COPY bot_bottle). Reuses the gateway payload rather than copying mitmproxy/ gitleaks into a third image; the docker backend keeps its two separate images. - infra_vm: build from source (gateway then infra image), and the PID-1 init now also launches `gateway_init` with BOT_BOTTLE_ORCHESTRATOR_URL=http://127.0.0.1:8099. Adds `gateway_ca_pem` (fetch the mitmproxy CA over SSH) and the agent-facing port constants. - init exports PATH — a bare-init shell resolves its own execs via a built-in default path, but that isn't in the environment, so gateway_init's `python3 ...` daemons would otherwise fail to spawn. Verified on a KVM host: infra VM boots, control plane /health -> 200, and egress:9099 / supervise:9100 / git-http:9420 all listen and are reachable from the host over the TAP link; the gateway CA is retrievable. (git-gate stays down until a bottle provisions its per-bottle entrypoint/creds, same as a fresh docker gateway — non-fatal, the supervisor keeps the rest up.) Next: agent->gateway VM-to-VM routing so bbfc* VMs reach these ports at the infra VM and nowhere else. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WBMWTEtQdJ4W5UrWuLHCck	2026-07-16 14:36:54 -04:00
didericis	1614172423	feat(firecracker): run the orchestrator control plane as a VM (Stage B, 1/n) ... First slice of Stage B: the orchestrator control plane runs as a persistent Firecracker infra VM instead of a Docker container. The host CLI reaches it over HTTP at the orchestrator link's guest IP; agent VMs will reach its gateway ports (added next) over VM-to-VM routing. - Dockerfile.orchestrator bakes the stdlib-only control-plane source (COPY bot_bottle) so the image is self-contained and runs from a built image with no runtime bind-mount — a guest VM can't bind-mount host source. (Build-from-source stays the default; a pull-from-registry mode lands later. The docker backend's dev bind-mount still overlays this.) - util.build_base_rootfs_dir / inject_guest_boot take a `variant` + `init_script`, so the same orchestrator image is prepared two ways without a cache collision: the builder VM keeps the SSH-only agent init; the infra VM gets a control-plane PID-1 init. - new firecracker/infra_vm.py: boot the infra VM on the orchestrator link, run `python -m bot_bottle.orchestrator` as PID 1, and poll /health. Verified on a KVM host: infra VM boots, control plane answers `GET /health -> 200 {"status":"ok"}` from the host over the TAP link. Next: fold gateway_init (egress/git-gate/supervise) into the same VM, then agent->gateway routing. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WBMWTEtQdJ4W5UrWuLHCck	2026-07-16 14:36:54 -04:00
didericis	4edd7803e8	feat(firecracker): build agent images in a builder VM, not host docker (Stage 3) ... Replace the host `docker build` + `docker export` behind the Firecracker agent rootfs with an in-VM buildah build. `image_builder.build_agent_rootfs_dir` boots a throwaway builder VM (the orchestrator image, which carries buildah) on the NAT'd orchestrator link, sends the Dockerfile over SSH, `buildah build`s it, smoke-tests the result with `buildah run` (the image's own PATH, so it catches an npm silent-failure stub), and streams the rootfs tar back into the content-addressed cache dir — the same base dir `util.build_rootfs_ext4` already turns into a bootable ext4 with `mke2fs -d`. No host Docker daemon, no root-equivalent `docker` group; an untrusted Dockerfile runs in a confined microVM, not on the host. - new firecracker/image_builder.py (boot → build → smoke → stream). - launch.py: `_build_agent_image` → `_build_agent_base`, returning the base dir from the builder VM. The committed-snapshot (freeze/migrate) path still exports via host docker until it too is ported. - util: `_inject_guest_boot` → public `inject_guest_boot` (shared with the builder). unit tests for the cache decision + smoke-test paths. Verified on a KVM host end-to-end: builds the real claude Dockerfile (node:22-slim + npm claude-code) in-VM in ~60s, the produced agent VM boots and `claude --version` returns 2.1.172; the content cache skips the rebuild on a repeat launch. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WBMWTEtQdJ4W5UrWuLHCck	2026-07-16 14:36:54 -04:00
didericis	81a2f15046	feat(firecracker): NAT'd egress link for the orchestrator/builder VM ... The orchestrator/gateway VM is trusted infra, not an isolated agent: it builds agent images in-VM (buildah must FROM-pull + apt/npm) and, in the Stage B cutover, forwards agent egress upstream. Give it a dedicated TAP (`bborch0`) on a /31 at the top of the IP_BASE /16 (clear of the bbfc* agent pool at the bottom), NAT'd out the host uplink — while agent VMs keep their fail-closed, gateway-only isolation table. - netpool.defaults.env / netpool.py: new BOT_BOTTLE_FC_ORCH_IFACE + `orch_slot()` (index -1 sentinel; host x.y.255.0 / guest x.y.255.1). - scripts/firecracker-netpool.sh: create + address the orchestrator TAP; `bot_bottle_fc_nat` table masquerades its /31 out the uplink and accepts its forward path. Because bootstrap still runs Docker (whose FORWARD policy is DROP), a best-effort, guarded, idempotent DOCKER-USER ACCEPT is added too (skipped once Docker is gone). down/status updated. - nix/firecracker-netpool.nix: mirror the option, pass it via the unit Environment= (the store-copied script can't read the defaults file), and add iptables to the unit path for the DOCKER-USER step. Agent isolation is unchanged: the new rules only ever accept/masquerade the orchestrator link and never drop, so they can't weaken the bbfc* drops. Applied by re-running `sudo ./scripts/firecracker-netpool.sh up`. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WBMWTEtQdJ4W5UrWuLHCck	2026-07-16 14:36:54 -04:00
didericis-claude	2e738c3338	fix(docker): use run_docker in docker_exec, docker_cp, verify_agent_image ... The rebase onto lazy-backend-imports converts existing helpers (image_exists, container_exists, etc.) to run_docker; the three new functions added in this branch still called subprocess.run directly. Switch them over for consistency.	2026-07-16 14:36:54 -04:00
didericis-claude	9369fb7de5	feat(firecracker): implement consolidated orchestrator launch (PRD 0070) ... Replace the per-bottle Docker sidecar bundle with the shared per-host orchestrator + gateway, mirroring what the Docker backend already has. - Add `bot_bottle/backend/firecracker/consolidated_launch.py`: `_FirecrackerOrchestratorService` (subclasses `OrchestratorService`, overrides `_gateway()` to return a `DockerGateway` with host port bindings so Firecracker VMs can reach it via their TAP link); `launch_consolidated()` registers the bottle by guest IP (attribution key), provisions git-gate into the shared gateway, and returns the shared CA + orchestrator URL for teardown; `teardown_consolidated()` deregisters and cleans up. - Rewrite `bot_bottle/backend/firecracker/launch.py`: removes the per-bottle sidecar bundle (`_start_sidecar_bundle`, `_stage_git_gate`, etc.) and `_mint_certs`; wires `launch_consolidated()` instead. The VM still sends to `host_tap_ip:PORT` — Docker's PREROUTING DNAT + the nft `ct status dnat accept` rule in the forward chain route the traffic to the shared gateway container. - Extend `DockerGateway` with `host_port_bindings` so the Firecracker gateway publishes its ports on the host (`0.0.0.0:PORT`). - Parameterise `OrchestratorService` with `orchestrator_name` / `orchestrator_label` so Docker and Firecracker orchestrators can coexist on the same host (`bot-bottle-orchestrator` vs `bot-bottle-fc-orchestrator`). Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>	2026-07-16 14:36:54 -04:00
didericis	9afdeff619	refactor(firecracker): use docker_mod instead of hand-rolled docker helpers ... firecracker/launch.py reimplemented docker build/image-exists/rm/exec/cp as private functions instead of the shared docker_mod used by the docker and macos-container backends. Switching to docker_mod dedupes the logic and gets --no-cache support for free (docker_mod.build_image already reads BOT_BOTTLE_NO_CACHE); docker_mod gains docker_exec/ docker_cp general-purpose helpers to cover what the private versions did.	2026-07-16 14:36:54 -04:00
didericis	e45df03bd9	fix: smoke-test agent images after build, add start --no-cache ... npm treats optionalDependencies failures as non-fatal, so a transient network blip fetching claude-code's platform-native binary during `npm install -g` left a stub CLI in an image that still "built" successfully — then got baked into the Docker/Container layer cache until forced to rebuild. Post-build smoke test (provider-declared argv, run in a throwaway container of the freshly built image) fails the launch loudly instead of shipping a broken image; --no-cache gives an escape hatch to force a from-scratch rebuild. Closes #353.	2026-07-16 14:36:54 -04:00
didericis	09393b354b	refactor(de-sidecar): purge the "sidecar" name from live code, tests, and current docs ... Completes the de-sidecar cleanup: no live code, test, current doc, script, or nix file mentions or is named "sidecar" any more. Only the dated PRD/research docs keep the term as historical record (agreed on the #385 thread). - Rename `sidecar_init.py`→`gateway_init.py` was done earlier; this pass sweeps the remaining descriptive uses: the egress / git-gate / supervise components are the gateway's *daemons*, the shared container is the *gateway*, the old per-bottle container was the *companion container*. - Rename `tests/integration/test_sidecar_bundle_image.py`→`test_gateway_image.py` and its class; update `docs/ci.md` + `tests/README.md` for the renamed/ removed integration tests. - `SIDECAR_PORTS` shell var in `scripts/firecracker-netpool.sh`→`GATEWAY_PORTS`. Full unit suite green (bar the pre-existing `/bin/sleep`-missing env errors in test_gateway_init); docker integration — gateway singleton, broker, real two-bottle multitenant isolation, and the gateway-image build — all pass. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WBMWTEtQdJ4W5UrWuLHCck	2026-07-14 17:07:56 -04:00
didericis	77948ef56c	refactor(de-sidecar): remove the per-bottle companion-container architecture ... The per-agent companion container (the egress/git-gate/supervise data plane run once per bottle) is the pre-consolidation architecture. Remove it and disable the backends that still depend on it, per the #385 thread. - Delete `backend/docker/sidecar_bundle.py`; docker's live path uses the consolidated shared gateway, not a per-bottle bundle. - Disable the firecracker and macos-container backends: their `launch()` fails closed (they launched a per-bottle companion; firecracker's consolidated relaunch is #354, macos follows). Their `enumerate` return empty and `cleanup` drop the companion-container discovery (firecracker keeps VMM/run-dir cleanup). - Fail-close both backends' `egress_apply` reload (it signalled the per-bottle container); consolidated egress policy resolves per-request against the orchestrator, so gateway-side apply is a follow-up. - Rename `egress_sidecar_env_entries` → `egress_gateway_env_entries`, `SIDECAR_PORTS` → `GATEWAY_PORTS`. - Move the shared DockerBottlePlan fixture to `tests/unit/_docker_bottle_plan.py`; delete tests for the removed launch paths; update cleanup/egress-apply tests. Docker consolidated launch verified end-to-end (multitenant isolation integration test passes). macos/firecracker are intentionally disabled. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WBMWTEtQdJ4W5UrWuLHCck	2026-07-14 17:02:08 -04:00
didericis	c10d1cb6e0	refactor(de-sidecar): rename sidecar_init→gateway_init, drop docker's dead per-bottle compose renderer ... Part of the de-sidecar cleanup (#385 discussion): the per-bottle companion container is the old architecture. - Rename `bot_bottle/sidecar_init.py` → `gateway_init.py` (it's the gateway image's PID-1 supervisor); env var `BOT_BOTTLE_SIDECAR_DAEMONS` → `BOT_BOTTLE_GATEWAY_DAEMONS`; log prefix `sidecar-init:` → `gateway-init:`. Update Dockerfile.gateway COPY/ENTRYPOINT and the test. - Remove the dead per-bottle compose renderer from `backend/docker/compose.py` (`bottle_plan_to_compose`, `_sidecar_bundle_service`, `_agent_service`, and the network/bind/proxy helpers). Docker's live path uses `consolidated_agent_compose`; only the compose *lifecycle* helpers (up/down/ls/write) remain. Trim `test_compose.py` to the surviving helpers. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WBMWTEtQdJ4W5UrWuLHCck	2026-07-14 16:46:46 -04:00
didericis	96b84eb84d	refactor(orchestrator): split conflated sidecar image into orchestrator + gateway images ... One image — `bot-bottle-sidecars:latest`, built from `Dockerfile.sidecars` — served two unrelated roles: the egress/git-gate/supervise *data plane* and the orchestrator *control plane* (which ran the same image with the entrypoint overridden to `python3 -m bot_bottle.orchestrator`). The control plane is stdlib-only, so it needed none of the mitmproxy/git/gitleaks payload it was riding on — while being the most secret-dense process on the host (PRD 0070's "secret concentration"). Split into two purpose-built images: - `Dockerfile.gateway` -> `bot-bottle-gateway:latest` — the data plane (renamed from Dockerfile.sidecars; identical contents). - `Dockerfile.orchestrator` -> `bot-bottle-orchestrator:latest` — a lean `python:3.12-slim` runtime; the bind-mounted `bot_bottle` package supplies the code (so the #381 source-hash recreate semantics are unchanged). `OrchestratorService` now takes distinct `image` (control plane, default `ORCHESTRATOR_IMAGE`) and `gateway_image` (data plane, default `GATEWAY_IMAGE`) instead of feeding one `self.image` to both, and builds the lean image (build-if-missing) before starting the container. The per-bottle bundle constants in `backend/docker/sidecar_bundle.py` now alias the gateway constants so a bundle and the shared gateway can never drift onto different images. The `bot-bottle-sidecars` *image* name and `Dockerfile.sidecars` are gone; the per-bottle *container* name prefix (`bot-bottle-sidecars-<slug>`) is intentionally left for a separate change. Verified end-to-end: both images build; the lean image runs the control plane; `ensure_running` brings up the orchestrator on `bot-bottle-orchestrator:latest` and the gateway on `bot-bottle-gateway:latest` (distinct images) and reports healthy. Closes #384. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WBMWTEtQdJ4W5UrWuLHCck	2026-07-14 16:24:16 -04:00
didericis	b2f61053ad	fix(egress+orchestrator): inject per-bottle auth tokens in the shared gateway ... The cut-over dropped the per-bottle token flow, so an authed egress route on the shared gateway failed with 'env var EGRESS_TOKEN_0 is unset' — the gateway reads the token from its env, but a shared gateway has no per-bottle env. Now the bottle's egress auth tokens travel to the gateway over /resolve and the addon injects from them, mirroring what the per-bottle sidecar's env did: - launch resolves the token values from the host env and hands them to the orchestrator, which holds them IN MEMORY (keyed by bottle_id, never written to the registry DB) and serves them on /resolve; - PolicyResolver.resolve_policy_and_bottle_id + resolve_client_context now return the token map alongside policy + bottle_id (one round-trip); - the egress addon overlays the process env with the bottle's tokens per request and uses that env for auth injection AND DLP — the agent never sees the credential. Secrets stay off disk (validated: /resolve returns the token, the registry DB does not contain it). SecretProvider (#355) is the future hardening. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WBMWTEtQdJ4W5UrWuLHCck	2026-07-14 02:38:20 -04:00
didericis	69db629e16	feat(backend): slice 13d — cut docker launch over to the consolidated gateway (e2e green) ... Rewire DockerBottleBackend.launch to the consolidated model, replacing the per-bottle sidecar bundle. VALIDATED END-TO-END on real docker: test_sandbox_escape passes all 5 attacks (egress DLP + git-gate gitleaks) through the shared gateway. launch now: - mints git-gate dynamic keys (if any), then launch_consolidated() to register the bottle + provision its git-gate state into the gateway and get the agent's attach context (pinned source IP, gateway address); - installs the SHARED gateway CA (gateway.ca_cert_pem) into the agent; - renders the agent-only consolidated compose on the gateway network at the pinned IP, proxied through the gateway; - points the agent's git-gate insteadOf (http://<gw>:9420) and supervise MCP (http://<gw>:9100) at the gateway (DockerBottlePlan.agent_git_gate_url / agent_supervise_url); - teardown = compose down + teardown_consolidated (dereg + deprovision). Dropped the per-bottle networks, per-bottle egress CA, and bundle service. Fixes surfaced by the real e2e (couldn't be caught by unit mocks): - provision_git_gate installs the bottle-agnostic pre-receive/access hooks into the gateway (were cp'd per-bundle before); - source-IP allocation reads the *actual* container IPs on the network (gateway + orchestrator + agents), not just gateway + registry — the orchestrator container sits on the network and was colliding. Unit tests for the old bundle launch rewritten against the new collaborators. NOTE for reviewers: the gateway/bundle image (bot-bottle-sidecars) must be rebuilt when its flat sources change — `ensure_built` only builds-if-missing, so a stale image silently runs the OLD single-tenant daemons. A content-hash / --rebuild path is a follow-up. pyright 0 errors; pylint 9.83/10; unit suite green (1760 tests; the 13 test_sidecar_init /bin/sleep errors are pre-existing NixOS-local noise); test_sandbox_escape green on real docker. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WBMWTEtQdJ4W5UrWuLHCck	2026-07-14 02:38:20 -04:00
didericis	96f75599a1	feat(orchestrator): slice 13d(i) — containerize the orchestrator (validated on docker) ... Real-on-host testing surfaced two issues the unit-mocked slices couldn't: 1. The host (NixOS) firewall DROPS container->host traffic, so a host-process orchestrator is unreachable from the gateway container. Fix: run the orchestrator AS a container on the shared gateway network (PRD 0070's "virtualize the orchestrator") — the gateway reaches it by container name over docker DNS (container<->container, no firewall), and the host CLI reaches it via a published loopback port. 2. The control plane crashed the connection on a dispatch error (e.g. a broker failure) instead of returning 500. Changes: - lifecycle: OrchestratorProcess (host process) -> OrchestratorService (containers). Runs the control plane in the bundle image with the repo bind-mounted (orchestrator is stdlib-only), register-only stub broker so it needs NO docker socket (the backend launches agents; the host manages both containers). Registry DB persists via a host-root mount. ensure_running is an idempotent singleton over both containers. - gateway: BOT_BOTTLE_ORCHESTRATOR_URL is now the orchestrator's *by-name* URL on the shared network (dropped the host.docker.internal hack). - control_plane: _serve wraps dispatch — a failure returns 500, never crashes the connection. - OrchestratorProcess default broker -> stub (register-only) for docker. Validated live end-to-end: both containers up, gateway->orchestrator by name OK, register -> resolve-by-source-IP returns the bottle's policy from inside the gateway. pyright 0 errors; pylint 9.83/10; unit suite green (1760 tests; the 13 test_sidecar_init /bin/sleep errors are pre-existing NixOS-local noise). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WBMWTEtQdJ4W5UrWuLHCck	2026-07-14 02:38:20 -04:00
didericis	3062fc9230	feat(orchestrator): slice 13c(viii) — agent-only consolidated compose render ... The per-bottle model rendered a compose project with the agent + a sidecar bundle on two per-bottle networks. Consolidated has no sidecars — one shared gateway serves everyone — so this renders JUST the agent, attached to the external shared gateway network at the orchestrator-allocated pinned IP, and pointed at the gateway's address for egress. - consolidated_agent_compose(plan, *, gateway_ip, source_ip, network): agent service only; networks {<gateway>: {ipv4_address: source_ip}} on the external gateway network; HTTPS_PROXY -> http://<gateway_ip>:9099; NO_PROXY includes the gateway address so git-http + supervise (also on the gateway) bypass the proxy and are reached directly. No depends_on/sidecars. - Pure (takes the launch-time LaunchContext values), so it's unit-testable without docker. pyright 0 errors; pylint 9.83/10; unit suite green (the 13 test_sidecar_init /bin/sleep errors are pre-existing NixOS-local noise). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WBMWTEtQdJ4W5UrWuLHCck	2026-07-14 02:38:20 -04:00
didericis	327e756eef	feat(orchestrator): slice 13c(vi) — consolidated launch sequence (composition) ... Composes the orchestrator primitives into the register/teardown sequence that replaces the per-bottle bundle: launch_consolidated(egress_plan, git_gate_plan): ensure orchestrator + gateway up -> read the gateway network's subnet + the gateway's own address -> allocate the bottle a pinned source IP (skip the gateway + live bottles) -> register (egress policy blob + slug metadata) -> provision its git-gate repos/creds into the gateway. Returns a LaunchContext (bottle_id, identity_token, source_ip, network, gateway_ip, orchestrator_url) — everything the agent container needs to attach. Rolls the registration back if provisioning fails, so no orphan. teardown_consolidated(bottle_id): deregister + deprovision (idempotent). The agent `docker run` itself stays the backend's job (provider provisioning); this owns the orchestrator-facing wiring so the sequence is unit-testable — collaborators mocked, next_free_ip + registration_inputs run for real (IP allocation + policy blob asserted end to end). pyright 0 errors; pylint 9.83/10; unit suite green (1755 tests; the 13 test_sidecar_init /bin/sleep errors are pre-existing NixOS-local noise). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WBMWTEtQdJ4W5UrWuLHCck	2026-07-14 02:38:20 -04:00
didericis	444924d95d	feat(git-gate+orchestrator): slice 13c(v) — provision git-gate into the running gateway ... Places a bottle's git-gate credentials + bare repos into the live shared gateway container when it registers — the execution side of the 13c(i) provisioning render. - provision_git_gate(gateway, bottle_id, plan): mkdir the per-bottle creds dir, docker cp each upstream's identity key (+ known_hosts when present) into /git-gate/creds/<bottle_id>/, then docker exec the namespaced, init-only script from git_gate_render_provision. No-op with no upstreams. - deprovision_git_gate(gateway, bottle_id): rm the bottle's /git/<id> + creds on teardown (idempotent). - bottle_id is validated (shell/path-safe alphabet) BEFORE any docker cp/rm, so a traversal id can never reach a path arg — the creds/repo dirs land in docker cp/rm arguments. Registry ids are token_hex; defense in depth. pyright 0 errors; pylint 9.83/10; unit suite green (the 13 test_sidecar_init /bin/sleep errors are pre-existing NixOS-local noise). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WBMWTEtQdJ4W5UrWuLHCck	2026-07-14 02:38:20 -04:00
didericis	40ad2364ab	feat(orchestrator): slice 13c(ii) — shared-gateway source-IP allocator ... Deterministic per-bottle address assignment on the consolidated docker gateway's shared network — the address the gateway uses as its attribution key. Pure ipaddress logic; the docker-specific inputs (subnet CIDR, the gateway container's own address) are gathered by the caller and passed as `taken`, keeping it testable without docker. - next_free_ip(cidr, taken): lowest host address not reserved (network + broadcast excluded by hosts(); docker's router .1 excluded here) and not already assigned (gateway container + live bottles from the registry). NoFreeAddressError when the subnet is exhausted. pyright 0 errors; pylint 9.83/10; unit suite green (the 13 test_sidecar_init /bin/sleep errors are pre-existing NixOS-local noise). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WBMWTEtQdJ4W5UrWuLHCck	2026-07-14 02:38:20 -04:00
didericis	a0e17d56de	refactor: move bot_bottle_root/host_db_path to paths; kill the monkeypatch (#352) ... Create bot_bottle/paths.py as the canonical home for the app-root path helpers (bot_bottle_root, host_db_path, HOST_DB_FILENAME) — foundational, not supervise- or db-specific. `bot_bottle_root()` now honours a BOT_BOTTLE_ROOT env override. Repoint every consumer (supervise, supervise_types, db_store, queue_store, audit_store, store_manager, bottle_state, cli/supervise, docker/cleanup, orchestrator/registry) at paths; remove the definitions (and supervise's duplicate host_db_path) and the now-dead `import sys`. Add paths.py to the sidecar bundle (Dockerfile.sidecars) for the flat-import copies. Tests: replace ~12 files' monkeypatching of supervise.bot_bottle_root (and the flat/pkg/supervise_types triple-patch dance) with a single `use_bottle_root()` helper that sets BOT_BOTTLE_ROOT — every module and flat/package copy reads the same env var, so one override covers them all. Net -97 lines. Behaviour-preserving: full unit suite unchanged (only the pre-existing /bin/sleep sidecar-init errors remain). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WBMWTEtQdJ4W5UrWuLHCck	2026-07-14 02:38:20 -04:00
didericis	2b970d1170	refactor(firecracker): single-source the network-pool defaults ... The pool params (size, IP base, iface prefix, nft table) were triplicated — hardcoded in netpool.py, scripts/firecracker-netpool.sh, and nix/firecracker-netpool.nix — plus the IP math (3x) and the nft ruleset (2x). Nothing enforced agreement; changing the base (ce3fad9, off CGNAT) forced a coordinated three-file edit, and a missed one would silently provision a range the launcher doesn't expect. Collapse to one source of truth: * netpool.defaults.env — a plain KEY=VALUE file (bash-sourceable, systemd EnvironmentFile-compatible, Python- and Nix-parseable) holding the four defaults. A BOT_BOTTLE_FC_* env var still overrides any key. * netpool.py reads it for the Python defaults (missing file = hard error, not confusing empty defaults). * the shell script falls back to it (no literal `:-8` / `10.243.0.0`), and its `up` is now non-destructive/idempotent (only creates a missing TAP), so re-running never cuts a live VM. * the Nix module readFile-parses it for its option defaults and delegates bring-up to the SAME shell script (dropping its duplicate IP math, nft ruleset, and TAP loop) — passing every value as Environment= so the store-detached script never needs the file. Net: defaults 3x -> 1x, nft ruleset 2x -> 1x, TAP loop 2x -> 1x. The one remaining IP-math dup (Python launch-addressing vs bash bring-up) is justified — different runtimes. Tests now guard the invariant (Python reads the shared file; the script/module hold no literals) instead of pinning duplicated strings. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WBMWTEtQdJ4W5UrWuLHCck	2026-07-13 04:27:48 -04:00
didericis	f71558aff6	fix(firecracker): run the agent from /home/node, not the /root SSH cwd ... The control SSH logs in as root, so the agent (dropped to node via runuser) inherited cwd=/root. That was harmless while the rootless rootfs left /root node-owned, but the dropbear fix (chown /root → root:root 0700) made /root unreadable to node — so the agent's cwd was inaccessible, breaking Node's process.cwd(), Claude Code's shell-snapshot machinery, and `/doctor` (which reported /root/.claude EACCES and failed every Bash command). Run the agent from its workdir (default /home/node) via `env --chdir`. Not a `sh -c 'cd … && exec "$@"'` wrapper: ssh space-joins everything after the host into one string for the guest shell, so the quoted script + $@ get re-split and mangled (it exec'd the $0 placeholder → exit 127). `env --chdir=DIR …` is all simple words, so it survives the join. Verified end-to-end: a real headless claude bottle now runs its Bash tool and exits 0 (was EACCES/127 before). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WBMWTEtQdJ4W5UrWuLHCck	2026-07-11 18:57:33 -04:00
didericis	a80356af75	feat(firecracker): portable systemd-unit install for the network pool ... Make the network pool a persistent, distro-uniform resource instead of a non-persistent per-distro shell command. systemd is the common denominator across Debian/Ubuntu/Fedora/RHEL/Arch/… (and NixOS), so `backend setup` on any systemd host now installs one bot-bottle-owned oneshot unit (bot-bottle-firecracker-netpool.service): params pinned via Environment= (so it doesn't depend on $SUDO_USER at boot), ExecStart/Stop delegating to the bundled bring-up script (single source of logic). - render_systemd_unit() in netpool.py (derived from the same constants as the shell script + nix module). - backend setup: install + enable the unit directly when run as root, else print a self-contained copy-paste block. NixOS keeps its declarative module (which produces the same-named unit); non-systemd hosts fall back to the raw imperative script. - backend teardown: symmetric — disable + remove the unit. - backend status: report the unit's active/inactive state so you can tell a persistent install from an imperative one. Net: one install path (`backend setup`), persistent, identical across distros; per-distro variance shrinks to installing nft + iproute2. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WBMWTEtQdJ4W5UrWuLHCck	2026-07-11 18:33:42 -04:00
didericis	3d7c508dc4	fix(firecracker): make the launch path work end-to-end ... First real end-to-end launch (there was no firecracker integration test) surfaced three bugs in the control/provision path, all now verified fixed by tests/integration/test_firecracker_launch: 1. Guest SSH rejected the correct key. The rootless rootfs build (`docker export | tar` as a non-root user) can't preserve uid 0, so every path — including /root — is owned by the build uid (node in guest). dropbear refuses root's authorized_keys when /root isn't root-owned, so auth fell back to password → denied. bb-init now `chown -R 0:0 /root`. 2. The SSH client (newer OpenSSH) didn't reliably present the -i key against the operator's ~/.ssh config; add `IdentitiesOnly=yes` to ssh_base_argv so only the per-bottle key is offered. 3. cp_in double-wrapped the remote command as `sh -c <remote>`, but ssh space-joins everything after the host into one string for the guest shell, collapsing `sh -c mkdir -p X && …` to `mkdir` with no operand ("missing operand"). Pass the command as a single arg and let the guest login shell run it (stdin still carries the tar). Also stop discarding dropbear's stderr (`-E` now reaches the host-side console.log) so future guest-auth issues are debuggable. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WBMWTEtQdJ4W5UrWuLHCck	2026-07-11 18:16:57 -04:00
didericis	f42a0dc7fe	fix(firecracker): status() defers unverifiable nft, like the preflight ... `status()` hard-failed when it couldn't confirm the nft table, but listing nftables usually needs root — so an unprivileged `backend status` reported "not ready" even with the pool fully up, making it useless as a launch gate (and skipping the firecracker integration test on a set-up host). Base readiness on what the launch preflight actually hard-requires: the TAP pool present (unprivileged, authoritative) + no range overlap. Report the nft table state (present / unverified / not-confirmable-unprivileged) but don't let it flip readiness — the post-boot isolation probe is the authoritative isolation check, same as the preflight's deferral. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WBMWTEtQdJ4W5UrWuLHCck	2026-07-11 17:28:17 -04:00
didericis	480269b116	refactor(firecracker): non-invasive NixOS module (no firewall switch) ... The module used `networking.nftables.tables.*` (which forces `networking.nftables.enable = true`, flipping the host firewall backend) and `systemd.network.enable` (handing interfaces to systemd-networkd) — disruptive on an iptables + Docker daily driver. Rewrite it to a single systemd oneshot that brings the pool up: idempotent `ip tuntap` for the TAPs + `nft -f` for the independent `inet bot_bottle_fc` table (its own hooks at priority -10), with ExecStop teardown. Same as the imperative script, but declarative. It touches neither the firewall backend nor networkd, so it coexists with iptables/Docker/ufw/firewalld. Verified through the NixOS module system (service present, firewall untouched). Drop the now-redundant `render_nixos_module()` paste generator (the module is a real importable file) and point `backend setup` at importing it (flake output or the file path), noting it's non-invasive. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WBMWTEtQdJ4W5UrWuLHCck	2026-07-11 17:25:50 -04:00
didericis	949b001464	feat(firecracker): show binary/KVM prerequisites in backend setup ... `backend setup --backend=firecracker` only covered the network pool. Lead with the other prerequisites: the firecracker binary (with a release install pointer when it's not on PATH, plus a NixOS note), a /dev/kvm check, and a reminder about the cached guest kernel + static dropbear and mke2fs. The network pool is now step 2. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WBMWTEtQdJ4W5UrWuLHCck	2026-07-11 17:03:04 -04:00