461 Commits

Author	SHA1	Message	Date
didericis-codex	240b540e11	fix: make config store schema explicit	2026-07-21 01:50:45 -04:00
didericis-codex	2cc090c184	Add cached image quickstart	2026-07-21 01:50:45 -04:00
didericis-claude	8e43c26ab4	fix(backend): extract poll_ca_cert helper and fix PRD port docs ... Extract the shared CA cert polling loop into `backend/util.poll_ca_cert` (firecracker and macos backends were duplicating deadline/sleep/raise logic). Each caller now wraps a fetch lambda and converts TimeoutError to its own error type. Also corrects the PRD port publication line from {port}:{port} to {host_port}:8099.	2026-07-21 04:54:40 +00:00
didericis-claude	14ff4fe186	fix(lifecycle): build gateway before infra and fix orchestrator port mapping ... - `_build_images()` now builds Dockerfile.gateway → Dockerfile.orchestrator → Dockerfile.infra in order; Dockerfile.infra starts FROM bot-bottle-gateway so the base must exist on clean hosts. - Publish mapping corrected from `self.port:self.port` to `self.port:DEFAULT_PORT` (8099) — gateway_init hardcodes the orchestrator on port 8099 inside the container, so the host-side published port must map to that fixed internal port. - BOT_BOTTLE_ORCHESTRATOR_URL inside the container now always points to 127.0.0.1:8099, not self.port, since gateway daemons reach the orchestrator over loopback at the fixed internal port. - Update test_ensure_running_builds_both_images → _all_images for the new three-step build sequence; add test_publish_maps_host_port_to_fixed_internal_port to lock in the port-mapping fix. Addresses the P1 findings from the didericis-codex review on PR #432.	2026-07-21 04:54:40 +00:00
didericis-claude	28766d7733	fix(pyright): resolve type errors introduced by lifecycle refactor ... - Remove unused INFRA_IMAGE import from test_orchestrator_lifecycle - Update integration test to use new single-container OrchestratorService API (infra_name/image replaces orchestrator_name/gateway_name/gateway_image) - Move type: ignore to the lambda line in gateway_init SIGHUP handler - Break two long lines in test_orchestrator_lifecycle	2026-07-21 04:54:40 +00:00
didericis-claude	819f967844	fix(lint): resolve pylint findings in gateway_init ... - Extract _sigkill_all() to cut nesting depth below the 5-block limit - Add pylint: disable=consider-using-with on Popen (process must outlive caller) - Break long SIGHUP signal line to stay within 100 chars	2026-07-21 04:54:40 +00:00
didericis-claude	2f45f5afec	feat(docker): consolidate to single infra container under gateway_init supervise tree ... Collapses the two-container Docker model (gateway + orchestrator) into one bot-bottle-infra container, matching the macOS and Firecracker backends. - Dockerfile.infra: now a shared gateway+orchestrator base (COPY bot_bottle from orchestrator build, no CMD override) - Dockerfile.infra.fc: new Firecracker-specific layer (buildah/crun/netavark) - gateway_init: adds orchestrator daemon with _OPT_IN_DAEMONS gating so it only starts when BOT_BOTTLE_GATEWAY_DAEMONS explicitly includes it - orchestrator/lifecycle: OrchestratorService manages one infra container; builds orchestrator (intermediate) then infra; live source bind-mounted at /bot-bottle-src with PYTHONPATH so the subprocess uses the checkout - backend/consolidated_util: extracts provision_bottle + teardown_consolidated shared across all three backends; removes duplication in docker/fc/macos consolidated_launch modules - firecracker/infra_vm: builds four images (orchestrator→gateway→infra→infra.fc) - All unit tests updated and passing (1878 tests) - PRD status: Draft → Active	2026-07-21 04:54:40 +00:00
didericis-claude	1f192d785a	refactor(config_store): replace generic key/value table with explicit schema ... Switch orchestrator_config from a generic key/text store to a typed single-row table with a dedicated teardown_timeout_seconds REAL column. The CHECK (id = 1) constraint enforces at most one config row. Remove TEARDOWN_TIMEOUT_KEY constant; update OrchestratorConfigStore to expose get/set/delete_teardown_timeout_seconds() instead of the generic get/set/delete(key). Resolving the DB value in resolve_teardown_timeout() no longer needs a float() conversion since the column is already REAL.	2026-07-21 00:40:59 -04:00
didericis-claude	cf9a53d582	fix: resolve teardown timeout before registration to prevent orphaned state ... If resolve_teardown_timeout() raised after launch_consolidated()/register_agent() returned, the teardown callback was never registered, leaving the bottle registered with no cleanup path. Resolve the timeout into a local variable before the registration call so any failure aborts before the bottle exists. Add tests covering OrchestratorConfigStore, resolve_teardown_timeout priority ordering (env > db > default), and the source-order invariant that the resolver runs before registration in all three backends.	2026-07-21 00:40:59 -04:00
didericis-claude	0b36c3eb48	fix: make orchestrator teardown timeout configurable (closes #435) ... Resolves via ENV VAR -> orchestrator DB config -> default (30 s, up from 5 s): BOT_BOTTLE_ORCHESTRATOR_TEARDOWN_TIMEOUT_SECONDS teardown_timeout_seconds key in new orchestrator_config table (bot-bottle.db) New OrchestratorConfigStore (same DbStore/TableMigrations pattern as the registry) stores the DB-level setting. resolve_teardown_timeout() implements the priority chain and is called at stack.callback registration time in all three backends (macos_container, docker, firecracker).	2026-07-21 00:40:59 -04:00
didericis-claude	571030b8e8	fix: require authoritative container snapshot before reconciling ... A failed container list or a partial per-container inspect were previously indistinguishable from a legitimately empty/partial live set, so reconciliation could silently unregister healthy bottles. - enumerate_active() now raises EnumerationError instead of returning [] when `container list` fails - Add inspect_container_network_ip() to util, which returns None on inspect failure (vs "" for "no DHCP address yet"), so live_source_ips can tell the two apart - live_source_ips() raises EnumerationError on either failure mode; register_agent() catches it and skips reconciliation, same as OrchestratorClientError - Update tests: rename test_empty_when_the_cli_fails to test_raises_when_the_cli_fails, update patching to the new function, add coverage for list failure, per-container inspect failure, and the launch-not-blocked path	2026-07-21 03:39:18 +00:00
didericis	0c1d27b605	fix(egress): the unattributed message must name the token-mismatch cause too ... `/resolve` fail-closes on a missing/ambiguous registry row *and* on a request whose identity token doesn't match. The message named only the first, so a bottle that was registered correctly but sent no token read as "not registered" and sent the reader looking for a deregistered bottle. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-07-21 03:27:16 +00:00
didericis	69361114d1	fix(egress): name the real fault when a deny-all is not an allowlist miss ... An unattributed bottle, an unreachable orchestrator, and an unparseable policy all become a deny-all Config, and a deny-all is indistinguishable from "policy loaded, host not allowed" at the decision point — both are just "no matching route". So every one of them reported `host X is not in the bottle's egress.routes allowlist`, which reads as a config problem and sends the operator hunting for a route that was never missing. Diagnosing a bricked registration cost hours for exactly this reason. Carry the structural reason on Config and prefer it in decide(). A genuine allowlist miss — a policy that loaded and simply lacks the host — keeps the original wording, so the message now tells the two cases apart. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-07-21 03:27:16 +00:00
didericis	e4d53fd360	fix(orchestrator): reap registry rows whose bottle is no longer running ... A registry row only ever left the registry two ways: an explicit teardown_bottle (the launcher's cleanup callback) or the same-IP supersede sweep in register(). Neither runs when the launching CLI dies hard, so the row outlives its container. That orphan is not inert. Source IPs are recycled by the backend's DHCP and by_source_ip fail-closes on ambiguity, so a leftover row at a reused address resolves no policy at all for the next bottle that lands there — and a bottle with no policy denies every host, which surfaces to the agent as "host X is not in the allowlist" for hosts that were never the problem. Add reap_absent/reconcile and call it from the macOS launch path before registering, so each launch self-heals the registry. Restores the invariant the data plane needs: at most one active row per live address, and none for a dead one. The second half matters as much as the first — when several rows claim a *live* address the newest wins and the rest are swept, otherwise a recycled address stays ambiguous, which is exactly the bricked state. The host supplies the live set because the orchestrator runs inside the infra container and cannot see the backend. A grace window exempts rows younger than it, so reconciliation cannot race a bottle still coming up, and a reconcile failure is logged rather than blocking an otherwise-fine launch. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-07-21 03:27:16 +00:00
didericis	2f8539c2c7	fix(macos): name the gateway instead of addressing it, so bottles survive it moving ... The shared gateway's address is DHCP-assigned and changes whenever the infra container is recreated — a source-hash bump, an image upgrade, a crash. Every agent-facing URL embedded that address, and the proxy URL reaches the agent as process environment at `container exec` time. A running process's environ cannot be rewritten from outside, so a moved gateway stranded every running bottle permanently: not degraded, unreachable, until relaunched and its agent session thrown away. Give the agent a stable name instead. `GATEWAY_HOSTNAME` replaces the address in the egress proxy URL, NO_PROXY, git-http, and supervise URLs, and resolves through the bottle's own /etc/hosts. Unlike environ that is a file, so it can be rewritten inside a container that is already running — which is the whole point: a gateway that returns at a new address is picked up by live bottles. Launch writes the entry before anything execs (every agent URL names the gateway, so it must resolve for the first connection), and re-points every running bottle once the gateway is up, so one stranded by an earlier restart re-attaches instead of needing a relaunch. The write needs root and the agent runs as `node`: the host can repoint a bottle's gateway name, the agent cannot repoint its own. Keep that asymmetry. Apple Container 1.0 has no container-name DNS on a user network and `container run` has no --add-host, so the entry is written by exec after start rather than declared at run. Does not address the other half of #443: per-bottle egress auth tokens are held in memory by the orchestrator and are still lost across a restart, so a re-attached bottle resolves its policy but not its injected credentials. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-07-20 23:08:59 -04:00
didericis	ad100b8a84	fix(macos): supply proxy env only at exec, so Codex keeps its identity token ... `container exec --env` does not override the run-time environment on Apple Container — it appends. The launch path baked a token-less `*_PROXY` into `container run` and relied on the exec-time, token-bearing value superseding it, so the agent's `environ` ended up with two `HTTPS_PROXY` entries, token-less first. Which entry a runtime reads is then luck. Node reads the last, so Claude bottles picked up the token and worked. Rust's `std::env::var` reads the first, so Codex proxied with no identity token at all; `/resolve` requires a matching (source_ip, identity_token) pair and fail-closes, so every request from a Codex bottle was denied — its model calls, its `wss://` reconnects, and its MCP servers alike. The registry row was correct the whole time, which is what made this read as a registration bug. Drop the run-time proxy vars entirely. A token-less proxy URL has no legitimate consumer: the init process is `sleep` and everything that egresses arrives by exec. Its only benefit was a tidy 403 for unattributed callers, which is not worth silently dropping attribution for — and a process that egresses before the exec-time env still fails closed, since the agent network is host-only and the gateway is the only route off it. Also corrects the two comments that asserted the false "exec --env wins" invariant, so the next reader doesn't rebuild the same assumption. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-07-20 22:54:52 -04:00
didericis-claude	c7375051fd	fix(lint): mark read_tty_line as an explicit re-export in _common.py ... Pyright reportUnusedImport fired because the bare import is not used within _common.py itself. The X as X idiom signals an intentional public re-export without requiring __all__.	2026-07-20 20:11:04 +00:00
didericis-claude	d9e685e860	fix(backend): don't prompt in headless/non-interactive backend selection ... _auto_select_backend gains a prompt parameter (default True). When prompt=False the docker-fallback [i/d/q] menu is skipped and the call dies immediately with an actionable message ("set BOT_BOTTLE_BACKEND=docker or install a VM backend"), preventing hangs in CI, webhook dispatch, and orchestrator launches. prepare_with_preflight passes prompt=not spec.headless so the headless start path can never block waiting for TTY input it cannot receive.	2026-07-20 19:58:14 +00:00
didericis-claude	b4b73a8acc	refactor: consolidate read_tty_line into bot_bottle/util.py ... Remove the private _read_tty_line duplicate from backend/__init__.py and the local definition from cli/_common.py. Both now import from the shared bot_bottle.util module.	2026-07-20 19:44:32 +00:00
didericis-claude	8b5b5730ae	fix: remove CLI backend assumptions and add docker fallback prompt ... - Remove hardcoded --backend=macos-container flag reference in firecracker/util.py require_firecracker() error message - Remove --backend flag from cli.py start; backend selection now driven exclusively by BOT_BOTTLE_BACKEND env var or auto-selection - Skip unavailable backends in cli.py cleanup (fixes crash on Linux when macos-container.prepare_cleanup calls require_container()) - Add two-tier auto-selection: VM backend first (macos-container on macOS, firecracker on Linux+KVM); fall back to docker with a security warning and interactive i/d/q prompt; exit if docker also unavailable and print VM install instructions Closes #344	2026-07-20 19:13:08 +00:00
didericis	44479f328e	fix(macos-container): launch gateway as a module, not /app/gateway_init.py ... The macOS infra container's init script still spawned the supervisor as `python3 /app/gateway_init.py`. 5ad3449 moved bot_bottle from flat files under /app into a pip-installed package, so that file no longer exists: the supervisor never started, mitmdump never generated its CA, and the launch failed downstream with a misleading "gateway CA not available" error pointing at TLS rather than at the supervisor. This is the same defect 127ba49 fixed for the firecracker backend. It survived on macOS because CI has no Apple Container runner — the KVM runner added in c193b04 covers firecracker, and ubuntu-latest covers docker, but nothing exercises macos_container. test_macos_infra asserted the substring "gateway_init.py", which the broken path satisfied; it now asserts the module form, matching the assertion 127ba49 already made in test_firecracker_infra_vm. Verified end to end on the macos-container backend: image builds, CA is generated, agent registers, and a headless bottle runs a prompt. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-07-20 14:37:09 -04:00
didericis	fa11ad9a4a	fix(ci,infra): repair post-merge publish + harden candidate handling ... Review follow-ups for the infra candidate-artifact flow: 1. publish-infra would fail on the first merge to main. The rootfs version now hashes the dropbear bytes, and build-infra sets BOT_BOTTLE_FC_DROPBEAR to the staged dropbear — but publish-infra (ubuntu-latest) set none, so _publish_ bundle re-derived the version with a "<missing>" dropbear and rejected the candidate as "does not match checkout". Download the same firecracker-inputs dropbear and export BOT_BOTTLE_FC_DROPBEAR in publish-infra (and add stage-firecracker-inputs to its needs, since it now consumes that artifact). 2. Guard stage-firecracker-inputs with the same fork-PR check as the other KVM-runner jobs, so a fork PR can't spin the privileged runner (it only copies a static binary, but keep the posture consistent; it gates the whole Firecracker chain via needs). 3. ensure_artifact_gz / _publish_bundle read version.txt before checking it exists — a bundle missing it raised a raw FileNotFoundError instead of the intended "bundle is incomplete" die. Guard the read (kept before the gz/sha completeness check so a wrong-version bundle still reports the version mismatch, not "incomplete"). 4. test_infra_artifact.setUp didn't isolate BOT_BOTTLE_INFRA_ARTIFACT_DIR, so an ambient candidate dir (the coverage job exports one) would send the registry-pull tests down the local-bundle path. Pin it off in setUp; the candidate-path cases set it explicitly. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01A9qa3xoavjQScufDfZaXKR	2026-07-20 02:07:08 -04:00
didericis-codex	4252ca3562	fix(ci): stage complete reproducible infra inputs	2026-07-19 22:25:45 +00:00
didericis-codex	701f5bf5e3	ci(infra): test and publish one candidate artifact	2026-07-19 22:22:42 +00:00
didericis-claude	9172bf3a42	fix(infra-artifact): include pyproject.toml in rootfs version digest ... Dockerfile.gateway COPYs pyproject.toml into /src and runs pip install /src, so it is a real input to the baked rootfs. A dependency-only change previously reused stale artifact versions, potentially booting a rootfs whose installed packages differed from the current checkout. Also adds _fake_repo fixture support and a regression test so this input can't silently drop out of the hash again. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>	2026-07-19 16:58:13 +00:00
didericis	d1aec706e3	fix(firecracker): fold guest init into the agent-rootfs cache key ... build_agent_rootfs_dir cached the built rootfs by Dockerfile content alone, but util.inject_guest_boot then writes util._GUEST_INIT into it. So a fix to the init — making /tmp world-writable (1777) so the agent can create scratch dirs / git worktrees there — did NOT bust the cache: the KVM runner kept reusing a stale agent-<dockerfilehash> rootfs built with the old init, and the sandbox-escape README-push test kept failing at `git init /tmp/...` with "Permission denied". Key the cache on Dockerfile content AND the injected init (_rootfs_digest), so an init change rebuilds. Self-busting: the new key yields a fresh cache dir, so no manual cache clear on the runner. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01A9qa3xoavjQScufDfZaXKR	2026-07-19 01:21:23 -04:00
didericis	4c01e31e96	fix(firecracker): version-aware infra-VM adoption + robust teardown ... The infra VM is a per-host singleton that outlives short-lived launchers, so `ensure_running` adopts it when its control plane is healthy. But it adopted ANY healthy VM regardless of the code that built it — so after an infra-code change the old VM kept being adopted and the new code never booted. The only way to dislodge it was an out-of-band `kill`, which then raced whatever launched next. On CI this meant every infra change needed a manual VM kill. Make adoption version-aware: * boot records the infra-artifact version it booted from in a `booted-version` marker beside the singleton; `stop` clears it. * `ensure_running` adopts only when the marker matches the current version (`_adoptable`); a missing/mismatched marker falls through to stop + reboot. So a stale VM is replaced automatically on the next launch — no manual kill, and it's concurrency-safe (the reboot happens under the singleton flock). Also harden teardown: the PID file drifts after crashes / out-of-band kills, so `stop` now also reaps any orphaned firecracker still bound to the infra config path (scoped to that path, so interactive-pool VMs are untouched) — otherwise a survivor holds the orchestrator TAP and the fresh boot dies with "tap … Resource busy". Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01A9qa3xoavjQScufDfZaXKR	2026-07-19 01:03:30 -04:00
didericis	6f885af4b4	fix(firecracker): make guest /tmp world-writable so agents can use it ... The rootless agent rootfs build can land /tmp as 0755/root-owned, so the agent (uid 1000 node) can't create scratch dirs there. The sandbox-escape suite's README-push test does `cd /tmp && git init sandbox-escape-repo` and died with "cannot mkdir sandbox-escape-repo: Permission denied" — before the git-gate gitleaks hook could run — so the test read it as a missing hook. On docker the agent inherits node:22-slim's 1777 /tmp, which is why only the Firecracker path was affected (and only now that the suite runs end-to-end). Set /tmp to 1777 in the guest PID-1 init, so every agent VM boots with a usable /tmp regardless of rootfs perm drift. Also update the infra-init unit test to assert the gateway launches via the `bot_bottle.gateway_init` module (matching the prior fix), not a file path. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01A9qa3xoavjQScufDfZaXKR	2026-07-19 00:49:00 -04:00
didericis	127ba49372	fix(firecracker): launch infra-VM gateway as a module, not /app/gateway_init.py ... The infra VM's init boots the orchestrator control plane AND the gateway data plane (egress/git-http/supervise). Since 5ad3449 moved the daemons into the installed `bot_bottle` package, there is no `/app/gateway_init.py` file — the gateway image's entrypoint is `python3 -m bot_bottle.gateway_init`. But the firecracker init still spawned the old file path, so the guest logged: python3: can't open file '/app/gateway_init.py': No such file or directory The control plane came up (it already used `python3 -m bot_bottle.orchestrator`) but the gateway never started, so mitmproxy never generated its CA and launch died with "gateway CA not available after 30s". Mirror the orchestrator line: run the gateway as `python3 -m bot_bottle.gateway_init` (bot_bottle resolves from the /app CWD, same as the control plane; egress-entrypoint.sh / egress_addon.py are present from the gateway base image). Third build/runtime bug from 5ad3449's package refactor that the Firecracker integration suite never exercised (the artifact publish was broken, so the job 404'd before boot). Changes the init, so the infra artifact version moves; the matching rootfs has been rebuilt and published. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01A9qa3xoavjQScufDfZaXKR	2026-07-19 00:35:19 -04:00
didericis-claude	e72ec71047	fix(tests): mock name_color_modal in test_cli_start_selector setUp ... On a self-hosted KVM runner the process has a real controlling terminal so name_color_modal successfully opens /dev/tty and enters a curses loop waiting for keyboard input, hanging the test indefinitely. Docker containers (ubuntu-latest runners) don't have a real /dev/tty, causing an OSError that triggers the existing fallback — this is why the hang was invisible in ubuntu-latest CI. Also add timeout=5 to _daemon_reachable() to match the same defensive fix already applied to docker_available() in tests/_docker.py. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>	2026-07-19 02:16:10 +00:00
didericis-claude	32e85de16f	fix(db): close SQLite connections explicitly to suppress ResourceWarning on Python 3.13 ... `sqlite3.Connection.__exit__` only commits/rolls back a transaction — it does not close the connection. Python 3.13 (the Nix env on the KVM runner) emits `ResourceWarning: unclosed database` for every connection GC'd without an explicit close, producing noisy output in the coverage job. Add `DbStore._connection()`, a `contextmanager` that calls `self._connect()`, wraps it in the existing transaction context manager, and closes the connection in a `finally` block. Change all `with self._connect() as conn:` call sites in `db_store.py`, `audit_store.py`, `queue_store.py`, and `orchestrator/registry.py` to `with self._connection() as conn:`. `_connect()` remains as the per-subclass hook (RegistryStore overrides it to set `busy_timeout`); `_connection()` delegates to `self._connect()` so the override is respected.	2026-07-19 02:16:10 +00:00
didericis	034f774529	feat(supervise): non-blocking MCP — pending carries proposal id + check-proposal poll tool ... Closes #412. The supervise MCP server blocked the agent's tool call polling for the operator's decision, and on timeout returned `status: pending` with no proposal id and no way to poll a specific proposal — so the only way to learn a late decision was to re-propose (a duplicate). - `handle_tools_call` pending timeout now returns the `proposal_id` and points the agent at `check-proposal`. - New `check-proposal` MCP tool: non-blocking status lookup by proposal id (pending | approved | modified | rejected | unknown). Reuses the queue's FileNotFoundError semantics; archives a decided proposal exactly like the synchronous path, so a pending proposal stays visible to the operator until it's both decided and polled. - `TOOL_CHECK_PROPOSAL` constant, re-exported from supervise; kept out of TOOLS since it never becomes a Proposal.tool. Enforcement is unchanged — the tools only propose policy; the egress proxy and git-gate still enforce — so returning early opens no hole. Follow-ups (git-gate reject-requeue, backpressure, notifications, web console) are in the PRD. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01YBCHap11yGAKuKfsehNPaD	2026-07-18 17:06:04 -04:00
didericis	5b359fe8d2	fix(git-gate): scan $new --not --all for every push, not $old..$new ... The pre-receive hook scanned existing-branch updates with the delta range $old..$new. On a rebase / non-fast-forward force-push onto an advanced main, $old is no longer an ancestor of $new, so $old..$new expands to all of main's new history — including the deliberate sandbox-escape gitleaks fixtures — and the push is rejected on commits that belong to main, not the branch. Unify the range on `$new --not --all` for every non-delete push (this is the deferred open question from PRD 0028, which already applied it to new refs for #106). It scans only the commits the push introduces and is security-equivalent: the bare repo's refs come only from trusted upstream mirror-fetch and gitleaks-gated pushes, so an excluded commit is already-upstream or already-scanned. It is also more correct for non-fast-forward pushes, where $old..$new can skip commits off the direct path. Fixes #421 Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01S1qRZTJC6qgBsUSjNrBdkX	2026-07-18 16:51:31 -04:00
didericis	015ff52eda	fix(cli): exempt backend command from DB migration gate ... `backend setup/status/teardown` manage host prerequisites only and never open the store, but the CLI dispatcher ran the schema-migration gate before every command. On a non-TTY runner the gate's `Migrate now? [y/N]` prompt reads EOF and refuses, so `backend status --backend=firecracker` exits 1 — breaking the Firecracker CI preflight on any host without a pre-migrated DB. Exempt `backend` from the gate; store-touching commands stay gated. Fixes #419 Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01S1qRZTJC6qgBsUSjNrBdkX	2026-07-18 15:27:29 -04:00
didericis-claude	410c19aaaf	fix(backend): fix pyright errors in lazy-load implementation ... - Rename _BACKENDS → _backends: pyright treats uppercase module-level names as constants and flags the reassignment in _get_backends() as reportConstantRedefinition; lowercase avoids this. - Add TYPE_CHECKING guard importing CommitCancelled/Freezer/get_freezer from .freeze: pyright cannot see module-level __getattr__ bindings, so reportUnsupportedDunderAll fired for those three __all__ entries; the guard makes them visible to the type checker without running at import time. - Update test_backend_selection.py to patch _backends (lowercase).	2026-07-18 05:14:27 -04:00
didericis-claude	f0ba399f17	fix(backend): silence pylint false positives from lazy-load pattern ... `undefined-all-variable` fires on CommitCancelled / Freezer / get_freezer in __all__ because pylint can't see module-level __getattr__ bindings; `global-statement` fires on the _BACKENDS singleton setter. Both are intentional patterns — add inline disables rather than suppress globally.	2026-07-18 05:14:27 -04:00
didericis-claude	8b442b8718	perf: lazy-load backend modules and consolidate docker subprocess helpers ... Importing backend.docker.util previously triggered eager loading of all three backend packages (~76 modules) because backend/__init__.py imported DockerBottleBackend, FirecrackerBottleBackend, and MacosContainerBottleBackend at module scope. This made the module prohibitively expensive to import from the orchestrator layer and elsewhere. The three backend imports are now deferred into _get_backends(), which loads all three on first call and caches the result in the module-level _BACKENDS variable (initially None). Module-level __getattr__ exposes backend classes and freeze symbols lazily for existing import/patch sites. backend/docker/util.py raw subprocess.run(["docker", ...]) calls are replaced with the shared run_docker primitive from docker_cmd, eliminating the duplication between the backend and orchestrator implementations. _silent_run() is removed; image_exists() is inlined directly onto run_docker. The commit_container test is updated to patch run_docker instead of subprocess.run.	2026-07-18 05:14:27 -04:00
didericis-claude	9a0dd821ef	refactor(gateway): invoke daemons via python3 -m instead of /app/ file copies ... supervise_server, git_http_backend, and gateway_init all have __main__ guards, so python3 -m bot_bottle.X replaces the individual COPY lines to /app/. egress_addon.py stays as a file copy because mitmdump -s requires a file path rather than a module reference.	2026-07-18 07:55:38 +00:00
didericis-claude	5ad3449e3b	refactor(gateway): replace flat-file import shims with installed package ... Install bot_bottle via pip in Dockerfile.gateway instead of COPYing individual .py files flat under /app/. This eliminates the try/except import shims in egress_addon_core, dlp_detectors, egress_addon, supervise, supervise_server, and git_http_backend that existed only to support the flat-bundle layout. Adds bot_bottle/constants.py as a single source of truth for IDENTITY_HEADER and GIT_GATE_TIMEOUT_SECS, removing the duplicated literal definitions in egress_addon.py, supervise_server.py, git_http_backend.py, and git_gate_render.py. Test files updated to match: test_supervise_server.py drops the sys.path.insert hack in favour of direct package imports; the egress_addon test shims no longer pre-populate sys.modules with a bare egress_addon_core alias.	2026-07-18 03:01:41 +00:00
didericis	ea1fbeeaa0	refactor(gateway): fail closed without an orchestrator URL; drop stale single-tenant comments ... Follow-ups from the #402 review of the single-tenant data-plane teardown. - egress_entrypoint.sh: refuse to launch mitmdump when BOT_BOTTLE_ORCHESTRATOR_URL is unset, so the fail-closed guarantee no longer rests solely on mitmproxy's errorcheck addon exiting on the addon's load-time raise. A misconfigured gateway can never come up as a bare TLS-bumping open proxy with no policy. - orchestrator/gateway.py: ensure_running() raises GatewayError on an empty orchestrator URL — a URL-less launch would only crash-loop the now-resolver-only daemons (egress raises, git-http exits 1, supervise exits 2). The env-injection branch is now unconditional. - Drop stale "single-tenant" / "reads routes.yaml" comments in gateway.py and egress_entrypoint.sh, and the /etc/egress/routes.yaml layout line in Dockerfile.gateway. - Tests: gateway fixtures supply an orchestrator URL; add a refuse-without-URL test and assert the URL env is injected. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UoEZHDjv84ChoZbozQERhJ	2026-07-17 22:34:28 -04:00
didericis	b601b663e2	refactor(gateway): remove the single-tenant data-plane paths (audit #400 finding 3) ... All three backends (docker, firecracker, macos-container) now launch through the consolidated orchestrator, and every production gateway sets BOT_BOTTLE_ORCHESTRATOR_URL — so the legacy single-tenant (`resolver is None`) branches in the shared gateway's data plane were unreachable dead code, a second security-relevant path to keep correct in parallel with the live one. Make the orchestrator resolver mandatory and delete the single-tenant paths from the three data-plane modules. egress_addon.py: drop the static routes file entirely — EGRESS_ROUTES, _reload, the SIGHUP handler, self.config, and the SUPERVISE_BOTTLE_SLUG env slug. The per-request /resolve is the only policy source; __init__ fail-closes if BOT_BOTTLE_ORCHESTRATOR_URL is unset. Introspection (`_egress.local/allowlist`) now reports the calling bottle's *resolved* routes. The block/redact log gates and _req_ctx redaction now read the per-flow config/env from the request-time stash, so they use each bottle's log level and token set (they silently used the empty static config before). Nothing sends `docker kill --signal HUP` to the gateway in the consolidated model (the egress applicators fail closed), so removing the SIGHUP reload is safe. git_http_backend.py: resolver mandatory; no flat-root fallback. main() refuses to start without an orchestrator URL; a request whose source resolves to no bottle 404s. supervise_server.py: resolver mandatory; every proposal is attributed to the source-IP-resolved bottle. Remove handle_list_egress_routes (the proxy-fetch introspection that only worked when the proxy carried one bottle's identity) — list-egress-routes is answered from the resolved policy. main() refuses to start without an orchestrator URL. Tests: a host-side fake resolver serves each test's Config through the real parse path (a small YAML-subset emitter round-trips route_to_yaml_dict); the response/websocket hooks stash it as request() would. Deletes the tests for the removed static-config, SIGHUP-reload, and single-tenant-passthrough paths; adds fail-closed-without-orchestrator coverage. Follow-up: gateway_init still forwards SIGHUP to the egress child (now dormant — no one sends it); the README still describes the docker backend's per-bottle topology. Both are outside the data-plane teardown. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UoEZHDjv84ChoZbozQERhJ	2026-07-17 22:34:28 -04:00
didericis	b1850be5d1	fix(git-gate): make the gateway access-hook executable regardless of copy transport ... Cloning/fetching from the git-gate on the Apple-container backend failed with "empty reply from server" (curl exit 52). Root cause: the git-http handler crashed on every upload-pack with PermissionError: [Errno 13] Permission denied: '/etc/git-gate/access-hook' The access-hook is exec'd directly, so it needs the x bit. prepare() stages it 0o700 and trusted the gateway copy to carry that mode. `docker cp` does; the Apple `container cp` (AppleGatewayTransport) does not, landing the hook 0o644 → EACCES. The unhandled exception killed the handler thread, closing the socket with no HTTP response — which the client sees as the opaque empty reply. - provision_git_gate now `chmod +x`es the access-hook on the gateway side after the copy, so it's executable under every transport (docker/apple/firecracker). - git-http handler wraps the access-hook subprocess.run: an OSError / SubprocessError (un-execable, timed out) now fails closed with a 503 instead of crashing the thread into an empty reply — a gate that can't run its hook should deny, visibly. - Updates the now-misleading "docker cp preserves source mode" comment in git_gate.prepare(). Regression tests: provisioning applies +x to the access-hook; the handler returns 503 (not an empty reply) when the hook can't be exec'd. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-07-17 21:49:30 -04:00
didericis	fd86e7fa99	fix(egress): redact the per-bottle token in the now-active multi-tenant response log ... Self-review of this PR: making `response()` run in the consolidated gateway also activates its `LOG_FULL` `_log_response` call there — previously unreachable, since the empty static config made `response()` return early. That logger redacted with `os.environ`, which in multi-tenant mode does NOT hold the bottle's per-request `/resolve` tokens (only the resolved `env` overlay does), so a non-token-shaped provisioned secret appearing in a response could be logged in the clear. Thread the resolved per-flow `env` into `_log_request` / `_log_response` so the LOG_FULL redaction scrubs the calling bottle's secrets. Adds a regression test (a non-token-shaped `/resolve` secret, absent from os.environ, must not appear in the response log) and updates the redaction-test helpers for the new arg. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UoEZHDjv84ChoZbozQERhJ	2026-07-17 17:14:52 -04:00
didericis	3bbd839917	fix(egress): scan response + websocket DLP against the resolved per-flow config ... In the consolidated (multi-tenant) gateway the addon's static `self.config` is empty — each request's real policy comes from the per-request `/resolve`. `response()` and `websocket_message()` still matched routes against that empty config, so inbound prompt-injection DLP and WebSocket credential/injection DLP silently skipped every scan (fail-open) whenever the gateway ran multi-tenant. This is backend-agnostic: the gateway image (and this addon) is shared by the Firecracker, macOS, and docker consolidated backends. Resolve the per-flow (config, slug, env) once in `request()`, stash it on `flow.metadata`, and have both hooks read it back — falling back to the static single-tenant values for a flow that never passed through `request()`. Reusing the request's one `/resolve` avoids a round-trip per response and per WebSocket frame. Adds multi-tenant regression tests for both hooks that fail against the old fall-open behaviour. Refs: audit issue #400 (finding #2) Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UoEZHDjv84ChoZbozQERhJ	2026-07-17 17:14:52 -04:00
didericis	5c526860bc	fix(test): resolve the remaining review findings on the control-plane auth test ... Addresses the 5 lower-priority findings left as follow-up in the earlier review, now that each has a concrete answer: - Add gateway_name: str = GATEWAY_NAME to OrchestratorService.__init__ (mirrors the existing orchestrator_name param) and thread it through _gateway(). Deletes the test's _IsolatedOrchestratorService subclass, which existed only to override a private method for this one kwarg — any caller needing gateway-name isolation can now use the public constructor. Backward compatible: every existing caller constructs OrchestratorService with keyword args and a sensible default is kept. - Give the test its own fixed image tags (bot-bottle-orchestrator:itest, bot-bottle-gateway:itest) instead of the production :latest ones. _running_image_is_current() keys gateway staleness off the image tag's ID, not per-instance identity, so rebuilding the shared :latest tag from whatever's on disk during a test run could make a real host's running production gateway look stale and get force-recreated. Fixed tags (not per-run-suffixed, so they don't accumulate) fully decouple the two. - setUp -> setUpClass/tearDownClass: all 5 tests are read-only checks against the same running control plane, so one shared container lifecycle replaces 5 (each of which paid its own container-start + image-build + health-poll cycle). Cuts the file's wall-clock roughly 4x (11.5s -> 2.9-4.3s) and, combined with the network-rm cleanup from the previous commit, means one cleanup instead of five. - Reuse OrchestratorClient (bot_bottle/orchestrator/client.py) instead of a hand-rolled urllib helper — the test now exercises the same request/response code path the real host CLI uses, rather than a private copy that could silently drift from it. - Add the chown workaround test_multitenant_isolation.py already needed for this exact bind-mount: the orchestrator container has no USER directive, so it writes the registry DB as root into the throwaway host_root; chown it back before tempdir cleanup so that doesn't raise PermissionError on native Linux Docker (no UID remap, unlike Docker Desktop's macOS VM). Verified: ran the suite twice in a row (idempotency — fixed image tags don't accumulate, 5/5 pass both times, 2.99-4.33s each), the real ~/.bot-bottle/control-plane-token is untouched, zero leaked networks or containers after either run, exactly 2 :itest images (not growing), the full orchestrator unit suite (93 tests) and the sibling docker gateway/broker integration tests still pass. pyright clean, pylint 10.00/10 on both changed files. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>	2026-07-17 17:00:31 -04:00
didericis	ca91fc4d91	fix(security): authenticate the orchestrator control plane (issue #400 Critical) ... The control plane had no caller authentication — dispatch() routed every request straight through, and the only guards anywhere were the (source_ip, identity_token) checks inside /resolve and /attribute, which authenticate the *bottle* a request is about, not the *caller*. On the consolidated backends the agent shares a network with the control-plane port (macOS: the infra container runs 8099 on the agent's --internal network; docker: the orchestrator container sits on the shared gateway network), so a prompt-injected agent could GET /bottles (enumerate siblings + policies), PUT /bottles/<id>/policy (rewrite its own egress allowlist to allow-all), POST /resolve (read the injected upstream credentials it's never meant to see), and POST /supervise/respond (self-approve its own proposals). Every README guarantee collapsed to the agent's discretion. Fix: a per-host control-plane secret required on every route but GET /health, compared with hmac.compare_digest. It is held only by the trusted callers and never handed to an agent: - minted + persisted 0600 at <root>/control-plane-token (paths.host_control_plane_token); - injected as $BOT_BOTTLE_CONTROL_PLANE_TOKEN into the orchestrator + gateway containers via bare `--env NAME` (value inherited from the launch process, so it never lands on argv or in `container/docker inspect`); - presented by the gateway's PolicyResolver (reads the env) on /resolve, and by the host CLI's OrchestratorClient (reads the host file) on every call. The agent container is never given the env var or the host file, so from a bottle every /bottles*, /resolve, /attribute, and /supervise/* call now returns 401 — closing the enumeration, allowlist-rewrite, credential-lift, and self-approval. The existing (source_ip, identity_token) checks stay as defense-in-depth. Enforced when configured: macOS + docker inject the secret (→ enforced). With no secret set the server runs open and warns loudly at startup — a fail-visible fallback for the unit suite and for Firecracker, whose port-scoped nft already blocks agents from 8099 (wiring the secret into its infra-VM init is a clean fast-follow, left out here to avoid churning the prebuilt-artifact hash). Verified end-to-end on real Apple Container: infra comes up healthy, the host CLI (with the secret) lists bottles while an unauthenticated GET /bottles gets 401, all five issue-#400 attacks from inside the agent get 401, and egress policy still works (200 allowed / 403 denied) — proving the gateway authenticates to /resolve with the secret. 1829 unit tests pass, pyright clean, pylint 9.91. Refs #400. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>	2026-07-17 05:15:16 -04:00
didericis	4a607ad098	refactor(macos): one infra container (control plane + gateway), fixes shared-DB races ... Adopts the firecracker infra-VM pattern for macOS: the orchestrator control plane and the gateway data plane now run in a SINGLE Apple container instead of two. Apple Containers are lightweight VMs with separate kernels, so the prior two-container design had both guests writing one bot-bottle.db over virtiofs, where fcntl locks are not coherent across kernels — concurrent writes (the orchestrator's registry vs the gateway supervise daemon's queue) could corrupt it. One container = one kernel = coherent locking. The DB moves onto a container-only Apple volume (bot-bottle-mac-db), never bind-mounted from the host, so no host process opens the live file either. The host CLI already reaches registry + supervise state over the control-plane HTTP surface (cli/supervise.py uses OrchestratorClient), exactly as firecracker's VM-only DB requires. Two simplifications fall out of the single container: - No DNS dance: the control plane and gateway daemons reach each other over 127.0.0.1, so the orchestrator-before-gateway ordering (a workaround for Apple having no container DNS) is gone, along with the moved-IP recreate logic it needed. - Net -243 lines. Mechanics: the infra container runs from the gateway image with the control-plane source bind-mounted read-only (like the docker orchestrator, so a code change needs no rebuild) and a small sh -c init that starts both processes (mirrors firecracker's _infra_init). Also implements the macOS backend's ensure_orchestrator() and adds it to discover_orchestrator_url, so operator tools (supervise) can bring up / find the control plane on demand — previously the macOS backend died with "no orchestrator control plane". Verified end-to-end on real Apple Container 1.0.0: the single infra container comes up healthy (one address for control plane + gateway), both processes run, the DB is written on the container-only volume, host-side supervise works over HTTP, and a registered agent gets 200 for an allowed host / 403 for a denied one. 1824 unit tests pass with `container` absent (CI parity), pyright clean, pylint 9.89. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>	2026-07-17 04:14:14 -04:00
didericis	e24b62b6b9	fix(macos): review fixes — token on plan, self-heal, symmetric digest, DHCP poll ... Addresses findings from a high-effort review of the PRD 0070 macOS backend. Correctness: - Stamp identity_token onto MacosContainerBottlePlan after registration. git's gitconfig extraHeader and the supervise MCP --header read getattr(plan,"identity_token","") at provision time, and both reach the gateway on NO_PROXY (bypassing the egress proxy that carries the token). The plan never carried it, so /resolve fail-closed and every git fetch/push and supervise call from a macOS bottle would have been denied. Registration precedes provision(), so — unlike the run-time env — the plan can carry it. - Self-heal the orchestrator: recreate when it is not (source-current AND answering /health), not on the source-hash label alone. A container running current code but with a wedged HTTP server was left alone and polled to death, failing every launch until manual deletion. - image_digest and container_image_digest now read the same descriptor.digest field; dropped image_digest's id/tag fallback that could yield a value the container side can't produce — a permanent mismatch would have recreated the shared gateway on every launch (severing every live bottle's egress, since the replacement gets a new DHCP address). - Poll for the agent's and gateway's DHCP address instead of a fatal read right after `container run` (there is no --ip; the address can lag start). Cleanup: - One _inspect_first + _descriptor_digest behind the four inspect readers. - Shared bind_mount_spec (util) and host_db_dir (paths) replace per-module copies; _GIT_HTTP_PORT now imports git_http_backend.DEFAULT_PORT. - Drop the dead _url cache / url property and the write-only agent_proxy_url. Deferred (noted on the PR, not fixed here): the gateway image rebuilding on every launch (needs source-hash-labeled build), SQLite shared across VM guests, and the sh -lc profile-override edge — each is design-level or behavior-risk beyond a review fix. Verified: real Apple Container bring-up is green and idempotent; 1826 unit tests pass with `container` absent (CI parity), pyright clean, pylint 9.86. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>	2026-07-17 03:26:33 -04:00
didericis	c69642e568	feat(macos): consolidated per-host gateway for the Apple backend (PRD 0070) ... Re-enables the macos-container backend on the shared per-host orchestrator + gateway, replacing the per-bottle companion container removed in #385. This is the last backend in PRD 0070's roadmap. Apple Container 1.0.0 forced three departures from the docker shape, each verified against the live CLI (findings recorded in the networking spike): - No `--ip`. The address is DHCP-assigned and knowable only once the container runs, so the order inverts: gateway up -> run agent -> read its address -> register. The identity token is minted by registration and therefore cannot be in the agent's run-time env; it rides the proxy URL applied at `container exec` time (bare `--env` names keep it off argv). - No container DNS. The gateway can only be handed the control plane's IP, so the orchestrator starts first and the gateway is pointed at its address. - No `network connect`. Networks are fixed at run time, so the shared host-only network is created up front; per-bottle networks would restart the gateway on every launch and defeat the consolidation. The agent runs with `--cap-drop CAP_NET_RAW`: Apple grants NET_RAW by default, which would let an agent forge a neighbour's source address on the shared segment. NET_ADMIN is already absent, so this closes the source-address half of PRD 0070's attribution invariant. Verified end-to-end on real Apple Container 1.0.0: both images build, the control plane comes up healthy, the gateway reaches it by IP, and a registered agent gets 200 for a host in its routes and 403 for one outside them. Bring-up is idempotent — a second launch does not churn the singletons. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-07-17 01:27:40 -04:00
didericis	dfc693e0b6	fix(firecracker): keep the snapshot partial private even if one was left behind ... Follow-up to the codex review on #398. os.open's mode arg only applies on creation, so a committed-rootfs.tar.partial left 0644 by an interrupted run would be opened/truncated (not re-moded) and stay world-readable for the whole SSH stream. Unlink any leftover and exclusively recreate it (O_EXCL|O_NOFOLLOW), then fchmod 0600 immediately so umask can't loosen it. Test pre-creates a 0644 partial and asserts the fd is 0600 mid-stream. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UoEZHDjv84ChoZbozQERhJ	2026-07-17 01:02:19 -04:00