pyright strict reportUnusedImport flagged BottleImages in
test_docker_launch_committed_image.py and test_macos_container_launch.py;
neither file references the type by name (they only use the returned
value's attributes).
`from __future__ import annotations` already defers all annotation
evaluation, so quoting `str | Path`, `BottleImages` inside the same
module was redundant and tripped pyright strict mode.
Addresses review comments 3098, 3099, 3100 on PR #336:
- Add BottleImages(agent, sidecar) dataclass to backend/__init__.py.
Docker/macOS backends use str image refs; smolmachines uses Path
artifacts. Replaces the singular `image` variable from the canonical
pattern in comment 3100.
- Replace _image_stale_checks/skip_stale with public prelaunch_checks().
CLI now calls backend.prelaunch_checks(plan) before backend.launch(plan);
if StaleImageError is raised and the operator confirms, launch proceeds
without re-checking. Removes the while-True/skip_stale retry loop.
- Add abstract _build_or_load_images(plan) -> BottleImages to
BottleBackend. launch() calls it then passes images to _launch_impl.
Each backend implements both methods.
- Fix comment 3098 (macos-container): _build_images is removed.
build_or_load_images() has separate fresh/cached code paths — the
cached path never calls a build helper.
- Update _start_bundle (smolmachines) to accept sidecar_artifact: Path
directly. Sidecar artifact resolution moves to _sidecar_from_path(),
called by build_or_load_images alongside _agent_from_path().
- image_cache: StaleImageError exception + check_stale/check_stale_path (raise instead of warn)
- BottleBackend.launch: template method (skip_stale flag) that calls _image_stale_checks then _launch_impl
- Each backend: _image_stale_checks delegates to a stale_checks() function in its launch module; _launch_impl replaces launch override
- macos_container: adds image_created_at to util, cached-image support in _build_images, stale_checks
- cli/start.py: catches StaleImageError, prompts interactively, retries with skip_stale=True; headless mode dies on it
Extract the shared CA cert polling loop into `backend/util.poll_ca_cert`
(firecracker and macos backends were duplicating deadline/sleep/raise logic).
Each caller now wraps a fetch lambda and converts TimeoutError to its own
error type. Also corrects the PRD port publication line from {port}:{port}
to {host_port}:8099.
- `_build_images()` now builds Dockerfile.gateway → Dockerfile.orchestrator
→ Dockerfile.infra in order; Dockerfile.infra starts FROM bot-bottle-gateway
so the base must exist on clean hosts.
- Publish mapping corrected from `self.port:self.port` to `self.port:DEFAULT_PORT`
(8099) — gateway_init hardcodes the orchestrator on port 8099 inside the
container, so the host-side published port must map to that fixed internal port.
- BOT_BOTTLE_ORCHESTRATOR_URL inside the container now always points to
127.0.0.1:8099, not self.port, since gateway daemons reach the orchestrator
over loopback at the fixed internal port.
- Update test_ensure_running_builds_both_images → _all_images for the new
three-step build sequence; add test_publish_maps_host_port_to_fixed_internal_port
to lock in the port-mapping fix.
Addresses the P1 findings from the didericis-codex review on PR #432.
- Remove unused INFRA_IMAGE import from test_orchestrator_lifecycle
- Update integration test to use new single-container OrchestratorService
API (infra_name/image replaces orchestrator_name/gateway_name/gateway_image)
- Move type: ignore to the lambda line in gateway_init SIGHUP handler
- Break two long lines in test_orchestrator_lifecycle
- Extract _sigkill_all() to cut nesting depth below the 5-block limit
- Add pylint: disable=consider-using-with on Popen (process must outlive caller)
- Break long SIGHUP signal line to stay within 100 chars
Collapses the two-container Docker model (gateway + orchestrator) into one
bot-bottle-infra container, matching the macOS and Firecracker backends.
- Dockerfile.infra: now a shared gateway+orchestrator base (COPY bot_bottle
from orchestrator build, no CMD override)
- Dockerfile.infra.fc: new Firecracker-specific layer (buildah/crun/netavark)
- gateway_init: adds orchestrator daemon with _OPT_IN_DAEMONS gating so it
only starts when BOT_BOTTLE_GATEWAY_DAEMONS explicitly includes it
- orchestrator/lifecycle: OrchestratorService manages one infra container;
builds orchestrator (intermediate) then infra; live source bind-mounted at
/bot-bottle-src with PYTHONPATH so the subprocess uses the checkout
- backend/consolidated_util: extracts provision_bottle + teardown_consolidated
shared across all three backends; removes duplication in docker/fc/macos
consolidated_launch modules
- firecracker/infra_vm: builds four images (orchestrator→gateway→infra→infra.fc)
- All unit tests updated and passing (1878 tests)
- PRD status: Draft → Active
Switch orchestrator_config from a generic key/text store to a typed
single-row table with a dedicated teardown_timeout_seconds REAL column.
The CHECK (id = 1) constraint enforces at most one config row.
Remove TEARDOWN_TIMEOUT_KEY constant; update OrchestratorConfigStore to
expose get/set/delete_teardown_timeout_seconds() instead of the generic
get/set/delete(key). Resolving the DB value in resolve_teardown_timeout()
no longer needs a float() conversion since the column is already REAL.
If resolve_teardown_timeout() raised after launch_consolidated()/register_agent()
returned, the teardown callback was never registered, leaving the bottle
registered with no cleanup path. Resolve the timeout into a local variable
before the registration call so any failure aborts before the bottle exists.
Add tests covering OrchestratorConfigStore, resolve_teardown_timeout priority
ordering (env > db > default), and the source-order invariant that the resolver
runs before registration in all three backends.
Resolves via ENV VAR -> orchestrator DB config -> default (30 s, up from 5 s):
BOT_BOTTLE_ORCHESTRATOR_TEARDOWN_TIMEOUT_SECONDS
teardown_timeout_seconds key in new orchestrator_config table (bot-bottle.db)
New OrchestratorConfigStore (same DbStore/TableMigrations pattern as the
registry) stores the DB-level setting. resolve_teardown_timeout() implements
the priority chain and is called at stack.callback registration time in all
three backends (macos_container, docker, firecracker).
A failed container list or a partial per-container inspect were
previously indistinguishable from a legitimately empty/partial live
set, so reconciliation could silently unregister healthy bottles.
- enumerate_active() now raises EnumerationError instead of returning []
when `container list` fails
- Add inspect_container_network_ip() to util, which returns None on
inspect failure (vs "" for "no DHCP address yet"), so live_source_ips
can tell the two apart
- live_source_ips() raises EnumerationError on either failure mode;
register_agent() catches it and skips reconciliation, same as
OrchestratorClientError
- Update tests: rename test_empty_when_the_cli_fails to
test_raises_when_the_cli_fails, update patching to the new function,
add coverage for list failure, per-container inspect failure, and the
launch-not-blocked path
`/resolve` fail-closes on a missing/ambiguous registry row *and* on a
request whose identity token doesn't match. The message named only the
first, so a bottle that was registered correctly but sent no token read as
"not registered" and sent the reader looking for a deregistered bottle.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
An unattributed bottle, an unreachable orchestrator, and an unparseable
policy all become a deny-all Config, and a deny-all is indistinguishable
from "policy loaded, host not allowed" at the decision point — both are just
"no matching route". So every one of them reported `host X is not in the
bottle's egress.routes allowlist`, which reads as a config problem and sends
the operator hunting for a route that was never missing. Diagnosing a
bricked registration cost hours for exactly this reason.
Carry the structural reason on Config and prefer it in decide(). A genuine
allowlist miss — a policy that loaded and simply lacks the host — keeps the
original wording, so the message now tells the two cases apart.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
A registry row only ever left the registry two ways: an explicit
teardown_bottle (the launcher's cleanup callback) or the same-IP supersede
sweep in register(). Neither runs when the launching CLI dies hard, so the
row outlives its container.
That orphan is not inert. Source IPs are recycled by the backend's DHCP and
by_source_ip fail-closes on ambiguity, so a leftover row at a reused address
resolves no policy at all for the next bottle that lands there — and a
bottle with no policy denies every host, which surfaces to the agent as
"host X is not in the allowlist" for hosts that were never the problem.
Add reap_absent/reconcile and call it from the macOS launch path before
registering, so each launch self-heals the registry. Restores the invariant
the data plane needs: at most one active row per live address, and none for
a dead one. The second half matters as much as the first — when several rows
claim a *live* address the newest wins and the rest are swept, otherwise a
recycled address stays ambiguous, which is exactly the bricked state.
The host supplies the live set because the orchestrator runs inside the
infra container and cannot see the backend. A grace window exempts rows
younger than it, so reconciliation cannot race a bottle still coming up, and
a reconcile failure is logged rather than blocking an otherwise-fine launch.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The shared gateway's address is DHCP-assigned and changes whenever the infra
container is recreated — a source-hash bump, an image upgrade, a crash. Every
agent-facing URL embedded that address, and the proxy URL reaches the agent
as process environment at `container exec` time. A running process's environ
cannot be rewritten from outside, so a moved gateway stranded every running
bottle permanently: not degraded, unreachable, until relaunched and its agent
session thrown away.
Give the agent a stable name instead. `GATEWAY_HOSTNAME` replaces the address
in the egress proxy URL, NO_PROXY, git-http, and supervise URLs, and resolves
through the bottle's own /etc/hosts. Unlike environ that is a file, so it can
be rewritten inside a container that is already running — which is the whole
point: a gateway that returns at a new address is picked up by live bottles.
Launch writes the entry before anything execs (every agent URL names the
gateway, so it must resolve for the first connection), and re-points every
running bottle once the gateway is up, so one stranded by an earlier restart
re-attaches instead of needing a relaunch.
The write needs root and the agent runs as `node`: the host can repoint a
bottle's gateway name, the agent cannot repoint its own. Keep that asymmetry.
Apple Container 1.0 has no container-name DNS on a user network and
`container run` has no --add-host, so the entry is written by exec after
start rather than declared at run.
Does not address the other half of #443: per-bottle egress auth tokens are
held in memory by the orchestrator and are still lost across a restart, so a
re-attached bottle resolves its policy but not its injected credentials.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
`container exec --env` does not override the run-time environment on Apple
Container — it appends. The launch path baked a token-less `*_PROXY` into
`container run` and relied on the exec-time, token-bearing value superseding
it, so the agent's `environ` ended up with two `HTTPS_PROXY` entries,
token-less first.
Which entry a runtime reads is then luck. Node reads the last, so Claude
bottles picked up the token and worked. Rust's `std::env::var` reads the
first, so Codex proxied with no identity token at all; `/resolve` requires a
matching (source_ip, identity_token) pair and fail-closes, so every request
from a Codex bottle was denied — its model calls, its `wss://` reconnects,
and its MCP servers alike. The registry row was correct the whole time,
which is what made this read as a registration bug.
Drop the run-time proxy vars entirely. A token-less proxy URL has no
legitimate consumer: the init process is `sleep` and everything that
egresses arrives by exec. Its only benefit was a tidy 403 for unattributed
callers, which is not worth silently dropping attribution for — and a
process that egresses before the exec-time env still fails closed, since the
agent network is host-only and the gateway is the only route off it.
Also corrects the two comments that asserted the false "exec --env wins"
invariant, so the next reader doesn't rebuild the same assumption.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Pyright reportUnusedImport fired because the bare import is not used
within _common.py itself. The X as X idiom signals an intentional
public re-export without requiring __all__.
_auto_select_backend gains a prompt parameter (default True). When
prompt=False the docker-fallback [i/d/q] menu is skipped and the call
dies immediately with an actionable message ("set
BOT_BOTTLE_BACKEND=docker or install a VM backend"), preventing hangs
in CI, webhook dispatch, and orchestrator launches.
prepare_with_preflight passes prompt=not spec.headless so the headless
start path can never block waiting for TTY input it cannot receive.
Remove the private _read_tty_line duplicate from backend/__init__.py and
the local definition from cli/_common.py. Both now import from the
shared bot_bottle.util module.
- Remove hardcoded --backend=macos-container flag reference in
firecracker/util.py require_firecracker() error message
- Remove --backend flag from cli.py start; backend selection now
driven exclusively by BOT_BOTTLE_BACKEND env var or auto-selection
- Skip unavailable backends in cli.py cleanup (fixes crash on Linux
when macos-container.prepare_cleanup calls require_container())
- Add two-tier auto-selection: VM backend first (macos-container on
macOS, firecracker on Linux+KVM); fall back to docker with a
security warning and interactive i/d/q prompt; exit if docker
also unavailable and print VM install instructions
Closes#344
The macOS infra container's init script still spawned the supervisor as
`python3 /app/gateway_init.py`. 5ad3449 moved bot_bottle from flat files
under /app into a pip-installed package, so that file no longer exists:
the supervisor never started, mitmdump never generated its CA, and the
launch failed downstream with a misleading "gateway CA not available"
error pointing at TLS rather than at the supervisor.
This is the same defect 127ba49 fixed for the firecracker backend. It
survived on macOS because CI has no Apple Container runner — the KVM
runner added in c193b04 covers firecracker, and ubuntu-latest covers
docker, but nothing exercises macos_container.
test_macos_infra asserted the substring "gateway_init.py", which the
broken path satisfied; it now asserts the module form, matching the
assertion 127ba49 already made in test_firecracker_infra_vm.
Verified end to end on the macos-container backend: image builds, CA is
generated, agent registers, and a headless bottle runs a prompt.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Review follow-ups for the infra candidate-artifact flow:
1. publish-infra would fail on the first merge to main. The rootfs version now
hashes the dropbear bytes, and build-infra sets BOT_BOTTLE_FC_DROPBEAR to the
staged dropbear — but publish-infra (ubuntu-latest) set none, so _publish_
bundle re-derived the version with a "<missing>" dropbear and rejected the
candidate as "does not match checkout". Download the same firecracker-inputs
dropbear and export BOT_BOTTLE_FC_DROPBEAR in publish-infra (and add
stage-firecracker-inputs to its needs, since it now consumes that artifact).
2. Guard stage-firecracker-inputs with the same fork-PR check as the other
KVM-runner jobs, so a fork PR can't spin the privileged runner (it only
copies a static binary, but keep the posture consistent; it gates the whole
Firecracker chain via needs).
3. ensure_artifact_gz / _publish_bundle read version.txt before checking it
exists — a bundle missing it raised a raw FileNotFoundError instead of the
intended "bundle is incomplete" die. Guard the read (kept before the gz/sha
completeness check so a wrong-version bundle still reports the version
mismatch, not "incomplete").
4. test_infra_artifact.setUp didn't isolate BOT_BOTTLE_INFRA_ARTIFACT_DIR, so
an ambient candidate dir (the coverage job exports one) would send the
registry-pull tests down the local-bundle path. Pin it off in setUp; the
candidate-path cases set it explicitly.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01A9qa3xoavjQScufDfZaXKR
Dockerfile.gateway COPYs pyproject.toml into /src and runs pip install
/src, so it is a real input to the baked rootfs. A dependency-only change
previously reused stale artifact versions, potentially booting a rootfs
whose installed packages differed from the current checkout.
Also adds _fake_repo fixture support and a regression test so this input
can't silently drop out of the hash again.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
build_agent_rootfs_dir cached the built rootfs by Dockerfile content alone,
but util.inject_guest_boot then writes util._GUEST_INIT into it. So a fix to
the init — making /tmp world-writable (1777) so the agent can create scratch
dirs / git worktrees there — did NOT bust the cache: the KVM runner kept
reusing a stale agent-<dockerfilehash> rootfs built with the old init, and the
sandbox-escape README-push test kept failing at `git init /tmp/...` with
"Permission denied".
Key the cache on Dockerfile content AND the injected init (_rootfs_digest), so
an init change rebuilds. Self-busting: the new key yields a fresh cache dir, so
no manual cache clear on the runner.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01A9qa3xoavjQScufDfZaXKR
The infra VM is a per-host singleton that outlives short-lived launchers, so
`ensure_running` adopts it when its control plane is healthy. But it adopted
ANY healthy VM regardless of the code that built it — so after an infra-code
change the old VM kept being adopted and the new code never booted. The only
way to dislodge it was an out-of-band `kill`, which then raced whatever
launched next. On CI this meant every infra change needed a manual VM kill.
Make adoption version-aware:
* boot records the infra-artifact version it booted from in a `booted-version`
marker beside the singleton; `stop` clears it.
* `ensure_running` adopts only when the marker matches the current version
(`_adoptable`); a missing/mismatched marker falls through to stop + reboot.
So a stale VM is replaced automatically on the next launch — no manual kill,
and it's concurrency-safe (the reboot happens under the singleton flock).
Also harden teardown: the PID file drifts after crashes / out-of-band kills,
so `stop` now also reaps any orphaned firecracker still bound to the infra
config path (scoped to that path, so interactive-pool VMs are untouched) —
otherwise a survivor holds the orchestrator TAP and the fresh boot dies with
"tap … Resource busy".
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01A9qa3xoavjQScufDfZaXKR
The rootless agent rootfs build can land /tmp as 0755/root-owned, so the
agent (uid 1000 node) can't create scratch dirs there. The sandbox-escape
suite's README-push test does `cd /tmp && git init sandbox-escape-repo` and
died with "cannot mkdir sandbox-escape-repo: Permission denied" — before the
git-gate gitleaks hook could run — so the test read it as a missing hook.
On docker the agent inherits node:22-slim's 1777 /tmp, which is why only the
Firecracker path was affected (and only now that the suite runs end-to-end).
Set /tmp to 1777 in the guest PID-1 init, so every agent VM boots with a
usable /tmp regardless of rootfs perm drift.
Also update the infra-init unit test to assert the gateway launches via the
`bot_bottle.gateway_init` module (matching the prior fix), not a file path.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01A9qa3xoavjQScufDfZaXKR
The infra VM's init boots the orchestrator control plane AND the gateway data
plane (egress/git-http/supervise). Since 5ad3449 moved the daemons into the
installed `bot_bottle` package, there is no `/app/gateway_init.py` file — the
gateway image's entrypoint is `python3 -m bot_bottle.gateway_init`. But the
firecracker init still spawned the old file path, so the guest logged:
python3: can't open file '/app/gateway_init.py': No such file or directory
The control plane came up (it already used `python3 -m bot_bottle.orchestrator`)
but the gateway never started, so mitmproxy never generated its CA and launch
died with "gateway CA not available after 30s". Mirror the orchestrator line:
run the gateway as `python3 -m bot_bottle.gateway_init` (bot_bottle resolves
from the /app CWD, same as the control plane; egress-entrypoint.sh /
egress_addon.py are present from the gateway base image).
Third build/runtime bug from 5ad3449's package refactor that the Firecracker
integration suite never exercised (the artifact publish was broken, so the
job 404'd before boot). Changes the init, so the infra artifact version moves;
the matching rootfs has been rebuilt and published.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01A9qa3xoavjQScufDfZaXKR
On a self-hosted KVM runner the process has a real controlling terminal
so name_color_modal successfully opens /dev/tty and enters a curses
loop waiting for keyboard input, hanging the test indefinitely.
Docker containers (ubuntu-latest runners) don't have a real /dev/tty,
causing an OSError that triggers the existing fallback — this is why
the hang was invisible in ubuntu-latest CI.
Also add timeout=5 to _daemon_reachable() to match the same defensive
fix already applied to docker_available() in tests/_docker.py.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
`sqlite3.Connection.__exit__` only commits/rolls back a transaction — it
does not close the connection. Python 3.13 (the Nix env on the KVM
runner) emits `ResourceWarning: unclosed database` for every connection
GC'd without an explicit close, producing noisy output in the coverage job.
Add `DbStore._connection()`, a `contextmanager` that calls `self._connect()`,
wraps it in the existing transaction context manager, and closes the
connection in a `finally` block. Change all `with self._connect() as conn:`
call sites in `db_store.py`, `audit_store.py`, `queue_store.py`, and
`orchestrator/registry.py` to `with self._connection() as conn:`.
`_connect()` remains as the per-subclass hook (RegistryStore overrides
it to set `busy_timeout`); `_connection()` delegates to `self._connect()` so
the override is respected.
Closes#412.
The supervise MCP server blocked the agent's tool call polling for the
operator's decision, and on timeout returned `status: pending` with no
proposal id and no way to poll a specific proposal — so the only way to
learn a late decision was to re-propose (a duplicate).
- `handle_tools_call` pending timeout now returns the `proposal_id` and
points the agent at `check-proposal`.
- New `check-proposal` MCP tool: non-blocking status lookup by proposal id
(pending | approved | modified | rejected | unknown). Reuses the queue's
FileNotFoundError semantics; archives a decided proposal exactly like the
synchronous path, so a pending proposal stays visible to the operator
until it's both decided and polled.
- `TOOL_CHECK_PROPOSAL` constant, re-exported from supervise; kept out of
TOOLS since it never becomes a Proposal.tool.
Enforcement is unchanged — the tools only propose policy; the egress proxy
and git-gate still enforce — so returning early opens no hole. Follow-ups
(git-gate reject-requeue, backpressure, notifications, web console) are in
the PRD.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YBCHap11yGAKuKfsehNPaD
The pre-receive hook scanned existing-branch updates with the delta range
$old..$new. On a rebase / non-fast-forward force-push onto an advanced main,
$old is no longer an ancestor of $new, so $old..$new expands to all of main's
new history — including the deliberate sandbox-escape gitleaks fixtures — and
the push is rejected on commits that belong to main, not the branch.
Unify the range on `$new --not --all` for every non-delete push (this is the
deferred open question from PRD 0028, which already applied it to new refs
for #106). It scans only the commits the push introduces and is
security-equivalent: the bare repo's refs come only from trusted upstream
mirror-fetch and gitleaks-gated pushes, so an excluded commit is
already-upstream or already-scanned. It is also more correct for
non-fast-forward pushes, where $old..$new can skip commits off the direct path.
Fixes#421
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01S1qRZTJC6qgBsUSjNrBdkX
`backend setup/status/teardown` manage host prerequisites only and never
open the store, but the CLI dispatcher ran the schema-migration gate before
every command. On a non-TTY runner the gate's `Migrate now? [y/N]` prompt
reads EOF and refuses, so `backend status --backend=firecracker` exits 1 —
breaking the Firecracker CI preflight on any host without a pre-migrated DB.
Exempt `backend` from the gate; store-touching commands stay gated.
Fixes#419
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01S1qRZTJC6qgBsUSjNrBdkX
- Rename _BACKENDS → _backends: pyright treats uppercase module-level
names as constants and flags the reassignment in _get_backends() as
reportConstantRedefinition; lowercase avoids this.
- Add TYPE_CHECKING guard importing CommitCancelled/Freezer/get_freezer
from .freeze: pyright cannot see module-level __getattr__ bindings, so
reportUnsupportedDunderAll fired for those three __all__ entries; the
guard makes them visible to the type checker without running at import
time.
- Update test_backend_selection.py to patch _backends (lowercase).
`undefined-all-variable` fires on CommitCancelled / Freezer / get_freezer
in __all__ because pylint can't see module-level __getattr__ bindings;
`global-statement` fires on the _BACKENDS singleton setter. Both are
intentional patterns — add inline disables rather than suppress globally.
Importing backend.docker.util previously triggered eager loading of all
three backend packages (~76 modules) because backend/__init__.py imported
DockerBottleBackend, FirecrackerBottleBackend, and MacosContainerBottleBackend
at module scope. This made the module prohibitively expensive to import
from the orchestrator layer and elsewhere.
The three backend imports are now deferred into _get_backends(), which
loads all three on first call and caches the result in the module-level
_BACKENDS variable (initially None). Module-level __getattr__ exposes
backend classes and freeze symbols lazily for existing import/patch sites.
backend/docker/util.py raw subprocess.run(["docker", ...]) calls are
replaced with the shared run_docker primitive from docker_cmd, eliminating
the duplication between the backend and orchestrator implementations.
_silent_run() is removed; image_exists() is inlined directly onto
run_docker. The commit_container test is updated to patch run_docker
instead of subprocess.run.
supervise_server, git_http_backend, and gateway_init all have __main__
guards, so python3 -m bot_bottle.X replaces the individual COPY lines
to /app/. egress_addon.py stays as a file copy because mitmdump -s
requires a file path rather than a module reference.
Install bot_bottle via pip in Dockerfile.gateway instead of COPYing
individual .py files flat under /app/. This eliminates the try/except
import shims in egress_addon_core, dlp_detectors, egress_addon,
supervise, supervise_server, and git_http_backend that existed only
to support the flat-bundle layout.
Adds bot_bottle/constants.py as a single source of truth for
IDENTITY_HEADER and GIT_GATE_TIMEOUT_SECS, removing the duplicated
literal definitions in egress_addon.py, supervise_server.py,
git_http_backend.py, and git_gate_render.py.
Test files updated to match: test_supervise_server.py drops the
sys.path.insert hack in favour of direct package imports; the
egress_addon test shims no longer pre-populate sys.modules with a
bare egress_addon_core alias.
Follow-ups from the #402 review of the single-tenant data-plane teardown.
- egress_entrypoint.sh: refuse to launch mitmdump when
BOT_BOTTLE_ORCHESTRATOR_URL is unset, so the fail-closed guarantee no
longer rests solely on mitmproxy's errorcheck addon exiting on the
addon's load-time raise. A misconfigured gateway can never come up as
a bare TLS-bumping open proxy with no policy.
- orchestrator/gateway.py: ensure_running() raises GatewayError on an
empty orchestrator URL — a URL-less launch would only crash-loop the
now-resolver-only daemons (egress raises, git-http exits 1, supervise
exits 2). The env-injection branch is now unconditional.
- Drop stale "single-tenant" / "reads routes.yaml" comments in
gateway.py and egress_entrypoint.sh, and the /etc/egress/routes.yaml
layout line in Dockerfile.gateway.
- Tests: gateway fixtures supply an orchestrator URL; add a
refuse-without-URL test and assert the URL env is injected.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UoEZHDjv84ChoZbozQERhJ
All three backends (docker, firecracker, macos-container) now launch through
the consolidated orchestrator, and every production gateway sets
BOT_BOTTLE_ORCHESTRATOR_URL — so the legacy single-tenant (`resolver is None`)
branches in the shared gateway's data plane were unreachable dead code, a second
security-relevant path to keep correct in parallel with the live one. Make the
orchestrator resolver mandatory and delete the single-tenant paths from the
three data-plane modules.
egress_addon.py: drop the static routes file entirely — EGRESS_ROUTES, _reload,
the SIGHUP handler, self.config, and the SUPERVISE_BOTTLE_SLUG env slug. The
per-request /resolve is the only policy source; __init__ fail-closes if
BOT_BOTTLE_ORCHESTRATOR_URL is unset. Introspection (`_egress.local/allowlist`)
now reports the calling bottle's *resolved* routes. The block/redact log gates
and _req_ctx redaction now read the per-flow config/env from the request-time
stash, so they use each bottle's log level and token set (they silently used the
empty static config before). Nothing sends `docker kill --signal HUP` to the
gateway in the consolidated model (the egress applicators fail closed), so
removing the SIGHUP reload is safe.
git_http_backend.py: resolver mandatory; no flat-root fallback. main() refuses
to start without an orchestrator URL; a request whose source resolves to no
bottle 404s.
supervise_server.py: resolver mandatory; every proposal is attributed to the
source-IP-resolved bottle. Remove handle_list_egress_routes (the proxy-fetch
introspection that only worked when the proxy carried one bottle's identity) —
list-egress-routes is answered from the resolved policy. main() refuses to start
without an orchestrator URL.
Tests: a host-side fake resolver serves each test's Config through the real
parse path (a small YAML-subset emitter round-trips route_to_yaml_dict); the
response/websocket hooks stash it as request() would. Deletes the tests for the
removed static-config, SIGHUP-reload, and single-tenant-passthrough paths; adds
fail-closed-without-orchestrator coverage.
Follow-up: gateway_init still forwards SIGHUP to the egress child (now dormant —
no one sends it); the README still describes the docker backend's per-bottle
topology. Both are outside the data-plane teardown.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UoEZHDjv84ChoZbozQERhJ
Cloning/fetching from the git-gate on the Apple-container backend failed with
"empty reply from server" (curl exit 52). Root cause: the git-http handler
crashed on every upload-pack with
PermissionError: [Errno 13] Permission denied: '/etc/git-gate/access-hook'
The access-hook is exec'd directly, so it needs the x bit. prepare() stages it
0o700 and trusted the gateway copy to carry that mode. `docker cp` does; the
Apple `container cp` (AppleGatewayTransport) does not, landing the hook 0o644 →
EACCES. The unhandled exception killed the handler thread, closing the socket
with no HTTP response — which the client sees as the opaque empty reply.
- provision_git_gate now `chmod +x`es the access-hook on the gateway side after
the copy, so it's executable under every transport (docker/apple/firecracker).
- git-http handler wraps the access-hook subprocess.run: an OSError /
SubprocessError (un-execable, timed out) now fails closed with a 503 instead
of crashing the thread into an empty reply — a gate that can't run its hook
should deny, visibly.
- Updates the now-misleading "docker cp preserves source mode" comment in
git_gate.prepare().
Regression tests: provisioning applies +x to the access-hook; the handler
returns 503 (not an empty reply) when the hook can't be exec'd.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Self-review of this PR: making `response()` run in the consolidated gateway
also activates its `LOG_FULL` `_log_response` call there — previously
unreachable, since the empty static config made `response()` return early. That
logger redacted with `os.environ`, which in multi-tenant mode does NOT hold the
bottle's per-request `/resolve` tokens (only the resolved `env` overlay does),
so a non-token-shaped provisioned secret appearing in a response could be logged
in the clear.
Thread the resolved per-flow `env` into `_log_request` / `_log_response` so the
LOG_FULL redaction scrubs the calling bottle's secrets. Adds a regression test
(a non-token-shaped `/resolve` secret, absent from os.environ, must not appear
in the response log) and updates the redaction-test helpers for the new arg.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UoEZHDjv84ChoZbozQERhJ
In the consolidated (multi-tenant) gateway the addon's static `self.config`
is empty — each request's real policy comes from the per-request `/resolve`.
`response()` and `websocket_message()` still matched routes against that empty
config, so inbound prompt-injection DLP and WebSocket credential/injection DLP
silently skipped every scan (fail-open) whenever the gateway ran multi-tenant.
This is backend-agnostic: the gateway image (and this addon) is shared by the
Firecracker, macOS, and docker consolidated backends.
Resolve the per-flow (config, slug, env) once in `request()`, stash it on
`flow.metadata`, and have both hooks read it back — falling back to the static
single-tenant values for a flow that never passed through `request()`. Reusing
the request's one `/resolve` avoids a round-trip per response and per WebSocket
frame.
Adds multi-tenant regression tests for both hooks that fail against the old
fall-open behaviour.
Refs: audit issue #400 (finding #2)
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UoEZHDjv84ChoZbozQERhJ
Addresses the 5 lower-priority findings left as follow-up in the earlier
review, now that each has a concrete answer:
- Add gateway_name: str = GATEWAY_NAME to OrchestratorService.__init__
(mirrors the existing orchestrator_name param) and thread it through
_gateway(). Deletes the test's _IsolatedOrchestratorService subclass,
which existed only to override a private method for this one kwarg —
any caller needing gateway-name isolation can now use the public
constructor. Backward compatible: every existing caller constructs
OrchestratorService with keyword args and a sensible default is kept.
- Give the test its own fixed image tags (bot-bottle-orchestrator:itest,
bot-bottle-gateway:itest) instead of the production :latest ones.
_running_image_is_current() keys gateway staleness off the image tag's
ID, not per-instance identity, so rebuilding the shared :latest tag from
whatever's on disk during a test run could make a real host's running
production gateway look stale and get force-recreated. Fixed tags (not
per-run-suffixed, so they don't accumulate) fully decouple the two.
- setUp -> setUpClass/tearDownClass: all 5 tests are read-only checks
against the same running control plane, so one shared container
lifecycle replaces 5 (each of which paid its own container-start +
image-build + health-poll cycle). Cuts the file's wall-clock roughly
4x (11.5s -> 2.9-4.3s) and, combined with the network-rm cleanup from
the previous commit, means one cleanup instead of five.
- Reuse OrchestratorClient (bot_bottle/orchestrator/client.py) instead of
a hand-rolled urllib helper — the test now exercises the same
request/response code path the real host CLI uses, rather than a
private copy that could silently drift from it.
- Add the chown workaround test_multitenant_isolation.py already needed
for this exact bind-mount: the orchestrator container has no USER
directive, so it writes the registry DB as root into the throwaway
host_root; chown it back before tempdir cleanup so that doesn't raise
PermissionError on native Linux Docker (no UID remap, unlike Docker
Desktop's macOS VM).
Verified: ran the suite twice in a row (idempotency — fixed image tags
don't accumulate, 5/5 pass both times, 2.99-4.33s each), the real
~/.bot-bottle/control-plane-token is untouched, zero leaked networks or
containers after either run, exactly 2 :itest images (not growing), the
full orchestrator unit suite (93 tests) and the sibling docker
gateway/broker integration tests still pass. pyright clean, pylint
10.00/10 on both changed files.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
The control plane had no caller authentication — dispatch() routed every
request straight through, and the only guards anywhere were the
(source_ip, identity_token) checks inside /resolve and /attribute, which
authenticate the *bottle* a request is about, not the *caller*. On the
consolidated backends the agent shares a network with the control-plane port
(macOS: the infra container runs 8099 on the agent's --internal network;
docker: the orchestrator container sits on the shared gateway network), so a
prompt-injected agent could GET /bottles (enumerate siblings + policies),
PUT /bottles/<id>/policy (rewrite its own egress allowlist to allow-all),
POST /resolve (read the injected upstream credentials it's never meant to
see), and POST /supervise/respond (self-approve its own proposals). Every
README guarantee collapsed to the agent's discretion.
Fix: a per-host control-plane secret required on every route but GET /health,
compared with hmac.compare_digest. It is held only by the trusted callers and
never handed to an agent:
- minted + persisted 0600 at <root>/control-plane-token (paths.host_control_plane_token);
- injected as $BOT_BOTTLE_CONTROL_PLANE_TOKEN into the orchestrator + gateway
containers via bare `--env NAME` (value inherited from the launch process,
so it never lands on argv or in `container/docker inspect`);
- presented by the gateway's PolicyResolver (reads the env) on /resolve, and by
the host CLI's OrchestratorClient (reads the host file) on every call.
The agent container is never given the env var or the host file, so from a
bottle every /bottles*, /resolve, /attribute, and /supervise/* call now
returns 401 — closing the enumeration, allowlist-rewrite, credential-lift, and
self-approval. The existing (source_ip, identity_token) checks stay as
defense-in-depth.
Enforced when configured: macOS + docker inject the secret (→ enforced). With
no secret set the server runs open and warns loudly at startup — a
fail-visible fallback for the unit suite and for Firecracker, whose
port-scoped nft already blocks agents from 8099 (wiring the secret into its
infra-VM init is a clean fast-follow, left out here to avoid churning the
prebuilt-artifact hash).
Verified end-to-end on real Apple Container: infra comes up healthy, the host
CLI (with the secret) lists bottles while an unauthenticated GET /bottles gets
401, all five issue-#400 attacks from inside the agent get 401, and egress
policy still works (200 allowed / 403 denied) — proving the gateway
authenticates to /resolve with the secret. 1829 unit tests pass, pyright
clean, pylint 9.91.
Refs #400.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
didericis-claude
didericis-codex
didericis