Addresses review comments 3098, 3099, 3100 on PR #336:
- Add BottleImages(agent, sidecar) dataclass to backend/__init__.py.
Docker/macOS backends use str image refs; smolmachines uses Path
artifacts. Replaces the singular `image` variable from the canonical
pattern in comment 3100.
- Replace _image_stale_checks/skip_stale with public prelaunch_checks().
CLI now calls backend.prelaunch_checks(plan) before backend.launch(plan);
if StaleImageError is raised and the operator confirms, launch proceeds
without re-checking. Removes the while-True/skip_stale retry loop.
- Add abstract _build_or_load_images(plan) -> BottleImages to
BottleBackend. launch() calls it then passes images to _launch_impl.
Each backend implements both methods.
- Fix comment 3098 (macos-container): _build_images is removed.
build_or_load_images() has separate fresh/cached code paths — the
cached path never calls a build helper.
- Update _start_bundle (smolmachines) to accept sidecar_artifact: Path
directly. Sidecar artifact resolution moves to _sidecar_from_path(),
called by build_or_load_images alongside _agent_from_path().
- image_cache: StaleImageError exception + check_stale/check_stale_path (raise instead of warn)
- BottleBackend.launch: template method (skip_stale flag) that calls _image_stale_checks then _launch_impl
- Each backend: _image_stale_checks delegates to a stale_checks() function in its launch module; _launch_impl replaces launch override
- macos_container: adds image_created_at to util, cached-image support in _build_images, stale_checks
- cli/start.py: catches StaleImageError, prompts interactively, retries with skip_stale=True; headless mode dies on it
Review follow-ups for the infra candidate-artifact flow:
1. publish-infra would fail on the first merge to main. The rootfs version now
hashes the dropbear bytes, and build-infra sets BOT_BOTTLE_FC_DROPBEAR to the
staged dropbear — but publish-infra (ubuntu-latest) set none, so _publish_
bundle re-derived the version with a "<missing>" dropbear and rejected the
candidate as "does not match checkout". Download the same firecracker-inputs
dropbear and export BOT_BOTTLE_FC_DROPBEAR in publish-infra (and add
stage-firecracker-inputs to its needs, since it now consumes that artifact).
2. Guard stage-firecracker-inputs with the same fork-PR check as the other
KVM-runner jobs, so a fork PR can't spin the privileged runner (it only
copies a static binary, but keep the posture consistent; it gates the whole
Firecracker chain via needs).
3. ensure_artifact_gz / _publish_bundle read version.txt before checking it
exists — a bundle missing it raised a raw FileNotFoundError instead of the
intended "bundle is incomplete" die. Guard the read (kept before the gz/sha
completeness check so a wrong-version bundle still reports the version
mismatch, not "incomplete").
4. test_infra_artifact.setUp didn't isolate BOT_BOTTLE_INFRA_ARTIFACT_DIR, so
an ambient candidate dir (the coverage job exports one) would send the
registry-pull tests down the local-bundle path. Pin it off in setUp; the
candidate-path cases set it explicitly.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01A9qa3xoavjQScufDfZaXKR
Dockerfile.gateway COPYs pyproject.toml into /src and runs pip install
/src, so it is a real input to the baked rootfs. A dependency-only change
previously reused stale artifact versions, potentially booting a rootfs
whose installed packages differed from the current checkout.
Also adds _fake_repo fixture support and a regression test so this input
can't silently drop out of the hash again.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
build_agent_rootfs_dir cached the built rootfs by Dockerfile content alone,
but util.inject_guest_boot then writes util._GUEST_INIT into it. So a fix to
the init — making /tmp world-writable (1777) so the agent can create scratch
dirs / git worktrees there — did NOT bust the cache: the KVM runner kept
reusing a stale agent-<dockerfilehash> rootfs built with the old init, and the
sandbox-escape README-push test kept failing at `git init /tmp/...` with
"Permission denied".
Key the cache on Dockerfile content AND the injected init (_rootfs_digest), so
an init change rebuilds. Self-busting: the new key yields a fresh cache dir, so
no manual cache clear on the runner.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01A9qa3xoavjQScufDfZaXKR
The infra VM is a per-host singleton that outlives short-lived launchers, so
`ensure_running` adopts it when its control plane is healthy. But it adopted
ANY healthy VM regardless of the code that built it — so after an infra-code
change the old VM kept being adopted and the new code never booted. The only
way to dislodge it was an out-of-band `kill`, which then raced whatever
launched next. On CI this meant every infra change needed a manual VM kill.
Make adoption version-aware:
* boot records the infra-artifact version it booted from in a `booted-version`
marker beside the singleton; `stop` clears it.
* `ensure_running` adopts only when the marker matches the current version
(`_adoptable`); a missing/mismatched marker falls through to stop + reboot.
So a stale VM is replaced automatically on the next launch — no manual kill,
and it's concurrency-safe (the reboot happens under the singleton flock).
Also harden teardown: the PID file drifts after crashes / out-of-band kills,
so `stop` now also reaps any orphaned firecracker still bound to the infra
config path (scoped to that path, so interactive-pool VMs are untouched) —
otherwise a survivor holds the orchestrator TAP and the fresh boot dies with
"tap … Resource busy".
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01A9qa3xoavjQScufDfZaXKR
The rootless agent rootfs build can land /tmp as 0755/root-owned, so the
agent (uid 1000 node) can't create scratch dirs there. The sandbox-escape
suite's README-push test does `cd /tmp && git init sandbox-escape-repo` and
died with "cannot mkdir sandbox-escape-repo: Permission denied" — before the
git-gate gitleaks hook could run — so the test read it as a missing hook.
On docker the agent inherits node:22-slim's 1777 /tmp, which is why only the
Firecracker path was affected (and only now that the suite runs end-to-end).
Set /tmp to 1777 in the guest PID-1 init, so every agent VM boots with a
usable /tmp regardless of rootfs perm drift.
Also update the infra-init unit test to assert the gateway launches via the
`bot_bottle.gateway_init` module (matching the prior fix), not a file path.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01A9qa3xoavjQScufDfZaXKR
The infra VM's init boots the orchestrator control plane AND the gateway data
plane (egress/git-http/supervise). Since 5ad3449 moved the daemons into the
installed `bot_bottle` package, there is no `/app/gateway_init.py` file — the
gateway image's entrypoint is `python3 -m bot_bottle.gateway_init`. But the
firecracker init still spawned the old file path, so the guest logged:
python3: can't open file '/app/gateway_init.py': No such file or directory
The control plane came up (it already used `python3 -m bot_bottle.orchestrator`)
but the gateway never started, so mitmproxy never generated its CA and launch
died with "gateway CA not available after 30s". Mirror the orchestrator line:
run the gateway as `python3 -m bot_bottle.gateway_init` (bot_bottle resolves
from the /app CWD, same as the control plane; egress-entrypoint.sh /
egress_addon.py are present from the gateway base image).
Third build/runtime bug from 5ad3449's package refactor that the Firecracker
integration suite never exercised (the artifact publish was broken, so the
job 404'd before boot). Changes the init, so the infra artifact version moves;
the matching rootfs has been rebuilt and published.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01A9qa3xoavjQScufDfZaXKR
On a self-hosted KVM runner the process has a real controlling terminal
so name_color_modal successfully opens /dev/tty and enters a curses
loop waiting for keyboard input, hanging the test indefinitely.
Docker containers (ubuntu-latest runners) don't have a real /dev/tty,
causing an OSError that triggers the existing fallback — this is why
the hang was invisible in ubuntu-latest CI.
Also add timeout=5 to _daemon_reachable() to match the same defensive
fix already applied to docker_available() in tests/_docker.py.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
`sqlite3.Connection.__exit__` only commits/rolls back a transaction — it
does not close the connection. Python 3.13 (the Nix env on the KVM
runner) emits `ResourceWarning: unclosed database` for every connection
GC'd without an explicit close, producing noisy output in the coverage job.
Add `DbStore._connection()`, a `contextmanager` that calls `self._connect()`,
wraps it in the existing transaction context manager, and closes the
connection in a `finally` block. Change all `with self._connect() as conn:`
call sites in `db_store.py`, `audit_store.py`, `queue_store.py`, and
`orchestrator/registry.py` to `with self._connection() as conn:`.
`_connect()` remains as the per-subclass hook (RegistryStore overrides
it to set `busy_timeout`); `_connection()` delegates to `self._connect()` so
the override is respected.
Closes#412.
The supervise MCP server blocked the agent's tool call polling for the
operator's decision, and on timeout returned `status: pending` with no
proposal id and no way to poll a specific proposal — so the only way to
learn a late decision was to re-propose (a duplicate).
- `handle_tools_call` pending timeout now returns the `proposal_id` and
points the agent at `check-proposal`.
- New `check-proposal` MCP tool: non-blocking status lookup by proposal id
(pending | approved | modified | rejected | unknown). Reuses the queue's
FileNotFoundError semantics; archives a decided proposal exactly like the
synchronous path, so a pending proposal stays visible to the operator
until it's both decided and polled.
- `TOOL_CHECK_PROPOSAL` constant, re-exported from supervise; kept out of
TOOLS since it never becomes a Proposal.tool.
Enforcement is unchanged — the tools only propose policy; the egress proxy
and git-gate still enforce — so returning early opens no hole. Follow-ups
(git-gate reject-requeue, backpressure, notifications, web console) are in
the PRD.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YBCHap11yGAKuKfsehNPaD
The pre-receive hook scanned existing-branch updates with the delta range
$old..$new. On a rebase / non-fast-forward force-push onto an advanced main,
$old is no longer an ancestor of $new, so $old..$new expands to all of main's
new history — including the deliberate sandbox-escape gitleaks fixtures — and
the push is rejected on commits that belong to main, not the branch.
Unify the range on `$new --not --all` for every non-delete push (this is the
deferred open question from PRD 0028, which already applied it to new refs
for #106). It scans only the commits the push introduces and is
security-equivalent: the bare repo's refs come only from trusted upstream
mirror-fetch and gitleaks-gated pushes, so an excluded commit is
already-upstream or already-scanned. It is also more correct for
non-fast-forward pushes, where $old..$new can skip commits off the direct path.
Fixes#421
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01S1qRZTJC6qgBsUSjNrBdkX
`backend setup/status/teardown` manage host prerequisites only and never
open the store, but the CLI dispatcher ran the schema-migration gate before
every command. On a non-TTY runner the gate's `Migrate now? [y/N]` prompt
reads EOF and refuses, so `backend status --backend=firecracker` exits 1 —
breaking the Firecracker CI preflight on any host without a pre-migrated DB.
Exempt `backend` from the gate; store-touching commands stay gated.
Fixes#419
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01S1qRZTJC6qgBsUSjNrBdkX
- Rename _BACKENDS → _backends: pyright treats uppercase module-level
names as constants and flags the reassignment in _get_backends() as
reportConstantRedefinition; lowercase avoids this.
- Add TYPE_CHECKING guard importing CommitCancelled/Freezer/get_freezer
from .freeze: pyright cannot see module-level __getattr__ bindings, so
reportUnsupportedDunderAll fired for those three __all__ entries; the
guard makes them visible to the type checker without running at import
time.
- Update test_backend_selection.py to patch _backends (lowercase).
`undefined-all-variable` fires on CommitCancelled / Freezer / get_freezer
in __all__ because pylint can't see module-level __getattr__ bindings;
`global-statement` fires on the _BACKENDS singleton setter. Both are
intentional patterns — add inline disables rather than suppress globally.
Importing backend.docker.util previously triggered eager loading of all
three backend packages (~76 modules) because backend/__init__.py imported
DockerBottleBackend, FirecrackerBottleBackend, and MacosContainerBottleBackend
at module scope. This made the module prohibitively expensive to import
from the orchestrator layer and elsewhere.
The three backend imports are now deferred into _get_backends(), which
loads all three on first call and caches the result in the module-level
_BACKENDS variable (initially None). Module-level __getattr__ exposes
backend classes and freeze symbols lazily for existing import/patch sites.
backend/docker/util.py raw subprocess.run(["docker", ...]) calls are
replaced with the shared run_docker primitive from docker_cmd, eliminating
the duplication between the backend and orchestrator implementations.
_silent_run() is removed; image_exists() is inlined directly onto
run_docker. The commit_container test is updated to patch run_docker
instead of subprocess.run.
supervise_server, git_http_backend, and gateway_init all have __main__
guards, so python3 -m bot_bottle.X replaces the individual COPY lines
to /app/. egress_addon.py stays as a file copy because mitmdump -s
requires a file path rather than a module reference.
Install bot_bottle via pip in Dockerfile.gateway instead of COPYing
individual .py files flat under /app/. This eliminates the try/except
import shims in egress_addon_core, dlp_detectors, egress_addon,
supervise, supervise_server, and git_http_backend that existed only
to support the flat-bundle layout.
Adds bot_bottle/constants.py as a single source of truth for
IDENTITY_HEADER and GIT_GATE_TIMEOUT_SECS, removing the duplicated
literal definitions in egress_addon.py, supervise_server.py,
git_http_backend.py, and git_gate_render.py.
Test files updated to match: test_supervise_server.py drops the
sys.path.insert hack in favour of direct package imports; the
egress_addon test shims no longer pre-populate sys.modules with a
bare egress_addon_core alias.
Follow-ups from the #402 review of the single-tenant data-plane teardown.
- egress_entrypoint.sh: refuse to launch mitmdump when
BOT_BOTTLE_ORCHESTRATOR_URL is unset, so the fail-closed guarantee no
longer rests solely on mitmproxy's errorcheck addon exiting on the
addon's load-time raise. A misconfigured gateway can never come up as
a bare TLS-bumping open proxy with no policy.
- orchestrator/gateway.py: ensure_running() raises GatewayError on an
empty orchestrator URL — a URL-less launch would only crash-loop the
now-resolver-only daemons (egress raises, git-http exits 1, supervise
exits 2). The env-injection branch is now unconditional.
- Drop stale "single-tenant" / "reads routes.yaml" comments in
gateway.py and egress_entrypoint.sh, and the /etc/egress/routes.yaml
layout line in Dockerfile.gateway.
- Tests: gateway fixtures supply an orchestrator URL; add a
refuse-without-URL test and assert the URL env is injected.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UoEZHDjv84ChoZbozQERhJ
All three backends (docker, firecracker, macos-container) now launch through
the consolidated orchestrator, and every production gateway sets
BOT_BOTTLE_ORCHESTRATOR_URL — so the legacy single-tenant (`resolver is None`)
branches in the shared gateway's data plane were unreachable dead code, a second
security-relevant path to keep correct in parallel with the live one. Make the
orchestrator resolver mandatory and delete the single-tenant paths from the
three data-plane modules.
egress_addon.py: drop the static routes file entirely — EGRESS_ROUTES, _reload,
the SIGHUP handler, self.config, and the SUPERVISE_BOTTLE_SLUG env slug. The
per-request /resolve is the only policy source; __init__ fail-closes if
BOT_BOTTLE_ORCHESTRATOR_URL is unset. Introspection (`_egress.local/allowlist`)
now reports the calling bottle's *resolved* routes. The block/redact log gates
and _req_ctx redaction now read the per-flow config/env from the request-time
stash, so they use each bottle's log level and token set (they silently used the
empty static config before). Nothing sends `docker kill --signal HUP` to the
gateway in the consolidated model (the egress applicators fail closed), so
removing the SIGHUP reload is safe.
git_http_backend.py: resolver mandatory; no flat-root fallback. main() refuses
to start without an orchestrator URL; a request whose source resolves to no
bottle 404s.
supervise_server.py: resolver mandatory; every proposal is attributed to the
source-IP-resolved bottle. Remove handle_list_egress_routes (the proxy-fetch
introspection that only worked when the proxy carried one bottle's identity) —
list-egress-routes is answered from the resolved policy. main() refuses to start
without an orchestrator URL.
Tests: a host-side fake resolver serves each test's Config through the real
parse path (a small YAML-subset emitter round-trips route_to_yaml_dict); the
response/websocket hooks stash it as request() would. Deletes the tests for the
removed static-config, SIGHUP-reload, and single-tenant-passthrough paths; adds
fail-closed-without-orchestrator coverage.
Follow-up: gateway_init still forwards SIGHUP to the egress child (now dormant —
no one sends it); the README still describes the docker backend's per-bottle
topology. Both are outside the data-plane teardown.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UoEZHDjv84ChoZbozQERhJ
Cloning/fetching from the git-gate on the Apple-container backend failed with
"empty reply from server" (curl exit 52). Root cause: the git-http handler
crashed on every upload-pack with
PermissionError: [Errno 13] Permission denied: '/etc/git-gate/access-hook'
The access-hook is exec'd directly, so it needs the x bit. prepare() stages it
0o700 and trusted the gateway copy to carry that mode. `docker cp` does; the
Apple `container cp` (AppleGatewayTransport) does not, landing the hook 0o644 →
EACCES. The unhandled exception killed the handler thread, closing the socket
with no HTTP response — which the client sees as the opaque empty reply.
- provision_git_gate now `chmod +x`es the access-hook on the gateway side after
the copy, so it's executable under every transport (docker/apple/firecracker).
- git-http handler wraps the access-hook subprocess.run: an OSError /
SubprocessError (un-execable, timed out) now fails closed with a 503 instead
of crashing the thread into an empty reply — a gate that can't run its hook
should deny, visibly.
- Updates the now-misleading "docker cp preserves source mode" comment in
git_gate.prepare().
Regression tests: provisioning applies +x to the access-hook; the handler
returns 503 (not an empty reply) when the hook can't be exec'd.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Self-review of this PR: making `response()` run in the consolidated gateway
also activates its `LOG_FULL` `_log_response` call there — previously
unreachable, since the empty static config made `response()` return early. That
logger redacted with `os.environ`, which in multi-tenant mode does NOT hold the
bottle's per-request `/resolve` tokens (only the resolved `env` overlay does),
so a non-token-shaped provisioned secret appearing in a response could be logged
in the clear.
Thread the resolved per-flow `env` into `_log_request` / `_log_response` so the
LOG_FULL redaction scrubs the calling bottle's secrets. Adds a regression test
(a non-token-shaped `/resolve` secret, absent from os.environ, must not appear
in the response log) and updates the redaction-test helpers for the new arg.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UoEZHDjv84ChoZbozQERhJ
In the consolidated (multi-tenant) gateway the addon's static `self.config`
is empty — each request's real policy comes from the per-request `/resolve`.
`response()` and `websocket_message()` still matched routes against that empty
config, so inbound prompt-injection DLP and WebSocket credential/injection DLP
silently skipped every scan (fail-open) whenever the gateway ran multi-tenant.
This is backend-agnostic: the gateway image (and this addon) is shared by the
Firecracker, macOS, and docker consolidated backends.
Resolve the per-flow (config, slug, env) once in `request()`, stash it on
`flow.metadata`, and have both hooks read it back — falling back to the static
single-tenant values for a flow that never passed through `request()`. Reusing
the request's one `/resolve` avoids a round-trip per response and per WebSocket
frame.
Adds multi-tenant regression tests for both hooks that fail against the old
fall-open behaviour.
Refs: audit issue #400 (finding #2)
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UoEZHDjv84ChoZbozQERhJ
Addresses the 5 lower-priority findings left as follow-up in the earlier
review, now that each has a concrete answer:
- Add gateway_name: str = GATEWAY_NAME to OrchestratorService.__init__
(mirrors the existing orchestrator_name param) and thread it through
_gateway(). Deletes the test's _IsolatedOrchestratorService subclass,
which existed only to override a private method for this one kwarg —
any caller needing gateway-name isolation can now use the public
constructor. Backward compatible: every existing caller constructs
OrchestratorService with keyword args and a sensible default is kept.
- Give the test its own fixed image tags (bot-bottle-orchestrator:itest,
bot-bottle-gateway:itest) instead of the production :latest ones.
_running_image_is_current() keys gateway staleness off the image tag's
ID, not per-instance identity, so rebuilding the shared :latest tag from
whatever's on disk during a test run could make a real host's running
production gateway look stale and get force-recreated. Fixed tags (not
per-run-suffixed, so they don't accumulate) fully decouple the two.
- setUp -> setUpClass/tearDownClass: all 5 tests are read-only checks
against the same running control plane, so one shared container
lifecycle replaces 5 (each of which paid its own container-start +
image-build + health-poll cycle). Cuts the file's wall-clock roughly
4x (11.5s -> 2.9-4.3s) and, combined with the network-rm cleanup from
the previous commit, means one cleanup instead of five.
- Reuse OrchestratorClient (bot_bottle/orchestrator/client.py) instead of
a hand-rolled urllib helper — the test now exercises the same
request/response code path the real host CLI uses, rather than a
private copy that could silently drift from it.
- Add the chown workaround test_multitenant_isolation.py already needed
for this exact bind-mount: the orchestrator container has no USER
directive, so it writes the registry DB as root into the throwaway
host_root; chown it back before tempdir cleanup so that doesn't raise
PermissionError on native Linux Docker (no UID remap, unlike Docker
Desktop's macOS VM).
Verified: ran the suite twice in a row (idempotency — fixed image tags
don't accumulate, 5/5 pass both times, 2.99-4.33s each), the real
~/.bot-bottle/control-plane-token is untouched, zero leaked networks or
containers after either run, exactly 2 :itest images (not growing), the
full orchestrator unit suite (93 tests) and the sibling docker
gateway/broker integration tests still pass. pyright clean, pylint
10.00/10 on both changed files.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
The control plane had no caller authentication — dispatch() routed every
request straight through, and the only guards anywhere were the
(source_ip, identity_token) checks inside /resolve and /attribute, which
authenticate the *bottle* a request is about, not the *caller*. On the
consolidated backends the agent shares a network with the control-plane port
(macOS: the infra container runs 8099 on the agent's --internal network;
docker: the orchestrator container sits on the shared gateway network), so a
prompt-injected agent could GET /bottles (enumerate siblings + policies),
PUT /bottles/<id>/policy (rewrite its own egress allowlist to allow-all),
POST /resolve (read the injected upstream credentials it's never meant to
see), and POST /supervise/respond (self-approve its own proposals). Every
README guarantee collapsed to the agent's discretion.
Fix: a per-host control-plane secret required on every route but GET /health,
compared with hmac.compare_digest. It is held only by the trusted callers and
never handed to an agent:
- minted + persisted 0600 at <root>/control-plane-token (paths.host_control_plane_token);
- injected as $BOT_BOTTLE_CONTROL_PLANE_TOKEN into the orchestrator + gateway
containers via bare `--env NAME` (value inherited from the launch process,
so it never lands on argv or in `container/docker inspect`);
- presented by the gateway's PolicyResolver (reads the env) on /resolve, and by
the host CLI's OrchestratorClient (reads the host file) on every call.
The agent container is never given the env var or the host file, so from a
bottle every /bottles*, /resolve, /attribute, and /supervise/* call now
returns 401 — closing the enumeration, allowlist-rewrite, credential-lift, and
self-approval. The existing (source_ip, identity_token) checks stay as
defense-in-depth.
Enforced when configured: macOS + docker inject the secret (→ enforced). With
no secret set the server runs open and warns loudly at startup — a
fail-visible fallback for the unit suite and for Firecracker, whose
port-scoped nft already blocks agents from 8099 (wiring the secret into its
infra-VM init is a clean fast-follow, left out here to avoid churning the
prebuilt-artifact hash).
Verified end-to-end on real Apple Container: infra comes up healthy, the host
CLI (with the secret) lists bottles while an unauthenticated GET /bottles gets
401, all five issue-#400 attacks from inside the agent get 401, and egress
policy still works (200 allowed / 403 denied) — proving the gateway
authenticates to /resolve with the secret. 1829 unit tests pass, pyright
clean, pylint 9.91.
Refs #400.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Adopts the firecracker infra-VM pattern for macOS: the orchestrator control
plane and the gateway data plane now run in a SINGLE Apple container instead of
two. Apple Containers are lightweight VMs with separate kernels, so the prior
two-container design had both guests writing one bot-bottle.db over virtiofs,
where fcntl locks are not coherent across kernels — concurrent writes (the
orchestrator's registry vs the gateway supervise daemon's queue) could corrupt
it. One container = one kernel = coherent locking.
The DB moves onto a container-only Apple volume (bot-bottle-mac-db), never
bind-mounted from the host, so no host process opens the live file either. The
host CLI already reaches registry + supervise state over the control-plane HTTP
surface (cli/supervise.py uses OrchestratorClient), exactly as firecracker's
VM-only DB requires.
Two simplifications fall out of the single container:
- No DNS dance: the control plane and gateway daemons reach each other over
127.0.0.1, so the orchestrator-before-gateway ordering (a workaround for
Apple having no container DNS) is gone, along with the moved-IP recreate
logic it needed.
- Net -243 lines.
Mechanics: the infra container runs from the gateway image with the
control-plane source bind-mounted read-only (like the docker orchestrator, so a
code change needs no rebuild) and a small sh -c init that starts both processes
(mirrors firecracker's _infra_init). Also implements the macOS backend's
ensure_orchestrator() and adds it to discover_orchestrator_url, so operator
tools (supervise) can bring up / find the control plane on demand — previously
the macOS backend died with "no orchestrator control plane".
Verified end-to-end on real Apple Container 1.0.0: the single infra container
comes up healthy (one address for control plane + gateway), both processes run,
the DB is written on the container-only volume, host-side supervise works over
HTTP, and a registered agent gets 200 for an allowed host / 403 for a denied
one. 1824 unit tests pass with `container` absent (CI parity), pyright clean,
pylint 9.89.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Addresses findings from a high-effort review of the PRD 0070 macOS backend.
Correctness:
- Stamp identity_token onto MacosContainerBottlePlan after registration. git's
gitconfig extraHeader and the supervise MCP --header read
getattr(plan,"identity_token","") at provision time, and both reach the
gateway on NO_PROXY (bypassing the egress proxy that carries the token). The
plan never carried it, so /resolve fail-closed and every git fetch/push and
supervise call from a macOS bottle would have been denied. Registration
precedes provision(), so — unlike the run-time env — the plan can carry it.
- Self-heal the orchestrator: recreate when it is not (source-current AND
answering /health), not on the source-hash label alone. A container running
current code but with a wedged HTTP server was left alone and polled to
death, failing every launch until manual deletion.
- image_digest and container_image_digest now read the same descriptor.digest
field; dropped image_digest's id/tag fallback that could yield a value the
container side can't produce — a permanent mismatch would have recreated the
shared gateway on every launch (severing every live bottle's egress, since
the replacement gets a new DHCP address).
- Poll for the agent's and gateway's DHCP address instead of a fatal read
right after `container run` (there is no --ip; the address can lag start).
Cleanup:
- One _inspect_first + _descriptor_digest behind the four inspect readers.
- Shared bind_mount_spec (util) and host_db_dir (paths) replace per-module
copies; _GIT_HTTP_PORT now imports git_http_backend.DEFAULT_PORT.
- Drop the dead _url cache / url property and the write-only agent_proxy_url.
Deferred (noted on the PR, not fixed here): the gateway image rebuilding on
every launch (needs source-hash-labeled build), SQLite shared across VM
guests, and the sh -lc profile-override edge — each is design-level or
behavior-risk beyond a review fix.
Verified: real Apple Container bring-up is green and idempotent; 1826 unit
tests pass with `container` absent (CI parity), pyright clean, pylint 9.86.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Re-enables the macos-container backend on the shared per-host orchestrator +
gateway, replacing the per-bottle companion container removed in #385. This is
the last backend in PRD 0070's roadmap.
Apple Container 1.0.0 forced three departures from the docker shape, each
verified against the live CLI (findings recorded in the networking spike):
- No `--ip`. The address is DHCP-assigned and knowable only once the container
runs, so the order inverts: gateway up -> run agent -> read its address ->
register. The identity token is minted by registration and therefore cannot
be in the agent's run-time env; it rides the proxy URL applied at
`container exec` time (bare `--env` names keep it off argv).
- No container DNS. The gateway can only be handed the control plane's IP, so
the orchestrator starts first and the gateway is pointed at its address.
- No `network connect`. Networks are fixed at run time, so the shared host-only
network is created up front; per-bottle networks would restart the gateway
on every launch and defeat the consolidation.
The agent runs with `--cap-drop CAP_NET_RAW`: Apple grants NET_RAW by default,
which would let an agent forge a neighbour's source address on the shared
segment. NET_ADMIN is already absent, so this closes the source-address half of
PRD 0070's attribution invariant.
Verified end-to-end on real Apple Container 1.0.0: both images build, the
control plane comes up healthy, the gateway reaches it by IP, and a registered
agent gets 200 for a host in its routes and 403 for one outside them. Bring-up
is idempotent — a second launch does not churn the singletons.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Follow-up to the codex review on #398. os.open's mode arg only applies on
creation, so a committed-rootfs.tar.partial left 0644 by an interrupted run
would be opened/truncated (not re-moded) and stay world-readable for the
whole SSH stream. Unlink any leftover and exclusively recreate it
(O_EXCL|O_NOFOLLOW), then fchmod 0600 immediately so umask can't loosen it.
Test pre-creates a 0644 partial and asserts the fd is 0600 mid-stream.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UoEZHDjv84ChoZbozQERhJ
Address the codex review on #398:
- P1: inject_guest_boot no longer follows a symlink at bb-init/bb-dropbear.
A committed snapshot is guest-controlled and could plant those paths as
symlinks aimed at a host file (e.g. bb-init -> ~/.bashrc); write_text /
copy2 would then overwrite the target as the host user during resume.
Replace any pre-existing entry and create the files with
O_EXCL|O_NOFOLLOW so the write stays inside the staging tree.
- P2: write the snapshot tar owner-only (0600). It can contain the bottle's
private workspace; it was being created world-readable (0644).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UoEZHDjv84ChoZbozQERhJ
The last host-Docker dependency in the Firecracker launch path. Freeze
and resume no longer touch the docker daemon, so the backend needs
firecracker + KVM only — completing #348.
Freeze: stream the guest rootfs over SSH straight into a persistent
committed-rootfs.tar (the resumable/migratable artifact) instead of
round-tripping through `docker build` from a scratch image. Written to
a .partial sibling and atomically renamed so a failed freeze leaves no
truncated artifact.
Resume: extract the snapshot tar into a cached base dir and feed it to
the existing rootless `mke2fs -d` pipeline, replacing the
`docker create` + `docker export | tar` path. Recreate the
proc/sys/dev/run mount points the freezer excludes so the guest init
can mount them.
`util.build_base_rootfs_dir` / `docker_image_id` stay — they still back
the opt-in BOT_BOTTLE_INFRA_BUILD=local dev path and off-host
publish_infra, which are out of scope.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UoEZHDjv84ChoZbozQERhJ
Rename the Gitea generic package from bot-bottle-infra to
bot-bottle-firecracker-infra so it's self-evident in the package list which
backend it serves (and leaves room for other artifacts, e.g. a shipped
kernel). The version slot stays the content hash — "firecracker" belongs in
the package name, not the version. Docker image / VM names are unchanged.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UoEZHDjv84ChoZbozQERhJ
Generic packages have no description field, so publish_infra now uploads a
short about.txt alongside the rootfs on every publish — it's what identifies
the package as the Firecracker backend's infra rootfs on the package page.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UoEZHDjv84ChoZbozQERhJ
Address PR #395 review (two P1s):
- Version hash covered only `bot_bottle/**.py`, but the image `COPY`s the
whole package — non-Python inputs baked in (egress_entrypoint.sh,
netpool.defaults.env) didn't change the version, so a launch host could
boot a stale rootfs whose code differs from its checkout. Hash every
regular file under bot_bottle/ (excluding __pycache__/.pyc). Regression
tests: a shell-script change bumps the version; .pyc/__pycache__ don't.
- publish_infra `_put` read the whole (hundreds-of-MB) gz into memory via
read_bytes(). Stream it from disk with an explicit Content-Length; the
tiny .sha256 stays in-memory. Test asserts the body is the file object,
not bytes.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UoEZHDjv84ChoZbozQERhJ
Drop the fallback to the general-purpose BOT_BOTTLE_CLAUDE_GITEA_TOKEN so
the artifact pull/publish uses a dedicated, package-scoped token that can
be granted (or revoked) independently of the general Gitea token.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UoEZHDjv84ChoZbozQERhJ
Stage 2 of the docker-free Firecracker backend (#348): stop building the
fixed infra image on the launch host. The infra VM's rootfs is host- and
bottle-agnostic (authorized_keys + guest IP ride the kernel cmdline, not the
rootfs), so it's built once off-host and published as a versioned, ready-to-
boot ext4; the launch host downloads + verifies + boots it — no Docker, no
image tooling, just HTTP + gunzip.
- infra_artifact.py: version = content hash of the rootfs inputs (the shipped
bot_bottle package + the three Dockerfiles + the init), so a launch host
pulls the artifact matching its code and a content change can't silently
boot a stale rootfs. Pull + sha256-verify (fail-closed) + gunzip from a
Gitea generic package; base/owner/token configurable, default this Gitea.
- infra_vm.ensure_built/boot default to the pull path; BOT_BOTTLE_INFRA_BUILD=
local keeps the docker build-from-source path for iterating on Dockerfiles.
- publish_infra.py: the off-host half — builds the images with Docker, mke2fs
the rootfs (with buildah slack), gzips, and PUTs it to the generic package.
Rollout note: default=pull means a launch 404s until an artifact is published;
until the Gitea packages endpoint is enabled + an artifact published, use
BOT_BOTTLE_INFRA_BUILD=local. Freeze/migrate's remaining docker use is a
separate PR.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UoEZHDjv84ChoZbozQERhJ
Adds globalize_slug(slug) to bottle_state alongside bottle_identity.
git_gate_provision now calls globalize_slug(slug) instead of inlining
socket.gethostname(), so the hostname-qualification logic has a single,
named home. Assumes slug is a mint_slug output.
Title format changes from bot-bottle:{host}:{slug}:{name}
to bot-bottle:{host}-{slug}:{name} to match the globalize_slug contract.
Closes#388 (part 1 of 3). Deploy key titles now carry the machine
hostname so keys provisioned on different hosts don't collide with
each other on the forge when a prior bottle was never torn down.
Title format: bot-bottle:<hostname>:<slug>:<repo-name>
In consolidated mode the gateway's static route table is empty (routes
are resolved per request by source IP), but `list-egress-routes` was
reading that static table via the `_egress.local/allowlist` introspection
endpoint — so it always returned an empty allowlist. Agents, told to call
it before composing an egress proposal, then sent a routes.yaml with only
the newly-needed host; approving it replaced the whole policy and silently
dropped base routes like api.anthropic.com, breaking the bottle's egress.
Answer `list-egress-routes` from the calling bottle's resolved policy
(same (source_ip, identity-token) attribution the proposal path uses,
same JSON shape the single-tenant introspection endpoint returns).
Fail-closed to an empty list on an unreachable orchestrator; falls back
to the introspection endpoint in single-tenant mode (no resolver).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UoEZHDjv84ChoZbozQERhJ
Consolidated proposals are keyed by the orchestrator-assigned bottle_id,
so the supervise TUI was rendering a hex id (e.g. 3601cbe883c2786d) as
the bottle name — and the resume hint printed `./cli.py resume <id>`,
which resume can't take (it wants the human identity/slug).
`supervise_pending` now tags each dict with `bottle_label` — the human
slug resolved from registry metadata, falling back to the id when the
bottle is gone. `QueuedProposal` carries the label; every display site
(list rows, detail view, status lines, resume hint) shows it, while
approve/reject still key on `proposal.bottle_slug` (the id).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UoEZHDjv84ChoZbozQERhJ
In consolidated mode the supervise server attributes each proposal to
the orchestrator-assigned bottle_id and stores that as the proposal's
`bottle_slug` (supervise_server `_attributed_config`). But
`_record_for_slug` only scanned registry metadata for a matching human
slug, so approving an egress proposal 409'd with "bottle <id> is no
longer registered; cannot apply the route change" — the id never
matched a human slug.
Resolve the record by bottle_id first (the consolidated reality),
keeping the metadata-slug scan as a fallback for legacy single-tenant
proposals. The prior tests keyed proposals by the human slug matching
the metadata, exercising only the fallback path — added a test that
mirrors production (proposal keyed by bottle_id, distinct human slug in
metadata).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UoEZHDjv84ChoZbozQERhJ
Add BottleBackend.ensure_orchestrator() -> str: the backend-agnostic
entry point that brings up the per-host orchestrator + shared gateway
(idempotent) and returns its host-reachable control-plane URL. Docker
starts the orchestrator + gateway containers (OrchestratorService);
firecracker boots the infra VM; macos-container dies with a pointer
(no orchestrator). Previously bring-up was reachable only through each
backend's consolidated_launch, with no shared handle.
Wire it into `bot-bottle supervise`: supervise is often the first thing
an operator runs, before any bottle has booted the control plane, so
`_resolve_orchestrator_url` now starts the selected backend's
orchestrator on demand when discovery finds nothing, instead of failing
with "launch a bottle first".
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UoEZHDjv84ChoZbozQERhJ
The docker gateway container ran the supervise daemon but bind-mounted no
DB and set no SUPERVISE_DB_PATH — so the daemon wrote proposals to a
container-local, ephemeral SQLite file, disconnected from the host DB the
orchestrator (and now the operator, over HTTP) uses. Same split-DB bug
firecracker had; Dockerfile.gateway even documents the mount
(`/run/supervise/bot-bottle.db bind-mounted at run time`) that
ensure_running never provided.
Bind-mount the host DB dir into the gateway at /run/supervise and set
SUPERVISE_DB_PATH, so the daemon queues into the same file the
orchestrator container opens (its BOT_BOTTLE_ROOT bind-mount) and the
operator reaches over the control plane. One DB per host, shared by
bind-mounts — docker now on the exact same supervise path as firecracker.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WBMWTEtQdJ4W5UrWuLHCck
The `bot-bottle supervise` operator TUI read and wrote the queue DB
directly and tried a backend-specific live "apply" (which was unwired —
it raised). It now talks only to the orchestrator control plane:
- OrchestratorClient gains supervise_pending() + supervise_respond().
- discover_orchestrator_url() finds the one running per-host control
plane by health-probing the backends' well-known :8099 addresses
(docker publishes on loopback; the firecracker infra VM serves it on
the orchestrator TAP) — no backend branching in the TUI.
- discover_pending/approve/reject call the client; the server does the
apply + response + audit atomically. The dead direct-DB apply/audit
helpers and the docker/macos applicator imports are gone.
- A missing control plane is now a clean one-line error up front, not a
mid-curses crash.
CLI tests move to mocking the client (the DB-write behaviour they used to
assert is now server-side, covered by test_orchestrator_service). Docker's
orchestrator-container DB wiring lands next so its /supervise endpoints hit
the same shared DB.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WBMWTEtQdJ4W5UrWuLHCck
The orchestrator owns the single DB *and* the live policy, so operator
decisions belong there — applied server-side, reached over HTTP. Adds:
GET /supervise/proposals -> pending proposals across bottles
POST /supervise/respond -> apply + record an operator decision
`supervise_respond` is one atomic server-side op on the one DB: approve/
modify on an egress tool rewrites the bottle's policy (so the gateway
serves the new routes on its next /resolve — the live "apply" that was a
documented TODO), then writes the queued Response (unblocking the agent's
MCP call) and an audit entry. reject records the response + audit only.
Fails closed (409) when the proposal is unknown or the bottle was torn
down before the operator acted (an egress apply would have no target).
This is the server half of unifying every backend onto one HTTP path for
supervise; the host TUI (direct-DB today) moves onto this client next.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WBMWTEtQdJ4W5UrWuLHCck
The firecracker supervise MCP daemon 500'd (-32603) on egress-allow/block:
it ran with no BOT_BOTTLE_ROOT/SUPERVISE_DB_PATH, so it targeted a stray,
unmigrated SQLite file and `write_proposal` hit "no such table:
supervise_proposals". Meanwhile the in-VM control plane migrated only the
registry table (orchestrator_bottles) into its own DB, and the host
operator reads a third, disconnected DB — three files, none shared.
Consolidate to one DB per host, owned by the control plane on the
persisted registry volume (/var/lib/bot-bottle/db/bot-bottle.db):
- orchestrator startup now migrates the supervise queue + audit tables
into the same file it migrates the registry into (StoreManager), so the
control-plane DB carries every table.
- the in-VM supervise daemon is pointed at that same file via
SUPERVISE_DB_PATH, so daemon and control plane share one queue.
`list-egress-routes` already worked (no DB); egress-allow now queues +
waits on the single persisted DB instead of erroring. Validated against a
live infra VM: migrate + write_proposal succeeds and the proposal is
queued. The host-operator HTTP bridge (so approvals complete from the
host, unifying docker onto the same path) is the follow-up.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WBMWTEtQdJ4W5UrWuLHCck
didericis-claude
didericis-codex
didericis