This is the layer that actually decides it, and it was in the script
from the start: ~/.docker/config.json's proxies block is what the Docker
CLI copies into every container it starts. Being client-side it beats
the podman service outright, which is why containers.conf `env`,
http_proxy=false, and the service environment each looked correct on
disk and changed nothing a nested container saw.
Also corrects the comment on the service-env substitution, which
claimed to be the one place the address sticks. It is not; it covers
podman's own pulls and non-CLI API clients.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Third attempt at the same symptom, and the first at the right layer.
podman's Docker-compatible API injects proxy settings from the *daemon*
environment after containers.conf is applied, so neither the env list
(892bfc16) nor http_proxy=false (21c144ae) could override it — the same
compat-path blind spot already seen with hosts_file. Nested containers
kept reporting `wget: bad address 'bot-bottle-gateway:9099'` through
both.
Substituting the resolved address into the service's own environment,
before `podman system service` starts, is the one place it sticks.
NO_PROXY keeps the name, which is matched against what a client asks
for.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The address-bearing proxy URLs from 892bfc16 were written correctly but
never took effect: podman copies the host's proxy environment into every
container by default, and that copy overrides containers.conf `env`,
restoring the `bot-bottle-gateway` name a nested container cannot
resolve. Verified on the macOS host — containers still reported
`wget: bad address 'bot-bottle-gateway:9099'` with the addresses in
place.
Setting http_proxy=false disables only podman's own passthrough; the
proxy vars still reach containers, by address, from the env list.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
podman's containers.conf `hosts_file` turns out to apply only to native
`podman run`; the Docker-compatible API ignores it, and the agent types
`docker`. (`base_hosts_file`, used before this, is a no-op in both
paths on podman 5.4.2.) So no config key can put the gateway name in a
nested container's /etc/hosts.
The name is therefore resolved in the bottle, where /etc/hosts does
carry it, and the address goes into the proxy URL handed to nested
containers. NO_PROXY keeps the name, since that is matched against what
a client asks for.
Verified on the macOS host: with the gateway reachable, a nested
container gets 200 from an allowlisted host and 403 from one that is
not — the egress boundary applies inside nested containers too.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Answers the last open question in #392: DNS from inside a nested
container was not a phantom, but it was also not a DNS problem. Public
resolution fails there by design — everything egresses through the
proxy — and the actual breakage was two missing pieces:
- The proxy URL names `bot-bottle-gateway`, which resolves only through
the bottle's own /etc/hosts. Containers got their own hosts file, so
they failed at "Could not resolve proxy", which reads like broken DNS.
base_hosts_file propagates the bottle's entries.
- The gateway TLS-intercepts, so a container that does not trust the
bottle's CA bundle fails with "unable to get local issuer
certificate". The bundle is now mounted read-only and the usual env
vars point at it, which covers curl, wget, python, and node without
distro-specific trust commands.
Verified on the macOS host by reproducing each failure and confirming
that these three conditions applied by hand produced HTTP 200 from
inside a nested container. podman already forwards the proxy env, so
that needed nothing.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
netavark shells out to `nft` for the bridge network every compose file
expects, and aardvark-dns serves name resolution inside nested
containers. Neither arrives with the base image's
`--no-install-recommends` podman, and each fails at a different and
misleading layer: without nft containers are created but never start;
without aardvark-dns DNS inside a container fails while the engine, the
pulls, and the bridge all look healthy. That last one is the likeliest
explanation for the "DNS inside nested containers fails entirely" note
in #392 — a missing package, not a design gap.
Verified on the macOS host: with passt alone, `docker run` reached
netavark and died on "unable to execute nft".
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
podman 5 uses pasta as the default rootless network helper; podman 4
used slirp4netns. The spike was written against bookworm's podman 4.3.1,
so its package list never included passt — and the trixie bump (#451)
changed the default out from under it. Every nested container failed to
start with "could not find pasta", while image pulls and the service
itself worked, which disguised a missing dependency as a compat-API bug.
The bootstrap now checks for pasta up front so this fails with a clear
message instead of surfacing at the first `docker run`.
Note the sun_path fix in a12d2191 did NOT cause or cure this; that path
was over the 108-byte limit independently and would have bitten as soon
as attach was reached.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
podman derives conmon's attach socket as
`$XDG_RUNTIME_DIR/libpod/tmp/socket/<64-hex-id>/attach`. With the
descriptive `/tmp/bot-bottle-podman-run` that path is 116 bytes, past
the 108-byte sun_path limit, so every attached `docker run` failed with
"unable to upgrade to tcp, received 500" — while image pulls and the
service itself worked, which is what made it look like a compat-API bug
rather than a path-length one.
Found on the macOS host: quay.io pulled fine and then no container
could be started. Test pins the budget so a more readable name cannot
silently reintroduce it.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Adds the `nested_containers` bottle flag. On the macOS backend it starts
a rootless podman service inside the bottle and exposes its
Docker-compatible API socket, so the agent still runs `docker` and
`docker compose`. No host daemon socket is mounted and the guest gains
no capabilities; backends that cannot run a guest-local engine reject
the flag in the shared prepare template rather than ignoring it.
Podman rather than rootless Docker because Apple Container's capability
bounding set omits CAP_SYS_ADMIN, which the kernel requires to write a
multi-range uid_map via newuidmap. With no subordinate UID range podman
falls back to a single-UID self-mapping an unprivileged process may
write itself, so the image build strips /etc/subuid and /etc/subgid
entries rather than adding them.
That mapping is also why nested containers are not an isolation layer:
root inside one is the agent user outside it. They are a build/test
convenience; the bottle remains the boundary.
Ports the spike branch onto main, renaming docker_access — it implied
Docker and granted access to nothing on the host — and drops podman
from the derived layer now that every built-in image ships it (#451).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
`cleanup()` enumerated *every* firecracker process under the run root and
*every* run dir, so `./cli.py cleanup` would kill running bottles and
delete their rootfs — despite being described as orphan cleanup. The
backend's `enumerate_active` registry is still a stub (#354), so there
was no live/dead signal.
Use the running firecracker processes themselves as that signal: a run
dir with a live VM is protected (never killed, never removed); only run
dirs with no live process (leaked by a hard-killed launch) are reaped,
plus firecracker pids whose run dir is already gone (lingering VMMs).
This makes cleanup safe to run alongside live bottles and lets it back a
periodic GC for the run-dir leak the launch.py teardown fix closes on the
clean-exit path.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Each Firecracker bottle created its run dir (holding an ~1G
rootfs.ext4) but never removed it — launch teardown only terminated
the VM. Every run leaked its rootfs, filling the CI runner's disk
(187 stale sandbox-tester dirs / 141G in one incident).
Register a run-dir rmtree callback on the ExitStack before the
vm.terminate callback so, LIFO, the VM is gone before its rootfs is
removed. ignore_errors keeps teardown resilient.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
FROM-scratch images (commit_container output) and registry images built
for reproducibility often omit the created field entirely. Both backends
were calling die() in that case, crashing the cached-image quickstart
before any container started.
image_created_at now returns datetime | None — None when the field is
absent or unparseable, die() only on real inspect failures (non-zero
exit, malformed JSON). stale_checks in both backends skips the staleness
check when None is returned.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
pyright strict reportUnusedImport flagged BottleImages in
test_docker_launch_committed_image.py and test_macos_container_launch.py;
neither file references the type by name (they only use the returned
value's attributes).
`from __future__ import annotations` already defers all annotation
evaluation, so quoting `str | Path`, `BottleImages` inside the same
module was redundant and tripped pyright strict mode.
Addresses review comments 3098, 3099, 3100 on PR #336:
- Add BottleImages(agent, sidecar) dataclass to backend/__init__.py.
Docker/macOS backends use str image refs; smolmachines uses Path
artifacts. Replaces the singular `image` variable from the canonical
pattern in comment 3100.
- Replace _image_stale_checks/skip_stale with public prelaunch_checks().
CLI now calls backend.prelaunch_checks(plan) before backend.launch(plan);
if StaleImageError is raised and the operator confirms, launch proceeds
without re-checking. Removes the while-True/skip_stale retry loop.
- Add abstract _build_or_load_images(plan) -> BottleImages to
BottleBackend. launch() calls it then passes images to _launch_impl.
Each backend implements both methods.
- Fix comment 3098 (macos-container): _build_images is removed.
build_or_load_images() has separate fresh/cached code paths — the
cached path never calls a build helper.
- Update _start_bundle (smolmachines) to accept sidecar_artifact: Path
directly. Sidecar artifact resolution moves to _sidecar_from_path(),
called by build_or_load_images alongside _agent_from_path().
- image_cache: StaleImageError exception + check_stale/check_stale_path (raise instead of warn)
- BottleBackend.launch: template method (skip_stale flag) that calls _image_stale_checks then _launch_impl
- Each backend: _image_stale_checks delegates to a stale_checks() function in its launch module; _launch_impl replaces launch override
- macos_container: adds image_created_at to util, cached-image support in _build_images, stale_checks
- cli/start.py: catches StaleImageError, prompts interactively, retries with skip_stale=True; headless mode dies on it
coverage run's --data-file flag can be overridden or ignored in some
runner environments (Nix Python, older act-based runners). Switching to
the COVERAGE_FILE env var with an absolute ${{ github.workspace }} path
ensures coverage.py writes to a known location in every runner context,
so upload-artifact can find the file.
Also adds --reuse-published to the infra build step: if the artifact for
this content hash already exists in the registry, download it instead of
running the full docker build → mke2fs → gzip pipeline.
Extract the shared CA cert polling loop into `backend/util.poll_ca_cert`
(firecracker and macos backends were duplicating deadline/sleep/raise logic).
Each caller now wraps a fetch lambda and converts TimeoutError to its own
error type. Also corrects the PRD port publication line from {port}:{port}
to {host_port}:8099.
Collapses the two-container Docker model (gateway + orchestrator) into one
bot-bottle-infra container, matching the macOS and Firecracker backends.
- Dockerfile.infra: now a shared gateway+orchestrator base (COPY bot_bottle
from orchestrator build, no CMD override)
- Dockerfile.infra.fc: new Firecracker-specific layer (buildah/crun/netavark)
- gateway_init: adds orchestrator daemon with _OPT_IN_DAEMONS gating so it
only starts when BOT_BOTTLE_GATEWAY_DAEMONS explicitly includes it
- orchestrator/lifecycle: OrchestratorService manages one infra container;
builds orchestrator (intermediate) then infra; live source bind-mounted at
/bot-bottle-src with PYTHONPATH so the subprocess uses the checkout
- backend/consolidated_util: extracts provision_bottle + teardown_consolidated
shared across all three backends; removes duplication in docker/fc/macos
consolidated_launch modules
- firecracker/infra_vm: builds four images (orchestrator→gateway→infra→infra.fc)
- All unit tests updated and passing (1878 tests)
- PRD status: Draft → Active
If resolve_teardown_timeout() raised after launch_consolidated()/register_agent()
returned, the teardown callback was never registered, leaving the bottle
registered with no cleanup path. Resolve the timeout into a local variable
before the registration call so any failure aborts before the bottle exists.
Add tests covering OrchestratorConfigStore, resolve_teardown_timeout priority
ordering (env > db > default), and the source-order invariant that the resolver
runs before registration in all three backends.
Resolves via ENV VAR -> orchestrator DB config -> default (30 s, up from 5 s):
BOT_BOTTLE_ORCHESTRATOR_TEARDOWN_TIMEOUT_SECONDS
teardown_timeout_seconds key in new orchestrator_config table (bot-bottle.db)
New OrchestratorConfigStore (same DbStore/TableMigrations pattern as the
registry) stores the DB-level setting. resolve_teardown_timeout() implements
the priority chain and is called at stack.callback registration time in all
three backends (macos_container, docker, firecracker).
A failed container list or a partial per-container inspect were
previously indistinguishable from a legitimately empty/partial live
set, so reconciliation could silently unregister healthy bottles.
- enumerate_active() now raises EnumerationError instead of returning []
when `container list` fails
- Add inspect_container_network_ip() to util, which returns None on
inspect failure (vs "" for "no DHCP address yet"), so live_source_ips
can tell the two apart
- live_source_ips() raises EnumerationError on either failure mode;
register_agent() catches it and skips reconciliation, same as
OrchestratorClientError
- Update tests: rename test_empty_when_the_cli_fails to
test_raises_when_the_cli_fails, update patching to the new function,
add coverage for list failure, per-container inspect failure, and the
launch-not-blocked path
A registry row only ever left the registry two ways: an explicit
teardown_bottle (the launcher's cleanup callback) or the same-IP supersede
sweep in register(). Neither runs when the launching CLI dies hard, so the
row outlives its container.
That orphan is not inert. Source IPs are recycled by the backend's DHCP and
by_source_ip fail-closes on ambiguity, so a leftover row at a reused address
resolves no policy at all for the next bottle that lands there — and a
bottle with no policy denies every host, which surfaces to the agent as
"host X is not in the allowlist" for hosts that were never the problem.
Add reap_absent/reconcile and call it from the macOS launch path before
registering, so each launch self-heals the registry. Restores the invariant
the data plane needs: at most one active row per live address, and none for
a dead one. The second half matters as much as the first — when several rows
claim a *live* address the newest wins and the rest are swept, otherwise a
recycled address stays ambiguous, which is exactly the bricked state.
The host supplies the live set because the orchestrator runs inside the
infra container and cannot see the backend. A grace window exempts rows
younger than it, so reconciliation cannot race a bottle still coming up, and
a reconcile failure is logged rather than blocking an otherwise-fine launch.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The shared gateway's address is DHCP-assigned and changes whenever the infra
container is recreated — a source-hash bump, an image upgrade, a crash. Every
agent-facing URL embedded that address, and the proxy URL reaches the agent
as process environment at `container exec` time. A running process's environ
cannot be rewritten from outside, so a moved gateway stranded every running
bottle permanently: not degraded, unreachable, until relaunched and its agent
session thrown away.
Give the agent a stable name instead. `GATEWAY_HOSTNAME` replaces the address
in the egress proxy URL, NO_PROXY, git-http, and supervise URLs, and resolves
through the bottle's own /etc/hosts. Unlike environ that is a file, so it can
be rewritten inside a container that is already running — which is the whole
point: a gateway that returns at a new address is picked up by live bottles.
Launch writes the entry before anything execs (every agent URL names the
gateway, so it must resolve for the first connection), and re-points every
running bottle once the gateway is up, so one stranded by an earlier restart
re-attaches instead of needing a relaunch.
The write needs root and the agent runs as `node`: the host can repoint a
bottle's gateway name, the agent cannot repoint its own. Keep that asymmetry.
Apple Container 1.0 has no container-name DNS on a user network and
`container run` has no --add-host, so the entry is written by exec after
start rather than declared at run.
Does not address the other half of #443: per-bottle egress auth tokens are
held in memory by the orchestrator and are still lost across a restart, so a
re-attached bottle resolves its policy but not its injected credentials.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
`container exec --env` does not override the run-time environment on Apple
Container — it appends. The launch path baked a token-less `*_PROXY` into
`container run` and relied on the exec-time, token-bearing value superseding
it, so the agent's `environ` ended up with two `HTTPS_PROXY` entries,
token-less first.
Which entry a runtime reads is then luck. Node reads the last, so Claude
bottles picked up the token and worked. Rust's `std::env::var` reads the
first, so Codex proxied with no identity token at all; `/resolve` requires a
matching (source_ip, identity_token) pair and fail-closes, so every request
from a Codex bottle was denied — its model calls, its `wss://` reconnects,
and its MCP servers alike. The registry row was correct the whole time,
which is what made this read as a registration bug.
Drop the run-time proxy vars entirely. A token-less proxy URL has no
legitimate consumer: the init process is `sleep` and everything that
egresses arrives by exec. Its only benefit was a tidy 403 for unattributed
callers, which is not worth silently dropping attribution for — and a
process that egresses before the exec-time env still fails closed, since the
agent network is host-only and the gateway is the only route off it.
Also corrects the two comments that asserted the false "exec --env wins"
invariant, so the next reader doesn't rebuild the same assumption.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
_auto_select_backend gains a prompt parameter (default True). When
prompt=False the docker-fallback [i/d/q] menu is skipped and the call
dies immediately with an actionable message ("set
BOT_BOTTLE_BACKEND=docker or install a VM backend"), preventing hangs
in CI, webhook dispatch, and orchestrator launches.
prepare_with_preflight passes prompt=not spec.headless so the headless
start path can never block waiting for TTY input it cannot receive.
Remove the private _read_tty_line duplicate from backend/__init__.py and
the local definition from cli/_common.py. Both now import from the
shared bot_bottle.util module.
- Remove hardcoded --backend=macos-container flag reference in
firecracker/util.py require_firecracker() error message
- Remove --backend flag from cli.py start; backend selection now
driven exclusively by BOT_BOTTLE_BACKEND env var or auto-selection
- Skip unavailable backends in cli.py cleanup (fixes crash on Linux
when macos-container.prepare_cleanup calls require_container())
- Add two-tier auto-selection: VM backend first (macos-container on
macOS, firecracker on Linux+KVM); fall back to docker with a
security warning and interactive i/d/q prompt; exit if docker
also unavailable and print VM install instructions
Closes#344
The macOS infra container's init script still spawned the supervisor as
`python3 /app/gateway_init.py`. 5ad3449 moved bot_bottle from flat files
under /app into a pip-installed package, so that file no longer exists:
the supervisor never started, mitmdump never generated its CA, and the
launch failed downstream with a misleading "gateway CA not available"
error pointing at TLS rather than at the supervisor.
This is the same defect 127ba49 fixed for the firecracker backend. It
survived on macOS because CI has no Apple Container runner — the KVM
runner added in c193b04 covers firecracker, and ubuntu-latest covers
docker, but nothing exercises macos_container.
test_macos_infra asserted the substring "gateway_init.py", which the
broken path satisfied; it now asserts the module form, matching the
assertion 127ba49 already made in test_firecracker_infra_vm.
Verified end to end on the macos-container backend: image builds, CA is
generated, agent registers, and a headless bottle runs a prompt.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Review follow-ups for the infra candidate-artifact flow:
1. publish-infra would fail on the first merge to main. The rootfs version now
hashes the dropbear bytes, and build-infra sets BOT_BOTTLE_FC_DROPBEAR to the
staged dropbear — but publish-infra (ubuntu-latest) set none, so _publish_
bundle re-derived the version with a "<missing>" dropbear and rejected the
candidate as "does not match checkout". Download the same firecracker-inputs
dropbear and export BOT_BOTTLE_FC_DROPBEAR in publish-infra (and add
stage-firecracker-inputs to its needs, since it now consumes that artifact).
2. Guard stage-firecracker-inputs with the same fork-PR check as the other
KVM-runner jobs, so a fork PR can't spin the privileged runner (it only
copies a static binary, but keep the posture consistent; it gates the whole
Firecracker chain via needs).
3. ensure_artifact_gz / _publish_bundle read version.txt before checking it
exists — a bundle missing it raised a raw FileNotFoundError instead of the
intended "bundle is incomplete" die. Guard the read (kept before the gz/sha
completeness check so a wrong-version bundle still reports the version
mismatch, not "incomplete").
4. test_infra_artifact.setUp didn't isolate BOT_BOTTLE_INFRA_ARTIFACT_DIR, so
an ambient candidate dir (the coverage job exports one) would send the
registry-pull tests down the local-bundle path. Pin it off in setUp; the
candidate-path cases set it explicitly.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01A9qa3xoavjQScufDfZaXKR
Dockerfile.gateway COPYs pyproject.toml into /src and runs pip install
/src, so it is a real input to the baked rootfs. A dependency-only change
previously reused stale artifact versions, potentially booting a rootfs
whose installed packages differed from the current checkout.
Also adds _fake_repo fixture support and a regression test so this input
can't silently drop out of the hash again.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
build_agent_rootfs_dir cached the built rootfs by Dockerfile content alone,
but util.inject_guest_boot then writes util._GUEST_INIT into it. So a fix to
the init — making /tmp world-writable (1777) so the agent can create scratch
dirs / git worktrees there — did NOT bust the cache: the KVM runner kept
reusing a stale agent-<dockerfilehash> rootfs built with the old init, and the
sandbox-escape README-push test kept failing at `git init /tmp/...` with
"Permission denied".
Key the cache on Dockerfile content AND the injected init (_rootfs_digest), so
an init change rebuilds. Self-busting: the new key yields a fresh cache dir, so
no manual cache clear on the runner.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01A9qa3xoavjQScufDfZaXKR
The infra VM is a per-host singleton that outlives short-lived launchers, so
`ensure_running` adopts it when its control plane is healthy. But it adopted
ANY healthy VM regardless of the code that built it — so after an infra-code
change the old VM kept being adopted and the new code never booted. The only
way to dislodge it was an out-of-band `kill`, which then raced whatever
launched next. On CI this meant every infra change needed a manual VM kill.
Make adoption version-aware:
* boot records the infra-artifact version it booted from in a `booted-version`
marker beside the singleton; `stop` clears it.
* `ensure_running` adopts only when the marker matches the current version
(`_adoptable`); a missing/mismatched marker falls through to stop + reboot.
So a stale VM is replaced automatically on the next launch — no manual kill,
and it's concurrency-safe (the reboot happens under the singleton flock).
Also harden teardown: the PID file drifts after crashes / out-of-band kills,
so `stop` now also reaps any orphaned firecracker still bound to the infra
config path (scoped to that path, so interactive-pool VMs are untouched) —
otherwise a survivor holds the orchestrator TAP and the fresh boot dies with
"tap … Resource busy".
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01A9qa3xoavjQScufDfZaXKR
The rootless agent rootfs build can land /tmp as 0755/root-owned, so the
agent (uid 1000 node) can't create scratch dirs there. The sandbox-escape
suite's README-push test does `cd /tmp && git init sandbox-escape-repo` and
died with "cannot mkdir sandbox-escape-repo: Permission denied" — before the
git-gate gitleaks hook could run — so the test read it as a missing hook.
On docker the agent inherits node:22-slim's 1777 /tmp, which is why only the
Firecracker path was affected (and only now that the suite runs end-to-end).
Set /tmp to 1777 in the guest PID-1 init, so every agent VM boots with a
usable /tmp regardless of rootfs perm drift.
Also update the infra-init unit test to assert the gateway launches via the
`bot_bottle.gateway_init` module (matching the prior fix), not a file path.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01A9qa3xoavjQScufDfZaXKR
The infra VM's init boots the orchestrator control plane AND the gateway data
plane (egress/git-http/supervise). Since 5ad3449 moved the daemons into the
installed `bot_bottle` package, there is no `/app/gateway_init.py` file — the
gateway image's entrypoint is `python3 -m bot_bottle.gateway_init`. But the
firecracker init still spawned the old file path, so the guest logged:
python3: can't open file '/app/gateway_init.py': No such file or directory
The control plane came up (it already used `python3 -m bot_bottle.orchestrator`)
but the gateway never started, so mitmproxy never generated its CA and launch
died with "gateway CA not available after 30s". Mirror the orchestrator line:
run the gateway as `python3 -m bot_bottle.gateway_init` (bot_bottle resolves
from the /app CWD, same as the control plane; egress-entrypoint.sh /
egress_addon.py are present from the gateway base image).
Third build/runtime bug from 5ad3449's package refactor that the Firecracker
integration suite never exercised (the artifact publish was broken, so the
job 404'd before boot). Changes the init, so the infra artifact version moves;
the matching rootfs has been rebuilt and published.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01A9qa3xoavjQScufDfZaXKR
On a self-hosted KVM runner the process has a real controlling terminal
so name_color_modal successfully opens /dev/tty and enters a curses
loop waiting for keyboard input, hanging the test indefinitely.
Docker containers (ubuntu-latest runners) don't have a real /dev/tty,
causing an OSError that triggers the existing fallback — this is why
the hang was invisible in ubuntu-latest CI.
Also add timeout=5 to _daemon_reachable() to match the same defensive
fix already applied to docker_available() in tests/_docker.py.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Rename _BACKENDS → _backends: pyright treats uppercase module-level
names as constants and flags the reassignment in _get_backends() as
reportConstantRedefinition; lowercase avoids this.
- Add TYPE_CHECKING guard importing CommitCancelled/Freezer/get_freezer
from .freeze: pyright cannot see module-level __getattr__ bindings, so
reportUnsupportedDunderAll fired for those three __all__ entries; the
guard makes them visible to the type checker without running at import
time.
- Update test_backend_selection.py to patch _backends (lowercase).
`undefined-all-variable` fires on CommitCancelled / Freezer / get_freezer
in __all__ because pylint can't see module-level __getattr__ bindings;
`global-statement` fires on the _BACKENDS singleton setter. Both are
intentional patterns — add inline disables rather than suppress globally.
Importing backend.docker.util previously triggered eager loading of all
three backend packages (~76 modules) because backend/__init__.py imported
DockerBottleBackend, FirecrackerBottleBackend, and MacosContainerBottleBackend
at module scope. This made the module prohibitively expensive to import
from the orchestrator layer and elsewhere.
The three backend imports are now deferred into _get_backends(), which
loads all three on first call and caches the result in the module-level
_BACKENDS variable (initially None). Module-level __getattr__ exposes
backend classes and freeze symbols lazily for existing import/patch sites.
backend/docker/util.py raw subprocess.run(["docker", ...]) calls are
replaced with the shared run_docker primitive from docker_cmd, eliminating
the duplication between the backend and orchestrator implementations.
_silent_run() is removed; image_exists() is inlined directly onto
run_docker. The commit_container test is updated to patch run_docker
instead of subprocess.run.
Cloning/fetching from the git-gate on the Apple-container backend failed with
"empty reply from server" (curl exit 52). Root cause: the git-http handler
crashed on every upload-pack with
PermissionError: [Errno 13] Permission denied: '/etc/git-gate/access-hook'
The access-hook is exec'd directly, so it needs the x bit. prepare() stages it
0o700 and trusted the gateway copy to carry that mode. `docker cp` does; the
Apple `container cp` (AppleGatewayTransport) does not, landing the hook 0o644 →
EACCES. The unhandled exception killed the handler thread, closing the socket
with no HTTP response — which the client sees as the opaque empty reply.
- provision_git_gate now `chmod +x`es the access-hook on the gateway side after
the copy, so it's executable under every transport (docker/apple/firecracker).
- git-http handler wraps the access-hook subprocess.run: an OSError /
SubprocessError (un-execable, timed out) now fails closed with a 503 instead
of crashing the thread into an empty reply — a gate that can't run its hook
should deny, visibly.
- Updates the now-misleading "docker cp preserves source mode" comment in
git_gate.prepare().
Regression tests: provisioning applies +x to the access-hook; the handler
returns 503 (not an empty reply) when the hook can't be exec'd.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The control plane had no caller authentication — dispatch() routed every
request straight through, and the only guards anywhere were the
(source_ip, identity_token) checks inside /resolve and /attribute, which
authenticate the *bottle* a request is about, not the *caller*. On the
consolidated backends the agent shares a network with the control-plane port
(macOS: the infra container runs 8099 on the agent's --internal network;
docker: the orchestrator container sits on the shared gateway network), so a
prompt-injected agent could GET /bottles (enumerate siblings + policies),
PUT /bottles/<id>/policy (rewrite its own egress allowlist to allow-all),
POST /resolve (read the injected upstream credentials it's never meant to
see), and POST /supervise/respond (self-approve its own proposals). Every
README guarantee collapsed to the agent's discretion.
Fix: a per-host control-plane secret required on every route but GET /health,
compared with hmac.compare_digest. It is held only by the trusted callers and
never handed to an agent:
- minted + persisted 0600 at <root>/control-plane-token (paths.host_control_plane_token);
- injected as $BOT_BOTTLE_CONTROL_PLANE_TOKEN into the orchestrator + gateway
containers via bare `--env NAME` (value inherited from the launch process,
so it never lands on argv or in `container/docker inspect`);
- presented by the gateway's PolicyResolver (reads the env) on /resolve, and by
the host CLI's OrchestratorClient (reads the host file) on every call.
The agent container is never given the env var or the host file, so from a
bottle every /bottles*, /resolve, /attribute, and /supervise/* call now
returns 401 — closing the enumeration, allowlist-rewrite, credential-lift, and
self-approval. The existing (source_ip, identity_token) checks stay as
defense-in-depth.
Enforced when configured: macOS + docker inject the secret (→ enforced). With
no secret set the server runs open and warns loudly at startup — a
fail-visible fallback for the unit suite and for Firecracker, whose
port-scoped nft already blocks agents from 8099 (wiring the secret into its
infra-VM init is a clean fast-follow, left out here to avoid churning the
prebuilt-artifact hash).
Verified end-to-end on real Apple Container: infra comes up healthy, the host
CLI (with the secret) lists bottles while an unauthenticated GET /bottles gets
401, all five issue-#400 attacks from inside the agent get 401, and egress
policy still works (200 allowed / 403 denied) — proving the gateway
authenticates to /resolve with the secret. 1829 unit tests pass, pyright
clean, pylint 9.91.
Refs #400.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Adopts the firecracker infra-VM pattern for macOS: the orchestrator control
plane and the gateway data plane now run in a SINGLE Apple container instead of
two. Apple Containers are lightweight VMs with separate kernels, so the prior
two-container design had both guests writing one bot-bottle.db over virtiofs,
where fcntl locks are not coherent across kernels — concurrent writes (the
orchestrator's registry vs the gateway supervise daemon's queue) could corrupt
it. One container = one kernel = coherent locking.
The DB moves onto a container-only Apple volume (bot-bottle-mac-db), never
bind-mounted from the host, so no host process opens the live file either. The
host CLI already reaches registry + supervise state over the control-plane HTTP
surface (cli/supervise.py uses OrchestratorClient), exactly as firecracker's
VM-only DB requires.
Two simplifications fall out of the single container:
- No DNS dance: the control plane and gateway daemons reach each other over
127.0.0.1, so the orchestrator-before-gateway ordering (a workaround for
Apple having no container DNS) is gone, along with the moved-IP recreate
logic it needed.
- Net -243 lines.
Mechanics: the infra container runs from the gateway image with the
control-plane source bind-mounted read-only (like the docker orchestrator, so a
code change needs no rebuild) and a small sh -c init that starts both processes
(mirrors firecracker's _infra_init). Also implements the macOS backend's
ensure_orchestrator() and adds it to discover_orchestrator_url, so operator
tools (supervise) can bring up / find the control plane on demand — previously
the macOS backend died with "no orchestrator control plane".
Verified end-to-end on real Apple Container 1.0.0: the single infra container
comes up healthy (one address for control plane + gateway), both processes run,
the DB is written on the container-only volume, host-side supervise works over
HTTP, and a registered agent gets 200 for an allowed host / 403 for a denied
one. 1824 unit tests pass with `container` absent (CI parity), pyright clean,
pylint 9.89.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Addresses findings from a high-effort review of the PRD 0070 macOS backend.
Correctness:
- Stamp identity_token onto MacosContainerBottlePlan after registration. git's
gitconfig extraHeader and the supervise MCP --header read
getattr(plan,"identity_token","") at provision time, and both reach the
gateway on NO_PROXY (bypassing the egress proxy that carries the token). The
plan never carried it, so /resolve fail-closed and every git fetch/push and
supervise call from a macOS bottle would have been denied. Registration
precedes provision(), so — unlike the run-time env — the plan can carry it.
- Self-heal the orchestrator: recreate when it is not (source-current AND
answering /health), not on the source-hash label alone. A container running
current code but with a wedged HTTP server was left alone and polled to
death, failing every launch until manual deletion.
- image_digest and container_image_digest now read the same descriptor.digest
field; dropped image_digest's id/tag fallback that could yield a value the
container side can't produce — a permanent mismatch would have recreated the
shared gateway on every launch (severing every live bottle's egress, since
the replacement gets a new DHCP address).
- Poll for the agent's and gateway's DHCP address instead of a fatal read
right after `container run` (there is no --ip; the address can lag start).
Cleanup:
- One _inspect_first + _descriptor_digest behind the four inspect readers.
- Shared bind_mount_spec (util) and host_db_dir (paths) replace per-module
copies; _GIT_HTTP_PORT now imports git_http_backend.DEFAULT_PORT.
- Drop the dead _url cache / url property and the write-only agent_proxy_url.
Deferred (noted on the PR, not fixed here): the gateway image rebuilding on
every launch (needs source-hash-labeled build), SQLite shared across VM
guests, and the sh -lc profile-override edge — each is design-level or
behavior-risk beyond a review fix.
Verified: real Apple Container bring-up is green and idempotent; 1826 unit
tests pass with `container` absent (CI parity), pyright clean, pylint 9.86.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
didericis
didericis-codex
didericis-claude