Answers the last open question in #392: DNS from inside a nested
container was not a phantom, but it was also not a DNS problem. Public
resolution fails there by design — everything egresses through the
proxy — and the actual breakage was two missing pieces:
- The proxy URL names `bot-bottle-gateway`, which resolves only through
the bottle's own /etc/hosts. Containers got their own hosts file, so
they failed at "Could not resolve proxy", which reads like broken DNS.
base_hosts_file propagates the bottle's entries.
- The gateway TLS-intercepts, so a container that does not trust the
bottle's CA bundle fails with "unable to get local issuer
certificate". The bundle is now mounted read-only and the usual env
vars point at it, which covers curl, wget, python, and node without
distro-specific trust commands.
Verified on the macOS host by reproducing each failure and confirming
that these three conditions applied by hand produced HTTP 200 from
inside a nested container. podman already forwards the proxy env, so
that needed nothing.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
netavark shells out to `nft` for the bridge network every compose file
expects, and aardvark-dns serves name resolution inside nested
containers. Neither arrives with the base image's
`--no-install-recommends` podman, and each fails at a different and
misleading layer: without nft containers are created but never start;
without aardvark-dns DNS inside a container fails while the engine, the
pulls, and the bridge all look healthy. That last one is the likeliest
explanation for the "DNS inside nested containers fails entirely" note
in #392 — a missing package, not a design gap.
Verified on the macOS host: with passt alone, `docker run` reached
netavark and died on "unable to execute nft".
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
podman 5 uses pasta as the default rootless network helper; podman 4
used slirp4netns. The spike was written against bookworm's podman 4.3.1,
so its package list never included passt — and the trixie bump (#451)
changed the default out from under it. Every nested container failed to
start with "could not find pasta", while image pulls and the service
itself worked, which disguised a missing dependency as a compat-API bug.
The bootstrap now checks for pasta up front so this fails with a clear
message instead of surfacing at the first `docker run`.
Note the sun_path fix in a12d2191 did NOT cause or cure this; that path
was over the 108-byte limit independently and would have bitten as soon
as attach was reached.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
podman derives conmon's attach socket as
`$XDG_RUNTIME_DIR/libpod/tmp/socket/<64-hex-id>/attach`. With the
descriptive `/tmp/bot-bottle-podman-run` that path is 116 bytes, past
the 108-byte sun_path limit, so every attached `docker run` failed with
"unable to upgrade to tcp, received 500" — while image pulls and the
service itself worked, which is what made it look like a compat-API bug
rather than a path-length one.
Found on the macOS host: quay.io pulled fine and then no container
could be started. Test pins the budget so a more readable name cannot
silently reintroduce it.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
merge_bottles_runtime sees resolved bottles, where a bottle that never
mentioned the key is indistinguishable from one that set it false. With
"later replaces" semantics, `--bottle with-containers --bottle claude-dev`
silently dropped the capability — the second bottle's default clobbered
it. OR instead: composing bottles adds capabilities, it does not remove
them. The file-based extends path still sees raw keys, so an explicit
`nested_containers: false` in a child continues to turn it back off.
Also surface the flag in the launch summary, where its absence was the
only reason the clobber went unnoticed.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Adds the `nested_containers` bottle flag. On the macOS backend it starts
a rootless podman service inside the bottle and exposes its
Docker-compatible API socket, so the agent still runs `docker` and
`docker compose`. No host daemon socket is mounted and the guest gains
no capabilities; backends that cannot run a guest-local engine reject
the flag in the shared prepare template rather than ignoring it.
Podman rather than rootless Docker because Apple Container's capability
bounding set omits CAP_SYS_ADMIN, which the kernel requires to write a
multi-range uid_map via newuidmap. With no subordinate UID range podman
falls back to a single-UID self-mapping an unprivileged process may
write itself, so the image build strips /etc/subuid and /etc/subgid
entries rather than adding them.
That mapping is also why nested containers are not an isolation layer:
root inside one is the agent user outside it. They are a build/test
convenience; the bottle remains the boundary.
Ports the spike branch onto main, renaming docker_access — it implied
Docker and granted access to nothing on the host — and drops podman
from the derived layer now that every built-in image ships it (#451).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
`cleanup()` enumerated *every* firecracker process under the run root and
*every* run dir, so `./cli.py cleanup` would kill running bottles and
delete their rootfs — despite being described as orphan cleanup. The
backend's `enumerate_active` registry is still a stub (#354), so there
was no live/dead signal.
Use the running firecracker processes themselves as that signal: a run
dir with a live VM is protected (never killed, never removed); only run
dirs with no live process (leaked by a hard-killed launch) are reaped,
plus firecracker pids whose run dir is already gone (lingering VMMs).
This makes cleanup safe to run alongside live bottles and lets it back a
periodic GC for the run-dir leak the launch.py teardown fix closes on the
clean-exit path.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Adds a per-route boolean field preserve_auth (default false) that skips
the gateway's Authorization header stripping for that host. Intended for
registry endpoints like Docker Hub (registry-1.docker.io) and GHCR
(ghcr.io) where the agent must supply its own per-scope bearer token.
Threaded through ManifestEgressRoute → EgressRoute → Route, serialized
in route_to_yaml_dict, and parsed in parse_routes. The strip at
egress_addon.py now checks route.preserve_auth before popping the header.
Closes#392
The shared gateway self-generates a mitmproxy CA that every bottle installs
to trust its TLS interception. It was persisted on a Docker named volume,
which survives `docker rm` but is silently wiped by `docker volume prune` /
`docker system prune --volumes` during routine host maintenance. When that
happens the gateway mints a fresh CA on restart, and every already-running
bottle fails the TLS handshake even after it re-resolves and reconnects to
the moved gateway — a re-attachment blocker distinct from #443/#445.
Move CA persistence to a host bind-mount under the app-data root
(`bot_bottle_root()/gateway-ca`, via `host_gateway_ca_dir()`), mirroring how
the shared DB and control-plane token already live on the host. Docker never
prunes a path under the root, and it stays inspectable + rotatable from the
host. mitmproxy already adopts an existing CA and generates one only on first
run, so the bind-mount gives adopt-existing/generate-on-first-run for free.
Add an explicit rollover path: `rotate_gateway_ca()` clears the persisted CA
so the next start remints it, and `python -m bot_bottle.orchestrator.rotate_ca`
wires that together with dropping the running gateway container (whose
mitmproxy still holds the old CA in memory). Rotation stays an operator action
— it doesn't auto-re-provision running bottles, which re-attach to pick up the
new anchor.
Scope: the Docker infra/gateway path (the "infra container" in the report).
The macOS (`container`-only volume) and Firecracker (VM-attached ext4) backends
persist the CA differently and are unaffected.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
`python -m bot_bottle.cli` failed with "is a package and cannot be
directly executed" — the package had a `if __name__ == "__main__"` block
in `__init__.py`, which never fires for a package and made the invocation
look supported when it wasn't. Add a real `__main__.py` and drop the dead
block.
Matters for `bb login`, whose docstring documents a `bb` entry point that
nothing installs; `-m` is the closest thing to it until there's a
console-script.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Replaces broad `except Exception` with specific types that reflect the
actual failure modes:
- _save_credentials: `except OSError` (IO-only path; re-raises for cleanup)
- _post error handler: `except (OSError, ValueError)` (network + bad JSON)
- poll-loop: `except (OSError, ValueError)` (network + bad JSON)
Also:
- Simplify _fake_get to use next(..., default) — removes uncovered
StopIteration branch
- Add encoding="utf-8" to open() in test_approved_flow_returns_0
- Add test_cleanup_on_write_failure to cover the except OSError block
in _save_credentials
All files score 10.00/10 on pylint (fail-under=10) and 0 errors on
pyright strict.
Write credentials via a 0600 temp file + os.replace() so the token file
never appears at its final path with world-readable permissions, even if
the process is interrupted between write and chmod.
Parse poll_interval from the authorization response (clamped to 1–60 s,
falling back to _POLL_SLEEP) so aggressive polling can't trigger console
rate limits.
Tests: add atomicity spy asserting the temp file is 0600 before replace;
patch time.sleep instead of _POLL_SLEEP; add explicit interval-passthrough
assertion.
Starts a device-authorization flow against a bot-bottle console, polls
until the operator approves, then writes access + refresh tokens to
$BOT_BOTTLE_ROOT/console.json. Console URL is read from --console-url
flag or BB_CONSOLE_URL env var.
Part of didericis/bot-bottle-platform#1
Two issues introduced by the rebase conflict resolution:
- test_cached_lookup_requires_ready_marker used _dockerfile_hash as the
cache-dir key, but cached_agent_rootfs_dir now uses _rootfs_digest (which
also folds in the guest init). Updated the test to match.
- test_pyproject_toml_change_bumps_version appeared twice in
test_infra_artifact.py (once from main, once from the PR commit that
added pyproject.toml support). Removed the duplicate.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Dockerfile.gateway COPYs pyproject.toml into /src and runs pip install
/src, so it is a real input to the baked rootfs. A dependency-only change
previously reused stale artifact versions, potentially booting a rootfs
whose installed packages differed from the current checkout.
Also adds _fake_repo fixture support and a regression test so this input
can't silently drop out of the hash again.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
FROM-scratch images (commit_container output) and registry images built
for reproducibility often omit the created field entirely. Both backends
were calling die() in that case, crashing the cached-image quickstart
before any container started.
image_created_at now returns datetime | None — None when the field is
absent or unparseable, die() only on real inspect failures (non-zero
exit, malformed JSON). stale_checks in both backends skips the staleness
check when None is returned.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- config_store.py imported host_db_path from supervise_types (wrong);
it lives in paths.py, matching all other stores (audit, queue, etc.)
- test_stale_checks: two tests expected check_stale called twice
(agent + sidecar) — consolidated arch has no sidecar, so once is
correct; update assertions and remove unused Path import
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
pyright strict reportUnusedImport flagged BottleImages in
test_docker_launch_committed_image.py and test_macos_container_launch.py;
neither file references the type by name (they only use the returned
value's attributes).
Addresses review comments 3098, 3099, 3100 on PR #336:
- Add BottleImages(agent, sidecar) dataclass to backend/__init__.py.
Docker/macOS backends use str image refs; smolmachines uses Path
artifacts. Replaces the singular `image` variable from the canonical
pattern in comment 3100.
- Replace _image_stale_checks/skip_stale with public prelaunch_checks().
CLI now calls backend.prelaunch_checks(plan) before backend.launch(plan);
if StaleImageError is raised and the operator confirms, launch proceeds
without re-checking. Removes the while-True/skip_stale retry loop.
- Add abstract _build_or_load_images(plan) -> BottleImages to
BottleBackend. launch() calls it then passes images to _launch_impl.
Each backend implements both methods.
- Fix comment 3098 (macos-container): _build_images is removed.
build_or_load_images() has separate fresh/cached code paths — the
cached path never calls a build helper.
- Update _start_bundle (smolmachines) to accept sidecar_artifact: Path
directly. Sidecar artifact resolution moves to _sidecar_from_path(),
called by build_or_load_images alongside _agent_from_path().
- image_cache: StaleImageError exception + check_stale/check_stale_path (raise instead of warn)
- BottleBackend.launch: template method (skip_stale flag) that calls _image_stale_checks then _launch_impl
- Each backend: _image_stale_checks delegates to a stale_checks() function in its launch module; _launch_impl replaces launch override
- macos_container: adds image_created_at to util, cached-image support in _build_images, stale_checks
- cli/start.py: catches StaleImageError, prompts interactively, retries with skip_stale=True; headless mode dies on it
Reads the host's Claude OAuth session key from ~/.claude.json at launch
and forwards it only to the egress sidecar (never to the agent), placing
a placeholder CLAUDE_CODE_OAUTH_TOKEN in the agent env so Claude Code
starts without seeing the real credential.
Mirrors the existing Codex forward_host_credentials flow (PRD 0029).
Adds claude_auth.py to extract and validate the sessionKey, a
CLAUDE_HOST_CREDENTIAL_TOKEN_REF constant in egress.py, and updates
manifest_agent.py to allow the flag for both 'codex' and 'claude'
templates. Also adds a mutual-exclusion check that rejects setting
both auth_token and forward_host_credentials together.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
fix(claude): read credentials from ~/.claude/.credentials.json
The actual OAuth token is in ~/.claude/.credentials.json under
claudeAiOauth.accessToken, not in ~/.claude.json.
~/.claude.json holds only UI state and profile metadata (oauthAccount
has no token fields). expiresAt in the credentials file is milliseconds,
not seconds.
Discovered after testing against Claude Code 2.1.198.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
fix(claude): fall back to macOS Keychain for credentials
On macOS, Claude Code stores credentials in the Keychain under
service "Claude Code-credentials" rather than in a file. When
~/.claude/.credentials.json is absent, shell out to:
security find-generic-password -s "Claude Code-credentials" -w
and parse the result as the same JSON schema.
~/.claude.json holds only profile/UI metadata (oauthAccount has
no token fields). expiresAt in the credentials is milliseconds.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
docs(prd): fix credential path references (~/.claude/.credentials.json)
fix(test): suppress gitleaks false positives on synthetic Claude tokens
Add tests for the three uncovered paths introduced by the poll_ca_cert
extraction: the timeout + sleep branches in backend/util, and the
TimeoutError → GatewayError and TimeoutError → die() conversions in the
macOS and Firecracker callers.
- `_build_images()` now builds Dockerfile.gateway → Dockerfile.orchestrator
→ Dockerfile.infra in order; Dockerfile.infra starts FROM bot-bottle-gateway
so the base must exist on clean hosts.
- Publish mapping corrected from `self.port:self.port` to `self.port:DEFAULT_PORT`
(8099) — gateway_init hardcodes the orchestrator on port 8099 inside the
container, so the host-side published port must map to that fixed internal port.
- BOT_BOTTLE_ORCHESTRATOR_URL inside the container now always points to
127.0.0.1:8099, not self.port, since gateway daemons reach the orchestrator
over loopback at the fixed internal port.
- Update test_ensure_running_builds_both_images → _all_images for the new
three-step build sequence; add test_publish_maps_host_port_to_fixed_internal_port
to lock in the port-mapping fix.
Addresses the P1 findings from the didericis-codex review on PR #432.
Add three new tests:
- noop when healthy but docker inspect fails (returns True → don't churn)
- build failure raises GatewayError
- _ensure_network creates the network when it doesn't exist
Also update the integration test to use new OrchestratorService API
(infra_name/image instead of orchestrator_name/gateway_name/gateway_image).
Brings diff-coverage from 86% to 90.3% against origin/main.
- Remove unused INFRA_IMAGE import from test_orchestrator_lifecycle
- Update integration test to use new single-container OrchestratorService
API (infra_name/image replaces orchestrator_name/gateway_name/gateway_image)
- Move type: ignore to the lambda line in gateway_init SIGHUP handler
- Break two long lines in test_orchestrator_lifecycle
Collapses the two-container Docker model (gateway + orchestrator) into one
bot-bottle-infra container, matching the macOS and Firecracker backends.
- Dockerfile.infra: now a shared gateway+orchestrator base (COPY bot_bottle
from orchestrator build, no CMD override)
- Dockerfile.infra.fc: new Firecracker-specific layer (buildah/crun/netavark)
- gateway_init: adds orchestrator daemon with _OPT_IN_DAEMONS gating so it
only starts when BOT_BOTTLE_GATEWAY_DAEMONS explicitly includes it
- orchestrator/lifecycle: OrchestratorService manages one infra container;
builds orchestrator (intermediate) then infra; live source bind-mounted at
/bot-bottle-src with PYTHONPATH so the subprocess uses the checkout
- backend/consolidated_util: extracts provision_bottle + teardown_consolidated
shared across all three backends; removes duplication in docker/fc/macos
consolidated_launch modules
- firecracker/infra_vm: builds four images (orchestrator→gateway→infra→infra.fc)
- All unit tests updated and passing (1878 tests)
- PRD status: Draft → Active
Switch orchestrator_config from a generic key/text store to a typed
single-row table with a dedicated teardown_timeout_seconds REAL column.
The CHECK (id = 1) constraint enforces at most one config row.
Remove TEARDOWN_TIMEOUT_KEY constant; update OrchestratorConfigStore to
expose get/set/delete_teardown_timeout_seconds() instead of the generic
get/set/delete(key). Resolving the DB value in resolve_teardown_timeout()
no longer needs a float() conversion since the column is already REAL.
If resolve_teardown_timeout() raised after launch_consolidated()/register_agent()
returned, the teardown callback was never registered, leaving the bottle
registered with no cleanup path. Resolve the timeout into a local variable
before the registration call so any failure aborts before the bottle exists.
Add tests covering OrchestratorConfigStore, resolve_teardown_timeout priority
ordering (env > db > default), and the source-order invariant that the resolver
runs before registration in all three backends.
The None/empty-string distinction is the invariant that prevents
live_source_ips from treating a failed inspect as a legitimately
no-address container. Add TestInspectContainerNetworkIp to util tests
covering: IP found, CIDR stripping, no address yet, absent network
list, non-zero exit, malformed JSON, and unexpected JSON shape.
A failed container list or a partial per-container inspect were
previously indistinguishable from a legitimately empty/partial live
set, so reconciliation could silently unregister healthy bottles.
- enumerate_active() now raises EnumerationError instead of returning []
when `container list` fails
- Add inspect_container_network_ip() to util, which returns None on
inspect failure (vs "" for "no DHCP address yet"), so live_source_ips
can tell the two apart
- live_source_ips() raises EnumerationError on either failure mode;
register_agent() catches it and skips reconciliation, same as
OrchestratorClientError
- Update tests: rename test_empty_when_the_cli_fails to
test_raises_when_the_cli_fails, update patching to the new function,
add coverage for list failure, per-container inspect failure, and the
launch-not-blocked path
didericis
didericis-claude
didericis-codex