Compare commits

..

108 Commits

Author SHA1 Message Date
didericis-claude 65a49a239e firecracker status(): open /dev/kvm O_RDWR; check kernel, dropbear, mke2fs
test / integration-docker (pull_request) Successful in 19s
tracker-policy-pr / check-pr (pull_request) Successful in 19s
test / unit (pull_request) Successful in 40s
lint / lint (push) Successful in 58s
test / integration-firecracker (pull_request) Successful in 3m20s
test / coverage (pull_request) Successful in 22s
test / publish-infra (pull_request) Has been skipped
Two reviewer findings addressed:

1. _kvm_accessible() now opens /dev/kvm with O_RDWR|O_CLOEXEC instead
   of read-only. VM creation requires write access; a read-only
   descriptor can satisfy KVM_GET_API_VERSION but fails at boot time.

2. status() now checks every hard prerequisite that require_firecracker()
   checks at launch: guest kernel image, static dropbear binary, and
   mke2fs. Previously a host with a configured TAP pool but missing
   artifacts could pass status() and then fail during launch.

Regression coverage: TestFirecrackerKvmCheck gains test_kvm_open_rdwr_fails;
TestFirecrackerArtifactCheck covers each missing artifact; existing
TestFirecrackerStatus fixtures updated to stub the new checks.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-07-24 16:03:51 +00:00
didericis-claude 755a11a608 firecracker status(): add binary and KVM readiness checks
tracker-policy-pr / check-pr (pull_request) Successful in 11s
test / integration-docker (pull_request) Successful in 19s
lint / lint (push) Successful in 1m0s
test / unit (pull_request) Successful in 1m43s
test / integration-firecracker (pull_request) Successful in 3m22s
test / coverage (pull_request) Successful in 16s
test / publish-infra (pull_request) Has been skipped
Adds `_firecracker_binary_ok()` (runs `firecracker --version`) and
`_kvm_accessible()` (opens /dev/kvm, issues KVM_GET_API_VERSION ioctl)
and gates `status()` on both, so the launch preflight reports missing
binary or inaccessible KVM before the caller ever attempts a boot.

Regression tests in TestFirecrackerBinaryCheck, TestFirecrackerKvmCheck,
and TestFirecrackerStatusRuntime cover all branches. Updated the
pre-existing TestFirecrackerStatus fixture to also stub the two new
helpers so it still exercises only the tap-pool/nft logic it was
designed for.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-07-24 06:51:58 +00:00
didericis-claude e074c6959f fix: add missing type annotations and remove unused variables in new tests
test / integration-docker (pull_request) Successful in 23s
test / unit (pull_request) Successful in 47s
lint / lint (push) Successful in 3m3s
test / integration-firecracker (pull_request) Successful in 3m24s
test / coverage (pull_request) Successful in 20s
test / publish-infra (pull_request) Has been skipped
tracker-policy-pr / check-pr (pull_request) Successful in 17s
Pyright flagged three inner class `status` methods missing a type
annotation on the `quiet` parameter, and two unused variables
(`written`, `real_status`) left over from an earlier draft of
test_docker_quiet_true_suppresses_stderr.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-07-24 06:18:51 +00:00
didericis-claude 05b62ce805 test: cover is_backend_available, is_backend_ready, and status(quiet=True)
test / integration-docker (pull_request) Successful in 21s
lint / lint (push) Failing after 59s
test / unit (pull_request) Successful in 1m47s
test / integration-firecracker (pull_request) Successful in 3m22s
test / coverage (pull_request) Successful in 18s
test / publish-infra (pull_request) Has been skipped
tracker-policy-pr / check-pr (pull_request) Failing after 13m0s
Add unit tests for the three new public symbols introduced by this PR:

- TestIsBackendAvailable — delegates to has_backend; returns True/False
  correctly for available and unavailable backends
- TestIsBackendReady — unknown names return False; known names return
  True/False based on status() return code; quiet flag is forwarded
- TestBackendStatusQuiet — verifies the quiet=True branch in
  DockerBottleBackend, FirecrackerBottleBackend, and
  MacosContainerBottleBackend: return code is propagated and
  diagnostic stderr is suppressed

Together these bring diff-coverage for the PR's changed lines to 100%.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-07-24 06:12:02 +00:00
didericis-claude 1f9a15dede fix: align skip-guard tests with is_backend_ready and drop unused import
tracker-policy-pr / check-pr (pull_request) Successful in 14s
test / integration-docker (pull_request) Successful in 16s
test / unit (pull_request) Successful in 37s
lint / lint (push) Successful in 53s
test / integration-firecracker (pull_request) Successful in 3m18s
test / coverage (pull_request) Failing after 35s
test / publish-infra (pull_request) Has been skipped
tests/_backend.py imported is_backend_available but never called it,
causing a pyright reportUnusedImport error. Remove the unused import.

test_backend_skip_guards.py was patching tests._backend.has_backend but
_backend.py calls is_backend_ready — the mock never intercepted the real
call, producing AttributeError at test runtime. Update all patch targets
to is_backend_ready and add quiet=False to the assert_called_once_with
assertion to match the actual call signature.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-07-24 04:25:04 +00:00
didericis-claude 0ca39cf4d5 feat(backend): BackendStatus enum, quiet status(), is_backend_ready()
tracker-policy-pr / check-pr (pull_request) Successful in 11s
test / integration-docker (pull_request) Successful in 17s
lint / lint (push) Failing after 54s
test / unit (pull_request) Failing after 1m33s
test / integration-firecracker (pull_request) Successful in 3m17s
test / coverage (pull_request) Has been skipped
test / publish-infra (pull_request) Has been skipped
Add a module-level BackendStatus(IntEnum) with READY=0 to replace magic
0 comparisons. Extend BottleBackend.status() with a quiet parameter:
quiet=False (default) prints diagnostics; quiet=True returns the code
silently for programmatic checks.

Add is_backend_available() (cheap PATH check, alias for has_backend) and
is_backend_ready(name, *, quiet=False) (full status() check) to the
backend package for callers that need to distinguish availability from
readiness.

Update tests/_backend.py guards to use is_backend_ready(quiet=False) so
diagnostic output is printed during test discovery, giving the operator a
concrete reason for each skip rather than a bare "prerequisites
unavailable" message.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-07-24 03:03:00 +00:00
didericis-claude d4d45f835e ci: reuse backend.has_backend / backend status for readiness
test / integration-docker (pull_request) Successful in 25s
test / unit (pull_request) Successful in 52s
lint / lint (push) Successful in 1m0s
test / integration-firecracker (pull_request) Successful in 2m5s
test / coverage (pull_request) Successful in 24s
test / publish-infra (pull_request) Has been skipped
tracker-policy-pr / check-pr (pull_request) Successful in 6s
Review feedback: the hand-rolled `/dev/kvm` + docker-daemon capability
probes duplicated logic the CLI already owns. Delegate instead.

- tests/_backend.py skip guards now gate on `bot_bottle.backend.has_backend`
  (each backend's `is_available()` classmethod — the same probe behind
  `./cli.py backend status`), dropping the bespoke `Capability` probes.
- Remove tests/backend_preflight.py; the docker integration job runs
  `./cli.py backend status --backend=docker` as its preflight (clear
  per-check summary, non-zero exit when unready), matching the firecracker
  job. The firecracker preflight reverts to its original binary/KVM checks
  (backend status doesn't cover those).
- Unit test + docs updated to match.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-24 01:42:04 +00:00
didericis-claude 9fdaba4bd4 ci: backend-agnostic integration guards + per-backend preflight
tracker-policy-pr / check-pr (pull_request) Successful in 15s
test / integration-docker (pull_request) Successful in 20s
lint / lint (push) Successful in 1m5s
test / unit (pull_request) Successful in 1m49s
test / integration-firecracker (pull_request) Successful in 2m0s
test / coverage (pull_request) Successful in 16s
test / publish-infra (pull_request) Has been skipped
Integration tests now select their backend from BOT_BOTTLE_BACKEND and
skip on the capability that backend actually needs, instead of gating
every backend on unrelated Docker availability.

Task 1 — backend-agnostic guards (tests/_backend.py):
- Capability probes: docker_capability() (reachable daemon) and
  firecracker_capability() (accessible /dev/kvm + firecracker on PATH,
  Docker-independent). backend_capability()/selected_backend() resolve
  the target from BOT_BOTTLE_BACKEND (default docker).
- skip_unless_selected_backend_available() for backend-agnostic tests
  (test_sandbox_escape) — runs through whichever backend is selected and
  checks that backend's real capability.
- skip_unless_backend("docker") for Docker-implementation tests
  (DockerBroker, DockerGateway, backend.docker.*) — they no-op under a
  non-Docker run rather than testing internals that run doesn't target.
- Retires tests/_docker.py; the KVM job no longer needs SKIP_DOCKER_TESTS
  to steer Docker-only classes.

Task 2 — explicit per-backend skip visibility:
- tests/backend_preflight.py prints a clear PASS/FAIL capability line and
  exits non-zero when the selected backend is missing.
- Both integration jobs run it as a preflight, so absent infrastructure
  is surfaced at the job level instead of hidden among unittest.skip
  lines. The docker job replaces its soft "Show environment" step; the
  firecracker job keeps its richer backend-status check.

Docs (tests/README.md, docs/ci.md) updated; unit coverage for the probes,
guards, and preflight in test_backend_skip_guards.py.

Closes #414

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-23 21:33:14 -04:00
didericis-codex f24ae45d13 test(gateway): cover single and simultaneous daemon crashes
test / integration-docker (push) Successful in 21s
lint / lint (push) Successful in 1m4s
test / unit (push) Successful in 1m44s
test / integration-firecracker (push) Successful in 4m55s
test / coverage (push) Successful in 21s
test / publish-infra (push) Successful in 1m50s
Update Quality Badges / update-badges (push) Failing after 14m42s
2026-07-23 20:15:18 -04:00
didericis-codex 594d07410a fix(gateway): restart dead children before completion check 2026-07-23 20:15:18 -04:00
didericis-claude c9c62f256d fix(egress): restart dead daemons and cap inbound scan body size (#455)
Two complementary fixes for the egress gateway OOM:

1. gateway_init: auto-restart any daemon that dies unexpectedly. The
   supervisor already had restart_daemon()/request_restart() logic; this
   wires it into tick() so an OOM-killed mitmdump is respawned without
   operator intervention. Implements the "eventual" failure policy
   described in the original module docstring.

2. egress_addon: cap response body bytes passed to the DLP inbound scan
   at EGRESS_INBOUND_SCAN_LIMIT_BYTES (default 1 MiB). mitmproxy buffers
   the full response before the hook fires; capping at scan time limits
   the additional amplification from decoded text and regex strings. Bodies
   over the limit emit an egress_scan_truncated log event. Set to 0 to
   disable the cap.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-07-23 20:15:18 -04:00
didericis 10150ae9f5 docs(paths): correct root docstring after SQLite queue migration
lint / lint (push) Successful in 2m37s
test / unit (push) Successful in 1m42s
test / integration-firecracker (push) Successful in 4m43s
test / integration-docker (push) Successful in 33s
test / coverage (push) Successful in 19s
Update Quality Badges / update-badges (push) Successful in 1m45s
test / publish-infra (push) Successful in 2m1s
The module docstring still listed "queue" and "audit logs" as things
living under the app data root, which read as directories. Both have
been tables in the shared SQLite DB since PRD 0067; the legacy `queue/`
directory was unplumbed in 29904609 and nothing has written there since.

Lists what the root actually holds and points at the stores that own the
queue/audit rows, so the next reader doesn't go looking for a directory
that isn't there.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-23 19:40:20 -04:00
Quality Badge Bot 86c7ac1843 chore: update quality badges
- Coverage: 84%
- Core coverage: 94%

[skip ci]
2026-07-23 23:29:52 +00:00
didericis-claude 2e0414f969 fix(test): restore headless unit tests broken by PTY check in e719022
test / integration-docker (push) Successful in 15s
test / unit (push) Successful in 39s
Update Quality Badges / update-badges (push) Successful in 45s
lint / lint (push) Successful in 2m36s
test / integration-firecracker (push) Successful in 5m15s
test / coverage (push) Successful in 16s
test / publish-infra (push) Successful in 1m39s
The PTY check added in e719022 calls sys.stdin.fileno() which raises
io.UnsupportedOperation under pytest's stdin capture. Guard the fileno()
call with a try/except so the check degrades cleanly to "not a tty"
instead of crashing, then stub os.isatty in the test setUp so headless
unit tests aren't blocked on a real TTY.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-07-23 23:28:55 +00:00
didericis-claude 12b071833d test: annotate _git_repo helper to satisfy pyright strict mode
test / integration-docker (push) Successful in 13s
Update Quality Badges / update-badges (push) Successful in 40s
lint / lint (push) Successful in 53s
test / integration-firecracker (push) Successful in 4m56s
test / unit (push) Failing after 33s
test / coverage (push) Has been skipped
test / publish-infra (push) Has been skipped
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-07-23 18:32:59 -04:00
didericis-claude bf72282f8e fix(manifest): avoid KeyError when override bottle has repos absent from base
merge_bottles_runtime used .get(n, base_repos_by_name[n]) which eagerly
evaluates the default, crashing when a repo exists only in the override
bottle. Replaced with a conditional expression so the base lookup only
runs when needed.

Adds three regression tests covering override-only, base-only, and
name-collision cases.

Closes #457

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-07-23 18:32:59 -04:00
Quality Badge Bot f2c3710d0d chore: update quality badges
- Coverage: 83%
- Core coverage: 94%

[skip ci]
2026-07-23 22:13:28 +00:00
didericis-claude e719022698 fix(cli): preflight PTY check in --headless before resource allocation
test / integration-docker (push) Successful in 19s
lint / lint (push) Successful in 52s
Update Quality Badges / update-badges (push) Successful in 48s
test / unit (push) Failing after 1m45s
test / integration-firecracker (push) Successful in 4m55s
test / coverage (push) Has been skipped
test / publish-infra (push) Has been skipped
Fails fast with a clear message and workaround hint instead of the
opaque ENODEV deep in the container exec layer after images are built,
the gateway is started, and deploy keys are provisioned.

Closes #460

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-07-23 22:12:31 +00:00
Quality Badge Bot 0fb9b04c01 chore: update quality badges
- Coverage: 84%
- Core coverage: 94%

[skip ci]
2026-07-23 21:49:28 +00:00
didericis-codex 26d0f5e3b2 refactor(secrets): keep ciphertext format minimal
tracker-policy-pr / check-pr (pull_request) Successful in 15s
test / integration-docker (pull_request) Successful in 35s
test / unit (pull_request) Successful in 46s
test / integration-firecracker (pull_request) Successful in 4m17s
test / coverage (pull_request) Successful in 16s
test / publish-infra (pull_request) Has been skipped
test / integration-docker (push) Successful in 14s
prd-number / assign-numbers (push) Failing after 31s
test / unit (push) Successful in 41s
Update Quality Badges / update-badges (push) Successful in 44s
lint / lint (push) Successful in 1m0s
test / integration-firecracker (push) Successful in 4m59s
test / coverage (push) Successful in 24s
test / publish-infra (push) Successful in 1m49s
2026-07-23 21:39:07 +00:00
didericis-codex bc4e559775 test(secrets): satisfy static coverage checks 2026-07-23 21:39:07 +00:00
didericis-codex 854f6b5696 fix(secrets): recover encrypted tokens on all backends 2026-07-23 21:39:07 +00:00
didericis-claude 28953bfe0b fix(types): remove unused new_env_var_secret import
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-07-23 21:38:33 +00:00
didericis 1ffc553ade test(secrets): cover per-bottle egress secret encryption
Add unit coverage for the encrypted at-rest egress secrets: the
secret_store round-trip (new_env_var_secret/encrypt_value/decrypt_value)
and the registry/service wiring that injects ENV_VAR_SECRET.

Recovered from the bot-bottle-claude-agent-1 VM after the CI runner's
disk filled and forced its rootfs read-only; committed work was already
on origin, these were the agent's uncommitted changes.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-23 21:38:33 +00:00
didericis-claude 9014c07b86 feat(secrets): encrypt egress tokens at rest with per-bottle ENV_VAR_SECRET
Implements the interim secret-provider design (PRD prd-new-secret-provider):
each agent receives a random ENV_VAR_SECRET injected into its container env
at launch. The host uses this key to encrypt each egress auth token value
(HMAC-SHA256 CTR mode, stdlib-only) and store it in a new
bottled_agent_secrets table (one row per env var, key column plaintext for
auditing). The key never touches the DB.

On infra container restart the in-memory token map is lost. launch_consolidated
now calls _reprovision_running_bottles after ensure_running: for each
registered bottle still alive on the gateway network it execs
`printenv ENV_VAR_SECRET` into the agent container and posts the result to the
new POST /bottles/<id>/reprovision_gateway control-plane endpoint, which
decrypts the stored rows and restores _tokens — no manual intervention needed.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-07-23 21:38:33 +00:00
didericis 8e2465e241 docs(prd): draft encrypted at-rest egress secrets (SecretProvider interim)
The orchestrator holds each bottle's egress auth tokens in process memory
only, so recreating the infra container strips every already-running
bottle of its upstream credentials. The registry row and the gateway CA
both survive; the tokens do not, so /resolve serves an intact policy with
an empty token map and the addon fails closed on `token_env unset`.

Drafts the interim slice of #355: persist the tokens encrypted so they
survive a restart, without regressing to plaintext at rest and without
foreclosing the per-request minting end state. Design section is left for
the author to fill in.
2026-07-23 21:38:33 +00:00
didericis-codex a8043be394 fix(types): annotate inspect settings fixture
tracker-policy-pr / check-pr (pull_request) Successful in 11s
test / integration-docker (pull_request) Successful in 18s
test / unit (pull_request) Successful in 1m41s
test / integration-firecracker (pull_request) Successful in 3m27s
test / coverage (pull_request) Successful in 22s
test / publish-infra (pull_request) Has been skipped
test / integration-docker (push) Successful in 34s
test / unit (push) Successful in 45s
Update Quality Badges / update-badges (push) Successful in 45s
lint / lint (push) Successful in 53s
test / integration-firecracker (push) Successful in 5m5s
test / coverage (push) Successful in 42s
test / publish-infra (push) Successful in 1m44s
2026-07-23 21:28:39 +00:00
didericis-codex 7d401a68c5 refactor(egress): make TLS inspection explicit 2026-07-23 21:28:39 +00:00
didericis-claude ce7a7c9915 refactor: resolve once per HTTPS connection, not per inner request
http_connect now stashes the resolved (config, slug, env) under
_FLOW_CTX_KEY after resolving, so request() can reuse it without
a second orchestrator round-trip. Plain-HTTP flows (no prior CONNECT
stash) still resolve in request() as before.
2026-07-23 21:28:22 +00:00
didericis-claude 83aa6768fc feat: add dlp: false passthrough option for egress routes
Routes with `dlp: false` skip all DLP scanning (including CRLF injection,
which cannot be disabled via `dlp.outbound_detectors: false`) and tunnel
HTTPS connections without TLS interception so the client sees the server's
real certificate. Fixes Docker image pulls, which fail when the proxy MITM's
the TLS handshake and the container doesn't trust the per-bottle CA.

Closes #462
2026-07-23 21:28:22 +00:00
didericis-codex 6fea44067f fix(manifest): preserve declared boolean fields
tracker-policy-pr / check-pr (pull_request) Successful in 11s
test / integration-docker (pull_request) Successful in 18s
test / unit (pull_request) Successful in 42s
test / integration-firecracker (pull_request) Successful in 3m35s
test / coverage (pull_request) Successful in 26s
test / publish-infra (pull_request) Has been skipped
prd-number / assign-numbers (push) Failing after 29s
test / integration-docker (push) Successful in 44s
lint / lint (push) Successful in 1m0s
test / unit (push) Successful in 56s
Update Quality Badges / update-badges (push) Successful in 1m45s
test / integration-firecracker (push) Successful in 6m54s
test / coverage (push) Successful in 26s
test / publish-infra (push) Successful in 2m22s
2026-07-22 18:35:31 +00:00
didericis-claude bf8ff91b31 fix(manifest): presence-aware runtime merge for nested_containers
test / integration-docker (pull_request) Successful in 17s
test / unit (pull_request) Successful in 44s
lint / lint (push) Successful in 57s
test / integration-firecracker (pull_request) Successful in 3m31s
test / coverage (pull_request) Successful in 19s
test / publish-infra (pull_request) Has been skipped
tracker-policy-pr / check-pr (pull_request) Successful in 13s
The runtime merge (merge_bottles_runtime / _fold_two_bottles) was using
OR semantics for nested_containers, which meant `nested_containers: false`
in a later bottle could never override an earlier `true`.

Fix: add `nested_containers_explicit: bool` to ManifestBottle, set to
True only when the key is present in the source dict. The runtime merge
functions now use the override's value only when it was explicitly
supplied; otherwise the base's value is preserved. This matches the
documented last-wins / presence-aware semantics used by supervise and
agent_provider, and aligns with _merge_bottles (the file-based extends
path) which already did presence checks via `"nested_containers" in
child_raw`.

Tests: rename existing test to clarify the preserved-when-omitted case,
add explicit-false-overrides-true and explicit-true-overrides-false cases.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-07-22 17:58:00 +00:00
didericis-claude 315ed04979 test: update unit tests to match podman-in-nested-containers-layer change
tracker-policy-pr / check-pr (pull_request) Successful in 8s
test / integration-docker (pull_request) Successful in 15s
test / unit (pull_request) Successful in 38s
lint / lint (push) Successful in 54s
test / integration-firecracker (pull_request) Successful in 4m21s
test / coverage (pull_request) Successful in 24s
test / publish-infra (pull_request) Has been skipped
- Replace test_all_install_podman with test_none_install_podman: agent
  images intentionally no longer include podman; the nested-containers
  derived layer is where it lives (see build_image in nested_containers.py).
- Add podman to the package assertions in test_macos_nested_containers so
  its presence in the derived Dockerfile is explicitly verified.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-07-22 17:36:51 +00:00
didericis-claude 9a04ab262b review: remove security-boundary note, move podman into nested-containers layer
test / integration-docker (pull_request) Successful in 19s
test / unit (pull_request) Failing after 39s
lint / lint (push) Successful in 56s
tracker-policy-pr / check-pr (pull_request) Failing after 13m51s
test / integration-firecracker (pull_request) Failing after 14m1s
test / coverage (pull_request) Has been skipped
test / publish-infra (pull_request) Has been skipped
- Drop the module-level paragraph emphasizing the single-UID security
  boundary (#issuecomment-5010): redundant given the safe environment.
- Move `podman` from each agent image (claude, codex, pi) into the
  `build_image` derived layer so bottles without `nested_containers`
  pay no image-size cost (#issuecomment-5013).
- Update `build_image` docstring and inline comments to reflect the move.
- Add TODO(#394) noting the planned docker-layer abstraction refactor.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-07-22 17:30:58 +00:00
didericis cc094765fd docs(readme): make the registry route example correct and complete
test / integration-docker (pull_request) Successful in 14s
lint / lint (push) Successful in 49s
test / unit (pull_request) Successful in 1m37s
test / integration-firecracker (pull_request) Successful in 3m25s
test / coverage (pull_request) Successful in 19s
test / publish-infra (pull_request) Has been skipped
tracker-policy-pr / check-pr (pull_request) Successful in 7s
The example previously left DLP enabled on the blob CDNs, which is
exactly backwards: those hosts carry the multi-hundred-MB layers whose
buffering triggers #455, while the registry hosts they hang off carry
little. It also listed one of quay's three CDNs and omitted the CDN
requirement from the prose, so a reader following it would authenticate
fine and then 403 partway through a pull.

Adds the by-design DNS behaviour and the BusyBox wget caveat, both of
which cost real time during the #392 verification: nslookup failing
inside a nested container looks like breakage but is expected, and
BusyBox wget reports "error getting response" for requests the proxy
served successfully.

Verified by feeding the snippet through parse_frontmatter +
ManifestBottle.from_dict: all nine routes parse with the intended
preserve_auth and DLP settings.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-21 22:54:26 -04:00
didericis 0ba25352b9 docs(nested-containers): record what live verification established
Documents the proxy-layer precedence that took four attempts to get
right, the blob-CDN routes every registry needs alongside its own, the
podman 5 networking packages and how each one fails when absent, and
the BusyBox wget caveat. Closes the DNS open question: public
resolution inside a nested container fails by design; reaching the
proxy was the real problem.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-21 22:54:26 -04:00
didericis dba48706de fix(macos): put the gateway address in the Docker CLI proxy config
This is the layer that actually decides it, and it was in the script
from the start: ~/.docker/config.json's proxies block is what the Docker
CLI copies into every container it starts. Being client-side it beats
the podman service outright, which is why containers.conf `env`,
http_proxy=false, and the service environment each looked correct on
disk and changed nothing a nested container saw.

Also corrects the comment on the service-env substitution, which
claimed to be the one place the address sticks. It is not; it covers
podman's own pulls and non-CLI API clients.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-21 22:54:26 -04:00
didericis 854a8956ad fix(macos): substitute the gateway address before the service starts
Third attempt at the same symptom, and the first at the right layer.
podman's Docker-compatible API injects proxy settings from the *daemon*
environment after containers.conf is applied, so neither the env list
(892bfc16) nor http_proxy=false (21c144ae) could override it — the same
compat-path blind spot already seen with hosts_file. Nested containers
kept reporting `wget: bad address 'bot-bottle-gateway:9099'` through
both.

Substituting the resolved address into the service's own environment,
before `podman system service` starts, is the one place it sticks.
NO_PROXY keeps the name, which is matched against what a client asks
for.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-21 22:54:26 -04:00
didericis 7a9628fc03 fix(macos): stop podman re-injecting the gateway name into containers
The address-bearing proxy URLs from 892bfc16 were written correctly but
never took effect: podman copies the host's proxy environment into every
container by default, and that copy overrides containers.conf `env`,
restoring the `bot-bottle-gateway` name a nested container cannot
resolve. Verified on the macOS host — containers still reported
`wget: bad address 'bot-bottle-gateway:9099'` with the addresses in
place.

Setting http_proxy=false disables only podman's own passthrough; the
proxy vars still reach containers, by address, from the env list.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-21 22:54:26 -04:00
didericis ca8b2a9f2c fix(macos): reach the egress proxy by address in nested containers
podman's containers.conf `hosts_file` turns out to apply only to native
`podman run`; the Docker-compatible API ignores it, and the agent types
`docker`. (`base_hosts_file`, used before this, is a no-op in both
paths on podman 5.4.2.) So no config key can put the gateway name in a
nested container's /etc/hosts.

The name is therefore resolved in the bottle, where /etc/hosts does
carry it, and the address goes into the proxy URL handed to nested
containers. NO_PROXY keeps the name, since that is matched against what
a client asks for.

Verified on the macOS host: with the gateway reachable, a nested
container gets 200 from an allowlisted host and 403 from one that is
not — the egress boundary applies inside nested containers too.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-21 22:54:26 -04:00
didericis 96f5be48a6 fix(egress): emit preserve_auth into the generated routes
The flag from #453 was parsed off the manifest and honored by the
addon, but `_route_to_yaml_fields` never wrote it, so it was dropped on
the way to the proxy and has never worked end to end. Every registry
route silently kept the unconditional Authorization strip.

That is the whole of the Docker Hub / GHCR "unauthorized" blocker in
#392: podman fetched a valid 2658-byte bearer from auth.docker.io, the
proxy stripped it, and registry-1.docker.io answered the resulting
anonymous request with a fresh WWW-Authenticate challenge — which reads
exactly like a credential problem rather than a serializer gap.

Found by diffing a bottle's rendered routes.yaml against its manifest:
preserve_auth was in the manifest and absent from the output.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-21 22:54:26 -04:00
didericis 82cf9bab5a feat(macos): give nested containers working egress
Answers the last open question in #392: DNS from inside a nested
container was not a phantom, but it was also not a DNS problem. Public
resolution fails there by design — everything egresses through the
proxy — and the actual breakage was two missing pieces:

- The proxy URL names `bot-bottle-gateway`, which resolves only through
  the bottle's own /etc/hosts. Containers got their own hosts file, so
  they failed at "Could not resolve proxy", which reads like broken DNS.
  base_hosts_file propagates the bottle's entries.
- The gateway TLS-intercepts, so a container that does not trust the
  bottle's CA bundle fails with "unable to get local issuer
  certificate". The bundle is now mounted read-only and the usual env
  vars point at it, which covers curl, wget, python, and node without
  distro-specific trust commands.

Verified on the macOS host by reproducing each failure and confirming
that these three conditions applied by hand produced HTTP 200 from
inside a nested container. podman already forwards the proxy env, so
that needed nothing.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-21 22:54:26 -04:00
didericis 3c92e79775 fix(macos): install podman 5's full networking stack
netavark shells out to `nft` for the bridge network every compose file
expects, and aardvark-dns serves name resolution inside nested
containers. Neither arrives with the base image's
`--no-install-recommends` podman, and each fails at a different and
misleading layer: without nft containers are created but never start;
without aardvark-dns DNS inside a container fails while the engine, the
pulls, and the bridge all look healthy. That last one is the likeliest
explanation for the "DNS inside nested containers fails entirely" note
in #392 — a missing package, not a design gap.

Verified on the macOS host: with passt alone, `docker run` reached
netavark and died on "unable to execute nft".

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-21 22:54:26 -04:00
didericis 220620bfcc fix(macos): install passt for podman 5 rootless networking
podman 5 uses pasta as the default rootless network helper; podman 4
used slirp4netns. The spike was written against bookworm's podman 4.3.1,
so its package list never included passt — and the trixie bump (#451)
changed the default out from under it. Every nested container failed to
start with "could not find pasta", while image pulls and the service
itself worked, which disguised a missing dependency as a compat-API bug.

The bootstrap now checks for pasta up front so this fails with a clear
message instead of surfacing at the first `docker run`.

Note the sun_path fix in a12d2191 did NOT cause or cure this; that path
was over the 108-byte limit independently and would have bitten as soon
as attach was reached.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-21 22:54:26 -04:00
didericis b0f012b8e6 fix(macos): shorten the podman runtime dir to fit sun_path
podman derives conmon's attach socket as
`$XDG_RUNTIME_DIR/libpod/tmp/socket/<64-hex-id>/attach`. With the
descriptive `/tmp/bot-bottle-podman-run` that path is 116 bytes, past
the 108-byte sun_path limit, so every attached `docker run` failed with
"unable to upgrade to tcp, received 500" — while image pulls and the
service itself worked, which is what made it look like a compat-API bug
rather than a path-length one.

Found on the macOS host: quay.io pulled fine and then no container
could be started. Test pins the budget so a more readable name cannot
silently reintroduce it.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-21 22:54:26 -04:00
didericis fa9fed4194 fix(manifest): OR nested_containers across composed bottles
merge_bottles_runtime sees resolved bottles, where a bottle that never
mentioned the key is indistinguishable from one that set it false. With
"later replaces" semantics, `--bottle with-containers --bottle claude-dev`
silently dropped the capability — the second bottle's default clobbered
it. OR instead: composing bottles adds capabilities, it does not remove
them. The file-based extends path still sees raw keys, so an explicit
`nested_containers: false` in a child continues to turn it back off.

Also surface the flag in the launch summary, where its absence was the
only reason the clobber went unnoticed.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-21 22:54:26 -04:00
didericis d0b595828f feat(macos): run containers inside a bottle via guest-local podman
Adds the `nested_containers` bottle flag. On the macOS backend it starts
a rootless podman service inside the bottle and exposes its
Docker-compatible API socket, so the agent still runs `docker` and
`docker compose`. No host daemon socket is mounted and the guest gains
no capabilities; backends that cannot run a guest-local engine reject
the flag in the shared prepare template rather than ignoring it.

Podman rather than rootless Docker because Apple Container's capability
bounding set omits CAP_SYS_ADMIN, which the kernel requires to write a
multi-range uid_map via newuidmap. With no subordinate UID range podman
falls back to a single-UID self-mapping an unprivileged process may
write itself, so the image build strips /etc/subuid and /etc/subgid
entries rather than adding them.

That mapping is also why nested containers are not an isolation layer:
root inside one is the agent user outside it. They are a build/test
convenience; the bottle remains the boundary.

Ports the spike branch onto main, renaming docker_access — it implied
Docker and granted access to nothing on the host — and drops podman
from the derived layer now that every built-in image ships it (#451).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-21 22:54:26 -04:00
didericis-claude 182a28d724 docs(research): firecracker image remote store
test / integration-docker (pull_request) Successful in 17s
test / unit (pull_request) Successful in 40s
tracker-policy-pr / check-pr (pull_request) Successful in 6s
test / integration-firecracker (pull_request) Successful in 3m33s
test / coverage (pull_request) Successful in 16s
test / publish-infra (pull_request) Has been skipped
test / integration-docker (push) Successful in 35s
test / unit (push) Successful in 39s
Update Quality Badges / update-badges (push) Successful in 47s
lint / lint (push) Successful in 51s
test / integration-firecracker (push) Successful in 5m8s
test / coverage (push) Successful in 44s
test / publish-infra (push) Successful in 1m53s
Investigates storage candidates, retention policy design, and
image-size reduction techniques in response to the PR #459 comment
on a secure remote store for committed Firecracker snapshots.
2026-07-21 22:47:44 -04:00
didericis cfb2284b99 fix(firecracker): make cleanup reap orphans only, never live VMs
`cleanup()` enumerated *every* firecracker process under the run root and
*every* run dir, so `./cli.py cleanup` would kill running bottles and
delete their rootfs — despite being described as orphan cleanup. The
backend's `enumerate_active` registry is still a stub (#354), so there
was no live/dead signal.

Use the running firecracker processes themselves as that signal: a run
dir with a live VM is protected (never killed, never removed); only run
dirs with no live process (leaked by a hard-killed launch) are reaped,
plus firecracker pids whose run dir is already gone (lingering VMMs).

This makes cleanup safe to run alongside live bottles and lets it back a
periodic GC for the run-dir leak the launch.py teardown fix closes on the
clean-exit path.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-21 22:47:44 -04:00
didericis 2cd06814e6 fix(firecracker): remove per-bottle run dir on teardown
Each Firecracker bottle created its run dir (holding an ~1G
rootfs.ext4) but never removed it — launch teardown only terminated
the VM. Every run leaked its rootfs, filling the CI runner's disk
(187 stale sandbox-tester dirs / 141G in one incident).

Register a run-dir rmtree callback on the ExitStack before the
vm.terminate callback so, LIFO, the VM is gone before its rootfs is
removed. ignore_errors keeps teardown resilient.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-21 22:47:44 -04:00
didericis-codex 8fed02fd1e chore(ci): gate tests on relevant runtime inputs 2026-07-22 01:24:02 +00:00
Quality Badge Bot a7b2befc06 chore: update quality badges
- Coverage: 83%
- Core coverage: 94%

[skip ci]
2026-07-22 01:19:07 +00:00
didericis-codex 6fdf090469 fix(ci): authenticate badge pushes with PAT
test / unit (push) Failing after 2s
test / integration-docker (push) Successful in 10s
test / integration-firecracker (push) Successful in 2m59s
test / coverage (push) Has been skipped
test / publish-infra (push) Has been skipped
2026-07-22 01:09:42 +00:00
didericis-codex d32e9cc3c3 fix(ci): use Gitea token for badge updates
test / integration-docker (push) Successful in 11s
test / unit (push) Successful in 36s
test / integration-firecracker (push) Successful in 3m13s
test / coverage (push) Successful in 43s
test / publish-infra (push) Successful in 44s
2026-07-22 00:53:32 +00:00
didericis-codex ef89ed084f fix(ci): allow badge workflow to push updates
test / integration-docker (push) Successful in 21s
test / unit (push) Successful in 47s
lint / lint (push) Successful in 3m3s
test / integration-firecracker (push) Successful in 3m39s
test / coverage (push) Successful in 29s
Update Quality Badges / update-badges (push) Failing after 49s
test / publish-infra (push) Successful in 1m17s
2026-07-21 19:19:43 +00:00
didericis-claude ccd987a501 feat(egress): add preserve_auth flag to pass agent Authorization through
test / integration-docker (push) Successful in 11s
test / unit (push) Successful in 37s
lint / lint (push) Successful in 55s
Update Quality Badges / update-badges (push) Failing after 37s
test / integration-firecracker (push) Successful in 4m49s
test / coverage (push) Successful in 14s
test / publish-infra (push) Successful in 1m42s
Adds a per-route boolean field preserve_auth (default false) that skips
the gateway's Authorization header stripping for that host. Intended for
registry endpoints like Docker Hub (registry-1.docker.io) and GHCR
(ghcr.io) where the agent must supply its own per-scope bearer token.

Threaded through ManifestEgressRoute → EgressRoute → Route, serialized
in route_to_yaml_dict, and parsed in parse_routes. The strip at
egress_addon.py now checks route.preserve_auth before popping the header.

Closes #392
2026-07-21 15:01:59 -04:00
didericis-codex 2cd44cf79a fix(macos): persist gateway CA on host
test / integration-docker (push) Successful in 19s
Update Quality Badges / update-badges (push) Failing after 42s
lint / lint (push) Successful in 52s
test / unit (push) Successful in 1m43s
test / integration-firecracker (push) Successful in 4m56s
test / coverage (push) Successful in 17s
test / publish-infra (push) Successful in 1m49s
2026-07-21 15:01:38 -04:00
didericis-claude 8a1b833aaa fix(gateway): persist mitmproxy CA on the host, not a named volume (#450)
The shared gateway self-generates a mitmproxy CA that every bottle installs
to trust its TLS interception. It was persisted on a Docker named volume,
which survives `docker rm` but is silently wiped by `docker volume prune` /
`docker system prune --volumes` during routine host maintenance. When that
happens the gateway mints a fresh CA on restart, and every already-running
bottle fails the TLS handshake even after it re-resolves and reconnects to
the moved gateway — a re-attachment blocker distinct from #443/#445.

Move CA persistence to a host bind-mount under the app-data root
(`bot_bottle_root()/gateway-ca`, via `host_gateway_ca_dir()`), mirroring how
the shared DB and control-plane token already live on the host. Docker never
prunes a path under the root, and it stays inspectable + rotatable from the
host. mitmproxy already adopts an existing CA and generates one only on first
run, so the bind-mount gives adopt-existing/generate-on-first-run for free.

Add an explicit rollover path: `rotate_gateway_ca()` clears the persisted CA
so the next start remints it, and `python -m bot_bottle.orchestrator.rotate_ca`
wires that together with dropping the running gateway container (whose
mitmproxy still holds the old CA in memory). Rotation stays an operator action
— it doesn't auto-re-provision running bottles, which re-attach to pick up the
new anchor.

Scope: the Docker infra/gateway path (the "infra container" in the report).
The macOS (`container`-only volume) and Firecracker (VM-attached ext4) backends
persist the CA differently and are unaffected.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-21 15:01:38 -04:00
didericis-codex 3b5c55bc8e test(git): cover provisioning failures
prd-number / assign-numbers (push) Failing after 17s
test / integration-docker (push) Successful in 19s
Update Quality Badges / update-badges (push) Failing after 44s
lint / lint (push) Successful in 52s
test / unit (push) Successful in 1m44s
test / integration-firecracker (push) Successful in 4m40s
test / coverage (push) Successful in 15s
test / publish-infra (push) Successful in 1m34s
2026-07-21 14:52:44 -04:00
didericis-codex 95220b4808 fix(firecracker): restore agent home ownership at boot 2026-07-21 14:52:44 -04:00
didericis-codex b032562d74 fix(firecracker): pin the provisioned Git config 2026-07-21 14:52:44 -04:00
didericis-codex 1d925172ec fix(firecracker): repair runtime Git config ownership 2026-07-21 14:52:44 -04:00
didericis-codex f6ae485b68 Revert "refactor(agent-images): use explicit Debian base"
This reverts commit 51b82f80d1.
2026-07-21 14:52:44 -04:00
didericis-codex e3258d0683 refactor(agent-images): use explicit Debian base 2026-07-21 14:52:44 -04:00
didericis-codex 0ff11d8ed7 fix(agent-images): own Git config directory 2026-07-21 14:52:44 -04:00
didericis-codex a970f974a2 fix(agent-images): retain SSH client 2026-07-21 14:52:44 -04:00
didericis-codex c845d3fed4 feat(agent-images): update Debian and add Podman 2026-07-21 14:52:44 -04:00
didericis c6a9419b95 feat(login): point the printed URL at the hosts-page modal
test / integration-docker (push) Successful in 13s
test / unit (push) Successful in 37s
Update Quality Badges / update-badges (push) Failing after 41s
lint / lint (push) Successful in 55s
test / integration-firecracker (push) Successful in 4m44s
test / coverage (push) Successful in 15s
test / publish-infra (push) Successful in 1m29s
Approval now renders over the hosts page at /hosts/authorize?code=…
rather than on a standalone page. The console keeps /authorize as a
redirect, so this is cosmetic for older consoles.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-21 14:25:38 -04:00
didericis 36fb019007 feat(cli): make bot_bottle.cli runnable with python -m
`python -m bot_bottle.cli` failed with "is a package and cannot be
directly executed" — the package had a `if __name__ == "__main__"` block
in `__init__.py`, which never fires for a package and made the invocation
look supported when it wasn't. Add a real `__main__.py` and drop the dead
block.

Matters for `bb login`, whose docstring documents a `bb` entry point that
nothing installs; `-m` is the closest thing to it until there's a
console-script.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-21 14:25:38 -04:00
didericis-claude adc033a902 fix(login): narrow exception types and cover cleanup path
Replaces broad `except Exception` with specific types that reflect the
actual failure modes:
- _save_credentials: `except OSError` (IO-only path; re-raises for cleanup)
- _post error handler: `except (OSError, ValueError)` (network + bad JSON)
- poll-loop: `except (OSError, ValueError)` (network + bad JSON)

Also:
- Simplify _fake_get to use next(..., default) — removes uncovered
  StopIteration branch
- Add encoding="utf-8" to open() in test_approved_flow_returns_0
- Add test_cleanup_on_write_failure to cover the except OSError block
  in _save_credentials

All files score 10.00/10 on pylint (fail-under=10) and 0 errors on
pyright strict.
2026-07-21 14:25:38 -04:00
didericis-codex 21b253c7eb test(login): satisfy type and coverage gates 2026-07-21 14:25:38 -04:00
didericis-claude d4e2bc5f93 fix(login): atomic credential write and respect server poll_interval
Write credentials via a 0600 temp file + os.replace() so the token file
never appears at its final path with world-readable permissions, even if
the process is interrupted between write and chmod.

Parse poll_interval from the authorization response (clamped to 1–60 s,
falling back to _POLL_SLEEP) so aggressive polling can't trigger console
rate limits.

Tests: add atomicity spy asserting the temp file is 0600 before replace;
patch time.sleep instead of _POLL_SLEEP; add explicit interval-passthrough
assertion.
2026-07-21 14:25:38 -04:00
didericis-claude 4998a5ec6a feat: add bb login command for console host registration
Starts a device-authorization flow against a bot-bottle console, polls
until the operator approves, then writes access + refresh tokens to
$BOT_BOTTLE_ROOT/console.json. Console URL is read from --console-url
flag or BB_CONSOLE_URL env var.

Part of didericis/bot-bottle-platform#1
2026-07-21 14:25:38 -04:00
didericis-codex efd413c1ba test(macos): cover cached image selection
tracker-policy-pr / check-pr (pull_request) Successful in 6s
test / integration-docker (pull_request) Successful in 13s
test / unit (pull_request) Successful in 38s
test / integration-firecracker (pull_request) Successful in 3m19s
test / coverage (pull_request) Successful in 39s
test / publish-infra (pull_request) Has been skipped
test / integration-docker (push) Successful in 13s
test / unit (push) Successful in 38s
lint / lint (push) Successful in 54s
Update Quality Badges / update-badges (push) Failing after 1m41s
test / integration-firecracker (push) Successful in 4m45s
test / coverage (push) Successful in 19s
test / publish-infra (push) Successful in 1m36s
2026-07-21 13:56:29 -04:00
didericis-claude d3d468532f fix(rebase): correct test_cached_lookup key and drop duplicate test
Two issues introduced by the rebase conflict resolution:

- test_cached_lookup_requires_ready_marker used _dockerfile_hash as the
  cache-dir key, but cached_agent_rootfs_dir now uses _rootfs_digest (which
  also folds in the guest init). Updated the test to match.
- test_pyproject_toml_change_bumps_version appeared twice in
  test_infra_artifact.py (once from main, once from the PR commit that
  added pyproject.toml support). Removed the duplicate.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-07-21 13:56:29 -04:00
didericis-claude ad2927b3b1 fix(infra-artifact): include pyproject.toml in rootfs version digest
Dockerfile.gateway COPYs pyproject.toml into /src and runs pip install
/src, so it is a real input to the baked rootfs. A dependency-only change
previously reused stale artifact versions, potentially booting a rootfs
whose installed packages differed from the current checkout.

Also adds _fake_repo fixture support and a regression test so this input
can't silently drop out of the hash again.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-07-21 13:56:29 -04:00
didericis-claude 2582373490 fix(pyright): assert not None before accessing datetime attributes in tests
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-07-21 13:56:29 -04:00
didericis-claude 17ac1be93b fix: return None from image_created_at when timestamp is absent
FROM-scratch images (commit_container output) and registry images built
for reproducibility often omit the created field entirely. Both backends
were calling die() in that case, crashing the cached-image quickstart
before any container started.

image_created_at now returns datetime | None — None when the field is
absent or unparseable, die() only on real inspect failures (non-zero
exit, malformed JSON). stale_checks in both backends skips the staleness
check when None is returned.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-07-21 13:56:29 -04:00
didericis-codex c53254e9d5 fix(firecracker): honor cached image policy 2026-07-21 13:56:29 -04:00
didericis-claude 58ecd8cb90 fix: correct host_db_path import and stale-check test expectations
- config_store.py imported host_db_path from supervise_types (wrong);
  it lives in paths.py, matching all other stores (audit, queue, etc.)
- test_stale_checks: two tests expected check_stale called twice
  (agent + sidecar) — consolidated arch has no sidecar, so once is
  correct; update assertions and remove unused Path import

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-07-21 13:56:29 -04:00
didericis-claude 310b36196d fix(lint): remove unused BottleImages imports from two test files
pyright strict reportUnusedImport flagged BottleImages in
test_docker_launch_committed_image.py and test_macos_container_launch.py;
neither file references the type by name (they only use the returned
value's attributes).
2026-07-21 13:56:29 -04:00
didericis-claude c473e5e5d8 fix: remove redundant quoted annotations in BottleImages and abstract methods
`from __future__ import annotations` already defers all annotation
evaluation, so quoting `str | Path`, `BottleImages` inside the same
module was redundant and tripped pyright strict mode.
2026-07-21 13:56:29 -04:00
didericis-claude 137426d9ac refactor: BottleImages dataclass, prelaunch_checks, build_or_load_images
Addresses review comments 3098, 3099, 3100 on PR #336:

- Add BottleImages(agent, sidecar) dataclass to backend/__init__.py.
  Docker/macOS backends use str image refs; smolmachines uses Path
  artifacts. Replaces the singular `image` variable from the canonical
  pattern in comment 3100.

- Replace _image_stale_checks/skip_stale with public prelaunch_checks().
  CLI now calls backend.prelaunch_checks(plan) before backend.launch(plan);
  if StaleImageError is raised and the operator confirms, launch proceeds
  without re-checking. Removes the while-True/skip_stale retry loop.

- Add abstract _build_or_load_images(plan) -> BottleImages to
  BottleBackend. launch() calls it then passes images to _launch_impl.
  Each backend implements both methods.

- Fix comment 3098 (macos-container): _build_images is removed.
  build_or_load_images() has separate fresh/cached code paths — the
  cached path never calls a build helper.

- Update _start_bundle (smolmachines) to accept sidecar_artifact: Path
  directly. Sidecar artifact resolution moves to _sidecar_from_path(),
  called by build_or_load_images alongside _agent_from_path().
2026-07-21 13:56:29 -04:00
didericis-claude 2a3a7dfb5c fix(tests): resolve pyright errors in stale-check test files
Remove unused imports, add missing type annotations, fix Die()
constructor calls (int code, not str), replace fake backend
subclass with patch.object approach to avoid reportMissingParameterType
and override-incompatibility errors in strict pyright mode.
2026-07-21 13:56:29 -04:00
didericis-claude 16c12177d2 test: add unit tests for stale-image checks to reach ≥90% diff-coverage
Covers StaleImageError / check_stale / check_stale_path, the
BottleBackend.launch template method (skip_stale flag), stale_checks
functions across docker / smolmachines / macos-container backends,
_build_images cached paths in the macOS backend, image_created_at edge
cases in both docker and container util modules, the CLI stale loop
(headless die, interactive decline, interactive confirm + retry), and
ConfigStore.cached_image_stale_warning_days fallback paths.

Diff-coverage: 488/507 changed lines covered (96.3%).
2026-07-21 13:56:29 -04:00
didericis-claude 0f1734b823 refactor(image-cache): replace warn_if_stale with StaleImageError; add launch template
- image_cache: StaleImageError exception + check_stale/check_stale_path (raise instead of warn)
- BottleBackend.launch: template method (skip_stale flag) that calls _image_stale_checks then _launch_impl
- Each backend: _image_stale_checks delegates to a stale_checks() function in its launch module; _launch_impl replaces launch override
- macos_container: adds image_created_at to util, cached-image support in _build_images, stale_checks
- cli/start.py: catches StaleImageError, prompts interactively, retries with skip_stale=True; headless mode dies on it
2026-07-21 13:56:29 -04:00
didericis-codex 2bf28e03f4 fix: make config store schema explicit 2026-07-21 13:56:29 -04:00
didericis-codex d3428b8c14 Add cached image quickstart 2026-07-21 13:56:29 -04:00
didericis-claude 4199de5e3e feat(claude): add forward_host_credentials support
tracker-policy-pr / check-pr (pull_request) Successful in 10s
test / integration-docker (pull_request) Successful in 17s
test / unit (pull_request) Successful in 1m41s
test / integration-firecracker (pull_request) Successful in 3m17s
test / coverage (pull_request) Successful in 21s
test / publish-infra (pull_request) Has been skipped
test / integration-docker (push) Successful in 19s
prd-number / assign-numbers (push) Failing after 23s
test / unit (push) Successful in 42s
Update Quality Badges / update-badges (push) Failing after 43s
lint / lint (push) Successful in 51s
test / integration-firecracker (push) Successful in 4m46s
test / coverage (push) Successful in 15s
test / publish-infra (push) Successful in 1m35s
Reads the host's Claude OAuth session key from ~/.claude.json at launch
and forwards it only to the egress sidecar (never to the agent), placing
a placeholder CLAUDE_CODE_OAUTH_TOKEN in the agent env so Claude Code
starts without seeing the real credential.

Mirrors the existing Codex forward_host_credentials flow (PRD 0029).
Adds claude_auth.py to extract and validate the sessionKey, a
CLAUDE_HOST_CREDENTIAL_TOKEN_REF constant in egress.py, and updates
manifest_agent.py to allow the flag for both 'codex' and 'claude'
templates. Also adds a mutual-exclusion check that rejects setting
both auth_token and forward_host_credentials together.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

fix(claude): read credentials from ~/.claude/.credentials.json

The actual OAuth token is in ~/.claude/.credentials.json under
claudeAiOauth.accessToken, not in ~/.claude.json.
~/.claude.json holds only UI state and profile metadata (oauthAccount
has no token fields). expiresAt in the credentials file is milliseconds,
not seconds.

Discovered after testing against Claude Code 2.1.198.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

fix(claude): fall back to macOS Keychain for credentials

On macOS, Claude Code stores credentials in the Keychain under
service "Claude Code-credentials" rather than in a file. When
~/.claude/.credentials.json is absent, shell out to:
  security find-generic-password -s "Claude Code-credentials" -w
and parse the result as the same JSON schema.

~/.claude.json holds only profile/UI metadata (oauthAccount has
no token fields). expiresAt in the credentials is milliseconds.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

docs(prd): fix credential path references (~/.claude/.credentials.json)

fix(test): suppress gitleaks false positives on synthetic Claude tokens
2026-07-21 13:15:05 -04:00
didericis-codex 8348714e3e test(firecracker): cover published artifact reuse
tracker-policy-pr / check-pr (pull_request) Successful in 9s
test / integration-docker (pull_request) Successful in 16s
test / unit (pull_request) Successful in 1m37s
test / integration-firecracker (pull_request) Successful in 3m30s
test / coverage (pull_request) Successful in 15s
test / publish-infra (pull_request) Has been skipped
prd-number / assign-numbers (push) Failing after 23s
test / integration-docker (push) Successful in 15s
lint / lint (push) Successful in 58s
test / unit (push) Successful in 38s
Update Quality Badges / update-badges (push) Failing after 1m40s
test / integration-firecracker (push) Successful in 5m9s
test / coverage (push) Successful in 44s
test / publish-infra (push) Successful in 2m7s
2026-07-21 16:55:06 +00:00
didericis-claude 26002b75ca fix(ci): stage coverage data under non-dot names so upload-artifact uploads them
tracker-policy-pr / check-pr (pull_request) Successful in 12s
test / integration-docker (pull_request) Successful in 18s
lint / lint (push) Successful in 52s
test / unit (pull_request) Successful in 1m59s
test / integration-firecracker (pull_request) Successful in 3m23s
test / coverage (pull_request) Failing after 40s
test / publish-infra (pull_request) Has been skipped
upload-artifact@v3's glob silently skips hidden files, so uploading a
bare `.coverage.unit` logged "No files were found. No artifacts will be
uploaded" and registered nothing — the coverage job's download then 404'd
("List Artifacts failed: 404"). The coverage report step read the same
file fine, confirming it existed; only the leading dot broke the upload.
The old pipeline's cross-job artifacts (infra-candidate/, firecracker-
inputs) worked precisely because they were non-dotfiles.

Each test job now copies its .coverage.<suffix> to a non-dot
coverage-<suffix>.dat before upload (the cp also fails loudly if coverage
never wrote the file), and the coverage job renames them back to
.coverage.* before `coverage combine`.
2026-07-21 12:43:26 -04:00
didericis-claude 0c91c75a05 fix(ci): use COVERAGE_FILE env var for reliable artifact paths; add --reuse-published to infra build
coverage run's --data-file flag can be overridden or ignored in some
runner environments (Nix Python, older act-based runners). Switching to
the COVERAGE_FILE env var with an absolute ${{ github.workspace }} path
ensures coverage.py writes to a known location in every runner context,
so upload-artifact can find the file.

Also adds --reuse-published to the infra build step: if the artifact for
this content hash already exists in the registry, download it instead of
running the full docker build → mke2fs → gzip pipeline.
2026-07-21 12:43:26 -04:00
didericis-claude 8ce8a8cc62 fix(ci): use absolute github.workspace paths for coverage artifact upload/download
The delphi-ci runner resolves relative paths in upload-artifact and
download-artifact from a different CWD than run: shell steps, so
'.coverage.unit' etc. were never found. Using ${{ github.workspace }}
gives an absolute path that does not depend on the JS action's CWD.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-07-21 12:43:26 -04:00
didericis-claude 3fba385513 ci: artifact-based coverage and local Firecracker candidate flow
Each test job now runs once under coverage and uploads a small .coverage.*
artifact. The coverage job combines them on ubuntu-latest — no test reruns,
no KVM dependency. The infra candidate is built directly on the KVM runner,
eliminating the build-infra job and the ~70 s upload + ~83 s combined
download. For PRs, no rootfs artifact is transferred at all. Main-branch
pushes upload the tested rootfs and matching dropbear so publish-infra
publishes the byte-identical artifact. relative_files = True in .coveragerc
lets coverage files from different runners combine without path remapping.

Closes #446
2026-07-21 12:43:26 -04:00
didericis-claude b25cd72fc3 test(backend): cover poll_ca_cert timeout paths for diff-coverage gate
test / integration-docker (pull_request) Successful in 12s
tracker-policy-pr / check-pr (pull_request) Successful in 10s
test / unit (pull_request) Successful in 35s
test / stage-firecracker-inputs (pull_request) Successful in 2s
test / build-infra (pull_request) Successful in 3m37s
test / integration-firecracker (pull_request) Successful in 1m30s
test / coverage (pull_request) Successful in 1m32s
test / publish-infra (pull_request) Has been skipped
test / integration-docker (push) Successful in 13s
prd-number / assign-numbers (push) Failing after 19s
lint / lint (push) Successful in 43s
Update Quality Badges / update-badges (push) Failing after 41s
test / unit (push) Successful in 1m42s
test / stage-firecracker-inputs (push) Successful in 2s
test / build-infra (push) Failing after 7s
test / integration-firecracker (push) Has been skipped
test / coverage (push) Has been skipped
test / publish-infra (push) Has been skipped
Add tests for the three uncovered paths introduced by the poll_ca_cert
extraction: the timeout + sleep branches in backend/util, and the
TimeoutError → GatewayError and TimeoutError → die() conversions in the
macOS and Firecracker callers.
2026-07-21 04:54:40 +00:00
didericis-claude 8e43c26ab4 fix(backend): extract poll_ca_cert helper and fix PRD port docs
Extract the shared CA cert polling loop into `backend/util.poll_ca_cert`
(firecracker and macos backends were duplicating deadline/sleep/raise logic).
Each caller now wraps a fetch lambda and converts TimeoutError to its own
error type. Also corrects the PRD port publication line from {port}:{port}
to {host_port}:8099.
2026-07-21 04:54:40 +00:00
didericis-claude 14ff4fe186 fix(lifecycle): build gateway before infra and fix orchestrator port mapping
- `_build_images()` now builds Dockerfile.gateway → Dockerfile.orchestrator
  → Dockerfile.infra in order; Dockerfile.infra starts FROM bot-bottle-gateway
  so the base must exist on clean hosts.
- Publish mapping corrected from `self.port:self.port` to `self.port:DEFAULT_PORT`
  (8099) — gateway_init hardcodes the orchestrator on port 8099 inside the
  container, so the host-side published port must map to that fixed internal port.
- BOT_BOTTLE_ORCHESTRATOR_URL inside the container now always points to
  127.0.0.1:8099, not self.port, since gateway daemons reach the orchestrator
  over loopback at the fixed internal port.
- Update test_ensure_running_builds_both_images → _all_images for the new
  three-step build sequence; add test_publish_maps_host_port_to_fixed_internal_port
  to lock in the port-mapping fix.

Addresses the P1 findings from the didericis-codex review on PR #432.
2026-07-21 04:54:40 +00:00
didericis-claude cae1215f63 test(lifecycle): cover edge paths to satisfy diff-coverage gate
Add three new tests:
- noop when healthy but docker inspect fails (returns True → don't churn)
- build failure raises GatewayError
- _ensure_network creates the network when it doesn't exist

Also update the integration test to use new OrchestratorService API
(infra_name/image instead of orchestrator_name/gateway_name/gateway_image).

Brings diff-coverage from 86% to 90.3% against origin/main.
2026-07-21 04:54:40 +00:00
didericis-claude 28766d7733 fix(pyright): resolve type errors introduced by lifecycle refactor
- Remove unused INFRA_IMAGE import from test_orchestrator_lifecycle
- Update integration test to use new single-container OrchestratorService
  API (infra_name/image replaces orchestrator_name/gateway_name/gateway_image)
- Move type: ignore to the lambda line in gateway_init SIGHUP handler
- Break two long lines in test_orchestrator_lifecycle
2026-07-21 04:54:40 +00:00
didericis-claude 819f967844 fix(lint): resolve pylint findings in gateway_init
- Extract _sigkill_all() to cut nesting depth below the 5-block limit
- Add pylint: disable=consider-using-with on Popen (process must outlive caller)
- Break long SIGHUP signal line to stay within 100 chars
2026-07-21 04:54:40 +00:00
didericis-claude 2f45f5afec feat(docker): consolidate to single infra container under gateway_init supervise tree
Collapses the two-container Docker model (gateway + orchestrator) into one
bot-bottle-infra container, matching the macOS and Firecracker backends.

- Dockerfile.infra: now a shared gateway+orchestrator base (COPY bot_bottle
  from orchestrator build, no CMD override)
- Dockerfile.infra.fc: new Firecracker-specific layer (buildah/crun/netavark)
- gateway_init: adds orchestrator daemon with _OPT_IN_DAEMONS gating so it
  only starts when BOT_BOTTLE_GATEWAY_DAEMONS explicitly includes it
- orchestrator/lifecycle: OrchestratorService manages one infra container;
  builds orchestrator (intermediate) then infra; live source bind-mounted at
  /bot-bottle-src with PYTHONPATH so the subprocess uses the checkout
- backend/consolidated_util: extracts provision_bottle + teardown_consolidated
  shared across all three backends; removes duplication in docker/fc/macos
  consolidated_launch modules
- firecracker/infra_vm: builds four images (orchestrator→gateway→infra→infra.fc)
- All unit tests updated and passing (1878 tests)
- PRD status: Draft → Active
2026-07-21 04:54:40 +00:00
didericis-claude 31a5ec2fc8 docs(prd): consolidate infra backend for docker
Adds PRD for collapsing the Docker backend from two containers
(gateway + orchestrator) to a single bot-bottle-infra container,
restructuring Dockerfile.infra as the shared base, and extracting
duplicated CA polling / teardown / provision helpers into a shared
backend utility module.

Closes #431
2026-07-21 04:49:18 +00:00
didericis-claude 1f192d785a refactor(config_store): replace generic key/value table with explicit schema
test / stage-firecracker-inputs (push) Successful in 2s
test / integration-docker (push) Successful in 17s
Update Quality Badges / update-badges (push) Failing after 42s
lint / lint (push) Successful in 47s
test / unit (push) Successful in 1m41s
test / build-infra (push) Successful in 3m43s
test / integration-firecracker (push) Successful in 1m28s
test / coverage (push) Successful in 1m29s
test / publish-infra (push) Successful in 1m49s
Switch orchestrator_config from a generic key/text store to a typed
single-row table with a dedicated teardown_timeout_seconds REAL column.
The CHECK (id = 1) constraint enforces at most one config row.

Remove TEARDOWN_TIMEOUT_KEY constant; update OrchestratorConfigStore to
expose get/set/delete_teardown_timeout_seconds() instead of the generic
get/set/delete(key). Resolving the DB value in resolve_teardown_timeout()
no longer needs a float() conversion since the column is already REAL.
2026-07-21 00:40:59 -04:00
didericis-codex 853b6d1678 test(orchestrator): satisfy config store type checking 2026-07-21 00:40:59 -04:00
didericis-claude cf9a53d582 fix: resolve teardown timeout before registration to prevent orphaned state
If resolve_teardown_timeout() raised after launch_consolidated()/register_agent()
returned, the teardown callback was never registered, leaving the bottle
registered with no cleanup path. Resolve the timeout into a local variable
before the registration call so any failure aborts before the bottle exists.

Add tests covering OrchestratorConfigStore, resolve_teardown_timeout priority
ordering (env > db > default), and the source-order invariant that the resolver
runs before registration in all three backends.
2026-07-21 00:40:59 -04:00
didericis-claude 0b36c3eb48 fix: make orchestrator teardown timeout configurable (closes #435)
Resolves via ENV VAR -> orchestrator DB config -> default (30 s, up from 5 s):
  BOT_BOTTLE_ORCHESTRATOR_TEARDOWN_TIMEOUT_SECONDS
  teardown_timeout_seconds key in new orchestrator_config table (bot-bottle.db)

New OrchestratorConfigStore (same DbStore/TableMigrations pattern as the
registry) stores the DB-level setting. resolve_teardown_timeout() implements
the priority chain and is called at stack.callback registration time in all
three backends (macos_container, docker, firecracker).
2026-07-21 00:40:59 -04:00
didericis-claude 28fcc3f2d2 test: cover inspect_container_network_ip parser directly
test / stage-firecracker-inputs (pull_request) Successful in 5s
test / unit (pull_request) Successful in 35s
test / integration-docker (pull_request) Successful in 9s
tracker-policy-pr / check-pr (pull_request) Successful in 8s
test / build-infra (pull_request) Successful in 3m48s
test / integration-firecracker (pull_request) Successful in 1m43s
test / coverage (pull_request) Successful in 1m53s
test / publish-infra (pull_request) Has been skipped
test / integration-docker (push) Successful in 11s
test / unit (push) Successful in 35s
test / stage-firecracker-inputs (push) Successful in 2s
Update Quality Badges / update-badges (push) Failing after 40s
lint / lint (push) Successful in 2m43s
test / build-infra (push) Successful in 3m25s
test / integration-firecracker (push) Successful in 1m45s
test / coverage (push) Successful in 1m50s
test / publish-infra (push) Successful in 2m7s
The None/empty-string distinction is the invariant that prevents
live_source_ips from treating a failed inspect as a legitimately
no-address container. Add TestInspectContainerNetworkIp to util tests
covering: IP found, CIDR stripping, no address yet, absent network
list, non-zero exit, malformed JSON, and unexpected JSON shape.
2026-07-21 03:49:19 +00:00
didericis-claude 571030b8e8 fix: require authoritative container snapshot before reconciling
test / stage-firecracker-inputs (pull_request) Successful in 5s
tracker-policy-pr / check-pr (pull_request) Successful in 12s
test / integration-docker (pull_request) Successful in 33s
test / unit (pull_request) Successful in 39s
lint / lint (push) Successful in 49s
test / build-infra (pull_request) Successful in 3m23s
test / integration-firecracker (pull_request) Successful in 1m42s
test / coverage (pull_request) Failing after 1m26s
test / publish-infra (pull_request) Has been skipped
A failed container list or a partial per-container inspect were
previously indistinguishable from a legitimately empty/partial live
set, so reconciliation could silently unregister healthy bottles.

- enumerate_active() now raises EnumerationError instead of returning []
  when `container list` fails
- Add inspect_container_network_ip() to util, which returns None on
  inspect failure (vs "" for "no DHCP address yet"), so live_source_ips
  can tell the two apart
- live_source_ips() raises EnumerationError on either failure mode;
  register_agent() catches it and skips reconciliation, same as
  OrchestratorClientError
- Update tests: rename test_empty_when_the_cli_fails to
  test_raises_when_the_cli_fails, update patching to the new function,
  add coverage for list failure, per-container inspect failure, and the
  launch-not-blocked path
2026-07-21 03:39:18 +00:00
153 changed files with 8955 additions and 1084 deletions
+4
View File
@@ -1,6 +1,10 @@
[run]
branch = True
source = .
# Store paths relative to the project root so .coverage.* files produced on
# different runners (ubuntu-latest vs self-hosted KVM) can be combined by the
# coverage job without a [paths] remapping section.
relative_files = True
[report]
# Coverage policy: see docs/decisions/0004-coverage-policy.md.
+160 -127
View File
@@ -1,4 +1,5 @@
# Run the project's test suite on every PR push and on push to main.
# Run the project's test suite when package or runtime inputs change on a PR
# or on push to main.
#
# The suite uses stdlib `unittest` discovery — no external Python
# dependencies are required to execute it. Tests are split by directory:
@@ -9,10 +10,12 @@
# tests/canaries/ — upstream regression canaries; run on a separate
# schedule (see canaries.yml), not here
#
# Integration tests run once per backend in separate jobs. Each job sets
# BOT_BOTTLE_BACKEND explicitly so the test suite uses the right backend.
# Backends that aren't available on the runner fail the preflight step
# rather than silently skipping inside the test output.
# Each test job runs once under coverage and uploads a small .coverage.*
# artifact. The `coverage` job combines them — no test reruns, no KVM
# dependency on that job. For main-branch pushes only, the tested rootfs
# and matching dropbear are uploaded so `publish-infra` can publish the
# byte-identical artifact that was tested. PRs avoid the ~194 MB rootfs
# transfer entirely.
name: test
@@ -21,72 +24,37 @@ on:
branches:
- main
paths:
- '**.py'
- '.gitea/workflows/**.yml'
- 'scripts/**'
- 'README.md'
# Dockerfiles and pyproject.toml are baked into the infra rootfs; a
# change here alters what the integration/coverage jobs build locally.
- 'bot_bottle/**'
- 'tests/**/*.py'
- 'cli.py'
- 'scripts/coverage.sh'
- 'scripts/critical-modules.txt'
- 'scripts/diff_coverage.py'
- 'scripts/tracker_policy.py'
- 'scripts/firecracker-netpool.sh'
- 'Dockerfile*'
- 'pyproject.toml'
- 'requirements-dev.txt'
- '.coveragerc'
- '.dockerignore'
pull_request:
paths:
- '**.py'
- '.gitea/workflows/**.yml'
- 'scripts/**'
- 'README.md'
- 'bot_bottle/**'
- 'tests/**/*.py'
- 'cli.py'
- 'scripts/coverage.sh'
- 'scripts/critical-modules.txt'
- 'scripts/diff_coverage.py'
- 'scripts/tracker_policy.py'
- 'scripts/firecracker-netpool.sh'
- 'Dockerfile*'
- 'pyproject.toml'
- 'requirements-dev.txt'
- '.coveragerc'
- '.dockerignore'
workflow_dispatch:
jobs:
stage-firecracker-inputs:
runs-on: [self-hosted, kvm]
# Same guard as the other KVM-runner jobs: don't spin the privileged
# runner for fork PRs (this only copies a non-secret static binary, but
# keep the posture consistent — build-infra/integration/coverage all
# depend on it, so gating here gates the whole Firecracker chain).
if: >-
github.event_name == 'push' ||
github.event_name == 'workflow_dispatch' ||
(github.event_name == 'pull_request' &&
github.event.pull_request.head.repo.full_name == github.repository)
steps:
- name: Stage the provisioned static dropbear
run: |
mkdir -p firecracker-inputs
cp /var/cache/bot-bottle-fc/dropbear firecracker-inputs/dropbear
- name: Upload Firecracker build inputs
uses: actions/upload-artifact@v3
with:
name: firecracker-inputs
path: firecracker-inputs/
build-infra:
needs: stage-firecracker-inputs
runs-on: ubuntu-latest
steps:
- name: Checkout
uses: actions/checkout@v4
- name: Download Firecracker build inputs
uses: actions/download-artifact@v3
with:
name: firecracker-inputs
path: firecracker-inputs
- name: Build infra candidate from this checkout
env:
BOT_BOTTLE_FC_DROPBEAR: ${{ github.workspace }}/firecracker-inputs/dropbear
run: python3 -m bot_bottle.backend.firecracker.publish_infra --output infra-candidate
- name: Upload infra candidate
uses: actions/upload-artifact@v3
with:
name: infra-candidate
path: infra-candidate/
unit:
runs-on: ubuntu-latest
steps:
@@ -101,12 +69,29 @@ jobs:
- name: Install dev requirements
run: python3 -m pip install --break-system-packages -r requirements-dev.txt
- name: Run unit tests
- name: Run unit tests with coverage
env:
COVERAGE_FILE: ${{ github.workspace }}/.coverage.unit
run: python3 -m coverage run -m unittest discover -t . -s tests/unit -v
- name: Report unit coverage
env:
COVERAGE_FILE: ${{ github.workspace }}/.coverage.unit
run: python3 -m coverage report -m
# upload-artifact@v3's glob skips dotfiles, so a bare `.coverage.unit`
# silently uploads nothing ("No files were found"). Stage it under a
# non-dot name; the coverage job renames it back before `coverage
# combine`. `cp` also fails loudly if coverage never wrote the file.
- name: Stage unit coverage for upload
run: cp .coverage.unit coverage-unit.dat
- name: Upload unit coverage artifact
uses: actions/upload-artifact@v3
with:
name: coverage-unit
path: coverage-unit.dat
integration-docker:
runs-on: ubuntu-latest
steps:
@@ -115,19 +100,35 @@ jobs:
# No actions/setup-python (see the note in the `unit` job); the
# container's system Python 3.12 runs the stdlib test suite directly.
- name: Show environment
- name: Install coverage
run: python3 -m pip install --break-system-packages coverage
# Fail loudly if the backend this job promises isn't actually usable,
# rather than letting every test silently `unittest.skip` and the job
# go green on zero coverage. `backend status` prints a clear per-check
# summary (docker on PATH, daemon reachable) and exits non-zero when a
# prerequisite is missing — the same readiness check the skip guards
# gate on via `has_backend`.
- name: Preflight — Docker backend is ready
run: |
python3 --version
if command -v docker >/dev/null 2>&1; then
docker version || true
else
echo "docker not on PATH — integration tests will skip"
fi
python3 cli.py backend status --backend=docker
- name: Run integration tests (docker)
- name: Run integration tests (docker) with coverage
env:
BOT_BOTTLE_BACKEND: docker
run: python3 -m unittest discover -t . -s tests/integration -v
COVERAGE_FILE: ${{ github.workspace }}/.coverage.docker
run: python3 -m coverage run -m unittest discover -t . -s tests/integration -v
# Non-dot name so upload-artifact's dotfile-skipping glob picks it up.
- name: Stage docker coverage for upload
run: cp .coverage.docker coverage-docker.dat
- name: Upload docker coverage artifact
uses: actions/upload-artifact@v3
with:
name: coverage-docker
path: coverage-docker.dat
# Integration tests against the Firecracker backend. Runs on a self-hosted
# KVM runner (label `kvm`) where /dev/kvm and the TAP/nft pool are available.
@@ -137,9 +138,16 @@ jobs:
#
# Runner prerequisites (provision once; see README "Firecracker on Linux"):
# `firecracker` on PATH, `/dev/kvm` accessible, cached kernel +
# static dropbear, and the pool as a persistent systemd unit.
# static dropbear at /var/cache/bot-bottle-fc/dropbear, and the pool as a
# persistent systemd unit.
#
# The infra candidate is built here directly (no artifact download) to
# eliminate the ~70 s ubuntu-latest upload + ~83 s combined download that
# the old build-infra → integration-firecracker + coverage chain incurred.
# For main-branch pushes the tested rootfs and matching dropbear are
# uploaded so publish-infra can publish the byte-identical artifact; PRs
# skip those uploads entirely.
integration-firecracker:
needs: build-infra
runs-on: [self-hosted, kvm]
if: >-
github.event_name == 'push' ||
@@ -159,49 +167,65 @@ jobs:
# range overlap; it prints the exact `backend setup` fix.
python3 cli.py backend status --backend=firecracker
- name: Download the candidate built from this checkout
uses: actions/download-artifact@v3
with:
name: infra-candidate
path: infra-candidate
- name: Build infra candidate from this checkout
env:
BOT_BOTTLE_FC_DROPBEAR: /var/cache/bot-bottle-fc/dropbear
run: python3 -m bot_bottle.backend.firecracker.publish_infra --output infra-candidate --reuse-published
- name: Replace the persistent infra VM with the candidate
run: python3 -c 'from bot_bottle.backend.firecracker import infra_vm; infra_vm.stop()'
# No dev-requirements install: the integration suite runs on stdlib
# `unittest` (pylint/pyright are lint.yml's concern, not this job's),
# and the self-hosted runner's Nix python env has no `pip` module
# (`python3 -m pip` → "No module named pip"). Nothing to install.
- name: Run integration tests (firecracker)
# No dev-requirements install: `coverage` is already provided by the
# self-hosted runner's Nix python env, and that env has no `pip`
# module to install into anyway.
- name: Run integration tests (firecracker) with coverage
env:
BOT_BOTTLE_BACKEND: firecracker
BOT_BOTTLE_INFRA_ARTIFACT_DIR: ${{ github.workspace }}/infra-candidate
run: python3 -m unittest discover -t . -s tests/integration -v
COVERAGE_FILE: ${{ github.workspace }}/.coverage.firecracker
run: python3 -m coverage run -m unittest discover -t . -s tests/integration -v
# Combined unit+integration coverage + the diff-coverage gate (the hard
# gate: new/changed lines >= 90%). See docs/decisions/0004-coverage-policy.md.
# Non-dot name so upload-artifact's dotfile-skipping glob picks it up.
- name: Stage firecracker coverage for upload
run: cp .coverage.firecracker coverage-firecracker.dat
- name: Upload firecracker coverage artifact
uses: actions/upload-artifact@v3
with:
name: coverage-firecracker
path: coverage-firecracker.dat
# Only upload the large rootfs artifact on main-branch pushes;
# PRs avoid the ~194 MB transfer. publish-infra only runs on main
# and downloads these to publish the byte-identical tested rootfs.
- name: Upload tested rootfs (main branch only)
if: github.event_name == 'push' && github.ref == 'refs/heads/main'
uses: actions/upload-artifact@v3
with:
name: infra-candidate
path: infra-candidate/
- name: Upload dropbear for publish verification (main branch only)
if: github.event_name == 'push' && github.ref == 'refs/heads/main'
uses: actions/upload-artifact@v3
with:
name: firecracker-inputs
path: /var/cache/bot-bottle-fc/dropbear
# Combined coverage gate: aggregates .coverage.* artifacts uploaded by each
# test job, then runs the diff-coverage gate (new/changed lines >= 90%).
#
# This runs on a self-hosted KVM runner (label `kvm`), NOT ubuntu-latest,
# because the Firecracker backend's subprocess/VM orchestration
# (launch/boot/SSH/isolation-probe) is covered by the integration suite,
# and that suite needs `/dev/kvm` + the provisioned TAP/nft pool — which a
# container-based runner doesn't have. On such a runner the firecracker
# integration test skips and its ~230 orchestration lines read as
# uncovered, so the gate can't pass there.
# Runs on ubuntu-latest — no KVM needed, no test reruns. Coverage files use
# relative_files = True (.coveragerc) so they combine cleanly across runners.
# Each test job sets COVERAGE_FILE to an absolute path so coverage.py writes
# to a known location that upload-artifact can find regardless of runner env.
#
# Restricted to the same events as integration-firecracker (same-repo PRs,
# push, workflow_dispatch) for the same security reason.
#
# See #414 for the planned follow-up: artifact-based coverage combination
# (run tests once in their respective jobs, combine .coverage files here).
#
# build-infra creates one candidate from the checkout. This job boots that
# same candidate after integration-firecracker has exercised it; the main
# push path publishes the identical bytes only after every required job.
# Restricted to the same events as integration-firecracker: it depends on
# that job's coverage artifact and skips for fork PRs alongside it.
coverage:
needs: [build-infra, integration-firecracker]
needs: [unit, integration-docker, integration-firecracker]
timeout-minutes: 15
runs-on: [self-hosted, kvm]
runs-on: ubuntu-latest
if: >-
github.event_name == 'push' ||
github.event_name == 'workflow_dispatch' ||
@@ -213,29 +237,37 @@ jobs:
with:
fetch-depth: 0
- name: Preflight — Firecracker host is ready
run: |
command -v firecracker >/dev/null || {
echo "firecracker not on PATH — provision the runner (README: Firecracker on Linux)"; exit 1; }
test -e /dev/kvm || { echo "/dev/kvm missing — KVM not available on this runner"; exit 1; }
# `backend status` exits non-zero unless the TAP pool is up + no
# range overlap; it prints the exact `backend setup` fix.
python3 cli.py backend status --backend=firecracker
- name: Install coverage
run: python3 -m pip install --break-system-packages coverage
- name: Download the candidate already exercised by integration
- name: Download unit coverage artifact
uses: actions/download-artifact@v3
with:
name: infra-candidate
path: infra-candidate
name: coverage-unit
path: ${{ github.workspace }}
- name: Download docker coverage artifact
uses: actions/download-artifact@v3
with:
name: coverage-docker
path: ${{ github.workspace }}
- name: Download firecracker coverage artifact
uses: actions/download-artifact@v3
with:
name: coverage-firecracker
path: ${{ github.workspace }}
# Rename the non-dot upload names back to the .coverage.* files that
# `coverage combine` discovers (see the staging steps in each test job).
- name: Reassemble coverage data files
run: |
mv coverage-unit.dat .coverage.unit
mv coverage-docker.dat .coverage.docker
mv coverage-firecracker.dat .coverage.firecracker
# No dev-requirements install: `coverage` is already provided by the
# self-hosted runner's Nix python env, and that env has no `pip`
# module to install into anyway. `scripts/coverage.sh` +
# `diff_coverage.py` need only `coverage` (not pylint/pyright).
- name: Combined coverage (unit + integration, incl. firecracker)
env:
BOT_BOTTLE_CI_INFRA_ARTIFACT_DIR: ${{ github.workspace }}/infra-candidate
run: PYTHON=python3 bash scripts/coverage.sh critical
run: PYTHON=python3 bash scripts/coverage.sh aggregate critical
- name: Diff-coverage gate (changed lines >= 90%)
run: |
@@ -243,14 +275,14 @@ jobs:
python3 scripts/diff_coverage.py --base origin/main --min 90
publish-infra:
needs: [stage-firecracker-inputs, build-infra, unit, integration-docker, integration-firecracker, coverage]
needs: [unit, integration-docker, integration-firecracker, coverage]
runs-on: ubuntu-latest
if: github.event_name == 'push' && github.ref == 'refs/heads/main'
steps:
- name: Checkout the tested revision
uses: actions/checkout@v4
- name: Download the tested candidate
- name: Download the tested rootfs
uses: actions/download-artifact@v3
with:
name: infra-candidate
@@ -258,9 +290,10 @@ jobs:
# publish_infra re-derives the version from the checkout to confirm the
# bundle matches before uploading, and the version hashes the dropbear
# bytes. Stage the SAME dropbear build-infra used, or the recheck
# computes a "<missing>"-dropbear version and rejects the candidate.
- name: Download the staged dropbear (matches build-infra's version)
# bytes. Download the SAME dropbear integration-firecracker used, or
# the recheck computes a "<missing>"-dropbear version and rejects the
# candidate.
- name: Download the staged dropbear (matches build's version)
uses: actions/download-artifact@v3
with:
name: firecracker-inputs
+4 -1
View File
@@ -14,11 +14,14 @@ on:
jobs:
update-badges:
runs-on: ubuntu-latest
permissions:
contents: write
steps:
- uses: actions/checkout@v3
with:
fetch-depth: 0
token: ${{ secrets.GITHUB_TOKEN }}
token: ${{ secrets.BADGE_PUSH_TOKEN }}
# No actions/setup-python: the runner image ships Python 3.12 and older
# act_runner engines mishandle setup-python's PATH. Install into the
+13 -36
View File
@@ -1,45 +1,22 @@
# Firecracker single infra-VM image (PRD 0070 Stage B).
# Shared infra image: gateway data plane + orchestrator control plane.
#
# The per-host infra VM runs the orchestrator control plane, the gateway
# data plane, AND builds agent images (buildah) — all in one microVM (see
# backend/firecracker/infra_vm.py). It composes:
# * FROM the gateway image (mitmproxy / git / gitleaks / supervise + the
# flat daemon modules) — now trixie-based, so buildah 1.39 is available;
# * `COPY --from` the orchestrator image's content (the single definition
# of the control-plane payload — see Dockerfile.orchestrator), so this
# VM and the docker backend share one orchestrator definition; and
# * buildah, installed HERE only (the docker orchestrator/gateway images
# never carry it).
# Used directly by the Docker backend (run as one `bot-bottle-infra`
# container, replacing the prior two-container split). The Firecracker
# backend extends this via Dockerfile.infra.fc, adding buildah/crun/
# netavark for in-VM agent-image building.
#
# Dockerfile.orchestrator is the single definition of the orchestrator
# content (the lean `bot_bottle` package on python:3.12-slim). Both this
# image and Dockerfile.infra.fc pull it in via `COPY --from`.
#
# multi-`FROM` can't union two bases (that's multi-stage, not multiple
# inheritance), so the orchestrator content is pulled in via `COPY --from`
# rather than a second base. Both images share the trixie `python:3.12-slim`
# base, so the copy is clean (same python; future installed deps copy too).
#
# The docker backend keeps orchestrator + gateway as separate images; this
# combined image exists only for the Firecracker single-VM cut. Splitting a
# service back into its own VM later is a routing change, not a repackaging
# (PRD 0070's "secret concentration"; a disposable builder can boot from
# this same image on its own TAP).
FROM bot-bottle-gateway:latest
# --- in-VM agent-image builder (PRD 0069 Stage 3) -------------------
# The Firecracker backend builds users' agent Dockerfiles *inside this VM*
# with buildah (rootless, daemonless) instead of on the host — no host
# Docker daemon, no root-equivalent `docker` group. `crun` is the OCI
# runtime; `netavark` + `aardvark-dns` are the network backend for `FROM`
# pulls + `RUN` egress. Requires the trixie base (buildah 1.39: bookworm's
# 1.28 can't parse Dockerfile heredocs that agent images use).
RUN apt-get update \
&& apt-get install -y --no-install-recommends \
buildah crun netavark aardvark-dns \
&& rm -rf /var/lib/apt/lists/*
# vfs + chroot: buildah works as root in the bare microVM (no
# fuse-overlayfs / overlay module / subuid maps). Matches image_builder.
ENV STORAGE_DRIVER=vfs \
BUILDAH_ISOLATION=chroot
# The orchestrator content, pulled from its single definition. The gateway
# image already has the flat daemon modules under /app; this adds the full
# `bot_bottle` package so `python3 -m bot_bottle.orchestrator` resolves.
# The orchestrator content, from its single definition. The gateway image
# already has the flat daemon modules under /app; this adds the full
# `bot_bottle` package so `python3 -m bot_bottle.orchestrator` resolves —
# used by gateway_init when BOT_BOTTLE_GATEWAY_DAEMONS includes `orchestrator`.
COPY --from=bot-bottle-orchestrator:latest /app/bot_bottle /app/bot_bottle
+23
View File
@@ -0,0 +1,23 @@
# Firecracker infra VM image (PRD 0070 Stage B).
#
# Extends the shared infra base (Dockerfile.infra: gateway + orchestrator
# control plane) with the in-VM agent-image builder. The Firecracker backend
# builds users' agent Dockerfiles *inside this VM* with buildah (rootless,
# daemonless) instead of on the host — no host Docker daemon, no
# root-equivalent `docker` group.
#
# Requires the trixie base from bot-bottle-gateway (buildah 1.39: bookworm's
# 1.28 can't parse Dockerfile heredocs that agent images use).
#
# `crun` is the OCI runtime; `netavark` + `aardvark-dns` are the network
# backend for `FROM` pulls + `RUN` egress. `vfs` + `chroot`: buildah works
# as root in the bare microVM (no fuse-overlayfs / overlay module / subuid
# maps). Matches image_builder.
FROM bot-bottle-infra:latest
RUN apt-get update \
&& apt-get install -y --no-install-recommends \
buildah crun netavark aardvark-dns \
&& rm -rf /var/lib/apt/lists/*
ENV STORAGE_DRIVER=vfs \
BUILDAH_ISOLATION=chroot
+91 -9
View File
@@ -5,7 +5,7 @@
# bot-bottle
[![test](https://gitea.dideric.is/didericis/bot-bottle/actions/workflows/test.yml/badge.svg?branch=main)](https://gitea.dideric.is/didericis/bot-bottle/actions?workflow=test.yml)
[![coverage](https://img.shields.io/badge/coverage-81%25-brightgreen)](https://coverage.readthedocs.io/)
[![coverage](https://img.shields.io/badge/coverage-84%25-brightgreen)](https://coverage.readthedocs.io/)
[![core coverage](https://img.shields.io/badge/core%20coverage-94%25-brightgreen)](https://gitea.dideric.is/didericis/bot-bottle/src/branch/main/docs/decisions/0004-coverage-policy.md)
**Problem:** Developer wants to run a coding agent without supervision, but they don't want a prompt injected or misbehaving agent wrecking their environment or exfiltrating sensitive data.
@@ -75,6 +75,88 @@ On compatible macOS hosts, the default backend requires Apple's `container` CLI
Use `BOT_BOTTLE_BACKEND=docker ./cli.py start <agent>` on hosts where neither Apple Container nor KVM is available and Docker is the desired backend.
### Containers inside a bottle
A bottle may set `nested_containers: true`. On the macOS backend this starts a
guest-local, rootless **podman** service after the bottle is registered and
exposes its Docker-compatible API socket, so the agent still runs `docker` and
`docker compose`. Nothing is mounted from the host: Docker Desktop's socket
stays out of the bottle and the guest gains no outer VM capabilities. Backends
that cannot do this (`docker`, `firecracker`) reject the flag rather than
silently ignore it.
Rootless Docker was tried first and does not work here at all: Apple
Container's capability bounding set omits `CAP_SYS_ADMIN`, which the kernel
requires to write a multi-range `uid_map`. See
[`docs/research/rootless-docker-in-apple-container-spike.md`](docs/research/rootless-docker-in-apple-container-spike.md).
The tradeoff to understand before enabling it: podman avoids that requirement
by falling back to a single-UID mapping, so nested containers provide **no
isolation from the agent itself** — `root` inside a nested container is the
agent user outside it. Nested containers are a build/test convenience, not a
security boundary. The bottle remains the boundary.
Pulling images goes through the bottle's egress proxy like every other
request, so each registry needs a route — **and so does the CDN it redirects
layer blobs to**, which is a different host. Without the CDN route the pull
authenticates, fetches the manifest, then 403s partway through.
Docker Hub and GHCR additionally need `preserve_auth: true`: their token dance
uses a client-fetched per-scope bearer token that the proxy would otherwise
strip. Turn DLP off on every registry and CDN route — the bodies are
compressed layer blobs that no detector can read, and buffering them is what
triggers the shared-proxy OOM in #455.
```yaml
nested_containers: true
egress:
routes:
# Docker Hub: registry, token endpoint, blob CDN.
- host: registry-1.docker.io
preserve_auth: true
dlp: { outbound_detectors: false, inbound_detectors: false }
- host: auth.docker.io
preserve_auth: true
dlp: { outbound_detectors: false, inbound_detectors: false }
- host: production.cloudfront.docker.com
dlp: { outbound_detectors: false, inbound_detectors: false }
# GHCR: registry + blob CDN.
- host: ghcr.io
preserve_auth: true
dlp: { outbound_detectors: false, inbound_detectors: false }
- host: pkg-containers.githubusercontent.com
dlp: { outbound_detectors: false, inbound_detectors: false }
# quay.io: registry + blob CDNs. No preserve_auth needed for public pulls.
- host: quay.io
dlp: { outbound_detectors: false, inbound_detectors: false }
- host: cdn01.quay.io
dlp: { outbound_detectors: false, inbound_detectors: false }
- host: cdn02.quay.io
dlp: { outbound_detectors: false, inbound_detectors: false }
- host: cdn03.quay.io
dlp: { outbound_detectors: false, inbound_detectors: false }
```
`mcr.microsoft.com` and `registry.k8s.io` follow the same shape and also
redirect blobs elsewhere (`*.data.mcr.microsoft.com` and
`us-*-docker.pkg.dev` respectively); route whichever host the 403 names.
Inside a nested container the same allowlist applies: an allowlisted host
returns 200 and anything else gets a 403 straight from the proxy. The
gateway's CA bundle and proxy settings are wired in automatically, so
`docker run … curl https://…` works with no extra flags — no `--add-host`,
`-e`, or `-v`.
Two things worth knowing when testing that:
- Public DNS inside a nested container fails **by design**. Everything
egresses through the proxy, so `nslookup` failing is expected and is not
evidence of a problem.
- Alpine's BusyBox `wget` drops the connection after the proxy's TLS
interception and reports `error getting response`, even though the proxy
logs the decrypted request and returns a response. Use `curl` to test
egress; BusyBox `wget` will lie to you.
### Firecracker on Linux
On Linux, a KVM-capable host defaults to the Firecracker backend. It needs:
@@ -123,14 +205,14 @@ git:
egress:
routes:
- host: gitea.dideric.is
auth:
scheme: token # Bearer | token
token_ref: BOT_BOTTLE_GITEA_TOKEN
matches: # optional — restrict to specific paths/methods/headers
- paths:
- {type: prefix, value: /api/v1/}
methods: [GET, POST, PATCH, DELETE]
dlp: # optional — per-route detector overrides (default: all on)
inspect:
auth:
scheme: token # Bearer | token
token_ref: BOT_BOTTLE_GITEA_TOKEN
matches: # optional — restrict to specific paths/methods/headers
- paths:
- {type: prefix, value: /api/v1/}
methods: [GET, POST, PATCH, DELETE]
outbound_detectors: [token_patterns, known_secrets]
inbound_detectors: false # disable response scanning for this host
---
+43 -2
View File
@@ -45,6 +45,10 @@ PROVIDER_TEMPLATES = frozenset({PROVIDER_CLAUDE, PROVIDER_CODEX, PROVIDER_PI})
# forward_host_credentials is enabled. Pipelock must pass these through
# (no TLS MITM) or its header DLP blocks the injected JWT.
CODEX_HOST_CREDENTIAL_HOSTS = ("api.openai.com", "chatgpt.com")
# Host that egress injects the host Claude bearer on when Claude
# forward_host_credentials is enabled.
CLAUDE_HOST_CREDENTIAL_HOSTS = ("api.anthropic.com",)
PromptMode = Literal[
"append_file",
"read_prompt_file",
@@ -257,7 +261,28 @@ class AgentProvider(ABC):
Default: Debian/node — writes the git-gate insteadOf gitconfig
and sets user.name/email as node. Workspace copy runs through
BottleBackend.provision_workspace against the running bottle."""
from .log import info
from .log import die, info
# Firecracker exports image rootfs files through an unprivileged host
# tar extraction, so image-time ownership of XDG directories is not
# preserved. Git consults ~/.config/git even when the actual config
# is ~/.gitconfig; an unreadable directory there can prevent the
# git-gate insteadOf rules below from taking effect. Repair this at
# runtime, after every backend's copy/export path has completed.
git_xdg_dir = f"{plan.guest_home}/.config/git"
repair = bottle.exec(
f"chown node:node {shlex.quote(plan.guest_home)} && "
f"chmod 755 {shlex.quote(plan.guest_home)} && "
f"mkdir -p {shlex.quote(git_xdg_dir)} && "
f"chown -R node:node {shlex.quote(f'{plan.guest_home}/.config')} && "
f"chmod -R u+rwX,go+rX {shlex.quote(f'{plan.guest_home}/.config')}",
user="root",
)
if repair.returncode != 0:
die(
"git provisioning: could not make the runtime Git config "
f"directory readable: {(repair.stderr or repair.stdout).strip()}"
)
manifest_bottle = plan.manifest.bottle
if manifest_bottle.git:
@@ -280,11 +305,27 @@ class AgentProvider(ABC):
f"{len(manifest_bottle.git)} insteadOf rule(s)"
)
bottle.cp_in(str(config_file), guest_gitconfig)
bottle.exec(
permissions = bottle.exec(
f"chown node:node {shlex.quote(guest_gitconfig)} && "
f"chmod 644 {shlex.quote(guest_gitconfig)}",
user="root",
)
if permissions.returncode != 0:
die(
"git provisioning: could not set ownership on "
f"{guest_gitconfig}: "
f"{(permissions.stderr or permissions.stdout).strip()}"
)
configured = bottle.exec(
"git config --global --get-regexp '^url\\..*\\.insteadof$'",
user="node",
)
if configured.returncode != 0:
die(
"git provisioning: the runtime user cannot read the "
f"git-gate insteadOf rules from {guest_gitconfig}: "
f"{(configured.stderr or configured.stdout).strip()}"
)
gu = manifest_bottle.git_user
if not gu.is_empty():
+89 -8
View File
@@ -33,14 +33,15 @@ backend field; the host picks.
from __future__ import annotations
import enum
import os
import shlex
import sys
from abc import ABC, abstractmethod
from contextlib import AbstractContextManager
from contextlib import AbstractContextManager, contextmanager
from dataclasses import dataclass
from pathlib import Path
from typing import TYPE_CHECKING, Any, Generic, Sequence, TypeVar
from typing import TYPE_CHECKING, Any, Generator, Generic, Sequence, TypeVar
from ..agent_provider import AgentProvisionPlan, get_provider, build_agent_provision_plan
from ..egress import EgressPlan
@@ -59,6 +60,12 @@ if TYPE_CHECKING:
from .freeze import CommitCancelled, Freezer, get_freezer
class BackendStatus(enum.IntEnum):
"""Return codes for BottleBackend.status(). READY == 0 so callsites
can compare against 0 or the named constant interchangeably."""
READY = 0
@dataclass(frozen=True)
class BottleSpec:
"""CLI-supplied intent. Backend-agnostic — each backend's prepare
@@ -83,6 +90,9 @@ class BottleSpec:
# True when launched via --headless (no TTY, no interactive prompts).
# The git-gate host-key preflight uses this to error rather than prompt.
headless: bool = False
# Image startup policy. "fresh" preserves the normal build path;
# "cached" reuses the current local image/artifact without rebuilding.
image_policy: str = "fresh"
@dataclass(frozen=True)
@@ -278,6 +288,18 @@ PlanT = TypeVar("PlanT", bound=BottlePlan)
CleanupT = TypeVar("CleanupT", bound=BottleCleanupPlan)
@dataclass(frozen=True)
class BottleImages:
"""Resolved image references (or artifact paths) for a bottle launch.
For Docker/macOS-container backends, `agent` and `sidecar` are string
image refs. For the smolmachines backend they are Path objects pointing
to pre-built `.smolmachine` artifacts."""
agent: str | Path
sidecar: str | Path = ""
class BottleBackend(ABC, Generic[PlanT, CleanupT]):
"""Abstract base for selectable bottle backends. Concrete subclasses
(e.g. DockerBottleBackend) own their own prepare/launch impls.
@@ -287,6 +309,11 @@ class BottleBackend(ABC, Generic[PlanT, CleanupT]):
name: str
# Whether this backend can run a container engine *inside* the bottle.
# Backends that cannot must reject `nested_containers: true` rather than
# reach for a host daemon socket (issue #392).
supports_nested_containers: bool = False
def prepare(self, spec: BottleSpec, stage_dir: Path) -> PlanT:
"""Template method: run cross-backend host-side validation, then
delegate to the subclass's `_resolve_plan` for the
@@ -300,12 +327,16 @@ class BottleBackend(ABC, Generic[PlanT, CleanupT]):
prepare_egress,
prepare_git_gate,
prepare_supervise,
reject_nested_containers,
resolve_manifest_dockerfile,
write_launch_metadata,
)
manifest = self._validate(spec)
if not self.supports_nested_containers:
reject_nested_containers(self.name, manifest)
self._preflight()
from ..git_gate_host_key import preflight_host_keys
@@ -437,9 +468,27 @@ class BottleBackend(ABC, Generic[PlanT, CleanupT]):
prompt file, Dockerfile path, and guest home all live on
`agent_provision_plan` — the source of truth."""
def prelaunch_checks(self, plan: PlanT) -> None:
"""Raise StaleImageError if any cached image used by this plan is stale.
No-op default; backends override to call the shared check_stale*
helpers on their image/artifact timestamps. Called by the CLI before
launch so the operator can be prompted outside the launch context."""
@contextmanager
def launch(self, plan: PlanT) -> Generator[Bottle, None, None]:
"""Template: build or load images, then delegate to _launch_impl."""
images = self._build_or_load_images(plan)
with self._launch_impl(plan, images) as bottle:
yield bottle
@abstractmethod
def launch(self, plan: PlanT) -> AbstractContextManager[Bottle]:
"""Build/run the bottle and yield a handle; tear down on exit."""
def _build_or_load_images(self, plan: PlanT) -> BottleImages:
"""Return the agent and sidecar image references (or artifact paths)
for this plan, building fresh images when the policy requires it."""
@abstractmethod
def _launch_impl(self, plan: PlanT, images: BottleImages) -> AbstractContextManager[Bottle]:
"""Bring up the bottle using pre-resolved images; yield a handle; tear down on exit."""
def provision(self, plan: PlanT, bottle: "Bottle") -> str | None:
"""Copy host-side files (CA cert, prompt, skills, .git) into
@@ -569,12 +618,18 @@ class BottleBackend(ABC, Generic[PlanT, CleanupT]):
@classmethod
@abstractmethod
def status(cls) -> int:
def status(cls, *, quiet: bool = False) -> int:
"""Report whether this backend's prerequisites are satisfied on
the host — binaries, daemon reachability, network pool, range
conflicts, etc. Prints a human-readable summary; returns 0 when
the backend is ready to launch and non-zero when something is
missing. Invoked by `./cli.py backend status [--backend=…]`."""
conflicts, etc. Returns BackendStatus.READY (0) when the backend
is ready to launch and non-zero when something is missing.
When quiet=False (default) prints a human-readable summary to
stderr. When quiet=True returns the status code silently —
useful for cheap programmatic checks.
Invoked by `./cli.py backend status [--backend=…]` (quiet=False)
and by is_backend_ready() (caller-controlled)."""
@classmethod
@abstractmethod
@@ -769,6 +824,29 @@ def has_backend(name: str) -> bool:
return backends[name].is_available()
def is_backend_available(name: str) -> bool:
"""Cheap availability check: is the backend's binary on PATH?
Suitable for cleanup enumeration and auto-selection — does NOT probe
the daemon or network pool. Use is_backend_ready() for a full
readiness check before launching tests."""
return has_backend(name)
def is_backend_ready(name: str, *, quiet: bool = False) -> bool:
"""Full readiness check: passes all of the backend's status() checks.
When quiet=False the backend prints diagnostic output explaining what
is missing — intended for test-suite guards that run at discovery time
so the operator sees a concrete failure reason for each skip.
Returns False for unknown backend names."""
backends = _get_backends()
if name not in backends:
return False
return backends[name].status(quiet=quiet) == BackendStatus.READY
def enumerate_active_agents() -> list[ActiveAgent]:
"""All currently-running agents, across every available
backend. Used by CLI `list active` and the dashboard's agents
@@ -793,6 +871,7 @@ def enumerate_active_agents() -> list[ActiveAgent]:
__all__ = [
"ActiveAgent",
"BackendStatus",
"Bottle",
"BottleBackend",
"BottleCleanupPlan",
@@ -805,5 +884,7 @@ __all__ = [
"get_bottle_backend",
"get_freezer",
"has_backend",
"is_backend_available",
"is_backend_ready",
"known_backend_names",
]
+69
View File
@@ -0,0 +1,69 @@
"""Shared helpers for the consolidated launch sequence (PRD 0070).
Logic that was duplicated across the docker, macos_container, and
firecracker consolidated_launch modules — extracted so each backend
imports it rather than re-implementing it.
"""
from __future__ import annotations
import dataclasses
from ..egress import EgressPlan
from ..git_gate import GitGatePlan
from ..orchestrator.client import OrchestratorClient, RegisteredBottle
from ..orchestrator.registration import registration_inputs
from ..orchestrator.secret_store import new_env_var_secret
from .docker.gateway_provision import GatewayTransport, deprovision_git_gate, provision_git_gate
def provision_bottle(
client: OrchestratorClient,
source_ip: str,
egress_plan: EgressPlan,
git_gate_plan: GitGatePlan,
transport: GatewayTransport,
*,
image_ref: str = "",
tokens: dict[str, str] | None = None,
env_var_secret: str | None = None,
) -> RegisteredBottle:
"""Register the bottle and provision its git-gate state. Rolls back the
registration if provisioning fails so no orphan is left.
Generates a fresh ENV_VAR_SECRET, passes it to the orchestrator so it can
encrypt the token values at rest, and stamps the secret onto the returned
``RegisteredBottle`` so callers can inject it into the agent container's
environment."""
inputs = registration_inputs(egress_plan)
env_var_secret = env_var_secret or new_env_var_secret()
reg = client.register_bottle(
source_ip, image_ref=image_ref, policy=inputs.policy,
metadata=inputs.metadata, tokens=tokens, env_var_secret=env_var_secret,
)
try:
provision_git_gate(transport, reg.bottle_id, git_gate_plan)
except Exception:
client.teardown_bottle(reg.bottle_id)
raise
return dataclasses.replace(reg, env_var_secret=env_var_secret)
def teardown_consolidated(
bottle_id: str,
transport: GatewayTransport,
*,
orchestrator_url: str,
timeout: float | None = None,
) -> None:
"""Deregister the bottle and remove its git-gate state. Both steps are
idempotent so this is safe from a cleanup trap."""
from ..orchestrator.config_store import DEFAULT_TEARDOWN_TIMEOUT_SECONDS
OrchestratorClient(
orchestrator_url,
timeout=timeout if timeout is not None else DEFAULT_TEARDOWN_TIMEOUT_SECONDS,
).teardown_bottle(bottle_id)
deprovision_git_gate(transport, bottle_id)
__all__ = ["provision_bottle", "teardown_consolidated"]
+15 -5
View File
@@ -20,7 +20,8 @@ infrastructure: CA install and git copy-in.
from __future__ import annotations
import shutil
from contextlib import contextmanager
import io
from contextlib import contextmanager, redirect_stderr
from pathlib import Path
from typing import Generator, Sequence
@@ -31,7 +32,7 @@ from ...env import ResolvedEnv
from ...git_gate import GitGatePlan
from ...supervise import SupervisePlan
from ...manifest import Manifest
from .. import ActiveAgent, BottleBackend, BottleSpec
from .. import ActiveAgent, BottleBackend, BottleImages, BottleSpec
from . import cleanup as _cleanup
from . import enumerate as _enumerate
from . import launch as _launch
@@ -60,8 +61,11 @@ class DockerBottleBackend(BottleBackend["DockerBottlePlan", "DockerBottleCleanup
return _setup.setup()
@classmethod
def status(cls) -> int:
def status(cls, *, quiet: bool = False) -> int:
from . import setup as _setup
if quiet:
with redirect_stderr(io.StringIO()):
return _setup.status()
return _setup.status()
@classmethod
@@ -100,9 +104,15 @@ class DockerBottleBackend(BottleBackend["DockerBottlePlan", "DockerBottleCleanup
stage_dir=stage_dir,
)
def prelaunch_checks(self, plan: DockerBottlePlan) -> None:
_launch.stale_checks(plan)
def _build_or_load_images(self, plan: DockerBottlePlan) -> BottleImages:
return _launch.build_or_load_images(plan)
@contextmanager
def launch(self, plan: DockerBottlePlan) -> Generator[DockerBottle, None, None]:
with _launch.launch(plan, provision=self.provision) as bottle:
def _launch_impl(self, plan: DockerBottlePlan, images: BottleImages) -> Generator[DockerBottle, None, None]:
with _launch.launch(plan, images, provision=self.provision) as bottle:
yield bottle
def ensure_orchestrator(self) -> str:
+4
View File
@@ -39,6 +39,10 @@ class DockerBottlePlan(BottlePlan):
# (egress proxy credentials, git-gate/supervise headers); set by launch
# from the orchestrator registration. Empty pre-registration.
identity_token: str = ""
# Encryption key for the agent's stored egress secrets; injected into the
# agent container as ENV_VAR_SECRET via the compose subprocess env (bare
# name — value never written to the compose file). Empty pre-registration.
env_var_secret: str = ""
@property
def container_name(self) -> str:
@@ -17,6 +17,7 @@ from __future__ import annotations
from typing import Any
from ...egress import egress_agent_env_entries
from ...orchestrator.secret_store import ENV_VAR_SECRET_NAME
from ..util import AGENT_CA_BUNDLE, AGENT_CA_PATH
from .bottle_plan import DockerBottlePlan
from .egress import EGRESS_PORT
@@ -58,6 +59,10 @@ def consolidated_agent_compose(
# the secret value never lands on argv or in the compose file.
for name in sorted(plan.forwarded_env.keys()):
env.append(name)
# ENV_VAR_SECRET: bare name so the value comes from the compose subprocess
# env (set in launch.py) and is never written to the compose file on disk.
if getattr(plan, "env_var_secret", ""):
env.append(ENV_VAR_SECRET_NAME)
env.extend(egress_agent_env_entries(plan.egress_plan))
service: dict[str, Any] = {
@@ -1,19 +1,13 @@
"""Consolidated bottle launch sequence for the docker backend (PRD 0070).
Composes the orchestrator primitives into the register/teardown sequence that
replaces the per-bottle gateway:
Composes the orchestrator primitives into the register/teardown sequence:
1. ensure the orchestrator control plane + shared gateway are up;
2. allocate the bottle a pinned source IP on the gateway network (the
attribution key), skipping the gateway's own address + live bottles;
3. register it (egress policy blob + slug metadata) → bottle id + identity
token;
4. provision its git-gate repos/creds into the running gateway.
1. ensure the single infra container (control plane + gateway) is up;
2. allocate the bottle a pinned source IP on the gateway network;
3. register it and provision its git-gate repos/creds into the gateway.
It returns a `LaunchContext` with everything the agent container needs to
attach — network, pinned IP, the gateway's address (its proxy target), the
orchestrator URL, and the identity token. The agent `docker run` itself is
the backend's job (it owns provider provisioning); this owns the
Returns a `LaunchContext` with everything the agent container needs to
attach. The agent `docker run` itself is the backend's job; this owns the
orchestrator-facing wiring so that sequence stays testable in isolation.
"""
@@ -21,19 +15,19 @@ from __future__ import annotations
from dataclasses import dataclass
from ... import log
from ...docker_cmd import run_docker
from ...egress import EgressPlan
from ...git_gate import GitGatePlan
from ...orchestrator.client import OrchestratorClient
from ...orchestrator.gateway import GATEWAY_NAME, GATEWAY_NETWORK
from ...orchestrator.lifecycle import OrchestratorService
from ...orchestrator.registration import registration_inputs
from ...orchestrator.gateway import GATEWAY_NETWORK
from ...orchestrator.lifecycle import INFRA_NAME, OrchestratorService
from ...orchestrator.secret_store import ENV_VAR_SECRET_NAME
from ...orchestrator.reprovision import reprovision_bottles
from ..consolidated_util import provision_bottle
from ..consolidated_util import teardown_consolidated as _teardown_util
from .gateway_provision import DockerGatewayTransport
from .gateway_net import next_free_ip
from .gateway_provision import (
DockerGatewayTransport,
deprovision_git_gate,
provision_git_gate,
)
class ConsolidatedLaunchError(RuntimeError):
@@ -50,6 +44,7 @@ class LaunchContext:
network: str # the shared gateway network to attach to
gateway_ip: str # the gateway's address — the agent's proxy target
orchestrator_url: str
env_var_secret: str = "" # encryption key injected into the agent's env
def _network_cidr(network: str) -> str:
@@ -75,28 +70,74 @@ def _container_ip(name: str, network: str) -> str:
ip = proc.stdout.strip()
if proc.returncode != 0 or not ip:
raise ConsolidatedLaunchError(
f"gateway {name} has no address on {network}: {proc.stderr.strip()}"
f"container {name} has no address on {network}: {proc.stderr.strip()}"
)
return ip
def _network_container_ips(network: str) -> list[str]:
"""Every address currently assigned on the gateway network — the ground
truth for "in use": the gateway + orchestrator infrastructure containers
and every live agent. Read from the network so a new bottle can't collide
with anything actually attached (a registry-only view would miss the
orchestrator/gateway containers)."""
truth for "in use": the infra container and every live agent. Read from
the network so a new bottle can't collide with anything actually attached."""
proc = run_docker([
"docker", "network", "inspect", "--format",
"{{range .Containers}}{{.IPv4Address}} {{end}}", network,
])
ips: list[str] = []
for entry in proc.stdout.split():
# entries look like "172.20.0.2/16" — keep the address.
ips.append(entry.split("/", 1)[0])
return ips
def _reprovision_running_bottles(
orchestrator_url: str,
network: str = GATEWAY_NETWORK,
infra_name: str = INFRA_NAME,
) -> None:
"""Re-inject egress tokens for any registered bottles that lost their
in-memory tokens (e.g., after an infra container restart).
For each registered bottle whose source IP maps to a live container on the
gateway network, reads ENV_VAR_SECRET via ``docker exec … printenv`` and
calls ``POST /bottles/<id>/reprovision_gateway``. Idempotent — a no-op
when the orchestrator already has all tokens loaded. Best-effort: a single
container exec failure never blocks a new bottle launch."""
client = OrchestratorClient(orchestrator_url)
# Build {source_ip: container_name} from live containers on the gateway
# network, excluding the infra container itself.
try:
proc = run_docker([
"docker", "network", "inspect",
"--format", "{{range .Containers}}{{.Name}} {{.IPv4Address}}\n{{end}}",
network,
])
except OSError as exc:
log.info(f"egress token reprovision skipped: {exc}")
return
ip_to_container: dict[str, str] = {}
for line in proc.stdout.splitlines():
parts = line.strip().split()
if len(parts) >= 2 and parts[0] != infra_name:
ip = parts[1].split("/", 1)[0]
if ip:
ip_to_container[ip] = parts[0]
secrets_by_ip: dict[str, str] = {}
for source_ip, container_name in ip_to_container.items():
proc = run_docker(
["docker", "exec", container_name, "printenv", ENV_VAR_SECRET_NAME]
)
if proc.returncode == 0 and proc.stdout.strip():
secrets_by_ip[source_ip] = proc.stdout.strip()
reprovisioned = reprovision_bottles(client, secrets_by_ip)
if reprovisioned:
log.info(
"reprovisioned egress tokens",
context={"count": reprovisioned},
)
def launch_consolidated(
egress_plan: EgressPlan,
git_gate_plan: GitGatePlan,
@@ -104,33 +145,29 @@ def launch_consolidated(
image_ref: str = "",
tokens: dict[str, str] | None = None,
service: OrchestratorService | None = None,
gateway_name: str = GATEWAY_NAME,
infra_name: str = INFRA_NAME,
network: str = GATEWAY_NETWORK,
) -> LaunchContext:
"""Ensure the orchestrator + gateway are up, allocate + register the
bottle, and provision its git-gate state. Returns the agent's attach
context. Raises `ConsolidatedLaunchError` (or the primitives' own errors)
if any step fails — the caller tears down on failure."""
"""Ensure the infra container is up, allocate + register the bottle, and
provision its git-gate state. Returns the agent's attach context.
Also reprovisiones egress tokens for any already-running bottles that lost
their in-memory credentials (e.g. after an infra container restart), so
they regain egress access before the new bottle is registered."""
service = service or OrchestratorService()
url = service.ensure_running()
_reprovision_running_bottles(url, network=network, infra_name=infra_name)
client = OrchestratorClient(url)
cidr = _network_cidr(network)
gateway_ip = _container_ip(gateway_name, network)
gateway_ip = _container_ip(infra_name, network)
source_ip = next_free_ip(cidr, _network_container_ips(network))
inputs = registration_inputs(egress_plan)
reg = client.register_bottle(
source_ip, image_ref=image_ref, policy=inputs.policy,
metadata=inputs.metadata, tokens=tokens,
transport = DockerGatewayTransport(infra_name)
reg = provision_bottle(
client, source_ip, egress_plan, git_gate_plan, transport,
image_ref=image_ref, tokens=tokens,
)
try:
provision_git_gate(
DockerGatewayTransport(gateway_name), reg.bottle_id, git_gate_plan)
except Exception:
# Roll the registration back so a provisioning failure leaves no orphan.
client.teardown_bottle(reg.bottle_id)
raise
return LaunchContext(
bottle_id=reg.bottle_id,
identity_token=reg.identity_token,
@@ -138,16 +175,17 @@ def launch_consolidated(
network=network,
gateway_ip=gateway_ip,
orchestrator_url=url,
env_var_secret=reg.env_var_secret,
)
def teardown_consolidated(
bottle_id: str, *, orchestrator_url: str, gateway_name: str = GATEWAY_NAME,
bottle_id: str, *, orchestrator_url: str, infra_name: str = INFRA_NAME,
timeout: float | None = None,
) -> None:
"""Deregister the bottle and remove its git-gate state from the gateway.
Both steps are idempotent so this is safe from a cleanup trap."""
OrchestratorClient(orchestrator_url).teardown_bottle(bottle_id)
deprovision_git_gate(DockerGatewayTransport(gateway_name), bottle_id)
"""Deregister the bottle and remove its git-gate state. Idempotent."""
_teardown_util(bottle_id, DockerGatewayTransport(infra_name),
orchestrator_url=orchestrator_url, timeout=timeout)
__all__ = [
+65 -23
View File
@@ -42,7 +42,9 @@ from ...git_gate import (
provision_git_gate_dynamic_keys,
revoke_git_gate_provisioned_keys,
)
from ...log import info, warn
from ...image_cache import check_stale
from ...log import die, info, warn
from .. import BottleImages
from . import util as docker_mod
from .bottle import DockerBottle
from .bottle_plan import DockerBottlePlan
@@ -62,6 +64,7 @@ from .compose import (
write_compose_file,
)
from .consolidated_compose import consolidated_agent_compose
from ...orchestrator.config_store import resolve_teardown_timeout
from .consolidated_launch import launch_consolidated, teardown_consolidated
from ...orchestrator.gateway import DockerGateway
@@ -70,16 +73,47 @@ from ...orchestrator.gateway import DockerGateway
_REPO_DIR = str(Path(__file__).resolve().parent.parent.parent.parent)
def build_or_load_images(plan: DockerBottlePlan) -> BottleImages:
"""Resolve the agent image ref for this plan.
Returns the committed snapshot if one exists, the cached image when the
policy is 'cached', or builds a fresh image and returns that."""
committed = read_committed_image(plan.slug)
if committed and docker_mod.image_exists(committed):
info(f"using committed image {committed!r}")
return BottleImages(agent=committed)
if plan.spec.image_policy == "cached":
if not docker_mod.image_exists(plan.image):
die(
f"cached agent image {plan.image!r} not found; "
"run without --cached-images to build it"
)
info(f"using cached agent image {plan.image!r}")
return BottleImages(agent=plan.image)
docker_mod.build_image(plan.image, _REPO_DIR, dockerfile=plan.dockerfile_path)
docker_mod.verify_agent_image(
plan.image, runtime_for(plan.agent_provider_template).smoke_test,
)
return BottleImages(agent=plan.image)
@contextmanager
def launch(
plan: DockerBottlePlan,
images: BottleImages,
*,
provision: Callable[[DockerBottlePlan, "DockerBottle"], str | None],
) -> Generator[DockerBottle, None, None]:
"""Build, launch, and provision a Docker bottle via compose.
Teardown on exit."""
"""Launch and provision a Docker bottle via compose. Teardown on exit."""
stack = ExitStack()
# Stamp the resolved agent image ref into the plan so compose rendering
# picks up the right image (may be a committed snapshot or cached ref).
plan = dataclasses.replace(
plan,
agent_provision=dataclasses.replace(plan.agent_provision, image=str(images.agent)),
)
_bottle_for_revoke = plan.manifest.bottle
_git_gate_dir_for_revoke = git_gate_state_dir(plan.slug)
@@ -96,25 +130,6 @@ def launch(
)
try:
# Step 1: agent image. Use a committed snapshot when one exists
# and is present in the local daemon; otherwise build from the
# Dockerfile. (The gateway image is built by the orchestrator.)
committed = read_committed_image(plan.slug)
if committed and docker_mod.image_exists(committed):
info(f"using committed image {committed!r}")
plan = dataclasses.replace(
plan,
agent_provision=dataclasses.replace(plan.agent_provision, image=committed),
)
else:
docker_mod.build_image(
plan.image, _REPO_DIR,
dockerfile=plan.dockerfile_path,
)
docker_mod.verify_agent_image(
plan.image, runtime_for(plan.agent_provider_template).smoke_test,
)
# Step 2: mint the git-gate dynamic (gitea) deploy keys, if any, before
# provisioning the bottle's repos into the shared gateway.
git_gate_plan = plan.git_gate_plan
@@ -133,11 +148,14 @@ def launch(
token_values = egress_resolve_token_values(
plan.egress_plan.token_env_map, effective_env,
)
teardown_timeout = resolve_teardown_timeout()
ctx = launch_consolidated(
plan.egress_plan, git_gate_plan, image_ref=plan.image, tokens=token_values,
)
stack.callback(
teardown_consolidated, ctx.bottle_id, orchestrator_url=ctx.orchestrator_url,
teardown_consolidated, ctx.bottle_id,
orchestrator_url=ctx.orchestrator_url,
timeout=teardown_timeout,
)
# Step 4: install the SHARED gateway CA into the agent (replaces the
@@ -168,6 +186,7 @@ def launch(
agent_git_gate_url=git_gate_url,
agent_supervise_url=supervise_url,
identity_token=ctx.identity_token,
env_var_secret=ctx.env_var_secret,
)
# Step 5: render + up the agent-only compose, pinned on the shared
@@ -180,7 +199,12 @@ def launch(
project = compose_project_name(plan.slug)
# Forwarded vars (OAuth token, host interpolations) flow through the
# subprocess env as bare names so values never land in the file.
# ENV_VAR_SECRET follows the same pattern: bare name in the compose
# spec, value only in the subprocess env so it is never written to disk.
compose_env: dict[str, str] = {**os.environ, **plan.forwarded_env}
if plan.env_var_secret:
from ...orchestrator.secret_store import ENV_VAR_SECRET_NAME
compose_env[ENV_VAR_SECRET_NAME] = plan.env_var_secret
info(
f"docker compose up -d (project {project}, agent on shared "
f"gateway {ctx.gateway_ip}, ip {ctx.source_ip})"
@@ -207,3 +231,21 @@ def launch(
yield bottle
finally:
teardown()
def stale_checks(plan: DockerBottlePlan) -> None:
"""Raise StaleImageError if a cached image is older than the configured
threshold. Only runs when image_policy is 'cached'. Called by the backend
class's _image_stale_checks before _launch_impl starts any resources."""
if plan.spec.image_policy != "cached":
return
committed = read_committed_image(plan.slug)
if committed and docker_mod.image_exists(committed):
ts = docker_mod.image_created_at(committed)
if ts is not None:
check_stale(f"agent image {committed!r}", ts)
return
if docker_mod.image_exists(plan.image):
ts = docker_mod.image_created_at(plan.image)
if ts is not None:
check_stale(f"agent image {plan.image!r}", ts)
+44
View File
@@ -5,6 +5,7 @@ existence, and building images."""
from __future__ import annotations
import os
from datetime import datetime, timezone
import re
import shutil
import subprocess
@@ -197,3 +198,46 @@ def commit_container(container_name: str, image_tag: str) -> None:
f"{(result.stderr or '').strip() or '<no stderr>'}"
)
info(f"committed {container_name!r}{image_tag!r}")
def image_created_at(ref: str) -> datetime | None:
"""Return Docker's image Created timestamp as an aware UTC datetime, or
None when the field is absent or unparseable. Callers should skip the
stale check when None is returned."""
r = subprocess.run(
["docker", "image", "inspect", "--format", "{{.Created}}", ref],
capture_output=True,
text=True,
check=False,
)
if r.returncode != 0:
die(
f"docker image inspect for {ref!r} failed: "
f"{(r.stderr or '').strip() or '<no stderr>'}"
)
raw = r.stdout.strip()
if not raw:
return None
try:
return _parse_docker_timestamp(raw)
except ValueError:
return None
def _parse_docker_timestamp(raw: str) -> datetime:
text = raw.strip()
if text.endswith("Z"):
text = text[:-1] + "+00:00"
dot = text.find(".")
if dot != -1:
tz_plus = text.find("+", dot)
tz_minus = text.find("-", dot)
tz_candidates = [pos for pos in (tz_plus, tz_minus) if pos != -1]
if tz_candidates:
tz_pos = min(tz_candidates)
frac = text[dot + 1:tz_pos]
text = text[:dot + 1] + frac[:6].ljust(6, "0") + text[tz_pos:]
dt = datetime.fromisoformat(text)
if dt.tzinfo is None:
dt = dt.replace(tzinfo=timezone.utc)
return dt.astimezone(timezone.utc)
+17 -6
View File
@@ -8,7 +8,8 @@ fail-closed nftables egress boundary. Selected by
from __future__ import annotations
from contextlib import contextmanager
import io
from contextlib import contextmanager, redirect_stderr
from pathlib import Path
from typing import Generator, Sequence
@@ -18,7 +19,7 @@ from ...env import ResolvedEnv
from ...git_gate import GitGatePlan
from ...manifest import Manifest
from ...supervise import SupervisePlan
from .. import ActiveAgent, BottleBackend, BottleSpec
from .. import ActiveAgent, BottleBackend, BottleImages, BottleSpec
from . import cleanup as _cleanup
from . import enumerate as _enumerate
from . import launch as _launch
@@ -52,8 +53,11 @@ class FirecrackerBottleBackend(
return _setup.setup()
@classmethod
def status(cls) -> int:
def status(cls, *, quiet: bool = False) -> int:
from . import setup as _setup
if quiet:
with redirect_stderr(io.StringIO()):
return _setup.status()
return _setup.status()
@classmethod
@@ -92,11 +96,18 @@ class FirecrackerBottleBackend(
stage_dir=stage_dir,
)
def _build_or_load_images(self, plan: FirecrackerBottlePlan) -> BottleImages:
return BottleImages(agent=_launch.build_or_load_agent_base(plan))
def prelaunch_checks(self, plan: FirecrackerBottlePlan) -> None:
_launch.stale_checks(plan)
@contextmanager
def launch(
self, plan: FirecrackerBottlePlan
def _launch_impl(
self, plan: FirecrackerBottlePlan, images: BottleImages,
) -> Generator[FirecrackerBottle, None, None]:
with _launch.launch(plan, provision=self.provision) as bottle:
assert isinstance(images.agent, Path)
with _launch.launch(plan, images.agent, provision=self.provision) as bottle:
yield bottle
def prepare_cleanup(self) -> FirecrackerBottleCleanupPlan:
@@ -22,6 +22,9 @@ class FirecrackerBottlePlan(BottlePlan):
# (egress proxy credentials, git-gate/supervise headers); set by launch
# from the orchestrator registration. Empty pre-registration.
identity_token: str = ""
# Applied to every agent SSH exec and mirrored into /run inside the VM so
# the host can recover it after the infra VM restarts.
env_var_secret: str = ""
@property
def container_name(self) -> str:
+72 -16
View File
@@ -1,8 +1,23 @@
"""Cleanup for the Firecracker backend.
Orphans are: firecracker VMM processes whose config lives under our run
dir, and the per-bottle run dirs. TAP slots free themselves (the flock
drops when the launcher exits), so there is nothing to reclaim there.
Reaps *orphans* only — resources with no live VM behind them:
* orphan run dirs: a per-bottle run dir (holding the ~1G rootfs.ext4)
whose firecracker process has exited. These leak when a launch is
hard-killed before its teardown runs (host OOM/crash, a cancelled CI
job, `kill -9`); the clean-exit path already removes its own dir in
launch.py.
* orphan VM pids: a firecracker process whose run dir is already gone
— a VMM left lingering after its dir was removed.
A run dir with a *live* firecracker process is a running bottle and is
left strictly alone: it is neither killed nor removed. (The backend's
`enumerate_active` registry is still a stub — #354 — so a live process
is the only reliable "this bottle is in use" signal we have. Once the
registry lands, registry-orphaned-but-running VMs can be reaped too.)
TAP slots free themselves (the flock drops when the launcher exits), so
there is nothing to reclaim there.
"""
from __future__ import annotations
@@ -22,38 +37,79 @@ def _run_root() -> Path:
return util.cache_dir() / "run"
def _orphan_vm_pids() -> list[int]:
"""firecracker processes whose --config-file is under our run dir."""
run_root = str(_run_root())
def _run_dir_of(cmd: str, run_root: Path) -> Path | None:
"""The bottle run dir a firecracker cmdline belongs to, or None.
A bottle VM is launched with `--config-file <run_root>/<slug>/config.json`,
so the run dir is the config file's parent when it sits directly under
the run root. Anything else (a builder VM, the infra VM elsewhere) is
not ours to reap here.
"""
toks = cmd.split()
for i, tok in enumerate(toks):
if tok == "--config-file" and i + 1 < len(toks):
parent = Path(toks[i + 1]).parent
if parent.parent == run_root:
return parent
return None
def _scan_processes(run_root: Path) -> tuple[set[str], list[int]]:
"""Inspect running firecracker VMs under ``run_root``.
Returns ``(live_run_dirs, orphan_pids)``:
* ``live_run_dirs`` — run dirs backed by a running VM (never reaped);
* ``orphan_pids`` — firecracker pids whose run dir no longer exists
(a lingering VMM to kill).
"""
result = subprocess.run(
["pgrep", "-a", "firecracker"],
capture_output=True, text=True, check=False,
)
if result.returncode != 0:
return []
pids: list[int] = []
return set(), []
live: set[str] = set()
orphan_pids: list[int] = []
for line in result.stdout.splitlines():
parts = line.split(None, 1)
if len(parts) != 2 or run_root not in parts[1]:
if len(parts) != 2:
continue
try:
pids.append(int(parts[0]))
pid = int(parts[0])
except ValueError:
continue
return pids
run_dir = _run_dir_of(parts[1], run_root)
if run_dir is None:
continue
if run_dir.is_dir():
live.add(str(run_dir))
else:
orphan_pids.append(pid)
return live, orphan_pids
def _run_dirs() -> list[str]:
run_root = _run_root()
def live_run_dirs() -> tuple[Path, ...]:
"""Run directories backed by currently running agent microVMs."""
live, _ = _scan_processes(_run_root())
return tuple(Path(path) for path in sorted(live))
def _orphan_run_dirs(run_root: Path, live: set[str]) -> list[str]:
"""Run dirs with no live VM behind them — the leaked ones to remove."""
if not run_root.is_dir():
return []
return sorted(str(p) for p in run_root.iterdir() if p.is_dir())
return sorted(
str(p) for p in run_root.iterdir()
if p.is_dir() and str(p) not in live
)
def prepare_cleanup() -> FirecrackerBottleCleanupPlan:
run_root = _run_root()
live, orphan_pids = _scan_processes(run_root)
return FirecrackerBottleCleanupPlan(
vm_pids=tuple(_orphan_vm_pids()),
run_dirs=tuple(_run_dirs()),
vm_pids=tuple(orphan_pids),
run_dirs=tuple(_orphan_run_dirs(run_root, live)),
)
@@ -25,17 +25,27 @@ The TAP slot allocation, rootfs build, and VM boot are the caller's job.
from __future__ import annotations
import json
import subprocess
from dataclasses import dataclass
from pathlib import Path
from ...egress import EgressPlan
from ...git_gate import GitGatePlan
from ...orchestrator.client import OrchestratorClient
from ...log import info
from ...orchestrator.client import OrchestratorClient, OrchestratorClientError
from ...orchestrator.lifecycle import (
OrchestratorStartError, # re-exported so callers can catch it
)
from ...orchestrator.registration import registration_inputs
from ..docker.gateway_provision import deprovision_git_gate, provision_git_gate
from . import infra_vm
from ...orchestrator.reprovision import reprovision_bottles
from ...orchestrator.secret_store import ENV_VAR_SECRET_NAME
from ..consolidated_util import (
provision_bottle,
teardown_consolidated as _teardown_util,
)
from . import cleanup, infra_vm, util
_ENV_VAR_SECRET_PATH = "/run/bot-bottle/env-var-secret"
class ConsolidatedLaunchError(RuntimeError):
@@ -51,6 +61,55 @@ class LaunchContext:
source_ip: str # the VM's guest IP — the attribution key
gateway_ca_pem: str # the shared gateway CA the provisioner installs
orchestrator_url: str
env_var_secret: str = "" # encryption key injected into the agent's env
def _guest_ip_from_config(config_path: Path) -> str:
"""Read the kernel's configured guest IP from a Firecracker config."""
try:
config = json.loads(config_path.read_text())
args = config["boot-source"]["boot_args"]
ip_arg = next(part for part in args.split() if part.startswith("ip="))
return ip_arg.removeprefix("ip=").split(":", 1)[0]
except (OSError, ValueError, KeyError, TypeError, StopIteration):
return ""
def persist_env_var_secret(private_key: Path, guest_ip: str, secret: str) -> None:
"""Mirror the exec-time key into guest tmpfs for restart recovery."""
proc = subprocess.run(
util.ssh_base_argv(private_key, guest_ip)
+ [f"umask 077; mkdir -p /run/bot-bottle; cat > {_ENV_VAR_SECRET_PATH}"],
input=secret, capture_output=True, text=True, check=False,
)
if proc.returncode != 0:
raise ConsolidatedLaunchError(
f"failed to persist {ENV_VAR_SECRET_NAME} in agent VM: "
f"{proc.stderr.strip() or '<no stderr>'}"
)
def _reprovision_running_bottles(client: OrchestratorClient) -> None:
"""Read keys from live agent VMs and restore the restarted gateway."""
try:
secrets_by_ip: dict[str, str] = {}
for run_dir in cleanup.live_run_dirs():
guest_ip = _guest_ip_from_config(run_dir / "config.json")
private_key = run_dir / "bottle_id_ed25519"
if not guest_ip or not private_key.is_file():
continue
proc = subprocess.run(
util.ssh_base_argv(private_key, guest_ip)
+ [f"cat {_ENV_VAR_SECRET_PATH}"],
capture_output=True, text=True, check=False,
)
if proc.returncode == 0 and proc.stdout.strip():
secrets_by_ip[guest_ip] = proc.stdout.strip()
count = reprovision_bottles(client, secrets_by_ip)
if count:
info(f"reprovisioned egress tokens for {count} Firecracker bottle(s)")
except (OSError, OrchestratorClientError) as exc:
info(f"egress token reprovision skipped: {exc}")
def launch_consolidated(
@@ -67,19 +126,13 @@ def launch_consolidated(
infra = infra_vm.ensure_running()
url = infra.control_plane_url
client = OrchestratorClient(url)
_reprovision_running_bottles(client)
inputs = registration_inputs(egress_plan)
reg = client.register_bottle(
guest_ip, image_ref=image_ref, policy=inputs.policy,
metadata=inputs.metadata, tokens=tokens,
transport = infra_vm.gateway_transport()
reg = provision_bottle(
client, guest_ip, egress_plan, git_gate_plan, transport,
image_ref=image_ref, tokens=tokens,
)
try:
provision_git_gate(
infra_vm.gateway_transport(), reg.bottle_id, git_gate_plan)
except Exception:
client.teardown_bottle(reg.bottle_id)
raise
# The shared gateway CA every agent on this host trusts for TLS
# interception — fetched from the infra VM over SSH.
return LaunchContext(
@@ -88,16 +141,19 @@ def launch_consolidated(
source_ip=guest_ip,
gateway_ca_pem=infra.gateway_ca_pem(),
orchestrator_url=url,
env_var_secret=reg.env_var_secret,
)
def teardown_consolidated(bottle_id: str, *, orchestrator_url: str) -> None:
def teardown_consolidated(
bottle_id: str, *, orchestrator_url: str, timeout: float | None = None,
) -> None:
"""Deregister the bottle and remove its git-gate state from the gateway
VM. Both steps are idempotent so this is safe from a cleanup trap. Does
NOT stop the infra VM — it's a persistent per-host singleton shared by
every bottle."""
OrchestratorClient(orchestrator_url).teardown_bottle(bottle_id)
deprovision_git_gate(infra_vm.gateway_transport(), bottle_id)
_teardown_util(bottle_id, infra_vm.gateway_transport(),
orchestrator_url=orchestrator_url, timeout=timeout)
__all__ = [
+1 -1
View File
@@ -5,7 +5,7 @@ backend — we stream the guest root filesystem out over the control
channel (SSH here). Unlike the other backends this needs no Docker: the
tar *is* the resumable artifact. `resume` extracts it and rebuilds a
fresh per-bottle ext4 with `mke2fs -d` (see `util.build_committed_rootfs_dir`
and `launch._build_agent_base`). The bottle keeps running after the
and `launch.build_or_load_agent_base`). The bottle keeps running after the
snapshot.
"""
@@ -58,6 +58,12 @@ def _rootfs_digest(dockerfile: Path) -> str:
return h.hexdigest()[:16]
def cached_agent_rootfs_dir(dockerfile: Path) -> Path | None:
"""Return the ready cached rootfs for ``dockerfile``, if one exists."""
base = util.cache_dir() / "rootfs" / f"agent-{_rootfs_digest(dockerfile)}"
return base if (base / ".bb-ready").is_file() else None
def build_agent_rootfs_dir(
dockerfile: Path, *, image_tag: str, smoke_test: tuple[str, ...] = (),
) -> Path:
@@ -72,7 +78,7 @@ def build_agent_rootfs_dir(
silent-failure image at build time rather than at first agent use."""
digest = _rootfs_digest(dockerfile)
base = util.cache_dir() / "rootfs" / f"agent-{digest}"
if (base / ".bb-ready").is_file():
if cached_agent_rootfs_dir(dockerfile) is not None:
info(f"using cached agent rootfs {base.name}")
return base
@@ -41,7 +41,7 @@ from . import util
_ARTIFACT_FORMAT = "1"
_REPO_ROOT = Path(__file__).resolve().parents[3]
_DOCKERFILES = ("Dockerfile.orchestrator", "Dockerfile.gateway", "Dockerfile.infra")
_DOCKERFILES = ("Dockerfile.orchestrator", "Dockerfile.gateway", "Dockerfile.infra", "Dockerfile.infra.fc")
_DEFAULT_BASE = "https://gitea.dideric.is"
_DEFAULT_OWNER = "didericis"
+16 -13
View File
@@ -33,6 +33,7 @@ from pathlib import Path
from typing import Generator
from ...log import die, info
from .. import util as backend_util
from ..docker import util as docker_mod
from ..docker.gateway_provision import GatewayProvisionError
from . import firecracker_vm, infra_artifact, netpool, util
@@ -93,19 +94,18 @@ class InfraVm:
"""The gateway's mitmproxy CA (PEM) that agents install to trust its
TLS interception. Generated a moment after boot, so this polls over
SSH until it appears (mirrors DockerGateway.ca_cert_pem)."""
deadline = time.monotonic() + timeout
while True:
def _fetch() -> str | None:
proc = subprocess.run(
util.ssh_base_argv(self.private_key, self.guest_ip)
+ [f"cat {_GATEWAY_CA_PATH}"],
capture_output=True, text=True, timeout=15, check=False,
)
if proc.returncode == 0 and "BEGIN CERTIFICATE" in proc.stdout:
return proc.stdout
if time.monotonic() >= deadline:
die(f"gateway CA not available after {timeout:g}s: "
f"{proc.stderr.strip() or 'empty'}")
time.sleep(_HEALTH_POLL_SECONDS)
ok = proc.returncode == 0 and "BEGIN CERTIFICATE" in proc.stdout
return proc.stdout if ok else None
try:
return backend_util.poll_ca_cert(_fetch, timeout=timeout)
except TimeoutError as exc:
die(str(exc))
def ensure_built() -> None:
@@ -125,16 +125,19 @@ def ensure_built() -> None:
def build_infra_images_with_docker() -> None:
"""Build the three fixed images from source with host Docker: orchestrator,
gateway, then the combined infra image (`COPY --from` orchestrator, `FROM`
gateway). The launch host uses this only in `BOT_BOTTLE_INFRA_BUILD=local`
mode; `publish_infra` uses it off-host to produce the published artifact."""
"""Build the four fixed images from source with host Docker: orchestrator,
gateway, the shared infra base (Dockerfile.infra), then the Firecracker
infra image (Dockerfile.infra.fc: FROM infra + buildah). The launch host
uses this only in `BOT_BOTTLE_INFRA_BUILD=local` mode; `publish_infra`
uses it off-host to produce the published artifact."""
docker_mod.build_image(
_ORCHESTRATOR_IMAGE, str(_REPO_ROOT), dockerfile="Dockerfile.orchestrator")
docker_mod.build_image(
_GATEWAY_IMAGE, str(_REPO_ROOT), dockerfile="Dockerfile.gateway")
docker_mod.build_image(
_INFRA_IMAGE, str(_REPO_ROOT), dockerfile="Dockerfile.infra")
"bot-bottle-infra:latest", str(_REPO_ROOT), dockerfile="Dockerfile.infra")
docker_mod.build_image(
_INFRA_IMAGE, str(_REPO_ROOT), dockerfile="Dockerfile.infra.fc")
def build_infra_rootfs_dir() -> Path:
+53 -13
View File
@@ -26,6 +26,7 @@ from __future__ import annotations
import dataclasses
import os
import shutil
from contextlib import ExitStack, contextmanager
from pathlib import Path
from typing import Callable, Generator
@@ -45,15 +46,19 @@ from ...git_gate import (
provision_git_gate_dynamic_keys,
revoke_git_gate_provisioned_keys,
)
from ...log import info, warn
from ...image_cache import check_stale_path
from ...log import die, info, warn
from ...supervise import SUPERVISE_PORT
from ..docker.egress import EGRESS_PORT
from ..util import AGENT_CA_BUNDLE, AGENT_CA_PATH
from . import firecracker_vm, image_builder, isolation_probe, netpool, util
from .bottle import FirecrackerBottle
from .bottle_plan import FirecrackerBottlePlan
from ...orchestrator.config_store import resolve_teardown_timeout
from ...orchestrator.secret_store import ENV_VAR_SECRET_NAME
from .consolidated_launch import (
launch_consolidated,
persist_env_var_secret,
teardown_consolidated,
)
@@ -64,6 +69,7 @@ _GIT_HTTP_PORT = 9420
@contextmanager
def launch(
plan: FirecrackerBottlePlan,
agent_base: Path,
*,
provision: Callable[[FirecrackerBottlePlan, "FirecrackerBottle"], str | None],
) -> Generator[FirecrackerBottle, None, None]:
@@ -85,11 +91,9 @@ def launch(
raise teardown_exc
try:
# Step 1: agent rootfs. Built from the Dockerfile inside a Firecracker
# builder VM (buildah, no host docker); a committed snapshot is reused
# when present. Returns the base dir the per-bottle ext4 is made from.
plan, agent_base = _build_agent_base(plan)
# Step 1 (rootfs resolution/build) runs in BottleBackend.launch before
# this context starts resources. ``agent_base`` is the selected cache,
# fresh build, or committed snapshot.
# Step 2: mint the git-gate dynamic (gitea) deploy keys, if any.
git_gate_plan = plan.git_gate_plan
if git_gate_plan.upstreams:
@@ -112,6 +116,7 @@ def launch(
token_values = egress_resolve_token_values(
plan.egress_plan.token_env_map, effective_env,
)
teardown_timeout = resolve_teardown_timeout()
ctx = launch_consolidated(
plan.egress_plan, git_gate_plan,
guest_ip=slot.guest_ip,
@@ -121,6 +126,7 @@ def launch(
stack.callback(
teardown_consolidated, ctx.bottle_id,
orchestrator_url=ctx.orchestrator_url,
timeout=teardown_timeout,
)
# Step 5: install the SHARED gateway CA (replaces the per-bottle CA).
@@ -149,6 +155,7 @@ def launch(
git_gate_plan=git_gate_plan,
egress_plan=egress_plan,
identity_token=ctx.identity_token,
env_var_secret=ctx.env_var_secret,
# Deliver the identity token as egress proxy credentials — clients
# honor `HTTPS_PROXY=http://id:token@gw` without app changes; the
# gateway reads Proxy-Authorization, validates the (source_ip,
@@ -164,6 +171,10 @@ def launch(
# Step 6: build the per-bottle rootfs + SSH key, then boot.
run_dir = util.cache_dir() / "run" / plan.slug
run_dir.mkdir(parents=True, exist_ok=True)
# Remove the run dir on teardown so the per-bottle rootfs.ext4 (~1G)
# doesn't leak. Registered before vm.terminate below so it runs *after*
# it (ExitStack is LIFO): the VM is gone before we rm its rootfs.
stack.callback(lambda: shutil.rmtree(run_dir, ignore_errors=True))
rootfs = run_dir / "rootfs.ext4"
util.build_rootfs_ext4(agent_base, rootfs)
private_key, pubkey = util.generate_keypair(run_dir)
@@ -179,6 +190,7 @@ def launch(
)
stack.callback(vm.terminate)
firecracker_vm.wait_for_ssh(vm, private_key)
persist_env_var_secret(private_key, slot.guest_ip, ctx.env_var_secret)
# Authoritative fail-closed egress-boundary check, before the agent
# runs: prove the VM cannot reach the host directly.
@@ -206,9 +218,7 @@ def launch(
teardown()
def _build_agent_base(
plan: FirecrackerBottlePlan,
) -> tuple[FirecrackerBottlePlan, Path]:
def build_or_load_agent_base(plan: FirecrackerBottlePlan) -> Path:
"""Produce the agent's base rootfs dir. Primary path: build the Dockerfile
inside a Firecracker builder VM (buildah, no host docker), smoke-testing
the image before export. A committed snapshot (freeze/migrate) is resumed
@@ -217,13 +227,36 @@ def _build_agent_base(
committed_tar = committed_rootfs_path(plan.slug)
if committed and committed_tar.is_file():
info(f"resuming from committed rootfs {committed_tar}")
return plan, util.build_committed_rootfs_dir(committed_tar)
base = image_builder.build_agent_rootfs_dir(
Path(plan.dockerfile_path),
return util.build_committed_rootfs_dir(committed_tar)
dockerfile = Path(plan.dockerfile_path)
if plan.spec.image_policy == "cached":
cached = image_builder.cached_agent_rootfs_dir(dockerfile)
if cached is None:
die(
f"cached agent rootfs for {plan.image!r} not found; "
"run without --cached-images to build it"
)
info(f"using cached agent rootfs {cached.name}")
return cached
return image_builder.build_agent_rootfs_dir(
dockerfile,
image_tag=plan.image,
smoke_test=runtime_for(plan.agent_provider_template).smoke_test,
)
return plan, base
def stale_checks(plan: FirecrackerBottlePlan) -> None:
"""Raise when the cached rootfs selected by this plan is stale."""
if plan.spec.image_policy != "cached":
return
committed = read_committed_image(plan.slug)
committed_tar = committed_rootfs_path(plan.slug)
if committed and committed_tar.is_file():
check_stale_path(f"agent rootfs {committed_tar}", committed_tar)
return
cached = image_builder.cached_agent_rootfs_dir(Path(plan.dockerfile_path))
if cached is not None:
check_stale_path(f"agent rootfs {cached}", cached / ".bb-ready")
# --- agent guest env -------------------------------------------------
@@ -239,6 +272,11 @@ def _agent_guest_env(plan: FirecrackerBottlePlan, host_ip: str) -> dict[str, str
"HTTPS_PROXY": proxy_url, "HTTP_PROXY": proxy_url,
"https_proxy": proxy_url, "http_proxy": proxy_url,
"NO_PROXY": no_proxy, "no_proxy": no_proxy,
# Rootfs export can leave Git's implicit XDG paths unreadable even
# after the runtime repair. Bypass that discovery and name the
# provisioned global config explicitly so insteadOf can never fall
# through to the credential-bearing upstream URL.
"GIT_CONFIG_GLOBAL": f"{plan.guest_home}/.gitconfig",
"NODE_EXTRA_CA_CERTS": AGENT_CA_PATH,
"SSL_CERT_FILE": AGENT_CA_BUNDLE,
"REQUESTS_CA_BUNDLE": AGENT_CA_BUNDLE,
@@ -247,6 +285,8 @@ def _agent_guest_env(plan: FirecrackerBottlePlan, host_ip: str) -> dict[str, str
env["GIT_GATE_URL"] = plan.agent_git_gate_url
if plan.agent_supervise_url:
env["MCP_SUPERVISE_URL"] = plan.agent_supervise_url
if plan.env_var_secret:
env[ENV_VAR_SECRET_NAME] = plan.env_var_secret
for entry in egress_agent_env_entries(plan.egress_plan):
key, _, value = entry.partition("=")
env[key] = value
@@ -131,6 +131,29 @@ def build_artifact(out_dir: Path) -> tuple[str, Path, Path]:
return version, gz, sha
def _try_download_published(out_dir: Path) -> tuple[str, Path, Path] | None:
"""If this version's artifact is already in the registry, download the gz
and sha to out_dir and return (version, gz_path, sha_path). Returns None
when not yet published."""
version = infra_artifact.infra_artifact_version(infra_vm._infra_init())
sha_url = infra_artifact.artifact_url(version, "rootfs.ext4.gz.sha256")
try:
with urllib.request.urlopen(infra_artifact._open(sha_url)):
pass
except urllib.error.HTTPError as e:
if e.code == 404:
return None
raise SystemExit(f"registry check failed (HTTP {e.code}): {sha_url}")
except urllib.error.URLError as e:
raise SystemExit(f"registry unreachable: {sha_url} ({e.reason})")
print(f"infra rootfs {version} already published — downloading instead of building")
gz = out_dir / "rootfs.ext4.gz"
sha = out_dir / "rootfs.ext4.gz.sha256"
infra_artifact._download(infra_artifact.artifact_url(version, "rootfs.ext4.gz"), gz)
infra_artifact._download(infra_artifact.artifact_url(version, "rootfs.ext4.gz.sha256"), sha)
return version, gz, sha
def _publish_bundle(root: Path, token: str) -> str:
version_file = root / "version.txt"
# Guard the read so a missing version.txt is a clean error, not a raw
@@ -187,6 +210,8 @@ def main(argv: list[str] | None = None) -> int:
help="build a candidate bundle in DIR without publishing")
mode.add_argument("--publish-dir", type=Path,
help="publish an already-built and tested candidate bundle")
parser.add_argument("--reuse-published", action="store_true",
help="with --output: download from registry if already published instead of building")
args = parser.parse_args(argv)
_, _, token = infra_artifact._config()
@@ -197,6 +222,14 @@ def main(argv: list[str] | None = None) -> int:
if args.output is not None:
args.output.mkdir(parents=True, exist_ok=True)
reused = None
if args.reuse_published:
reused = _try_download_published(args.output)
if reused is not None:
version, _, _ = reused
(args.output / "version.txt").write_text(version + "\n", encoding="utf-8")
print(f"reused published infra rootfs candidate {version}")
return 0
version, _gz, _sha = build_artifact(args.output)
(args.output / "version.txt").write_text(version + "\n", encoding="utf-8")
print(f"built infra rootfs candidate {version}")
+86 -6
View File
@@ -13,6 +13,7 @@ generic `./cli.py backend {setup,status}` command dispatches to.
from __future__ import annotations
import fcntl
import os
import shutil
import subprocess
@@ -22,6 +23,9 @@ from pathlib import Path
from . import netpool
from . import util
# KVM_GET_API_VERSION = _IO(KVMIO=0xAE, 0x00): cheapest proof of KVM access.
_KVM_GET_API_VERSION = 0xAE00
_FC_RELEASES = "https://github.com/firecracker-microvm/firecracker/releases"
_UNIT_PATH = Path("/etc/systemd/system") / netpool.SYSTEMD_UNIT
@@ -219,14 +223,90 @@ def teardown() -> int:
return 0
def _firecracker_binary_ok() -> bool:
"""True iff the firecracker binary is on PATH and `--version` exits 0."""
if shutil.which("firecracker") is None:
return False
try:
return subprocess.run(
["firecracker", "--version"],
capture_output=True, check=False, timeout=5,
).returncode == 0
except (OSError, subprocess.TimeoutExpired):
return False
def _kvm_accessible() -> bool:
"""True iff /dev/kvm can be opened read-write and responds to KVM_GET_API_VERSION.
VM creation requires write access; opening read-only may satisfy the
ioctl but fails at boot time, so O_RDWR is the permission check."""
if not os.path.exists(util._KVM_DEVICE):
return False
try:
fd = os.open(util._KVM_DEVICE, os.O_RDWR | os.O_CLOEXEC)
try:
fcntl.ioctl(fd, _KVM_GET_API_VERSION)
finally:
os.close(fd)
return True
except OSError:
return False
def status() -> int:
# Readiness == what the launch preflight hard-requires: the TAP pool
# present (unprivileged, authoritative) and no range overlap. Listing
# the nft table usually needs root, so — like the preflight — an
# unconfirmable table is reported but NOT treated as not-ready; the
# post-boot isolation probe is the authoritative check. This keeps an
# unprivileged `backend status` usable as a launch gate.
# Readiness == what the launch preflight hard-requires: the binary
# executable, /dev/kvm accessible, the TAP pool present, and no range
# overlap. Listing the nft table usually needs root, so — like the
# preflight — an unconfirmable table is reported but NOT treated as
# not-ready; the post-boot isolation probe is the authoritative check.
# This keeps an unprivileged `backend status` usable as a launch gate.
ok = True
if _firecracker_binary_ok():
sys.stderr.write(f"firecracker binary: ok ({shutil.which('firecracker')})\n")
else:
fc_path = shutil.which("firecracker")
if fc_path is None:
sys.stderr.write("firecracker binary: NOT found on PATH\n")
else:
sys.stderr.write(
f"firecracker binary: found ({fc_path}) but `--version` failed\n"
)
ok = False
if _kvm_accessible():
sys.stderr.write(f"KVM: {util._KVM_DEVICE} accessible\n")
else:
if not os.path.exists(util._KVM_DEVICE):
sys.stderr.write(f"KVM: {util._KVM_DEVICE} not present\n")
else:
sys.stderr.write(
f"KVM: {util._KVM_DEVICE} not accessible (open/ioctl failed)\n"
)
ok = False
kernel = util.kernel_path()
if kernel.is_file():
sys.stderr.write(f"guest kernel: {kernel}\n")
else:
sys.stderr.write(
f"guest kernel: NOT found at {kernel} "
f"(set BOT_BOTTLE_FC_KERNEL or cache a vmlinux there)\n"
)
ok = False
dropbear = util.dropbear_path()
if dropbear.is_file():
sys.stderr.write(f"dropbear: {dropbear}\n")
else:
sys.stderr.write(
f"dropbear: NOT found at {dropbear} "
f"(set BOT_BOTTLE_FC_DROPBEAR or cache a static binary)\n"
)
ok = False
mke2fs = shutil.which("mke2fs")
if mke2fs is not None:
sys.stderr.write(f"mke2fs: {mke2fs}\n")
else:
sys.stderr.write("mke2fs: NOT found on PATH (install e2fsprogs)\n")
ok = False
missing = netpool.missing_taps()
total = netpool.pool_size()
if missing:
+9
View File
@@ -373,6 +373,12 @@ mount -o remount,rw / 2>/dev/null
# scratch dirs there — git worktrees, build temp, `git init /tmp/...`, etc.
mkdir -p /tmp && chmod 1777 /tmp
# Rootfs export also maps the image's original owners to the unprivileged
# host build uid. That uid is not guaranteed to be node's uid in the guest;
# restore the home-directory boundary before any SSH provisioning runs.
chown node:node /home/node 2>/dev/null || true
chmod 755 /home/node 2>/dev/null || true
# Install the per-bottle SSH pubkey from the kernel cmdline.
KEY=$(sed -n 's/.*bb_pubkey=\([^ ]*\).*/\1/p' /proc/cmdline | base64 -d 2>/dev/null)
if [ -n "$KEY" ]; then
@@ -393,6 +399,9 @@ fi
chown -R 0:0 /root 2>/dev/null || true
mkdir -p /etc/dropbear /run
# Keep restart-recovery key material memory-backed, separate from both the
# agent rootfs and the infra VM's persistent registry volume.
mount -t tmpfs -o mode=0755 tmpfs /run 2>/dev/null || true
# -R: generate host keys on demand. -E: log auth failures to stderr,
# captured in the host-side console.log for debugging.
/bb-dropbear -R -E -p 22 &
+17 -6
View File
@@ -2,7 +2,8 @@
from __future__ import annotations
from contextlib import contextmanager
import io
from contextlib import contextmanager, redirect_stderr
from pathlib import Path
from typing import Generator, Sequence
@@ -12,7 +13,7 @@ from ...env import ResolvedEnv
from ...git_gate import GitGatePlan
from ...supervise import SupervisePlan
from ...manifest import Manifest
from .. import ActiveAgent, BottleBackend, BottleSpec
from .. import ActiveAgent, BottleBackend, BottleImages, BottleSpec
from . import cleanup as _cleanup
from . import enumerate as _enumerate
from . import launch as _launch
@@ -31,6 +32,7 @@ class MacosContainerBottleBackend(
`--backend=macos-container`."""
name = "macos-container"
supports_nested_containers = True
@classmethod
def is_available(cls) -> bool:
@@ -42,8 +44,11 @@ class MacosContainerBottleBackend(
return _setup.setup()
@classmethod
def status(cls) -> int:
def status(cls, *, quiet: bool = False) -> int:
from . import setup as _setup
if quiet:
with redirect_stderr(io.StringIO()):
return _setup.status()
return _setup.status()
@classmethod
@@ -82,11 +87,17 @@ class MacosContainerBottleBackend(
stage_dir=stage_dir,
)
def prelaunch_checks(self, plan: MacosContainerBottlePlan) -> None:
_launch.stale_checks(plan)
def _build_or_load_images(self, plan: MacosContainerBottlePlan) -> BottleImages:
return _launch.build_or_load_images(plan)
@contextmanager
def launch(
self, plan: MacosContainerBottlePlan
def _launch_impl(
self, plan: MacosContainerBottlePlan, images: BottleImages
) -> Generator[MacosContainerBottle, None, None]:
with _launch.launch(plan, provision=self.provision) as bottle:
with _launch.launch(plan, images, provision=self.provision) as bottle:
yield bottle
def ensure_orchestrator(self) -> str:
@@ -20,6 +20,12 @@ class MacosContainerBottlePlan(BottlePlan):
# bottle is registered. See launch.py's stamp for why it lives here and not
# only in the exec-time proxy env.
identity_token: str = ""
# Guest-local container engine (issue #392). Gates the derived image, the
# device-mode relaxation, and the resident podman service.
nested_containers: bool = False
# Generated before `container run` so it becomes part of the container's
# configured environment and can be read back after an infra restart.
env_var_secret: str = ""
@property
def container_name(self) -> str:
@@ -38,10 +38,14 @@ from ...egress import EgressPlan
from ...git_gate import GitGatePlan
from ...log import info
from ...orchestrator.client import OrchestratorClient, OrchestratorClientError
from ...orchestrator.registration import registration_inputs
from ..docker.gateway_provision import deprovision_git_gate, provision_git_gate
from ...orchestrator.reprovision import reprovision_bottles
from ...orchestrator.secret_store import ENV_VAR_SECRET_NAME
from ..consolidated_util import (
provision_bottle,
teardown_consolidated as _teardown_util,
)
from . import util as container_mod
from .enumerate import CONTAINER_NAME_PREFIX, enumerate_active
from .enumerate import CONTAINER_NAME_PREFIX, EnumerationError, enumerate_active
from .gateway import GATEWAY_NETWORK
from .gateway_provision import AppleGatewayTransport
from .infra import MacosInfraService, OrchestratorStartError
@@ -73,6 +77,7 @@ class LaunchContext:
gateway_ip: str
network: str
orchestrator_url: str
env_var_secret: str = "" # encryption key injected into the agent's env
def ensure_gateway(
@@ -84,12 +89,35 @@ def ensure_gateway(
needs `gateway_ip` at run time."""
service = service or MacosInfraService()
infra = service.ensure_running()
return GatewayEndpoint(
endpoint = GatewayEndpoint(
orchestrator_url=infra.control_plane_url,
gateway_ip=infra.gateway_ip,
gateway_ca_pem=service.ca_cert_pem(),
network=service.network,
)
_reprovision_running_bottles(endpoint)
return endpoint
def _reprovision_running_bottles(endpoint: GatewayEndpoint) -> None:
"""Recover keys from live Apple containers and restore gateway tokens."""
try:
secrets_by_ip: dict[str, str] = {}
for agent in enumerate_active():
name = f"{CONTAINER_NAME_PREFIX}{agent.slug}"
source_ip = container_mod.inspect_container_network_ip(name, endpoint.network)
if not source_ip:
continue
secret = container_mod.read_container_env(name, ENV_VAR_SECRET_NAME)
if secret:
secrets_by_ip[source_ip] = secret
count = reprovision_bottles(
OrchestratorClient(endpoint.orchestrator_url), secrets_by_ip,
)
if count:
info(f"reprovisioned egress tokens for {count} macOS bottle(s)")
except (OrchestratorClientError, EnumerationError, OSError) as exc:
info(f"egress token reprovision skipped: {exc}")
def live_source_ips(network: str) -> list[str]:
@@ -98,14 +126,21 @@ def live_source_ips(network: str) -> list[str]:
The reconciliation input: the orchestrator lives inside the infra
container and cannot enumerate the host's containers, so the host has to
tell it which bottles are actually up. Containers that have not been
assigned an address yet contribute nothing (the read is non-fatal) the
reap's grace window, not this list, is what protects an in-flight
launch."""
assigned an address yet contribute nothing the reap's grace window, not
this list, is what protects an in-flight launch.
Raises `EnumerationError` when the live set cannot be determined
authoritatively: either the container listing fails or any individual
inspect fails. Callers must skip reconciliation in that case to avoid
unregistering healthy bottles."""
ips: list[str] = []
for agent in enumerate_active():
ip = container_mod.try_container_ipv4_on_network(
f"{CONTAINER_NAME_PREFIX}{agent.slug}", network,
)
name = f"{CONTAINER_NAME_PREFIX}{agent.slug}"
ip = container_mod.inspect_container_network_ip(name, network)
if ip is None:
raise EnumerationError(
f"container inspect {name!r} failed; live set is not authoritative"
)
if ip:
ips.append(ip)
return ips
@@ -119,6 +154,7 @@ def register_agent(
endpoint: GatewayEndpoint,
image_ref: str = "",
tokens: dict[str, str] | None = None,
env_var_secret: str | None = None,
) -> LaunchContext:
"""Register the (already running) agent by its address and provision its
git-gate state into the gateway. `source_ip` must be read from the live
@@ -133,19 +169,13 @@ def register_agent(
# reconciliation failure must not block an otherwise-fine launch.
try:
client.reconcile(live_source_ips(endpoint.network))
except OrchestratorClientError as e:
except (OrchestratorClientError, EnumerationError) as e:
info(f"registry reconciliation skipped: {e}")
inputs = registration_inputs(egress_plan)
reg = client.register_bottle(
source_ip, image_ref=image_ref, policy=inputs.policy,
metadata=inputs.metadata, tokens=tokens,
reg = provision_bottle(
client, source_ip, egress_plan, git_gate_plan, AppleGatewayTransport(),
image_ref=image_ref, tokens=tokens,
env_var_secret=env_var_secret,
)
try:
provision_git_gate(AppleGatewayTransport(), reg.bottle_id, git_gate_plan)
except Exception:
# Roll the registration back so a provisioning failure leaves no orphan.
client.teardown_bottle(reg.bottle_id)
raise
return LaunchContext(
bottle_id=reg.bottle_id,
identity_token=reg.identity_token,
@@ -153,15 +183,18 @@ def register_agent(
gateway_ip=endpoint.gateway_ip,
network=endpoint.network,
orchestrator_url=endpoint.orchestrator_url,
env_var_secret=reg.env_var_secret,
)
def teardown_consolidated(bottle_id: str, *, orchestrator_url: str) -> None:
def teardown_consolidated(
bottle_id: str, *, orchestrator_url: str, timeout: float | None = None,
) -> None:
"""Deregister the bottle and remove its git-gate state from the gateway.
Both steps are idempotent so this is safe from a cleanup trap. Does NOT
stop the gateway it's a persistent per-host singleton."""
OrchestratorClient(orchestrator_url).teardown_bottle(bottle_id)
deprovision_git_gate(AppleGatewayTransport(), bottle_id)
_teardown_util(bottle_id, AppleGatewayTransport(),
orchestrator_url=orchestrator_url, timeout=timeout)
__all__ = [
@@ -19,6 +19,10 @@ CONTAINER_NAME_PREFIX = "bot-bottle-"
_INFRA_NAMES = frozenset({INFRA_NAME})
class EnumerationError(RuntimeError):
"""container list failed; the resulting live set is not authoritative."""
def enumerate_active() -> list[ActiveAgent]:
result = subprocess.run(
["container", "list", "--quiet"],
@@ -27,7 +31,10 @@ def enumerate_active() -> list[ActiveAgent]:
check=False,
)
if result.returncode != 0:
return []
raise EnumerationError(
f"container list failed: "
f"{(result.stderr or '').strip() or '<no stderr>'}"
)
out: list[ActiveAgent] = []
for name in sorted(line.strip() for line in result.stdout.splitlines()):
if not name.startswith(CONTAINER_NAME_PREFIX) or name in _INFRA_NAMES:
+22 -14
View File
@@ -41,7 +41,7 @@ from dataclasses import dataclass
from pathlib import Path
from ... import log
from ...orchestrator.gateway import GATEWAY_CA_CERT
from ...orchestrator.gateway import GATEWAY_CA_CERT, MITMPROXY_HOME
from ...orchestrator.lifecycle import (
DEFAULT_PORT,
DEFAULT_STARTUP_TIMEOUT_SECONDS,
@@ -52,7 +52,9 @@ from ...paths import (
CONTROL_PLANE_TOKEN_ENV,
HOST_DB_FILENAME,
host_control_plane_token,
host_gateway_ca_dir,
)
from .. import util as backend_util
from . import util as container_mod
from .gateway import (
DEFAULT_CA_TIMEOUT_SECONDS,
@@ -217,6 +219,14 @@ class MacosInfraService:
# Container-only DB volume: one kernel writes bot-bottle.db, never
# shared with the host or another guest.
"--volume", f"{self._db_volume}:{_DB_ROOT_IN_CONTAINER}",
# The DB needs a container-only ext4 volume for coherent SQLite
# locking, but the CA has no such constraint. Keep it in the host
# app-data root so infra-container recreation and Apple Container
# volume pruning cannot silently rotate every bottle's trust
# anchor (issue #450).
"--mount",
container_mod.bind_mount_spec(
str(host_gateway_ca_dir()), MITMPROXY_HOME),
# Bind-mount the control-plane source (read-only); a code change
# takes effect on relaunch with no image rebuild.
"--mount",
@@ -260,21 +270,19 @@ class MacosInfraService:
def ca_cert_pem(self, *, timeout: float = DEFAULT_CA_TIMEOUT_SECONDS) -> str:
"""The gateway's mitmproxy CA (PEM) agents install to trust its TLS
interception. Read out of the container (the CA lives on a
container-internal path, not a host mount); polls because mitmproxy
writes it a beat after start."""
deadline = time.monotonic() + timeout
while True:
interception. Read through the container path backed by the persistent
host CA directory; polls because mitmproxy writes it a beat after
start."""
def _fetch() -> str | None:
result = container_mod.run_container_argv(
["container", "exec", self._name, "cat", GATEWAY_CA_CERT])
if result.returncode == 0 and result.stdout.strip():
return result.stdout
if time.monotonic() >= deadline:
raise GatewayError(
f"gateway CA not available in {self._name} after {timeout:g}s: "
f"{(result.stderr or '').strip() or 'empty'}"
)
time.sleep(_CA_POLL_SECONDS)
return result.stdout if result.returncode == 0 and result.stdout.strip() else None
try:
return backend_util.poll_ca_cert(_fetch, timeout=timeout)
except TimeoutError as exc:
raise GatewayError(
f"gateway CA not available in {self._name} after {timeout:g}s"
) from exc
def stop(self) -> None:
"""Remove the infra container (idempotent). The DB volume persists."""
+90 -19
View File
@@ -53,7 +53,9 @@ from ...git_gate import (
revoke_git_gate_provisioned_keys,
)
from ...git_http_backend import DEFAULT_PORT as _GIT_HTTP_PORT
from ...image_cache import check_stale
from ...log import die, info, warn
from .. import BottleImages
from ...supervise import SUPERVISE_PORT
from ..docker.egress import EGRESS_PORT
from ..util import AGENT_CA_BUNDLE, AGENT_CA_PATH
@@ -64,7 +66,10 @@ from .gateway_hosts import (
refresh_gateway_host,
set_gateway_host,
)
from . import nested_containers as nested_containers_mod
from .bottle_plan import MacosContainerBottlePlan
from ...orchestrator.config_store import resolve_teardown_timeout
from ...orchestrator.secret_store import ENV_VAR_SECRET_NAME, new_env_var_secret
from .consolidated_launch import (
GatewayEndpoint,
ensure_gateway,
@@ -76,18 +81,69 @@ _REPO_DIR = str(Path(__file__).resolve().parent.parent.parent.parent)
_AGENT_SLEEP_SECONDS = "2147483647"
def build_or_load_images(plan: MacosContainerBottlePlan) -> BottleImages:
"""Resolve the agent image ref for this plan. The gateway's own image is
built by `ensure_gateway` it belongs to the shared singleton."""
return BottleImages(agent=_layer_nested_containers(plan, _agent_image(plan)))
def _agent_image(plan: MacosContainerBottlePlan) -> str:
committed = read_committed_image(plan.slug)
if committed and container_mod.image_exists(committed):
info(f"using committed image {committed!r}")
return committed
if plan.spec.image_policy == "cached":
if not container_mod.image_exists(plan.image):
die(
f"cached agent image {plan.image!r} not found; "
"run without --cached-images to build it"
)
info(f"using cached agent image {plan.image!r}")
return plan.image
container_mod.build_image(plan.image, _REPO_DIR, dockerfile=plan.dockerfile_path)
return plan.image
def _layer_nested_containers(
plan: MacosContainerBottlePlan, agent_image: str,
) -> str:
"""Add the guest-local container tooling on top of the agent image.
A separate derived tag, not the provider Dockerfile, so bottles that never
ask for nested containers carry none of its weight.
"""
if not plan.nested_containers:
return agent_image
derived = f"{agent_image}{nested_containers_mod.IMAGE_SUFFIX}"
if plan.spec.image_policy == "cached":
if not container_mod.image_exists(derived):
die(
f"cached nested-container image {derived!r} not found; "
"run without --cached-images to build it"
)
info(f"using cached nested-container image {derived!r}")
return derived
return nested_containers_mod.build_image(agent_image, container_mod.build_image)
@contextmanager
def launch(
plan: MacosContainerBottlePlan,
images: BottleImages,
*,
provision: Callable[[MacosContainerBottlePlan, "MacosContainerBottle"], str | None],
) -> Generator[MacosContainerBottle, None, None]:
"""Build, run, register, provision, and yield an Apple Container bottle on
the shared per-host gateway."""
"""Run, register, provision, and yield an Apple Container bottle on the
shared per-host gateway."""
stack = ExitStack()
bottle_for_revoke = plan.manifest.bottle
git_gate_dir_for_revoke = git_gate_state_dir(plan.slug)
plan = dataclasses.replace(
plan,
agent_provision=dataclasses.replace(plan.agent_provision, image=str(images.agent)),
)
def teardown() -> None:
teardown_exc: BaseException | None = None
try:
@@ -100,8 +156,6 @@ def launch(
raise teardown_exc
try:
plan = _build_images(plan)
# Step 1: the per-host singletons. Must precede the agent run — its
# proxy env needs the gateway's address at `container run` time.
endpoint = ensure_gateway()
@@ -116,6 +170,7 @@ def launch(
plan = _provision_git_gate_keys(plan)
plan = _install_gateway_ca(plan, endpoint)
plan = _stamp_agent_urls(plan, endpoint)
plan = dataclasses.replace(plan, env_var_secret=new_env_var_secret())
# Step 3: run the agent. It has no identity token yet — registration
# needs the address this run assigns.
@@ -142,6 +197,7 @@ def launch(
token_values = egress_resolve_token_values(
plan.egress_plan.token_env_map, effective_env,
)
teardown_timeout = resolve_teardown_timeout()
ctx = register_agent(
plan.egress_plan,
plan.git_gate_plan,
@@ -149,10 +205,12 @@ def launch(
endpoint=endpoint,
image_ref=plan.image,
tokens=token_values,
env_var_secret=plan.env_var_secret,
)
stack.callback(
teardown_consolidated, ctx.bottle_id,
orchestrator_url=ctx.orchestrator_url,
timeout=teardown_timeout,
)
info(
f"agent {plan.container_name} registered "
@@ -168,6 +226,10 @@ def launch(
# token above, so — unlike the run-time env — the plan CAN carry it.
plan = dataclasses.replace(plan, identity_token=ctx.identity_token)
exec_env = {
**_identity_proxy_env(endpoint, ctx.identity_token),
**nested_containers_mod.guest_env(plan.nested_containers),
}
bottle = MacosContainerBottle(
plan.container_name,
teardown,
@@ -181,31 +243,38 @@ def launch(
),
terminal_color=plan.spec.color,
agent_workdir=plan.workspace_plan.workdir,
exec_env=_identity_proxy_env(endpoint, ctx.identity_token),
exec_env=exec_env,
)
bottle.prompt_path = provision(plan, bottle)
if plan.nested_containers:
nested_containers_mod.prepare_guest_devices(
plan.container_name, container_mod.exec_container_as_root,
)
nested_containers_mod.start(bottle)
yield bottle
finally:
teardown()
def _build_images(plan: MacosContainerBottlePlan) -> MacosContainerBottlePlan:
"""Build the agent image. The gateway's own image is built by
`ensure_gateway` it belongs to the shared singleton, not to a bottle."""
def stale_checks(plan: MacosContainerBottlePlan) -> None:
"""Raise StaleImageError if a cached image is older than the configured
threshold. Only runs when image_policy is 'cached'. Called by the backend
class's _image_stale_checks before _launch_impl starts any resources."""
if plan.spec.image_policy != "cached":
return
committed = read_committed_image(plan.slug)
if committed and container_mod.image_exists(committed):
info(f"using committed image {committed!r}")
return dataclasses.replace(
plan,
agent_provision=dataclasses.replace(
plan.agent_provision, image=committed,
),
)
container_mod.build_image(
plan.image, _REPO_DIR, dockerfile=plan.dockerfile_path,
)
return plan
ts = container_mod.image_created_at(committed)
if ts is not None:
check_stale(f"agent image {committed!r}", ts)
return
if container_mod.image_exists(plan.image):
ts = container_mod.image_created_at(plan.image)
if ts is not None:
check_stale(f"agent image {plan.image!r}", ts)
def _provision_git_gate_keys(
@@ -377,6 +446,8 @@ def _agent_env_entries(
env.append(f"GIT_GATE_URL={plan.agent_git_gate_url}")
if plan.agent_supervise_url:
env.append(f"MCP_SUPERVISE_URL={plan.agent_supervise_url}")
if getattr(plan, "env_var_secret", ""):
env.append(f"{ENV_VAR_SECRET_NAME}={plan.env_var_secret}")
for name, value in sorted(plan.agent_provision.guest_env.items()):
env.append(f"{name}={value}")
# Forwarded vars: bare name → inherits from the `container run` process env
@@ -0,0 +1,210 @@
#!/bin/sh
set -eu
uid="$(id -u)"
if [ "$uid" -eq 0 ]; then
echo "refusing to run the guest container engine as root" >&2
exit 1
fi
# Every piece of podman 5's networking stack is checked here, because each
# one fails at a different and misleading layer if it is absent: no pasta and
# nothing starts at all; no nft and netavark cannot build the bridge every
# compose file expects; no aardvark-dns and DNS inside nested containers fails
# while everything else looks healthy.
for command in podman docker fuse-overlayfs pasta nft slirp4netns; do
command -v "$command" >/dev/null 2>&1 || {
echo "missing nested-container prerequisite: $command" >&2
exit 1
}
done
# The inverse of the rootless-Docker check, and the whole point of the podman
# variant: a subordinate range would push podman onto newuidmap, which cannot
# write a multi-range uid_map without CAP_SYS_ADMIN in this guest. An empty
# range keeps it on the single-UID self-mapping an unprivileged process may
# write itself.
if grep -q "^$(id -un):" /etc/subuid 2>/dev/null; then
echo "unexpected subordinate UID range for $(id -un): podman would" >&2
echo "require CAP_SYS_ADMIN via newuidmap in this guest" >&2
exit 1
fi
for device in /dev/fuse /dev/net/tun; do
[ -r "$device" ] && [ -w "$device" ] || {
echo "device $device is not readable/writable by $(id -un)" >&2
exit 1
}
done
# Short by necessity, not by accident: conmon's attach socket lives under
# this directory and must fit in a 108-byte sun_path. See nested_containers.py.
# Must stay in step with AGENT_CA_BUNDLE in bot_bottle/backend/util.py; a unit
# test pins the two together.
CA_BUNDLE="/etc/ssl/certs/ca-certificates.crt"
[ -r "$CA_BUNDLE" ] || {
echo "gateway CA bundle $CA_BUNDLE is missing or unreadable" >&2
exit 1
}
# The proxy URL the agent inherits names `bot-bottle-gateway`, which resolves
# only through this bottle's /etc/hosts. A nested container gets its own hosts
# file, so it cannot resolve the name and dies at "Could not resolve proxy".
#
# podman's containers.conf `hosts_file` would fix that, except the
# Docker-compatible API ignores it — it only takes effect for native
# `podman run`, and the agent types `docker`. So the name is resolved *here*
# and the address, not the name, goes into the proxy URL the nested container
# receives. Verified on macOS 26 / podman 5.4.2: with the address in place,
# https://quay.io returns 200 and a non-allowlisted host still gets 403, so
# the egress boundary applies inside nested containers too.
GATEWAY_NAME="bot-bottle-gateway"
gateway_ip="$(
awk -v name="$GATEWAY_NAME" '$2 == name { print $1; exit }' /etc/hosts
)"
[ -n "$gateway_ip" ] || {
echo "no /etc/hosts entry for $GATEWAY_NAME; the gateway address is" >&2
echo "needed so nested containers can reach the egress proxy" >&2
exit 1
}
export XDG_RUNTIME_DIR="${XDG_RUNTIME_DIR:-/tmp/bbp}"
config="$HOME/.config/containers"
mkdir -p "$XDG_RUNTIME_DIR" "$config"
chmod 700 "$XDG_RUNTIME_DIR"
# ignore_chown_errors is required, not incidental: with a single-UID mapping
# there is no second UID for image layers to be chowned to, so layers that
# record other owners would otherwise fail to extract.
cat > "$config/storage.conf" <<'CONF'
[storage]
driver="overlay"
[storage.options.overlay]
mount_program="/usr/bin/fuse-overlayfs"
ignore_chown_errors="true"
CONF
# No cgroup delegation reaches this guest, so asking podman to manage cgroups
# fails; events_logger=file avoids the journald socket that is equally absent.
#
# The rest of this config is what lets a nested container reach the network:
#
# hosts_file only takes effect for native `podman run` — the
# Docker-compatible API ignores it, and the agent types
# `docker`. Kept anyway because it costs nothing and makes
# podman-native use behave; the compat path is covered by the
# address-bearing proxy URL below.
# volumes/env the gateway TLS-intercepts, so a container that does not
# trust the bottle's CA bundle gets "unable to get local issuer
# certificate". Mounting the bundle read-only and pointing the
# usual env vars at it covers curl, wget, python, and node
# without distro-specific trust commands.
#
# The proxy URL carries the bottle's identity token. podman already forwards
# that same URL into every nested container from the agent's own environment,
# so writing it to a 0600 file inside this disposable VM hands it to nobody
# new. It is never echoed.
CA_BUNDLE="$CA_BUNDLE" GATEWAY_NAME="$GATEWAY_NAME" GATEWAY_IP="$gateway_ip" \
CONTAINERS_CONF="$config/containers.conf" python3 - <<'PY'
import os
from pathlib import Path
ca = os.environ["CA_BUNDLE"]
name = os.environ["GATEWAY_NAME"]
ip = os.environ["GATEWAY_IP"]
entries = [
f"SSL_CERT_FILE={ca}",
f"CURL_CA_BUNDLE={ca}",
f"REQUESTS_CA_BUNDLE={ca}",
f"NODE_EXTRA_CA_CERTS={ca}",
]
# The gateway name resolves only through the bottle's /etc/hosts, which a
# nested container does not inherit, so hand it the address instead.
for var in ("HTTP_PROXY", "HTTPS_PROXY", "http_proxy", "https_proxy"):
value = os.environ.get(var)
if value:
entries.append(f"{var}={value.replace(name, ip)}")
# NO_PROXY keeps the name: it is matched against what a client asks for, and
# code inside a nested container still says "bot-bottle-gateway".
for var in ("NO_PROXY", "no_proxy"):
value = os.environ.get(var)
if value:
entries.append(f"{var}={value}")
path = Path(os.environ["CONTAINERS_CONF"])
path.write_text("\n".join([
"[containers]",
'cgroups="disabled"',
# podman copies the host's proxy vars into every container by default,
# and that copy *wins* over the env below — putting the unresolvable
# gateway name back. Turn it off so the address-bearing URLs stand.
"http_proxy=false",
'hosts_file="/etc/hosts"',
f'volumes=["{ca}:{ca}:ro"]',
"env=[",
*[f' "{entry}",' for entry in entries],
"]",
"[engine]",
'cgroup_manager="cgroupfs"',
'events_logger="file"',
"",
]), encoding="utf-8")
path.chmod(0o600)
PY
# Registry pulls egress through the bottle's proxy like everything else. The
# token-bearing proxy URL is already in the agent's environment; persisting it
# inside this disposable VM does not broaden its authority.
#
# This file is also what the Docker CLI copies into every container it starts,
# and being client-side it beats anything the podman service does — it is why
# containers.conf `env`, `http_proxy=false`, and the service's own environment
# all failed to change what a nested container saw. The address goes in here
# for the same reason it goes everywhere else: `bot-bottle-gateway` resolves
# in the bottle, never inside a nested container.
GATEWAY_NAME="$GATEWAY_NAME" GATEWAY_IP="$gateway_ip" python3 - <<'PY'
import json
import os
from pathlib import Path
name = os.environ["GATEWAY_NAME"]
ip = os.environ["GATEWAY_IP"]
proxy = os.environ.get("HTTPS_PROXY") or os.environ.get("https_proxy", "")
# NO_PROXY keeps the name: it is matched against what a client asks for, and
# code inside a nested container still says "bot-bottle-gateway".
no_proxy = os.environ.get("NO_PROXY") or os.environ.get("no_proxy", "")
config = {"proxies": {"default": {
"httpProxy": proxy.replace(name, ip),
"httpsProxy": proxy.replace(name, ip),
"noProxy": no_proxy,
}}}
path = Path.home() / ".docker" / "config.json"
path.parent.mkdir(parents=True, exist_ok=True)
path.write_text(json.dumps(config), encoding="utf-8")
path.chmod(0o600)
PY
if docker info >/dev/null 2>&1; then
exit 0
fi
# Belt to the ~/.docker/config.json braces above, which is what actually
# decides this for `docker run`. The service environment is what podman falls
# back to for anything the CLI does not stamp — its own registry pulls, and
# containers created through the API by something other than the Docker CLI.
# Cheap, and it keeps the address consistent across both paths.
#
# Assigned via parameter expansion, never echoed: these carry the bottle's
# identity token.
for var in HTTP_PROXY HTTPS_PROXY http_proxy https_proxy; do
eval "value=\${$var:-}"
[ -n "$value" ] || continue
eval "export $var=\"\${value%%$GATEWAY_NAME*}$gateway_ip\${value#*$GATEWAY_NAME}\""
done
log=/tmp/bot-bottle-nested-containers.log
nohup podman system service --time=0 \
"unix://$XDG_RUNTIME_DIR/podman.sock" \
>"$log" 2>&1 </dev/null &
@@ -0,0 +1,161 @@
"""Guest-local container engine for Apple-container bottles (issue #392).
The service and every nested container remain inside the existing per-bottle
VM. This module refuses to compensate for missing prerequisites with outer
capabilities, a privileged container, or a host Docker socket.
Podman is used rather than rootless Docker for one specific reason: Apple
Container's capability bounding set omits `CAP_SYS_ADMIN`, which the kernel
requires to write a multi-range `uid_map` via `newuidmap`. Rootless Docker
has no path that avoids that write. Podman does with no subordinate UID
range configured it falls back to a single-UID self-mapping, which an
unprivileged process may write itself. See
`docs/research/rootless-docker-in-apple-container-spike.md`.
That fallback is why `build_image` *removes* the agent user's `/etc/subuid`
and `/etc/subgid` entries instead of adding them: their presence is precisely
what would send podman down the `newuidmap` path that cannot work here.
The agent still talks to `docker` and `docker compose`; those speak to
podman's Docker-compatible API socket, so nothing in the agent's habits
changes.
"""
from __future__ import annotations
import shlex
import shutil
import tempfile
import time
from pathlib import Path
from typing import Callable
from ...log import die, info
_INIT = "/usr/local/libexec/bot-bottle/nested-containers-init"
# Deliberately cryptic and short. podman derives conmon's attach socket as
# `$XDG_RUNTIME_DIR/libpod/tmp/socket/<64-hex-id>/attach`, and a Unix socket
# path may not exceed 108 bytes (`sun_path`). The descriptive
# `/tmp/bot-bottle-podman-run` produced a 116-byte path — over the limit, so
# attach would have broken as soon as anything got far enough to attach. Do
# not lengthen this for readability; it buys 8 bytes of headroom.
_RUNTIME_DIR = "/tmp/bbp"
_SOCKET = f"{_RUNTIME_DIR}/podman.sock"
_LOG = "/tmp/bot-bottle-nested-containers.log"
IMAGE_SUFFIX = "-nested-containers"
READY_RETRIES = 30
# Apple Container creates both device nodes 0600 root:root, so the agent user
# cannot open them: /dev/fuse blocks the fuse-overlayfs storage driver and
# /dev/net/tun blocks slirp4netns, which rootless podman uses for the default
# bridge network that stock compose files expect. Relaxing the modes needs no
# capability the bottle does not already hold — unlike CAP_SYS_ADMIN, which is
# what killed the rootless-Docker approach.
_GUEST_DEVICES = ("/dev/fuse", "/dev/net/tun")
def build_image(
base_image: str,
build: Callable[..., None],
) -> str:
"""Layer the nested-container tooling onto an already-built agent image.
Podman and its networking stack live here rather than in the base agent
images so that bottles without the flag pay no image-size cost.
# TODO(#394): replace this hand-rolled Dockerfile with a docker-layer
# abstraction once that infrastructure exists.
"""
image = f"{base_image}{IMAGE_SUFFIX}"
init_script = Path(__file__).with_name("nested-containers-init.sh")
with tempfile.TemporaryDirectory(prefix="bot-bottle-nested-containers.") as tmp:
context = Path(tmp)
shutil.copy2(init_script, context / "nested-containers-init.sh")
(context / "Dockerfile").write_text(
"FROM docker:28-cli AS docker_cli\n"
f"FROM {base_image}\n"
"USER root\n"
"COPY --from=docker_cli /usr/local/bin/docker /usr/local/bin/docker\n"
"COPY --from=docker_cli /usr/local/libexec/docker/cli-plugins/"
"docker-compose /usr/local/libexec/docker/cli-plugins/docker-compose\n"
"RUN apt-get update \\\n"
# podman 5's networking stack, installed explicitly because
# --no-install-recommends omits it and each missing piece fails
# at a different, misleading layer:
# podman -> moved here from the base agent images so that
# bottles without nested_containers pay no cost
# passt -> `pasta`, the default rootless netns helper
# (podman 4 used slirp4netns); without it
# nothing starts: "could not find pasta"
# nftables -> `nft`, which netavark shells out to for the
# bridge network every compose file expects
# aardvark-dns -> name resolution *inside* nested containers;
# without it DNS fails while everything else
# looks healthy
# slirp4netns stays as the documented fallback for pasta.
" && apt-get install -y --no-install-recommends "
"aardvark-dns fuse-overlayfs netavark nftables passt podman "
"slirp4netns uidmap \\\n"
" && rm -rf /var/lib/apt/lists/* \\\n"
# Deliberate: an empty subordinate range keeps podman on the
# single-UID mapping that needs no CAP_SYS_ADMIN. Adding ranges
# here would reintroduce the newuidmap failure this design exists
# to route around.
" && sed -i '/^node:/d' /etc/subuid /etc/subgid\n"
"COPY nested-containers-init.sh "
"/usr/local/libexec/bot-bottle/nested-containers-init\n"
"RUN chmod 0755 /usr/local/libexec/bot-bottle/nested-containers-init\n"
"USER node\n",
encoding="utf-8",
)
build(image, str(context), dockerfile=str(context / "Dockerfile"))
return image
def guest_env(enabled: bool) -> dict[str, str]:
"""Environment consumed by the Docker CLI inside an enabled bottle."""
if not enabled:
return {}
return {
"DOCKER_HOST": f"unix://{_SOCKET}",
"XDG_RUNTIME_DIR": _RUNTIME_DIR,
}
def prepare_guest_devices(container_name: str, exec_as_root: Callable[..., None]) -> None:
"""Make /dev/fuse and /dev/net/tun openable by the agent user.
Runs as root inside the bottle because the agent must not be able to
re-mode device nodes itself. No outer capability is involved.
"""
exec_as_root(
container_name,
["sh", "-c", f"chmod 0666 {' '.join(_GUEST_DEVICES)}"],
)
def start(bottle: object) -> None:
"""Start and verify the unprivileged service through the bottle exec API."""
info("starting guest-local container engine")
result = bottle.exec(shlex.quote(_INIT)) # type: ignore[attr-defined]
if result.returncode != 0:
detail = (result.stderr or result.stdout or "").strip()
die(f"nested-container bootstrap failed: {detail or '<no output>'}")
for _ in range(READY_RETRIES):
result = bottle.exec("docker info >/dev/null 2>&1") # type: ignore[attr-defined]
if result.returncode == 0:
info("guest-local container engine is ready")
return
time.sleep(0.2)
logs = bottle.exec( # type: ignore[attr-defined]
f"tail -n 80 {_LOG} 2>/dev/null || true"
)
die(
"guest-local container engine did not become ready without additional "
f"outer privileges:\n{(logs.stdout or logs.stderr or '<no log>').strip()}"
)
__all__ = ["build_image", "guest_env", "prepare_guest_devices", "start"]
@@ -44,4 +44,5 @@ def resolve_plan(
egress_plan=egress_plan,
supervise_plan=supervise_plan,
agent_provision=agent_provision_plan,
nested_containers=manifest.bottle.nested_containers,
)
@@ -10,6 +10,7 @@ import shutil
import subprocess
import tempfile
import time
from datetime import datetime, timezone
from typing import Iterable
from ...log import die, info
@@ -360,6 +361,12 @@ def exec_container(name: str, argv: list[str]) -> None:
)
def read_container_env(name: str, env_name: str) -> str:
"""Read one configured env value from a running container, or ``""``."""
result = _run_container_op([_CONTAINER, "exec", name, "printenv", env_name])
return result.stdout.strip() if result.returncode == 0 else ""
def exec_container_as_root(name: str, argv: list[str]) -> None:
"""`exec_container`, but as uid 0 inside the container.
@@ -572,6 +579,41 @@ def try_container_ipv4_on_network(name: str, network: str) -> str:
return ""
def inspect_container_network_ip(name: str, network: str) -> str | None:
"""IP of `name` on `network`, distinguishing inspect failure from "not yet".
Returns:
- the IP string when the container has one on `network`
- "" when inspect succeeds but no address is assigned yet (in-flight DHCP)
- None when the inspect command itself fails (authoritative list impossible)
"""
result = subprocess.run(
[_CONTAINER, "inspect", name],
capture_output=True, text=True, check=False,
)
if result.returncode != 0:
return None
try:
data = json.loads(result.stdout or "[]")
except json.JSONDecodeError:
return None
if isinstance(data, list):
data = data[0] if data else {}
if not isinstance(data, dict):
return None
status = data.get("status")
networks = status.get("networks") if isinstance(status, dict) else None
if not isinstance(networks, list):
return ""
for entry in networks:
if not isinstance(entry, dict) or entry.get("network") != network:
continue
raw = entry.get("ipv4Address")
if isinstance(raw, str) and raw:
return raw.split("/", 1)[0]
return ""
def wait_container_ipv4_on_network(
name: str, network: str, *, timeout: float = 15.0, poll: float = 0.25,
) -> str:
@@ -626,6 +668,39 @@ def image_id(ref: str) -> str:
raise AssertionError("unreachable")
def image_created_at(ref: str) -> datetime | None:
"""Return the image creation timestamp as an aware UTC datetime, or None
when the field is absent or unparseable (e.g. FROM-scratch images, images
pulled from registries that omit the field). Callers should skip the stale
check when None is returned rather than treating it as an error."""
result = subprocess.run(
[_CONTAINER, "image", "inspect", ref],
capture_output=True,
text=True,
check=False,
)
if result.returncode != 0:
die(
f"container image inspect for {ref!r} failed: "
f"{(result.stderr or '').strip() or '<no stderr>'}"
)
try:
data = json.loads(result.stdout or "{}")
except json.JSONDecodeError as exc:
die(f"container image inspect for {ref!r} returned malformed JSON: {exc}")
if isinstance(data, list) and data:
data = data[0]
if isinstance(data, dict):
value = data.get("created") or data.get("Created")
if isinstance(value, str) and value:
try:
ts = value.rstrip("Z")
return datetime.fromisoformat(ts).replace(tzinfo=timezone.utc)
except ValueError:
pass
return None
def save(ref: str, output: str) -> None:
subprocess.run([_CONTAINER, "image", "save", ref, "-o", output], check=True)
+18
View File
@@ -26,6 +26,7 @@ from ..bottle_state import (
)
from ..egress import Egress, EgressPlan
from ..git_gate import GitGate, GitGatePlan
from ..log import die
from ..manifest import Manifest, ManifestBottle
from ..supervise import Supervise, SupervisePlan
from . import BottleSpec
@@ -112,6 +113,22 @@ def merge_provision_env_vars(provision: AgentProvisionPlan) -> AgentProvisionPla
return replace(provision, guest_env=merged)
def reject_nested_containers(backend: str, manifest: Manifest) -> None:
"""Fail loudly when a backend cannot honor `nested_containers: true`.
Silently ignoring it would hand the agent a bottle where `docker` is not
there and the only sound alternatives on these backends (a host daemon
socket, a privileged container) are exactly what issue #392 rules out.
"""
if not manifest.bottle.nested_containers:
return
die(
f"nested_containers is not supported on the {backend} backend. "
"Only macos-container runs a guest-local container engine today; "
"mounting the host Docker socket is not an option bot-bottle offers."
)
def resolve_manifest_dockerfile(path_value: str, spec: BottleSpec) -> str:
"""Resolve a manifest-supplied dockerfile path relative to user_cwd."""
path = Path(os.path.expanduser(path_value))
@@ -122,6 +139,7 @@ def resolve_manifest_dockerfile(path_value: str, spec: BottleSpec) -> str:
__all__ = [
"merge_provision_env_vars",
"reject_nested_containers",
"mint_slug",
"prepare_agent_state_dir",
"prepare_egress",
+20
View File
@@ -7,6 +7,8 @@ from __future__ import annotations
import hashlib
import os
import ssl
import time
from collections.abc import Callable
from pathlib import Path
from typing import TYPE_CHECKING
@@ -15,6 +17,24 @@ from ..log import die, info
if TYPE_CHECKING:
from ..egress import EgressPlan
_CA_POLL_INTERVAL = 0.5
def poll_ca_cert(fetch: Callable[[], str | None], *, timeout: float) -> str:
"""Poll `fetch` until it returns a non-empty PEM string or `timeout` expires.
`fetch` should return the PEM on success and `None` (or empty string) when
the cert is not yet available. Raises `TimeoutError` if the cert never
appears within `timeout` seconds."""
deadline = time.monotonic() + timeout
while True:
result = fetch()
if result:
return result
if time.monotonic() >= deadline:
raise TimeoutError(f"CA cert not available after {timeout:g}s")
time.sleep(_CA_POLL_INTERVAL)
# Debian-family CA layout, shared by every backend (all guest images
# are Debian-family). AGENT_CA_PATH is the source path that
+4 -5
View File
@@ -19,6 +19,7 @@ from .commit import cmd_commit
from .edit import cmd_edit
from .info import cmd_info
from .init import cmd_init
from .login import cmd_login
from .resume import cmd_resume
from .start import cmd_start
from .supervise import cmd_supervise
@@ -33,6 +34,7 @@ COMMANDS = {
"info": cmd_info,
"init": cmd_init,
"list": cmd_list,
"login": cmd_login,
"resume": cmd_resume,
"start": cmd_start,
"supervise": cmd_supervise,
@@ -43,7 +45,7 @@ COMMANDS = {
# the host (TAP pool, /dev/kvm, firecracker) and never opens the store, so
# gating it on the schema breaks preflight on a fresh CI runner where stdin
# isn't a TTY and the migration prompt can't be answered.
NO_MIGRATION_COMMANDS = frozenset({"backend"})
NO_MIGRATION_COMMANDS = frozenset({"backend", "login"})
def usage() -> None:
@@ -56,6 +58,7 @@ def usage() -> None:
sys.stderr.write(" info print env, skills, and prompt details for a named agent\n")
sys.stderr.write(" init interactively create a new agent and add it to bot-bottle.json\n")
sys.stderr.write(" list list available agents or active containers\n")
sys.stderr.write(" login register this host with a bot-bottle console\n")
sys.stderr.write(
" resume re-launch a bottle by its identity "
"(continues state from PRD 0016)\n"
@@ -111,7 +114,3 @@ def main(argv: list[str] | None = None) -> int:
return e.code if isinstance(e.code, int) else 1
except KeyboardInterrupt:
return 130
if __name__ == "__main__":
sys.exit(main())
+15
View File
@@ -0,0 +1,15 @@
"""Entry point for `python -m bot_bottle.cli`.
`cli.py` at the repo root is the usual way in; this makes the package
runnable too, so the CLI works from an installed copy where there is no
`cli.py` on disk to point at.
"""
from __future__ import annotations
import sys
from . import main
if __name__ == "__main__":
sys.exit(main())
+168
View File
@@ -0,0 +1,168 @@
"""bb login — register this host with a bot-bottle console.
Opens a device-authorization flow against the target console, waits for the
operator to approve, then writes access and refresh tokens to
~/.bot-bottle/console.json (or $BOT_BOTTLE_ROOT/console.json).
Usage:
bb login [--console-url URL] [--label LABEL]
Flags:
--console-url URL Target console URL (overrides BB_CONSOLE_URL env var)
--label LABEL Host label shown in the console (default: hostname)
"""
from __future__ import annotations
import json
import os
import socket
import sys
import tempfile
import time
import urllib.error
import urllib.request
from pathlib import Path
from typing import Any
from ..paths import bot_bottle_root
_CONSOLE_URL_ENV = "BB_CONSOLE_URL"
_POLL_SLEEP = 2 # seconds between polls; matches console's poll_interval default
def _usage() -> None:
sys.stderr.write(
"usage: bb login [--console-url URL] [--label LABEL]\n"
"\n"
"Options:\n"
" --console-url URL Console base URL (or BB_CONSOLE_URL env var)\n"
" --label LABEL Host label shown in the console (default: hostname)\n"
)
def _flag(argv: list[str], name: str) -> str | None:
for i, arg in enumerate(argv):
if arg == name and i + 1 < len(argv):
return argv[i + 1]
if arg.startswith(f"{name}="):
return arg[len(name) + 1:]
return None
def _post(url: str, payload: dict[str, Any]) -> dict[str, Any]:
data = json.dumps(payload).encode()
req = urllib.request.Request(
url, data=data, headers={"Content-Type": "application/json"}
)
with urllib.request.urlopen(req, timeout=10) as resp:
return json.loads(resp.read())
def _get(url: str) -> tuple[int, dict[str, Any]]:
req = urllib.request.Request(url)
try:
with urllib.request.urlopen(req, timeout=10) as resp:
return resp.status, json.loads(resp.read())
except urllib.error.HTTPError as e:
return e.code, {}
def _save_credentials(
console_url: str, host_id: str, access_token: str, refresh_token: str
) -> Path:
path = bot_bottle_root() / "console.json"
path.parent.mkdir(parents=True, exist_ok=True)
content = (
json.dumps(
{
"url": console_url,
"host_id": host_id,
"access_token": access_token,
"refresh_token": refresh_token,
},
indent=2,
)
+ "\n"
)
fd, tmp_path_str = tempfile.mkstemp(dir=path.parent, prefix=".console-")
tmp = Path(tmp_path_str)
try:
tmp.chmod(0o600)
with os.fdopen(fd, "w") as f:
f.write(content)
os.replace(tmp, path)
except OSError:
try:
tmp.unlink()
except OSError:
pass
raise
return path
def cmd_login(argv: list[str]) -> int:
if "--help" in argv or "-h" in argv:
_usage()
return 0
console_url = _flag(argv, "--console-url") or os.environ.get(_CONSOLE_URL_ENV)
if not console_url:
sys.stderr.write(
"bb login: --console-url or BB_CONSOLE_URL is required\n"
)
return 1
console_url = console_url.rstrip("/")
label = _flag(argv, "--label") or socket.gethostname()
try:
resp = _post(f"{console_url}/api/v1/hosts/authorize", {"label": label})
except (OSError, ValueError) as exc:
sys.stderr.write(f"bb login: failed to start authorization: {exc}\n")
return 1
device_code = resp["device_code"]
user_code = resp["user_code"]
expires_in = resp.get("expires_in", 300)
poll_sleep = max(1, min(int(resp.get("poll_interval", _POLL_SLEEP)), 60))
sys.stderr.write(
f"\nOpen this URL in your browser to authorize this host:\n\n"
f" {console_url}/hosts/authorize?code={user_code}\n\n"
f"Waiting for approval"
)
deadline = time.monotonic() + expires_in
while time.monotonic() < deadline:
sys.stderr.write(".")
sys.stderr.flush()
time.sleep(poll_sleep)
try:
code, result = _get(
f"{console_url}/api/v1/hosts/authorize/{device_code}"
)
except (OSError, ValueError):
continue
if code == 410:
break
st = result.get("status")
if st == "approved":
sys.stderr.write("\n\nApproved.\n")
path = _save_credentials(
console_url,
result["host_id"],
result["access_token"],
result["refresh_token"],
)
sys.stderr.write(f"Credentials saved to {path}\n")
return 0
if st == "denied":
sys.stderr.write("\n\nDenied by operator.\n")
return 1
sys.stderr.write("\n\nAuthorization timed out.\n")
return 1
+46 -4
View File
@@ -14,6 +14,7 @@ the private orchestrator `_launch_bottle`.
from __future__ import annotations
import argparse
import io
import os
import shutil
import sys
@@ -35,6 +36,7 @@ from ..bottle_state import (
is_preserved,
mark_preserved,
)
from ..image_cache import StaleImageError
from ..log import info, die
from ..manifest import Manifest, ManifestIndex
from ._common import PROG, USER_CWD, read_tty_line
@@ -64,6 +66,14 @@ def cmd_start(argv: list[str]) -> int:
"skip all prompts. For orchestrators, CI, and webhooks."
),
)
parser.add_argument(
"--cached-images",
action="store_true",
help=(
"quickstart with existing local agent and sidecar images; "
"only valid with --headless"
),
)
parser.add_argument(
"--bottle",
action="append",
@@ -96,6 +106,8 @@ def cmd_start(argv: list[str]) -> int:
help="agent name defined in bot-bottle.json (omit to pick interactively)",
)
args = parser.parse_args(argv)
if args.cached_images and not args.headless:
die("--cached-images is only supported with --headless")
dry_run = args.dry_run or os.environ.get("BOT_BOTTLE_DRY_RUN") == "1"
if args.no_cache or os.environ.get("BOT_BOTTLE_NO_CACHE") == "1":
@@ -147,6 +159,10 @@ def cmd_start(argv: list[str]) -> int:
label, color = tui.name_color_modal(default_label=agent_name)
label, color = _resolve_unique_label(label, color)
image_policy = _select_image_policy()
if image_policy is None:
return 0
spec = BottleSpec(
manifest=manifest,
agent_name=agent_name,
@@ -155,6 +171,7 @@ def cmd_start(argv: list[str]) -> int:
label=label,
color=color,
bottle_names=bottle_names,
image_policy=image_policy,
)
return _launch_bottle(
spec,
@@ -179,6 +196,16 @@ def _start_headless(
path, so the agent still execs on the inherited stdio/PTY an
orchestrator allocates that PTY and relays it to its
desktop/mobile clients."""
try:
stdin_fd = sys.stdin.fileno()
except io.UnsupportedOperation:
stdin_fd = -1
if not os.isatty(stdin_fd):
die(
"--headless requires a PTY on stdin; run via:\n"
" script -q /dev/null ./cli.py start ..."
)
agent_name = args.name
if not agent_name:
die("--headless requires an agent name: ./cli.py start <agent> --headless")
@@ -213,6 +240,7 @@ def _start_headless(
color=args.color or "",
bottle_names=bottle_names,
headless=True,
image_policy="cached" if args.cached_images else "fresh",
)
return _launch_bottle(
spec,
@@ -395,6 +423,13 @@ def _text_prompt_yes() -> bool:
return reply in ("y", "Y", "yes", "YES")
def _select_image_policy() -> str | None:
return tui.filter_select(
["fresh", "cached"],
title="Select image startup mode",
)
def _text_render_preflight():
def _render(plan: DockerBottlePlan, backend_name: str) -> None:
print(file=sys.stderr)
@@ -500,6 +535,8 @@ def _manifest_to_yaml(manifest: Manifest) -> str:
lines.append(f" scheme: {r.AuthScheme}")
lines.append(f" supervise: {'true' if bottle.supervise else 'false'}")
if bottle.nested_containers:
lines.append(" nested_containers: true")
return "\n".join(lines)
@@ -537,6 +574,15 @@ def _launch_bottle(
return 0
backend = get_bottle_backend(backend_name)
try:
backend.prelaunch_checks(plan)
except StaleImageError as exc:
if assume_yes:
die(str(exc))
sys.stderr.write(f"bot-bottle: {exc}\nLaunch anyway? [y/N] ")
sys.stderr.flush()
if read_tty_line() not in ("y", "Y", "yes", "YES"):
return 0
with backend.launch(plan) as bottle:
agent_provider_template = getattr(plan, "agent_provider_template", "claude")
extra_args: tuple[str, ...] = ()
@@ -555,10 +601,6 @@ def _launch_bottle(
f"session ended (exit {exit_code}); "
f"container {bottle.name} will be removed"
)
# While the container is still alive: always snapshot the
# transcript and — if the agent exited non-zero — mark
# the state for preservation. This picks up crashes /
# Ctrl-Cs / OOM kills before cleanup removes the state dir.
if agent_provider_template == "claude":
capture_claude_session_state(identity, exit_code)
return 0
+71
View File
@@ -0,0 +1,71 @@
"""SQLite-backed bot-bottle configuration store."""
from __future__ import annotations
from pathlib import Path
try:
from .db_store import DbStore
from .migrations import TableMigrations
from .paths import host_db_path
except ImportError:
from db_store import DbStore # type: ignore[import-not-found] # pylint: disable=import-error,no-name-in-module
from migrations import TableMigrations # type: ignore[import-not-found] # pylint: disable=import-error,no-name-in-module
from paths import host_db_path # type: ignore[import-not-found] # pylint: disable=import-error,no-name-in-module
DEFAULT_CACHED_IMAGE_STALE_WARNING_DAYS = 1
class ConfigStore(DbStore):
"""SQLite configuration for host-side bot-bottle settings."""
def __init__(self, db_path: Path | None = None) -> None:
migrations = TableMigrations("config_store", [
# v1 — host-side bot-bottle settings
"""
CREATE TABLE IF NOT EXISTS bot_bottle_config (
id INTEGER PRIMARY KEY CHECK (id = 1),
cached_image_stale_warning_days INTEGER NOT NULL DEFAULT 1
)
""",
])
super().__init__(db_path or host_db_path(), migrations)
def cached_image_stale_warning_days(self) -> int:
if not self.db_path.is_file():
return DEFAULT_CACHED_IMAGE_STALE_WARNING_DAYS
with self._connect() as conn:
row = conn.execute(
"""
SELECT cached_image_stale_warning_days
FROM bot_bottle_config
WHERE id = 1
""",
).fetchone()
if row is None:
return DEFAULT_CACHED_IMAGE_STALE_WARNING_DAYS
try:
return int(row["cached_image_stale_warning_days"])
except (TypeError, ValueError):
return DEFAULT_CACHED_IMAGE_STALE_WARNING_DAYS
def set_cached_image_stale_warning_days(self, days: int) -> Path:
with self._connect() as conn:
conn.execute(
"""
INSERT INTO bot_bottle_config (id, cached_image_stale_warning_days)
VALUES (1, ?)
ON CONFLICT(id) DO UPDATE SET
cached_image_stale_warning_days = excluded.cached_image_stale_warning_days
""",
(days,),
)
self._chmod()
return self.db_path
__all__ = [
"DEFAULT_CACHED_IMAGE_STALE_WARNING_DAYS",
"ConfigStore",
]
+14 -2
View File
@@ -10,7 +10,7 @@
# Current Node LTS; slim variant keeps the image small while still
# providing apt-get for any future additions.
FROM node:22-slim
FROM node:22-trixie-slim
# Install runtime system deps. claude-code shells out to git for several
# features (status checks, commits, PR creation) — without git in the
@@ -21,7 +21,14 @@ FROM node:22-slim
# to it) works against egress's bumped TLS without the agent needing
# local DNS.
RUN apt-get update \
&& apt-get install -y --no-install-recommends git ca-certificates curl ripgrep iproute2 dnsutils \
&& apt-get install -y --no-install-recommends \
git \
ca-certificates \
curl \
openssh-client \
ripgrep \
iproute2 \
dnsutils \
&& rm -rf /var/lib/apt/lists/*
# App-specific deps. Python isn't required by claude-code itself
@@ -39,6 +46,11 @@ RUN apt-get update \
RUN npm install -g --no-fund --no-audit @anthropic-ai/claude-code@2.1.172 \
&& npm cache clean --force
# Git reads both ~/.gitconfig and ~/.config/git/config. Keep its XDG config
# path traversable by the non-root runtime user so permission errors do not
# suppress bot-bottle's git-gate insteadOf rules.
RUN install -d -o node -g node -m 755 /home/node/.config /home/node/.config/git
# Run as a non-root user. The node image already provides a `node` user
# (uid 1000) with a home directory, which is where claude-code will write
# its session state.
+17 -5
View File
@@ -23,8 +23,9 @@ from ...agent_provider import (
provider_startup_args,
)
from ...backend.docker import util as docker_mod
from ...egress import EgressRoute
from ...egress import CLAUDE_HOST_CREDENTIAL_TOKEN_REF, EgressRoute
from ...log import die, info, warn
from .claude_auth import claude_host_access_token
if TYPE_CHECKING:
@@ -118,7 +119,6 @@ class ClaudeAgentProvider(AgentProvider):
color: str = "",
provider_settings: dict[str, object] | None = None,
) -> AgentProvisionPlan:
del forward_host_credentials, host_env
resolved_guest_env = dict(guest_env or {})
startup_args = provider_startup_args(provider_settings)
guest_home = self.guest_home
@@ -180,13 +180,24 @@ class ClaudeAgentProvider(AgentProvider):
claude_settings,
f"{guest_home}/.claude/settings.json",
))
provisioned_env: dict[str, str] = {}
if forward_host_credentials:
_host_env = host_env or dict(os.environ)
provisioned_env[CLAUDE_HOST_CREDENTIAL_TOKEN_REF] = (
claude_host_access_token(_host_env)
)
cred_token_ref = (
CLAUDE_HOST_CREDENTIAL_TOKEN_REF if forward_host_credentials
else auth_token
)
egress_routes = (EgressRoute(
host="api.anthropic.com",
auth_scheme="Bearer" if auth_token else "",
token_ref=auth_token,
auth_scheme="Bearer" if (auth_token or forward_host_credentials) else "",
token_ref=cred_token_ref,
),)
hidden_env_names: frozenset[str] = frozenset()
if auth_token:
if auth_token or forward_host_credentials:
env_vars["CLAUDE_CODE_OAUTH_TOKEN"] = "egress-placeholder"
hidden_env_names = frozenset({"CLAUDE_CODE_OAUTH_TOKEN"})
@@ -208,6 +219,7 @@ class ClaudeAgentProvider(AgentProvider):
files=tuple(files),
egress_routes=egress_routes,
hidden_env_names=hidden_env_names,
provisioned_env=provisioned_env,
)
def provision_skills(self, plan: "BottlePlan", bottle: "Bottle") -> None:
+114
View File
@@ -0,0 +1,114 @@
"""Host Claude auth helpers.
Reads the host's Claude Code credentials and returns only the access
token needed by egress. Does not expose refresh tokens or raw payloads.
Credential storage by platform:
Linux ~/.claude/.credentials.json
macOS macOS Keychain, service "Claude Code-credentials"
(file path is tried first; Keychain is the fallback)
"""
from __future__ import annotations
import json
import os
import subprocess
import sys
from datetime import datetime, timezone
from pathlib import Path
from ...log import die
_KEYCHAIN_SERVICE = "Claude Code-credentials"
def claude_auth_path(host_env: dict[str, str] | None = None) -> Path:
env = os.environ if host_env is None else host_env
home = env.get("HOME")
if home:
return Path(home) / ".claude" / ".credentials.json"
return Path.home() / ".claude" / ".credentials.json"
def _read_keychain() -> dict[str, object] | None:
"""Try the macOS Keychain. Returns parsed JSON dict or None."""
if sys.platform != "darwin":
return None
try:
result = subprocess.run(
["security", "find-generic-password", "-s", _KEYCHAIN_SERVICE, "-w"],
capture_output=True,
text=True,
timeout=10,
)
except (FileNotFoundError, subprocess.TimeoutExpired):
return None
if result.returncode != 0 or not result.stdout.strip():
return None
try:
raw = json.loads(result.stdout.strip())
except json.JSONDecodeError:
return None
return raw if isinstance(raw, dict) else None
def claude_host_access_token(
host_env: dict[str, str] | None = None,
*,
now: datetime | None = None,
) -> str:
path = claude_auth_path(host_env)
raw: dict[str, object] | None = None
if path.is_file():
try:
raw = json.loads(path.read_text())
except (OSError, json.JSONDecodeError) as e:
die(f"claude host credentials: could not read valid JSON at {path}: {e}")
if not isinstance(raw, dict):
die(f"claude host credentials: {path} must contain a JSON object")
else:
raw = _read_keychain()
if raw is None:
die(
f"claude host credentials: auth file missing at {path} and "
f"macOS Keychain lookup for '{_KEYCHAIN_SERVICE}' failed. "
"Run `claude login` on the host or disable "
"agent_provider.forward_host_credentials."
)
oauth = raw.get("claudeAiOauth")
if not isinstance(oauth, dict):
die(
"claude host credentials: claudeAiOauth is missing from credentials. "
"Run `claude login` on the host or disable "
"agent_provider.forward_host_credentials."
)
access_token = oauth.get("accessToken")
if not isinstance(access_token, str) or not access_token:
die(
"claude host credentials: claudeAiOauth.accessToken is missing or empty. "
"Run `claude login` on the host and restart the bottle."
)
# expiresAt is in milliseconds
expires_at = oauth.get("expiresAt")
if isinstance(expires_at, (int, float)):
check_now = now or datetime.now(timezone.utc)
exp_dt = datetime.fromtimestamp(float(expires_at) / 1000.0, timezone.utc)
if exp_dt <= check_now:
die(
"claude host credentials: host Claude access token is expired. "
"Run `claude login` on the host and restart the bottle."
)
return access_token
__all__ = [
"claude_auth_path",
"claude_host_access_token",
]
+10 -2
View File
@@ -3,10 +3,16 @@
# Mirrors the default Claude image shape: Node LTS, git/network tooling,
# non-root node user, and the provider CLI installed for that user.
FROM node:22-slim
FROM node:22-trixie-slim
RUN apt-get update \
&& apt-get install -y --no-install-recommends git ca-certificates curl procps ripgrep \
&& apt-get install -y --no-install-recommends \
git \
ca-certificates \
curl \
openssh-client \
procps \
ripgrep \
&& rm -rf /var/lib/apt/lists/*
# App-specific deps. Python isn't required by codex itself
@@ -17,6 +23,8 @@ RUN apt-get update \
&& apt-get install -y --no-install-recommends python3 python3-pip python3-venv \
&& rm -rf /var/lib/apt/lists/*
RUN install -d -o node -g node -m 755 /home/node/.config /home/node/.config/git
USER node
WORKDIR /home/node
+4 -2
View File
@@ -2,7 +2,7 @@
#
# Node LTS, git/network tooling, and the Pi coding-agent CLI installed globally.
FROM node:22-slim
FROM node:22-trixie-slim
RUN apt-get update \
&& apt-get install -y --no-install-recommends \
@@ -10,6 +10,7 @@ RUN apt-get update \
ca-certificates \
curl \
fd-find \
openssh-client \
ripgrep \
&& ln -s /usr/bin/fdfind /usr/local/bin/fd \
&& rm -rf /var/lib/apt/lists/*
@@ -21,7 +22,8 @@ RUN apt-get update \
RUN npm install -g --ignore-scripts --no-fund --no-audit @earendil-works/pi-coding-agent \
&& npm cache clean --force
RUN mkdir -p /home/node/.pi/agent \
RUN install -d -o node -g node -m 755 /home/node/.config /home/node/.config/git \
&& mkdir -p /home/node/.pi/agent \
/home/node/.pi/context-mode/sessions \
/tmp/pi-subagents-uid-1000 \
&& chown -R node:node /home/node/.pi /tmp \
+50 -32
View File
@@ -30,6 +30,7 @@ if TYPE_CHECKING:
from .manifest import ManifestBottle
CODEX_HOST_CREDENTIAL_TOKEN_REF = "BOT_BOTTLE_CODEX_HOST_ACCESS_TOKEN"
CLAUDE_HOST_CREDENTIAL_TOKEN_REF = "BOT_BOTTLE_CLAUDE_HOST_ACCESS_TOKEN"
EGRESS_HOSTNAME = "egress"
@@ -145,6 +146,8 @@ def egress_manifest_routes(
outbound_detectors=r.OutboundDetectors,
inbound_detectors=r.InboundDetectors,
outbound_on_match=r.OutboundOnMatch,
preserve_auth=r.PreserveAuth,
inspect=r.Inspect,
))
return tuple(out)
@@ -224,9 +227,13 @@ def _yaml_str_escape(s: str) -> str:
def _route_to_yaml_fields(r: Route) -> dict[str, object]:
fields: dict[str, object] = {"host": r.host}
if not r.inspect:
fields["inspect"] = False
return fields
inspect: dict[str, object] = {}
if r.auth_scheme and r.token_env:
fields["auth_scheme"] = r.auth_scheme
fields["token_env"] = r.token_env
inspect["auth_scheme"] = r.auth_scheme
inspect["token_env"] = r.token_env
if r.matches:
matches_data: list[dict[str, object]] = []
for entry in r.matches:
@@ -250,28 +257,30 @@ def _route_to_yaml_fields(r: Route) -> dict[str, object]:
headers_data.append(hd)
entry_data["headers"] = headers_data
matches_data.append(entry_data)
fields["matches"] = matches_data
inspect["matches"] = matches_data
if r.git_fetch:
fields["git"] = {"fetch": True}
inspect["git"] = {"fetch": True}
if r.preserve_auth:
inspect["preserve_auth"] = True
if (
r.outbound_detectors is not None
or r.inbound_detectors is not None
or r.outbound_on_match
):
dlp: dict[str, object] = {}
if r.outbound_detectors is not None:
dlp["outbound_detectors"] = (
inspect["outbound_detectors"] = (
False if not r.outbound_detectors
else list(r.outbound_detectors)
)
if r.inbound_detectors is not None:
dlp["inbound_detectors"] = (
inspect["inbound_detectors"] = (
False if not r.inbound_detectors
else list(r.inbound_detectors)
)
if r.outbound_on_match:
dlp["outbound_on_match"] = r.outbound_on_match
fields["dlp"] = dlp
inspect["outbound_on_match"] = r.outbound_on_match
if inspect:
fields["inspect"] = inspect
return fields
@@ -279,30 +288,30 @@ def _render_match_entry(entry: dict[str, object]) -> list[str]:
lines: list[str] = []
first_key = True
if "paths" in entry:
lines.append(" - paths:")
lines.append(" - paths:")
first_key = False
for pd in entry["paths"]: # type: ignore[union-attr]
pd_dict: dict[str, str] = pd # type: ignore[assignment]
if "type" in pd_dict:
lines.append(f' - type: "{_yaml_str_escape(pd_dict["type"])}"')
lines.append(f' value: "{_yaml_str_escape(pd_dict["value"])}"')
lines.append(f' - type: "{_yaml_str_escape(pd_dict["type"])}"')
lines.append(f' value: "{_yaml_str_escape(pd_dict["value"])}"')
else:
lines.append(f' - value: "{_yaml_str_escape(pd_dict["value"])}"')
lines.append(f' - value: "{_yaml_str_escape(pd_dict["value"])}"')
if "methods" in entry:
methods_str = ", ".join(f'"{_yaml_str_escape(m)}"' for m in entry["methods"]) # type: ignore[union-attr]
prefix = " - " if first_key else " "
prefix = " - " if first_key else " "
lines.append(f'{prefix}methods: [{methods_str}]')
first_key = False
if "headers" in entry:
prefix = " - " if first_key else " "
prefix = " - " if first_key else " "
lines.append(f"{prefix}headers:")
first_key = False
for hd in entry["headers"]: # type: ignore[union-attr]
hd_dict: dict[str, str] = hd # type: ignore[assignment]
lines.append(f' - name: "{_yaml_str_escape(hd_dict["name"])}"')
lines.append(f' value: "{_yaml_str_escape(hd_dict["value"])}"')
lines.append(f' - name: "{_yaml_str_escape(hd_dict["name"])}"')
lines.append(f' value: "{_yaml_str_escape(hd_dict["value"])}"')
if first_key:
lines.append(" - {}")
lines.append(" - {}")
return lines
@@ -321,22 +330,30 @@ def egress_render_routes(
for r in routes:
f = _route_to_yaml_fields(r)
lines.append(f' - host: "{_yaml_str_escape(str(f["host"]))}"')
if "auth_scheme" in f:
lines.append(f' auth_scheme: "{_yaml_str_escape(str(f["auth_scheme"]))}"')
lines.append(f' token_env: "{_yaml_str_escape(str(f["token_env"]))}"')
if "matches" in f:
lines.append(" matches:")
for entry in f["matches"]: # type: ignore[union-attr]
if f.get("inspect") is False:
lines.append(" inspect: false")
continue
inspect: dict[str, object] = f.get("inspect", {}) # type: ignore[assignment]
if not inspect:
continue
lines.append(" inspect:")
if "auth_scheme" in inspect:
lines.append(f' auth_scheme: "{_yaml_str_escape(str(inspect["auth_scheme"]))}"')
lines.append(f' token_env: "{_yaml_str_escape(str(inspect["token_env"]))}"')
if "matches" in inspect:
lines.append(" matches:")
for entry in inspect["matches"]: # type: ignore[union-attr]
lines.extend(_render_match_entry(entry)) # type: ignore[arg-type]
if "git" in f:
git_dict: dict[str, object] = f["git"] # type: ignore
lines.append(" git:")
if "git" in inspect:
git_dict: dict[str, object] = inspect["git"] # type: ignore
lines.append(" git:")
if git_dict.get("fetch") is True:
lines.append(" fetch: true")
if "dlp" in f:
dlp_dict: dict[str, object] = f["dlp"] # type: ignore
lines.append(" dlp:")
for dk, dv in dlp_dict.items():
lines.append(" fetch: true")
if inspect.get("preserve_auth") is True:
lines.append(" preserve_auth: true")
for dk in ("outbound_detectors", "inbound_detectors", "outbound_on_match"):
if dk in inspect:
dv = inspect[dk]
if dv is False:
lines.append(f" {dk}: false")
elif isinstance(dv, list):
@@ -400,6 +417,7 @@ class Egress(ABC):
)
__all__ = [
"CLAUDE_HOST_CREDENTIAL_TOKEN_REF",
"CODEX_HOST_CREDENTIAL_TOKEN_REF",
"EGRESS_HOSTNAME",
"EGRESS_ROUTES_FILENAME",
+107 -16
View File
@@ -78,6 +78,15 @@ def _token_from_proxy_auth(header: str) -> str:
# Seconds the egress proxy holds a token-blocked request open waiting for the
# operator's supervisor decision (PRD 0062), overridable via env.
DEFAULT_TOKEN_ALLOW_TIMEOUT_SECONDS = 300.0
# Maximum bytes of a response body passed to the DLP inbound scan. mitmproxy
# buffers the full response before the hook fires; capping at scan time limits
# the additional memory amplification from decoded text and regex match strings.
# A cap is a security trade-off (content above the threshold is not scanned),
# but without it a single large download OOM-kills the shared egress process
# (issue #455). Override with EGRESS_INBOUND_SCAN_LIMIT_BYTES; set to 0 to
# disable the cap.
DEFAULT_INBOUND_SCAN_LIMIT_BYTES = 1 * 1024 * 1024 # 1 MiB
# Filesystem poll cadence while awaiting the operator's response.
TOKEN_ALLOW_POLL_INTERVAL_SECONDS = 0.5
@@ -97,10 +106,12 @@ class EgressAddon:
# comes from the orchestrator's /resolve (PRD 0070); there is no static
# per-bottle routes file, SIGHUP reload, or single-tenant fallback.
_resolver: "PolicyResolver"
# Class default so __new__-built addons have it (real runs get a fresh
# per-instance dict in __init__; only http_connect mutates it, which the
# request-flow tests don't exercise).
# Class defaults so __new__-built addons have them (real runs get fresh
# per-instance collections in __init__; only http_connect mutates them,
# which request-flow tests don't exercise unless they call http_connect).
_conn_tokens: "dict[str, str]" = {}
_passthrough_conns: "set[str]" = set()
_inbound_scan_limit: int = DEFAULT_INBOUND_SCAN_LIMIT_BYTES
def __init__(self) -> None:
# Resolver-only: the gateway is always multi-tenant, resolving each
@@ -125,7 +136,12 @@ class EgressAddon:
# `Proxy-Authorization` (HTTPS tunnels don't repeat it on the bumped
# inner requests). Keyed by client_conn.id; cleared on disconnect.
self._conn_tokens: dict[str, str] = {}
# Connections whose route carries `inspect: false` — mitmproxy tunnels
# these without TLS interception so the client sees the server's real
# cert. Keyed by client_conn.id; cleared on disconnect.
self._passthrough_conns: set[str] = set()
self._token_allow_timeout = _token_allow_timeout_from_env(os.environ)
self._inbound_scan_limit = _inbound_scan_limit_from_env(os.environ)
@staticmethod
def _supervise_available(slug: str) -> bool:
@@ -305,25 +321,68 @@ class EgressAddon:
def http_connect(self, flow: http.HTTPFlow) -> None:
"""Capture the identity token from an HTTPS tunnel's CONNECT (the inner
bumped requests won't carry `Proxy-Authorization`), keyed by client
connection, and strip it so it never reaches upstream."""
connection, and strip it so it never reaches upstream.
For `inspect: false` routes, also resolve the policy here to make the
allowlist decision before the TLS handshake: the tunnel is either
blocked immediately or marked for passthrough in `_passthrough_conns`
so `tls_clienthello` skips interception."""
token = _token_from_proxy_auth(
flow.request.headers.get("Proxy-Authorization", ""))
flow.request.headers.pop("Proxy-Authorization", None)
conn = flow.client_conn
if conn is not None and getattr(conn, "id", ""):
self._conn_tokens[conn.id] = token
conn_id = getattr(conn, "id", "") if conn is not None else ""
if conn_id:
self._conn_tokens[conn_id] = token
# Resolve the policy here for all HTTPS connections and stash it so
# request() reuses it without a second orchestrator round-trip. For
# passthrough hosts we also make the allowlist decision now because
# inner requests never reach request() after the TLS bypass.
client_ip = conn.peername[0] if conn is not None and conn.peername else ""
config, slug, env = resolve_client_context(self._resolver, client_ip, token)
self._stash_flow_ctx(flow, config, slug, env)
host = flow.request.pretty_host
route = match_route(config.routes, host)
if route is not None and not route.inspect:
decision = decide(config.routes, host, "/", env, deny_reason=config.deny_reason)
if decision.action == "block":
flow.response = http.Response.make(
403,
decision.reason.encode("utf-8"),
{"Content-Type": "text/plain; charset=utf-8"},
)
return
if conn_id:
self._passthrough_conns.add(conn_id)
def tls_clienthello(self, client_hello: typing.Any) -> None:
"""Skip TLS interception for `inspect: false` routes so the client sees
the server's real certificate rather than the MITM CA's leaf."""
conn_id = getattr(client_hello.context.client, "id", "")
if conn_id in self._passthrough_conns:
client_hello.ignore_connection = True
def client_disconnected(self, client: typing.Any) -> None:
"""Drop the per-connection token when the client goes away."""
self._conn_tokens.pop(getattr(client, "id", ""), None)
"""Drop the per-connection token and passthrough flag when the client
goes away."""
conn_id = getattr(client, "id", "")
self._conn_tokens.pop(conn_id, None)
self._passthrough_conns.discard(conn_id)
async def request(self, flow: http.HTTPFlow) -> None:
request_path, _, query = flow.request.path.partition("?")
config, slug, env = self._resolve_flow(flow)
# Stash for the response / websocket hooks so their DLP scans reuse this
# bottle's resolved policy (one /resolve per flow — see _flow_ctx).
self._stash_flow_ctx(flow, config, slug, env)
# Reuse the context stashed by http_connect for HTTPS flows (one
# orchestrator round-trip per connection). Plain-HTTP flows have no
# prior CONNECT stash, so resolve now and stash for response/websocket.
meta = getattr(flow, "metadata", None)
if isinstance(meta, dict) and _FLOW_CTX_KEY in meta:
config, slug, env = meta[_FLOW_CTX_KEY]
self._request_token(flow) # strip identity headers; token already resolved
else:
config, slug, env = self._resolve_flow(flow)
self._stash_flow_ctx(flow, config, slug, env)
# Introspection ("_egress.local/allowlist") reports the calling bottle's
# own resolved routes — served after resolution so it reflects this
@@ -335,8 +394,10 @@ class EgressAddon:
# DLP outbound scan BEFORE stripping auth — catches tokens the
# agent tried to smuggle in any header, path, query param, or body.
# Hostname is included to catch DNS-tunnelling exfiltration attempts.
# `inspect: false` routes skip scanning entirely (TLS is also not
# intercepted for HTTPS, so this branch only fires for plain HTTP).
route = match_route(config.routes, flow.request.pretty_host)
if route is not None:
if route is not None and route.inspect:
if not await self._handle_outbound_dlp(flow, route, slug, env):
return
# The redact policy may have rewritten the request line; recompute
@@ -367,7 +428,10 @@ class EgressAddon:
# Strip agent-set Authorization after DLP scan so smuggled tokens
# are caught above; the route may inject gateway-owned auth below.
flow.request.headers.pop("authorization", None)
# Routes with preserve_auth=True pass the header through as-is so the
# agent's own credentials (e.g. registry bearer tokens) reach the upstream.
if route is None or not route.preserve_auth:
flow.request.headers.pop("authorization", None)
# Build headers mapping for match evaluation
req_headers = {k.lower(): v for k, v in flow.request.headers.items()}
@@ -603,7 +667,7 @@ class EgressAddon:
bottle's resolved config (`request()` stashed it — see `_flow_ctx`)."""
config, _slug, env = self._flow_ctx(flow)
route = match_route(config.routes, flow.request.pretty_host)
if route is None:
if route is None or not route.inspect:
return
if flow.response is None:
return
@@ -611,6 +675,14 @@ class EgressAddon:
self._log_response(flow, env)
resp_headers = {k.lower(): v for k, v in flow.response.headers.items()}
body = flow.response.get_text(strict=False) or ""
if self._inbound_scan_limit and len(body) > self._inbound_scan_limit:
sys.stderr.write(json.dumps({
"event": "egress_scan_truncated",
"host": flow.request.pretty_host,
"body_bytes": len(body),
"scan_limit_bytes": self._inbound_scan_limit,
}) + "\n")
body = body[:self._inbound_scan_limit]
scan_text = build_inbound_scan_text(resp_headers, body)
if not scan_text:
return
@@ -649,7 +721,7 @@ class EgressAddon:
return
config, slug, env = self._flow_ctx(flow)
route = match_route(config.routes, flow.request.pretty_host)
if route is None:
if route is None or not route.inspect:
return
message = flow.websocket.messages[-1] # type: ignore[union-attr]
content = message.content.decode("utf-8", errors="replace")
@@ -673,6 +745,25 @@ class EgressAddon:
sys.stderr.write(f"egress DLP warn: {result.reason}\n")
def _inbound_scan_limit_from_env(env: "os._Environ[str]") -> int:
"""Read EGRESS_INBOUND_SCAN_LIMIT_BYTES; fall back to the default on an
unset or invalid value. Returns 0 to disable the cap."""
raw = env.get("EGRESS_INBOUND_SCAN_LIMIT_BYTES", "").strip()
if not raw:
return DEFAULT_INBOUND_SCAN_LIMIT_BYTES
try:
value = int(raw)
except ValueError:
value = -1
if value < 0:
sys.stderr.write(
"egress: invalid EGRESS_INBOUND_SCAN_LIMIT_BYTES="
f"{raw!r}; using default {DEFAULT_INBOUND_SCAN_LIMIT_BYTES}\n"
)
return DEFAULT_INBOUND_SCAN_LIMIT_BYTES
return value
def _token_allow_timeout_from_env(env: "os._Environ[str]") -> float:
"""Read EGRESS_TOKEN_ALLOW_TIMEOUT_SECONDS; fall back to the default on an
unset or invalid value (a bad value should not wedge egress at boot)."""
+70 -20
View File
@@ -28,7 +28,7 @@ from .egress_dlp_config import (
ON_MATCH_SUPERVISE,
OUTBOUND_DETECTOR_NAMES,
OUTBOUND_ON_MATCH_VALUES,
parse_dlp_block,
parse_inspect_block,
)
@@ -78,6 +78,9 @@ class Route:
inbound_detectors: tuple[str, ...] | None = None
# "" means unset → DEFAULT_OUTBOUND_ON_MATCH. See OUTBOUND_ON_MATCH_VALUES.
outbound_on_match: str = ""
preserve_auth: bool = False
# False tunnels HTTPS without TLS interception or HTTP-level controls.
inspect: bool = True
LOG_OFF = 0 # no logging
@@ -258,10 +261,31 @@ def _parse_one(idx: int, raw: object) -> Route:
host: object = raw_dict.get("host")
if not isinstance(host, str) or not host:
raise ValueError(f"{label}: 'host' must be a non-empty string")
legacy_flat = "inspect" not in raw_dict
inspect_raw = raw_dict.get("inspect", {})
if inspect_raw is False:
inspect = False
settings: dict[str, object] = {}
elif isinstance(inspect_raw, dict):
inspect = True
settings = (
{k: v for k, v in raw_dict.items() if k != "host"}
if legacy_flat
else typing.cast(dict[str, object], inspect_raw)
)
legacy_dlp = settings.pop("dlp", None)
if isinstance(legacy_dlp, dict):
settings.update(typing.cast(dict[str, object], legacy_dlp))
elif legacy_dlp is not None:
raise ValueError(
f"{label} ({host}): legacy 'dlp' must be an object"
)
else:
raise ValueError(f"{label} ({host}): 'inspect' must be false or an object")
# matches
matches: tuple[MatchEntry, ...] = ()
matches_raw = raw_dict.get("matches")
matches_raw = settings.get("matches")
if matches_raw is not None:
if not isinstance(matches_raw, list):
raise ValueError(f"{label} ({host}): 'matches' must be a list")
@@ -271,8 +295,8 @@ def _parse_one(idx: int, raw: object) -> Route:
)
# auth (unchanged wire format)
auth_scheme: object = raw_dict.get("auth_scheme", "")
token_env: object = raw_dict.get("token_env", "")
auth_scheme: object = settings.get("auth_scheme", "")
token_env: object = settings.get("token_env", "")
if not isinstance(auth_scheme, str):
raise ValueError(f"{label} ({host}): 'auth_scheme' must be a string")
if not isinstance(token_env, str):
@@ -286,7 +310,7 @@ def _parse_one(idx: int, raw: object) -> Route:
# git-over-HTTPS policy
git_fetch = False
git_raw = raw_dict.get("git")
git_raw = settings.get("git")
if git_raw is not None:
if not isinstance(git_raw, dict):
raise ValueError(f"{label} ({host}): 'git' must be an object")
@@ -304,15 +328,30 @@ def _parse_one(idx: int, raw: object) -> Route:
)
# dlp detectors
outbound_detectors, inbound_detectors, outbound_on_match = parse_dlp_block(
idx, host, raw_dict,
outbound_detectors, inbound_detectors, outbound_on_match = parse_inspect_block(
idx, host, settings,
)
preserve_auth_raw = settings.get("preserve_auth", False)
if preserve_auth_raw is not True and preserve_auth_raw is not False:
raise ValueError(
f"{label} ({host}): 'preserve_auth' must be a boolean"
)
preserve_auth: bool = preserve_auth_raw
for k in settings:
if k not in (
"matches", "auth_scheme", "token_env", "git", "preserve_auth",
"outbound_detectors", "inbound_detectors", "outbound_on_match",
):
raise ValueError(
f"{label} ({host}): inspect has unknown key {k!r}"
)
for k in raw_dict:
if k not in ("host", "matches", "auth_scheme", "token_env", "dlp", "git"):
if not legacy_flat and k not in ("host", "inspect"):
raise ValueError(
f"{label} ({host}): unknown key {k!r}; accepted keys "
f"are 'host', 'matches', 'auth_scheme', 'token_env', 'dlp', 'git'"
f"are 'host' and 'inspect'"
)
return Route(
@@ -324,6 +363,8 @@ def _parse_one(idx: int, raw: object) -> Route:
outbound_detectors=outbound_detectors,
inbound_detectors=inbound_detectors,
outbound_on_match=outbound_on_match,
preserve_auth=preserve_auth,
inspect=inspect,
)
@@ -360,22 +401,27 @@ def route_to_yaml_dict(r: Route) -> dict[str, object]:
proposal without translation. Fields that are empty/default are
omitted so the agent doesn't copy irrelevant keys."""
d: dict[str, object] = {"host": r.host}
if not r.inspect:
d["inspect"] = False
return d
inspected: dict[str, object] = {}
if r.auth_scheme:
d["auth_scheme"] = r.auth_scheme
d["token_env"] = r.token_env
inspected["auth_scheme"] = r.auth_scheme
inspected["token_env"] = r.token_env
if r.matches:
d["matches"] = [_match_entry_to_dict(m) for m in r.matches]
inspected["matches"] = [_match_entry_to_dict(m) for m in r.matches]
if r.git_fetch:
d["git"] = {"fetch": True}
dlp: dict[str, object] = {}
inspected["git"] = {"fetch": True}
if r.outbound_detectors is not None:
dlp["outbound_detectors"] = list(r.outbound_detectors)
inspected["outbound_detectors"] = list(r.outbound_detectors)
if r.inbound_detectors is not None:
dlp["inbound_detectors"] = list(r.inbound_detectors)
inspected["inbound_detectors"] = list(r.inbound_detectors)
if r.outbound_on_match:
dlp["outbound_on_match"] = r.outbound_on_match
if dlp:
d["dlp"] = dlp
inspected["outbound_on_match"] = r.outbound_on_match
if r.preserve_auth:
inspected["preserve_auth"] = True
if inspected:
d["inspect"] = inspected
return d
@@ -747,6 +793,8 @@ def scan_outbound(
safe_tokens: typing.AbstractSet[str] | None = None,
crlf_text: str | None = None,
) -> ScanResult | None:
if not route.inspect:
return None
# Lazy import to avoid circular deps and keep dlp_detectors optional
# at import time (the gateway copies it flat alongside this file).
try:
@@ -844,6 +892,8 @@ def scan_inbound(
route: Route,
body: str | bytes,
) -> ScanResult | None:
if not route.inspect:
return None
try:
from dlp_detectors import scan_naive_injection # type: ignore[import-not-found]
except ImportError: # pragma: no cover - host-side path
@@ -871,7 +921,7 @@ __all__ = [
"DEFAULT_OUTBOUND_ON_MATCH",
"OUTBOUND_DETECTOR_NAMES",
"INBOUND_DETECTOR_NAMES",
"parse_dlp_block",
"parse_inspect_block",
"Config",
"Decision",
"HeaderMatch",
+10 -23
View File
@@ -1,6 +1,6 @@
"""DLP detector-config parsing for egress routes (PRD 0053, PRD 0062).
"""Inspection and DLP configuration parsing for egress routes.
A route's optional `dlp:` block names which outbound/inbound detectors run
A route's optional `inspect:` object names which outbound/inbound detectors run
and what the proxy does when an outbound detector matches a token
(`outbound_on_match`). This module owns parsing and validating that block,
kept apart from the request-time scan/decision flow in `egress_addon_core`
@@ -26,20 +26,14 @@ OUTBOUND_ON_MATCH_VALUES = (ON_MATCH_BLOCK, ON_MATCH_REDACT, ON_MATCH_SUPERVISE)
DEFAULT_OUTBOUND_ON_MATCH = ON_MATCH_SUPERVISE
def parse_dlp_block(
def parse_inspect_block(
idx: int,
host: str,
raw_dict: dict[str, object],
inspect: dict[str, object],
) -> tuple[tuple[str, ...] | None, tuple[str, ...] | None, str]:
"""Parse the optional `dlp` block on a route, returning
(outbound_detectors, inbound_detectors, outbound_on_match)."""
dlp_raw = raw_dict.get("dlp")
if dlp_raw is None:
return None, None, ""
"""Parse DLP settings from an inspected route."""
label = f"route[{idx}] ({host})"
if not isinstance(dlp_raw, dict):
raise ValueError(f"{label}: 'dlp' must be an object")
dlp = typing.cast(dict[str, object], dlp_raw)
dlp = inspect
def _parse_detector_field(
field: str,
@@ -52,18 +46,18 @@ def parse_dlp_block(
return ()
if not isinstance(val, list):
raise ValueError(
f"{label}: dlp.{field} must be false, a list, or omitted"
f"{label}: inspect.{field} must be false, a list, or omitted"
)
items = typing.cast(list[object], val)
names: list[str] = []
for j, item in enumerate(items):
if not isinstance(item, str):
raise ValueError(
f"{label}: dlp.{field}[{j}] must be a string"
f"{label}: inspect.{field}[{j}] must be a string"
)
if item not in valid_names:
raise ValueError(
f"{label}: dlp.{field}[{j}] {item!r} is not a valid "
f"{label}: inspect.{field}[{j}] {item!r} is not a valid "
f"detector name; valid names: {', '.join(sorted(valid_names))}"
)
names.append(item)
@@ -77,16 +71,9 @@ def parse_dlp_block(
if on_match_raw is not None:
if not isinstance(on_match_raw, str) or on_match_raw not in OUTBOUND_ON_MATCH_VALUES:
raise ValueError(
f"{label}: dlp.outbound_on_match must be one of "
f"{label}: inspect.outbound_on_match must be one of "
f"{', '.join(OUTBOUND_ON_MATCH_VALUES)} (got {on_match_raw!r})"
)
on_match = on_match_raw
for k in dlp:
if k not in ("outbound_detectors", "inbound_detectors", "outbound_on_match"):
raise ValueError(
f"{label}: dlp has unknown key {k!r}; accepted keys "
f"are 'outbound_detectors', 'inbound_detectors', "
f"'outbound_on_match'"
)
return outbound, inbound, on_match
+57 -36
View File
@@ -5,18 +5,11 @@ the configured daemons (egress, git-gate, supervise),
forwards SIGTERM/SIGINT to each child, and propagates per-daemon
stdout+stderr to the container log with a `[name] ` prefix.
Failure policy (interim): when a child dies unexpectedly, the
supervisor logs the death and leaves the surviving children
running. The gateway stays up; whatever the dead daemon served
will start failing, surfacing in the agent's own error path.
The supervisor itself exits only when (a) the operator sends
SIGTERM/SIGINT, or (b) every child has died.
Failure policy (eventual): on unexpected death, the supervisor
restarts the daemon and emits a notification to the supervise
daemon so the operator sees the event. That lands in a later
PR; the interim policy is "don't take the gateway down for one
sick daemon."
Failure policy: when a child dies unexpectedly, the supervisor
restarts it automatically and logs the restart. The gateway stays
up; a temporary loss of one daemon (e.g. egress OOM-killed) is
recovered without manual container recreation. The supervisor
itself exits only when the operator sends SIGTERM/SIGINT.
Daemon subset is env-driven via `BOT_BOTTLE_GATEWAY_DAEMONS=egress`
for callers that don't use git-gate or supervise. Default: all
@@ -61,6 +54,11 @@ class _DaemonSpec:
_EGRESS_ONLY_ENV_PREFIXES: tuple[str, ...] = ("EGRESS_TOKEN_",)
_READY_GATED_DAEMONS: tuple[str, ...] = ("git-gate", "git-http")
# Daemons that must be requested explicitly via BOT_BOTTLE_GATEWAY_DAEMONS
# and are NOT started in the default (env-var-unset) case. The orchestrator
# only runs in the combined infra container, never in a standalone gateway.
_OPT_IN_DAEMONS: frozenset[str] = frozenset({"orchestrator"})
def _env_for_daemon(name: str, base_env: dict[str, str]) -> dict[str, str]:
"""Egress sees the full bundle env. Everyone else gets a copy
@@ -75,7 +73,14 @@ def _env_for_daemon(name: str, base_env: dict[str, str]) -> dict[str, str]:
}
# The orchestrator is listed first so it starts before the gateway daemons,
# giving the control plane a head start to accept /resolve calls. The gateway
# daemons tolerate early /resolve failures and retry per-request.
_DAEMONS: tuple[_DaemonSpec, ...] = (
_DaemonSpec("orchestrator", (
"python3", "-m", "bot_bottle.orchestrator",
"--host", "0.0.0.0", "--port", "8099", "--broker", "stub",
)),
_DaemonSpec("egress", ("/bin/sh", "/app/egress-entrypoint.sh")),
_DaemonSpec("git-gate", ("/bin/sh", "/git-gate-entrypoint.sh")),
_DaemonSpec("git-http", ("python3", "-m", "bot_bottle.git_http_backend")),
@@ -103,18 +108,20 @@ def _selected_daemons(
env: dict[str, str],
all_daemons: Sequence[_DaemonSpec] | None = None,
) -> tuple[_DaemonSpec, ...]:
"""Filter the daemon set by the BOT_BOTTLE_GATEWAY_DAEMONS env
var. Unknown names in the list are ignored the caller is the
source of truth for which daemons are wired.
"""Filter the daemon set by the BOT_BOTTLE_GATEWAY_DAEMONS env var.
`all_daemons` defaults to `_DAEMONS` resolved at call time (not
at definition time), so tests can monkey-patch the module-level
`_DAEMONS` and have the new value take effect."""
When the var is unset/empty, return all non-opt-in daemons (the
standard gateway subset). Opt-in daemons (e.g. `orchestrator`) only
run when explicitly named they never start in a plain gateway
container that doesn't set the env var. Unknown names are ignored.
`all_daemons` defaults to `_DAEMONS` resolved at call time (not at
definition time), so tests can pass a custom list."""
if all_daemons is None:
all_daemons = _DAEMONS
raw = env.get("BOT_BOTTLE_GATEWAY_DAEMONS", "").strip()
if not raw:
return tuple(all_daemons)
return tuple(d for d in all_daemons if d.name not in _OPT_IN_DAEMONS)
wanted = {n.strip() for n in raw.split(",") if n.strip()}
return tuple(d for d in all_daemons if d.name in wanted)
@@ -136,7 +143,7 @@ def _pump(name: str, stream: IO[bytes]) -> None:
def _spawn(spec: _DaemonSpec) -> subprocess.Popen[bytes]:
env = _env_for_daemon(spec.name, dict(os.environ))
proc = subprocess.Popen(
proc = subprocess.Popen( # pylint: disable=consider-using-with
_argv_for_daemon(spec.name, spec.argv, env),
stdout=subprocess.PIPE,
stderr=subprocess.STDOUT,
@@ -183,6 +190,14 @@ class _Supervisor:
except ProcessLookupError:
pass
def _sigkill_all(self) -> None:
for _, p in self.procs:
if p.poll() is None:
try:
p.kill()
except ProcessLookupError:
pass
def request_restart(self, daemon_name: str) -> bool:
"""Queue a daemon restart for the main loop to process.
@@ -205,9 +220,10 @@ class _Supervisor:
"""One iteration of the watch loop. Returns True when every
child has exited and the supervisor can return.
A child dying unexpectedly is logged but does NOT initiate
shutdown see the module docstring's failure-policy
section. Shutdown is signal-driven only."""
A child dying unexpectedly is logged and restarted but does
NOT initiate shutdown see the module docstring's
failure-policy section. Shutdown is signal-driven only."""
restarted_children = bool(self._restart_requested)
self._drain_restart_requests()
for spec, p in self.procs:
@@ -216,14 +232,18 @@ class _Supervisor:
continue
self._logged_dead.add(spec.name)
if self.shutdown_at is None:
_log(
f"{spec.name} exited with code {rc}; leaving "
f"surviving daemons running (operator-visible "
f"via agent-side failure)"
)
_log(f"{spec.name} exited with code {rc}; scheduling restart")
self._restart_requested.add(spec.name)
else:
_log(f"{spec.name} exited with code {rc}")
# Restart deaths discovered above before checking whether all
# processes are done. Deferring this until the next tick would make a
# single-daemon supervisor return True and exit with the restart still
# queued.
restarted_children |= bool(self._restart_requested)
self._drain_restart_requests()
if self.shutdown_at is not None:
elapsed = time.monotonic() - self.shutdown_at
if elapsed > _GRACE_SECONDS:
@@ -235,14 +255,12 @@ class _Supervisor:
f"grace ({_GRACE_SECONDS:.0f}s) elapsed; SIGKILL on "
f"{', '.join(still_running)}"
)
for _, p in self.procs:
if p.poll() is None:
try:
p.kill()
except ProcessLookupError:
pass
self._sigkill_all()
done = all(p.poll() is not None for _, p in self.procs)
done = (
not restarted_children
and all(p.poll() is not None for _, p in self.procs)
)
if done:
for _, p in self.procs:
if p.stdout is not None:
@@ -361,7 +379,10 @@ def main(argv: Sequence[str] | None = None) -> int:
# --signal HUP <bundle>` after writing routes.yaml. The kernel
# delivers SIGHUP to PID 1 (this supervisor); forward it to
# mitmdump so it reloads its addon.
signal.signal(signal.SIGHUP, lambda *_: sup.forward_signal(signal.SIGHUP, "egress")) # type: ignore
signal.signal(
signal.SIGHUP,
lambda *_: sup.forward_signal(signal.SIGHUP, "egress"), # type: ignore[misc]
)
while not sup.tick():
time.sleep(_POLL_INTERVAL)
+42
View File
@@ -0,0 +1,42 @@
"""Shared helpers for cached-image quickstart stale checks."""
from __future__ import annotations
from datetime import datetime, timezone
from pathlib import Path
try:
from .config_store import ConfigStore
except ImportError:
from config_store import ConfigStore # type: ignore[import-not-found] # pylint: disable=import-error,no-name-in-module
class StaleImageError(Exception):
"""Raised when a cached image or artifact exceeds the configured staleness
threshold. Callers can catch this to prompt interactively; headless paths
let it propagate as a fatal error."""
def check_stale(label: str, created_at: datetime) -> None:
"""Raise StaleImageError if `created_at` is older than the configured
stale-warning threshold. Negative threshold disables the check."""
threshold_days = ConfigStore().cached_image_stale_warning_days()
if threshold_days < 0:
return
now = datetime.now(timezone.utc)
created = created_at.astimezone(timezone.utc)
age = now - created
if age.total_seconds() <= threshold_days * 86400:
return
raise StaleImageError(
f"cached {label} is {age.days} day(s) old; "
"quickstart does not verify it matches the current Dockerfile/context"
)
def check_stale_path(label: str, path: Path) -> None:
"""Raise StaleImageError if `path`'s mtime exceeds the staleness threshold."""
check_stale(label, datetime.fromtimestamp(path.stat().st_mtime, tz=timezone.utc))
__all__ = ["StaleImageError", "check_stale", "check_stale_path"]
+1
View File
@@ -20,6 +20,7 @@ Bottle schema (frontmatter):
egress: { routes: [ <egress-route>, ... ] }
# route keys: host, matches, auth, role, dlp
supervise: <bool> # optional (default true)
nested_containers: <bool> # optional (default false)
Agent schema (frontmatter):
bottle: <bottle-name> # required
+10 -4
View File
@@ -25,8 +25,9 @@ class ManifestAgentProvider:
header, and sets a placeholder CLAUDE_CODE_OAUTH_TOKEN in the agent
so the Claude Code CLI starts.
`forward_host_credentials` forwards the host Codex auth token into
the egress daemon (Codex only).
`forward_host_credentials` forwards the host provider auth token into
the egress sidecar (Codex and Claude). For Codex this reads
`~/.codex/auth.json`; for Claude it reads `~/.claude/.credentials.json`.
"""
template: str = "claude"
@@ -92,10 +93,15 @@ class ManifestAgentProvider:
f"is only supported for built-in templates "
f"({', '.join(sorted(PROVIDER_TEMPLATES))})"
)
if forward_host_credentials and template != "codex":
if forward_host_credentials and template not in {"codex", "claude"}:
raise ManifestError(
f"bottle '{bottle_name}' agent_provider.forward_host_credentials "
"is currently only supported for template 'codex'"
"is only supported for templates 'codex' and 'claude'"
)
if forward_host_credentials and auth_token:
raise ManifestError(
f"bottle '{bottle_name}' agent_provider.forward_host_credentials "
"and auth_token both set; use one or the other"
)
settings = _parse_provider_settings(bottle_name, template, d.get("settings"))
return cls(
+18
View File
@@ -44,6 +44,15 @@ class ManifestBottle:
# daemon that exposes egress MCP tools to the agent. Set
# `supervise: false` to skip the gateway.
supervise: bool = True
# Guest-local container engine (issue #392). Not a host-daemon grant:
# backends implement it inside the bottle or reject it. Gated because it
# costs image weight, a resident service, and relaxed guest device modes
# that the majority of bottles never need.
nested_containers: bool = False
# Source fields retained across extends/runtime composition. Boolean
# defaults otherwise erase the distinction between "omitted" and an
# explicitly declared value (especially False).
declared_fields: frozenset[str] = frozenset()
@classmethod
def from_dict(cls, name: str, raw: object) -> "ManifestBottle":
@@ -123,7 +132,16 @@ class ManifestBottle:
f"(was {type(supervise_raw).__name__})"
)
nested_raw = d.get("nested_containers", False)
if not isinstance(nested_raw, bool):
raise ManifestError(
f"bottle '{name}' nested_containers must be a boolean "
f"(was {type(nested_raw).__name__})"
)
return cls(
env=env, agent_provider=agent_provider, git=git,
git_user=git_user, egress=egress, supervise=supervise_raw,
nested_containers=nested_raw,
declared_fields=frozenset(d),
)
+51 -23
View File
@@ -71,6 +71,8 @@ class ManifestEgressRoute:
OutboundDetectors: tuple[str, ...] | None = None
InboundDetectors: tuple[str, ...] | None = None
OutboundOnMatch: str = ""
PreserveAuth: bool = False
Inspect: bool = True
@classmethod
def from_dict(cls, bottle_name: str, idx: int, raw: object) -> "ManifestEgressRoute":
@@ -80,9 +82,17 @@ class ManifestEgressRoute:
if not isinstance(host, str) or not host:
raise ManifestError(f"{label} missing required string field 'host'")
inspect_raw = d.get("inspect", {})
if inspect_raw is False:
inspect = False
inspect_d: dict[str, object] = {}
else:
inspect = True
inspect_d = as_json_object(inspect_raw, f"{label} inspect")
# --- matches ---
matches: tuple[ManifestMatchEntry, ...] = ()
matches_raw = d.get("matches")
matches_raw = inspect_d.get("matches")
if matches_raw is not None:
if not isinstance(matches_raw, list):
raise ManifestError(
@@ -100,9 +110,9 @@ class ManifestEgressRoute:
# --- auth ---
auth_scheme = ""
token_ref = ""
if "auth" in d:
auth_raw = d.get("auth")
auth_d = as_json_object(auth_raw, f"{label} auth")
if "auth" in inspect_d:
auth_raw = inspect_d.get("auth")
auth_d = as_json_object(auth_raw, f"{label} inspect.auth")
if not auth_d:
raise ManifestError(
f"{label} auth is empty ({{}}); omit the 'auth' key "
@@ -162,19 +172,19 @@ class ManifestEgressRoute:
f"the 'role' field is reserved for future use"
)
# --- dlp ---
# --- DLP settings (inspection-only) ---
outbound_detectors: tuple[str, ...] | None = None
inbound_detectors: tuple[str, ...] | None = None
outbound_on_match = ""
if "dlp" in d:
outbound_detectors, inbound_detectors, outbound_on_match = _parse_dlp_block(
label, d.get("dlp"),
if inspect:
outbound_detectors, inbound_detectors, outbound_on_match = _parse_inspect_block(
label, inspect_d,
)
# --- git-over-HTTPS policy ---
git_fetch = False
if "git" in d:
git_d = as_json_object(d.get("git"), f"{label} git")
if "git" in inspect_d:
git_d = as_json_object(inspect_d.get("git"), f"{label} inspect.git")
raw_fetch = git_d.get("fetch", False)
if isinstance(raw_fetch, bool):
git_fetch = raw_fetch
@@ -190,11 +200,33 @@ class ManifestEgressRoute:
f"only 'fetch' is accepted"
)
# --- preserve_auth ---
preserve_auth = False
if "preserve_auth" in inspect_d:
raw_preserve_auth = inspect_d.get("preserve_auth")
if not isinstance(raw_preserve_auth, bool):
raise ManifestError(
f"{label} preserve_auth must be a boolean "
f"(was {type(raw_preserve_auth).__name__})"
)
preserve_auth = raw_preserve_auth
for k in inspect_d:
if k not in (
"matches", "auth", "git", "preserve_auth",
"outbound_detectors", "inbound_detectors", "outbound_on_match",
):
raise ManifestError(
f"{label} inspect has unknown key {k!r}; accepted keys are "
f"'matches', 'auth', 'git', 'preserve_auth', "
f"'outbound_detectors', 'inbound_detectors', "
f"'outbound_on_match'"
)
for k in d:
if k not in ("host", "matches", "auth", "role", "dlp", "git"):
if k not in ("host", "role", "inspect"):
raise ManifestError(
f"{label} has unknown key {k!r}; accepted keys are "
f"'host', 'matches', 'auth', 'role', 'dlp', 'git'"
f"'host', 'role', and 'inspect'"
)
return cls(
@@ -207,6 +239,8 @@ class ManifestEgressRoute:
OutboundDetectors=outbound_detectors,
InboundDetectors=inbound_detectors,
OutboundOnMatch=outbound_on_match,
PreserveAuth=preserve_auth,
Inspect=inspect,
)
@@ -326,12 +360,13 @@ def _parse_header_match(
return ManifestHeaderMatch(Name=name, Value=value, Type=htype)
def _parse_dlp_block(
def _parse_inspect_block(
route_label: str,
raw: object,
inspect: dict[str, object],
) -> tuple[tuple[str, ...] | None, tuple[str, ...] | None, str]:
label = f"{route_label} dlp"
d = as_json_object(raw, label)
"""Parse DLP settings from an inspected route."""
label = f"{route_label} inspect"
d = inspect
def _parse_field(
field: str,
@@ -374,13 +409,6 @@ def _parse_dlp_block(
)
on_match = on_match_raw
for k in d:
if k not in ("outbound_detectors", "inbound_detectors", "outbound_on_match"):
raise ManifestError(
f"{label} has unknown key {k!r}; accepted keys are "
f"'outbound_detectors', 'inbound_detectors', "
f"'outbound_on_match'"
)
return outbound, inbound, on_match
+34 -6
View File
@@ -8,6 +8,17 @@ from .manifest_git import ManifestGitUser, parse_git_gate_config
from .manifest_util import ManifestError, as_json_object
def _overlay_declared_bool(
base: ManifestBottle,
override: ManifestBottle,
field: str,
) -> bool:
"""Overlay a defaulted boolean only when override declared it."""
value = getattr(override if field in override.declared_fields else base, field)
assert isinstance(value, bool)
return value
def merge_bottles_runtime(bottles: "list[ManifestBottle]") -> "ManifestBottle":
"""Merge an ordered list of pre-resolved ManifestBottle objects.
@@ -15,7 +26,13 @@ def merge_bottles_runtime(bottles: "list[ManifestBottle]") -> "ManifestBottle":
the same field-merge rules as the file-based extends machinery:
env: dict merge, later wins; git_user: per-field overlay, later
wins on non-empty; git (repos): union by name, later wins; egress
routes: concatenate; agent_provider, supervise: later replaces.
routes: concatenate; agent_provider, supervise, nested_containers:
later replaces (presence-aware).
Defaulted booleans use presence-aware replacement: if the later bottle
was loaded from a source that explicitly declared the key, its value
wins (so an explicit `false` can override an earlier `true`). If the
later bottle never mentioned the key, the earlier value is preserved.
"""
if not bottles:
raise ValueError("merge_bottles_runtime requires at least one bottle")
@@ -40,7 +57,7 @@ def _merge_two_bottles_runtime(base: "ManifestBottle", override: "ManifestBottle
n for n in override_repos_by_name if n not in base_repos_by_name
]
merged_git = tuple(
override_repos_by_name.get(n, base_repos_by_name[n])
override_repos_by_name[n] if n in override_repos_by_name else base_repos_by_name[n]
for n in merged_repos_names
)
@@ -53,7 +70,11 @@ def _merge_two_bottles_runtime(base: "ManifestBottle", override: "ManifestBottle
git=merged_git,
git_user=merged_git_user,
egress=merged_egress,
supervise=override.supervise,
supervise=_overlay_declared_bool(base, override, "supervise"),
nested_containers=_overlay_declared_bool(
base, override, "nested_containers"
),
declared_fields=base.declared_fields | override.declared_fields,
)
@@ -205,7 +226,11 @@ def _fold_two_bottles(
git=merged_git,
git_user=merged_git_user,
egress=merged_egress,
supervise=later.supervise,
supervise=_overlay_declared_bool(earlier, later, "supervise"),
nested_containers=_overlay_declared_bool(
earlier, later, "nested_containers"
),
declared_fields=earlier.declared_fields | later.declared_fields,
), merged_repos_raw
@@ -263,8 +288,9 @@ def _merge_bottles(
if "agent_provider" in child_raw
else parent.agent_provider
)
merged_supervise = (
child.supervise if "supervise" in child_raw else parent.supervise
merged_supervise = _overlay_declared_bool(parent, child, "supervise")
merged_nested_containers = _overlay_declared_bool(
parent, child, "nested_containers"
)
validate_egress_routes(name, merged_egress.routes)
@@ -275,6 +301,8 @@ def _merge_bottles(
git_user=merged_git_user,
egress=merged_egress,
supervise=merged_supervise,
nested_containers=merged_nested_containers,
declared_fields=parent.declared_fields | child.declared_fields,
)
+4 -1
View File
@@ -16,7 +16,10 @@ _FILENAME_RX = re.compile(r"^[a-z][a-z0-9-]*$")
# sets dies with a "did you mean" pointer: typos should not silently
# ghost into an empty config.
BOTTLE_KEYS = frozenset(
{"env", "extends", "agent_provider", "git-gate", "egress", "supervise"}
{
"env", "extends", "agent_provider", "git-gate", "egress",
"supervise", "nested_containers",
}
)
AGENT_KEYS_REQUIRED: frozenset[str] = frozenset()
AGENT_KEYS_OPTIONAL = frozenset({"bottle", "skills", "git-gate"})
+28 -3
View File
@@ -41,10 +41,13 @@ class OrchestratorClientError(RuntimeError):
@dataclass(frozen=True)
class RegisteredBottle:
"""What `POST /bottles` returns: the minted bottle id and the per-bottle
identity token the agent presents for app-layer attribution."""
identity token the agent presents for app-layer attribution. `env_var_secret`
is set by the caller (not from the server response) and carries the
encryption key so it can be injected into the agent container's env."""
bottle_id: str
identity_token: str
env_var_secret: str = ""
class OrchestratorClient:
@@ -120,17 +123,21 @@ class OrchestratorClient:
metadata: str = "",
policy: str = "",
tokens: dict[str, str] | None = None,
env_var_secret: str = "",
) -> RegisteredBottle:
"""Register a bottle and broker its launch (`POST /bottles`). `tokens`
are the per-bottle egress auth values (env_name -> value) the
orchestrator holds in memory for the gateway to inject. Returns the
minted id + identity token."""
orchestrator holds in memory for the gateway to inject. When
*env_var_secret* is provided, the orchestrator also encrypts the token
values and stores them in ``bottled_agent_secrets`` for restart
recovery. Returns the minted id + identity token."""
payload = self._ok("POST", "/bottles", {
"source_ip": source_ip,
"image_ref": image_ref,
"metadata": metadata,
"policy": policy,
"tokens": tokens or {},
"env_var_secret": env_var_secret,
})
bottle_id = payload.get("bottle_id")
token = payload.get("identity_token")
@@ -138,6 +145,24 @@ class OrchestratorClient:
raise OrchestratorClientError("register: response missing bottle_id/identity_token")
return RegisteredBottle(bottle_id=bottle_id, identity_token=token)
def reprovision_gateway(self, bottle_id: str, env_var_secret: str) -> bool:
"""Re-inject a bottle's egress tokens from its ENV_VAR_SECRET
(`POST /bottles/<id>/reprovision_gateway`). Returns True when the
orchestrator successfully decrypted and restored the tokens, False
when it had no stored secrets for this bottle (404)."""
status, _ = self._request(
"POST",
f"/bottles/{bottle_id}/reprovision_gateway",
{"env_var_secret": env_var_secret},
)
if status == 404:
return False
if not 200 <= status < 300:
raise OrchestratorClientError(
f"reprovision_gateway {bottle_id}: HTTP {status}"
)
return True
def teardown_bottle(self, bottle_id: str) -> bool:
"""Tear a bottle down (`DELETE /bottles/<id>`). False if the
orchestrator didn't know it (404) — idempotent for cleanup paths."""
+107
View File
@@ -0,0 +1,107 @@
"""Per-host orchestrator configuration store (settings in bot-bottle.db).
Co-tenants the shared `bot-bottle.db` via the `DbStore` framework. Settings
are readable by the host launch path directly (no HTTP round-trip to the
orchestrator), so they take effect even before the orchestrator is reachable.
"""
from __future__ import annotations
import os
import sqlite3
from pathlib import Path
from ..db_store import DbStore
from ..migrations import TableMigrations
from ..paths import host_db_path
TEARDOWN_TIMEOUT_ENV = "BOT_BOTTLE_ORCHESTRATOR_TEARDOWN_TIMEOUT_SECONDS"
DEFAULT_TEARDOWN_TIMEOUT_SECONDS = 30.0
_MIGRATIONS = TableMigrations(
"orchestrator_config",
[
"""
CREATE TABLE IF NOT EXISTS orchestrator_config (
id INTEGER PRIMARY KEY CHECK (id = 1),
teardown_timeout_seconds REAL
)
""",
],
)
class OrchestratorConfigStore(DbStore):
"""Orchestrator settings in the shared host DB."""
def __init__(self, db_path: Path | None = None) -> None:
super().__init__(db_path or host_db_path(), _MIGRATIONS)
def _connect(self) -> sqlite3.Connection:
conn = super()._connect()
conn.execute("PRAGMA busy_timeout=5000")
return conn
def get_teardown_timeout_seconds(self) -> float | None:
"""Return the configured teardown timeout, or None if not set."""
try:
with self._connection() as conn:
row = conn.execute(
"SELECT teardown_timeout_seconds FROM orchestrator_config WHERE id = 1"
).fetchone()
except sqlite3.OperationalError:
return None
return row["teardown_timeout_seconds"] if row else None
def set_teardown_timeout_seconds(self, value: float) -> None:
"""Persist the teardown timeout."""
with self._connection() as conn:
conn.execute(
"INSERT OR REPLACE INTO orchestrator_config"
" (id, teardown_timeout_seconds) VALUES (1, ?)",
(value,),
)
self._chmod()
def delete_teardown_timeout_seconds(self) -> bool:
"""Clear the stored teardown timeout. Returns True if a value existed."""
with self._connection() as conn:
cur = conn.execute(
"UPDATE orchestrator_config SET teardown_timeout_seconds = NULL"
" WHERE id = 1 AND teardown_timeout_seconds IS NOT NULL"
)
return cur.rowcount > 0
def resolve_teardown_timeout(db_path: Path | None = None) -> float:
"""Return the teardown timeout to use, in priority order:
1. ``BOT_BOTTLE_ORCHESTRATOR_TEARDOWN_TIMEOUT_SECONDS`` env var
2. ``teardown_timeout_seconds`` in the orchestrator config DB
3. ``DEFAULT_TEARDOWN_TIMEOUT_SECONDS`` (30 s)
"""
raw = os.environ.get(TEARDOWN_TIMEOUT_ENV, "").strip()
if raw:
try:
value = float(raw)
if value > 0:
return value
except ValueError:
pass
store = OrchestratorConfigStore(db_path)
if not store.is_migrated():
store.migrate()
db_value = store.get_teardown_timeout_seconds()
if db_value is not None and db_value > 0:
return db_value
return DEFAULT_TEARDOWN_TIMEOUT_SECONDS
__all__ = [
"OrchestratorConfigStore",
"resolve_teardown_timeout",
"TEARDOWN_TIMEOUT_ENV",
"DEFAULT_TEARDOWN_TIMEOUT_SECONDS",
]
+24 -1
View File
@@ -9,9 +9,13 @@ vsock / unix-socket portability caveats):
GET /bottles -> 200 {"bottles": [ <redacted record>, ...]}
POST /bottles -> 201 {"bottle_id","identity_token"} (launch)
body: {"source_ip", ["image_ref"],
["metadata"], ["policy"]}
["metadata"], ["policy"],
["tokens"], ["env_var_secret"]}
PUT /bottles/<bottle_id>/policy -> 200 {"updated": true} | 404 (live reload)
body: {"policy"}
POST /bottles/<bottle_id>/reprovision_gateway
-> 200 {"reprovisioned": true} | 404
body: {"env_var_secret"}
DELETE /bottles/<bottle_id> -> 200 {"torn_down": true} | 404 (teardown)
POST /reconcile -> 200 {"reaped": [bottle_id, ...]}
body: {"live_source_ips": [...],
@@ -116,12 +120,14 @@ def dispatch( # pylint: disable=too-many-return-statements,too-many-branches
tokens = {
k: v for k, v in raw_tokens.items() if isinstance(k, str) and isinstance(v, str)
} if isinstance(raw_tokens, dict) else {}
env_var_secret = data.get("env_var_secret", "")
rec = orch.launch_bottle(
source_ip,
image_ref=image_ref if isinstance(image_ref, str) else "",
metadata=metadata if isinstance(metadata, str) else "",
policy=policy if isinstance(policy, str) else "",
tokens=tokens,
env_var_secret=env_var_secret if isinstance(env_var_secret, str) else "",
)
return 201, {"bottle_id": rec.bottle_id, "identity_token": rec.identity_token}
@@ -138,6 +144,23 @@ def dispatch( # pylint: disable=too-many-return-statements,too-many-branches
return 200, {"updated": True}
return 404, {"error": "no such bottle"}
if (
method == "POST"
and route.startswith("/bottles/")
and route.endswith("/reprovision_gateway")
):
bottle_id = route[len("/bottles/") : -len("/reprovision_gateway")]
try:
data = _parse_json_object(body)
except ValueError as e:
return 400, {"error": f"invalid JSON: {e}"}
env_var_secret = data.get("env_var_secret")
if not isinstance(env_var_secret, str) or not env_var_secret:
return 400, {"error": "env_var_secret (string) is required"}
if orch.reprovision_from_secret(bottle_id, env_var_secret):
return 200, {"reprovisioned": True}
return 404, {"error": "no stored secrets for this bottle"}
if method == "DELETE" and route.startswith("/bottles/"):
bottle_id = route[len("/bottles/"):]
if orch.teardown_bottle(bottle_id):
+41 -10
View File
@@ -27,6 +27,7 @@ from ..paths import (
CONTROL_PLANE_TOKEN_ENV,
host_control_plane_token,
host_db_path,
host_gateway_ca_dir,
)
from ..supervise import DB_PATH_IN_CONTAINER
@@ -48,14 +49,23 @@ GATEWAY_LABEL = "bot-bottle-orch-gateway=1"
# the source IP the gateway attributes by is the address on this network.
GATEWAY_NETWORK = "bot-bottle-gateway"
# mitmproxy's CA dir in the bundle. A persistent named volume here keeps the
# gateway's self-generated CA STABLE across container recreation — every agent
# installs this one CA to trust the shared gateway's TLS interception, so it
# must not rotate when the gateway restarts.
# mitmproxy's CA dir in the bundle. The host's gateway-CA dir (see
# `host_gateway_ca_dir`) is bind-mounted here so the gateway's self-generated
# CA stays STABLE across container recreation — every agent installs this one
# CA to trust the shared gateway's TLS interception, so it must not rotate when
# the gateway restarts. A host bind-mount rather than a named volume: a named
# volume is silently wiped by `docker volume prune`, minting a fresh CA that
# breaks every running bottle (issue #450).
MITMPROXY_HOME = "/home/mitmproxy/.mitmproxy"
GATEWAY_CA_VOLUME = "bot-bottle-gateway-mitmproxy"
GATEWAY_CA_CERT = f"{MITMPROXY_HOME}/mitmproxy-ca-cert.pem"
# The CA material mitmproxy writes into its confdir. mitmproxy reuses these on
# startup when present and generates them only on first run, so persisting them
# is what makes the CA stable; deleting them (see `rotate_gateway_ca`) forces a
# fresh CA on the next start. `mitmproxy-ca.pem` (cert + private key) is the
# signing identity; the rest are derived encodings agents/clients consume.
GATEWAY_CA_GLOB = "mitmproxy-ca*"
# The gateway data-plane image + its Dockerfile. Kept as a local constant
# rather than imported from the backend layer, which would drag
# the whole backend layer into the lean orchestrator (see #359); unify when
@@ -73,6 +83,26 @@ def _host_db_dir() -> str:
return str(db_dir)
def rotate_gateway_ca(ca_dir: Path | None = None) -> list[Path]:
"""Delete the persisted mitmproxy CA so the next gateway start mints a
fresh one the explicit, deliberate CA-rollover path (issue #450).
Persistence keeps the CA stable across restarts precisely because mitmproxy
reuses the on-disk CA; rotation is therefore just removing that material.
Returns the files removed (empty when there was no CA yet); idempotent.
This only clears the on-disk CA. It does NOT stop the running gateway (whose
mitmproxy still holds the old CA in memory) or re-provision agents the
caller recreates the gateway to mint the new CA and re-attaches bottles.
`rotate-ca` on the orchestrator CLI wires those steps together."""
ca_dir = ca_dir if ca_dir is not None else host_gateway_ca_dir()
removed: list[Path] = []
for path in sorted(ca_dir.glob(GATEWAY_CA_GLOB)):
path.unlink()
removed.append(path)
return removed
class GatewayError(Exception):
"""The shared gateway failed to build/start/stop (non-zero `docker` exit)."""
@@ -221,9 +251,10 @@ class DockerGateway(Gateway):
"--name", self.name,
"--label", GATEWAY_LABEL,
"--network", self.network,
# Persist the self-generated CA so it survives restarts (agents
# trust it) — see GATEWAY_CA_VOLUME.
"--volume", f"{GATEWAY_CA_VOLUME}:{MITMPROXY_HOME}",
# Persist the self-generated CA on the host so it survives both
# container recreation AND docker volume pruning (agents trust it)
# — see host_gateway_ca_dir / issue #450.
"--volume", f"{host_gateway_ca_dir()}:{MITMPROXY_HOME}",
# Share the one host DB: the supervise daemon queues proposals
# into the same file the orchestrator (and the operator, over
# HTTP) reads — no second, disconnected DB in the container.
@@ -272,7 +303,7 @@ class DockerGateway(Gateway):
__all__ = [
"Gateway", "DockerGateway", "GatewayError",
"Gateway", "DockerGateway", "GatewayError", "rotate_gateway_ca",
"GATEWAY_NAME", "GATEWAY_LABEL", "GATEWAY_IMAGE", "GATEWAY_NETWORK",
"GATEWAY_CA_VOLUME", "GATEWAY_CA_CERT",
"GATEWAY_CA_CERT", "GATEWAY_CA_GLOB",
]
+164 -169
View File
@@ -1,17 +1,15 @@
"""Orchestrator + gateway lifecycle (PRD 0070, docker slice).
Runs the orchestrator control plane **as a container** on the shared gateway
network, alongside the gateway container. This is the PRD's "virtualize the
orchestrator": container↔container between the gateway and the orchestrator
avoids the host firewall (which drops containerhost traffic), and the gateway
reaches the control plane by container name over docker DNS. The host CLI
reaches it via a published loopback port.
Runs both the orchestrator control plane and the gateway data plane inside
a single `bot-bottle-infra` container on the shared gateway network
matching the structure already used by the macOS and Firecracker backends.
`gateway_init` is PID 1 and supervises both; the infra container is an
idempotent per-host singleton.
The orchestrator runs with the **register-only broker** the *backend*
launches agent containers (compose), so the orchestrator needs no docker
socket. That keeps this control-plane container unprivileged; the host manages
both containers. `ensure_running` is an idempotent singleton (fixed container
names + the published port).
The combined container replaces the prior two-container split
(bot-bottle-orchestrator + bot-bottle-orch-gateway). The host CLI reaches
the control plane via a published loopback port; gateway daemons reach it
over 127.0.0.1 (same container).
"""
from __future__ import annotations
@@ -25,50 +23,75 @@ from pathlib import Path
from .. import log
from ..docker_cmd import run_docker
from ..paths import CONTROL_PLANE_TOKEN_ENV, bot_bottle_root, host_control_plane_token
from .gateway import GATEWAY_IMAGE, GATEWAY_NAME, GATEWAY_NETWORK, DockerGateway, GatewayError
from ..paths import (
CONTROL_PLANE_TOKEN_ENV,
bot_bottle_root,
host_control_plane_token,
host_gateway_ca_dir,
)
from ..supervise import DB_PATH_IN_CONTAINER
from .gateway import (
GATEWAY_DOCKERFILE,
GATEWAY_IMAGE,
GATEWAY_NETWORK,
GatewayError,
MITMPROXY_HOME,
_host_db_dir,
)
DEFAULT_PORT = 8099
ORCHESTRATOR_NAME = "bot-bottle-orchestrator"
ORCHESTRATOR_LABEL = "bot-bottle-orchestrator=1"
# The control-plane's own runtime image — lean (python + the stdlib-only
# `bot_bottle` package, bind-mounted at run time), distinct from the heavy
# gateway data-plane image it used to borrow (#384). Env override for
# operators pinning a published build.
DEFAULT_STARTUP_TIMEOUT_SECONDS = 45.0
INFRA_NAME = "bot-bottle-infra"
INFRA_LABEL = "bot-bottle-infra=1"
# The combined infra image: gateway data plane + orchestrator content.
# Built from Dockerfile.infra (FROM gateway + COPY --from orchestrator).
INFRA_IMAGE = os.environ.get("BOT_BOTTLE_INFRA_IMAGE", "bot-bottle-infra:latest")
INFRA_DOCKERFILE = "Dockerfile.infra"
# Baked as a container label so `ensure_running` can detect whether the
# running container is executing the current bind-mounted source.
INFRA_SOURCE_HASH_LABEL = "bot-bottle-infra-source-hash"
# Orchestrator image: the single canonical definition of the control-plane
# content (lean: python:3.12-slim + bot_bottle package, no mitmproxy/git).
# Used as a build intermediate: `Dockerfile.infra` COPY --from this image.
ORCHESTRATOR_IMAGE = os.environ.get(
"BOT_BOTTLE_ORCHESTRATOR_IMAGE", "bot-bottle-orchestrator:latest"
)
ORCHESTRATOR_DOCKERFILE = "Dockerfile.orchestrator"
# Baked onto the container as a label so `ensure_running` can tell whether the
# running process is executing the *current* bind-mounted source — see
# `source_hash`.
ORCHESTRATOR_SOURCE_HASH_LABEL = "bot-bottle-orchestrator-source-hash"
# The repo root is bind-mounted into the control-plane container so
# `python -m bot_bottle.orchestrator` resolves the package (the orchestrator
# is stdlib-only, so the lean orchestrator image's python is enough).
_REPO_ROOT = Path(__file__).resolve().parents[2]
_APP_DIR = "/app"
# The gateway daemons + orchestrator the infra container runs.
# BOT_BOTTLE_GATEWAY_DAEMONS listing `orchestrator` opts it in to
# gateway_init's supervise tree (see gateway_init._OPT_IN_DAEMONS).
_INFRA_DAEMONS = "egress,git-http,supervise,orchestrator"
# The bind-mount path for the live control-plane source inside the
# container. Separate from /app so the gateway's baked scripts
# (egress_addon.py, egress-entrypoint.sh) are not overlaid.
_SRC_IN_CONTAINER = "/bot-bottle-src"
# Bot-bottle host-root bind-mount inside the container (DB + state).
_ROOT_IN_CONTAINER = "/bot-bottle-root"
# The supervise daemon writes proposals into the host DB directory.
_SUPERVISE_DB_DIR_IN_CONTAINER = os.path.dirname(DB_PATH_IN_CONTAINER)
_HEALTH_POLL_SECONDS = 0.25
DEFAULT_STARTUP_TIMEOUT_SECONDS = 45.0
_HEALTH_REQUEST_TIMEOUT_SECONDS = 1.0
_REPO_ROOT = Path(__file__).resolve().parents[2]
class OrchestratorStartError(RuntimeError):
"""The orchestrator container did not become healthy within the timeout."""
"""The infra container did not become healthy within the timeout."""
def source_hash(repo_root: Path) -> str:
"""Content hash of the orchestrator's bind-mounted Python source (the
`bot_bottle` package the control-plane process imports). This only
changes when the code that would actually run inside the container
changes `ensure_running` recreates the container on a mismatch and
otherwise leaves a healthy one alone, so a bottle launch that isn't
accompanied by a code change doesn't restart the process and drop every
*other* active bottle's in-memory egress tokens (`Orchestrator._tokens`
in `service.py`, never persisted to disk by design)."""
`bot_bottle` package the control-plane process imports). Changes only
when the code that would actually run changes `ensure_running`
recreates the container on a mismatch so a code change takes effect,
but leaves a healthy up-to-date container alone to preserve in-memory
egress tokens."""
h = hashlib.sha256()
for path in sorted((repo_root / "bot_bottle").rglob("*.py")):
h.update(str(path.relative_to(repo_root)).encode())
@@ -77,57 +100,37 @@ def source_hash(repo_root: Path) -> str:
class OrchestratorService:
"""Manages the orchestrator control-plane container + the shared gateway.
"""Manages the single per-host infra container (control plane + gateway).
Callers only need `ensure_running()` + `url`.
`orchestrator_name` / `orchestrator_label` let backends run independent
orchestrators on the same host without name collisions (e.g. the
Firecracker backend uses `bot-bottle-fc-orchestrator` alongside the Docker
backend's `bot-bottle-orchestrator`); `gateway_name` gives the paired
gateway container the same treatment (e.g. isolated integration tests
that can't share the production `GATEWAY_NAME` singleton). Subclass and
override `_gateway()` for anything `_gateway_image`/`gateway_name` can't
express (a genuinely backend-specific gateway variant)."""
`infra_name` / `infra_label` let backends run independent infra containers
on the same host without name collisions (e.g. isolated integration tests
that can't share the production INFRA_NAME singleton)."""
def __init__(
self,
*,
port: int = DEFAULT_PORT,
network: str = GATEWAY_NETWORK,
image: str = ORCHESTRATOR_IMAGE,
gateway_image: str = GATEWAY_IMAGE,
gateway_name: str = GATEWAY_NAME,
image: str = INFRA_IMAGE,
repo_root: Path = _REPO_ROOT,
host_root: Path | None = None,
orchestrator_name: str = ORCHESTRATOR_NAME,
orchestrator_label: str = ORCHESTRATOR_LABEL,
infra_name: str = INFRA_NAME,
infra_label: str = INFRA_LABEL,
) -> None:
self.port = port
self.network = network
# Two distinct images (#384): `image` is the lean control-plane
# runtime this container runs; `_gateway_image` is the heavy egress /
# git-gate / supervise data plane the gateway container runs. They
# were one conflated image before the split.
self.image = image
self._gateway_image = gateway_image
self._gateway_name = gateway_name
self._repo_root = repo_root
self._host_root = host_root or bot_bottle_root()
self._orchestrator_name = orchestrator_name
self._orchestrator_label = orchestrator_label
self._infra_name = infra_name
self._infra_label = infra_label
@property
def url(self) -> str:
"""Host-side control-plane URL (published loopback port)."""
return f"http://127.0.0.1:{self.port}"
@property
def internal_url(self) -> str:
"""Control-plane URL as the gateway container reaches it — by name over
docker DNS on the shared network. This is the gateway's
BOT_BOTTLE_ORCHESTRATOR_URL."""
return f"http://{self._orchestrator_name}:{self.port}"
def is_healthy(self, *, timeout: float = _HEALTH_REQUEST_TIMEOUT_SECONDS) -> bool:
try:
with urllib.request.urlopen(f"{self.url}/health", timeout=timeout) as resp:
@@ -139,139 +142,131 @@ class OrchestratorService:
proc = run_docker(["docker", "ps", "--filter", f"name=^/{name}$", "--format", "{{.Names}}"])
return name in proc.stdout.split()
def _run_orchestrator_container(self, current_hash: str) -> None:
"""Start the control-plane container (idempotent: clears a stale
fixed-name container first). Register-only broker no docker socket.
Labels the container with `current_hash` so a later `ensure_running`
can detect a real code change (see `source_hash`)."""
run_docker(["docker", "rm", "--force", self._orchestrator_name])
proc = run_docker([
"docker", "run", "--detach",
"--name", self._orchestrator_name,
"--label", self._orchestrator_label,
"--label", f"{ORCHESTRATOR_SOURCE_HASH_LABEL}={current_hash}",
"--network", self.network,
# Host CLI reaches the control plane here; bound to loopback so it
# is not exposed on the host's external interfaces. NOTE: the
# container is still on `self.network` (the shared gateway network),
# so agents can reach it by container IP — which is exactly why the
# control plane requires the secret below rather than trusting the
# network boundary.
"--publish", f"127.0.0.1:{self.port}:{self.port}",
"--volume", f"{self._repo_root}:{_APP_DIR}:ro",
"--workdir", _APP_DIR,
# Persist the registry DB on the host (sole-owner: only the
# orchestrator opens bot-bottle.db).
"--volume", f"{self._host_root}:{_ROOT_IN_CONTAINER}",
"--env", f"BOT_BOTTLE_ROOT={_ROOT_IN_CONTAINER}",
# The control-plane secret it requires on every route but /health.
# Bare `--env NAME` → docker inherits the value from the run env
# below, so the secret never lands on argv / `docker inspect`.
"--env", CONTROL_PLANE_TOKEN_ENV,
"--entrypoint", "python3",
self.image,
"-m", "bot_bottle.orchestrator",
"--host", "0.0.0.0", "--port", str(self.port), "--broker", "stub",
], env={**os.environ, CONTROL_PLANE_TOKEN_ENV: host_control_plane_token()})
if proc.returncode != 0:
raise OrchestratorStartError(
f"orchestrator container failed to start: {proc.stderr.strip()}"
)
def _gateway(self) -> DockerGateway:
return DockerGateway(
self._gateway_image,
name=self._gateway_name,
network=self.network,
orchestrator_url=self.internal_url,
)
def _ensure_orchestrator_image(self) -> None:
"""Build the lean control-plane image from `Dockerfile.orchestrator`
when it's missing (#384). Cheap — a `FROM python:*-slim` base with no
deps to install, so the layer cache makes rebuilds a no-op. Unlike the
gateway image this is build-if-missing, not build-every-time: the
control plane bind-mounts its source, so a code change is caught by the
source-hash recreate (below), not by an image rebuild."""
if run_docker(["docker", "image", "inspect", self.image]).returncode == 0:
return
argv = ["docker", "build", "-t", self.image,
"-f", str(self._repo_root / ORCHESTRATOR_DOCKERFILE),
str(self._repo_root)]
if os.environ.get("BOT_BOTTLE_NO_CACHE"):
argv.insert(2, "--no-cache")
proc = run_docker(argv)
if proc.returncode != 0:
raise GatewayError(
f"orchestrator image build failed: {proc.stderr.strip()}"
)
def _orchestrator_source_current(self, current_hash: str) -> bool:
"""True iff the running orchestrator container was created from the
*current* bind-mounted source. Mirrors `DockerGateway`'s
image-staleness check, but by content hash rather than image id since
the orchestrator runs bind-mounted source, not a built image."""
if not self._container_running(self._orchestrator_name):
def _infra_source_current(self, current_hash: str) -> bool:
"""True iff the running infra container was started from the current
bind-mounted source. Mirrors the macOS backend's `_source_current`."""
if not self._container_running(self._infra_name):
return False
proc = run_docker([
"docker", "inspect", "--format",
"{{ index .Config.Labels \"" + ORCHESTRATOR_SOURCE_HASH_LABEL + "\" }}",
self._orchestrator_name,
"{{ index .Config.Labels \"" + INFRA_SOURCE_HASH_LABEL + "\" }}",
self._infra_name,
])
if proc.returncode != 0:
return True # can't compare -> don't churn a working container
return True # can't compare don't churn a working container
return proc.stdout.strip() == current_hash
def _ensure_network(self) -> None:
if run_docker(["docker", "network", "inspect", self.network]).returncode == 0:
return
proc = run_docker(["docker", "network", "create", self.network])
if proc.returncode != 0 and "already exists" not in proc.stderr:
raise GatewayError(
f"gateway network {self.network} failed to create: {proc.stderr.strip()}"
)
def _build_images(self) -> None:
"""Build the gateway base, the orchestrator intermediate, then the
infra image. All are cache-aware: a no-op when nothing changed."""
for tag, dockerfile in (
(GATEWAY_IMAGE, GATEWAY_DOCKERFILE),
(ORCHESTRATOR_IMAGE, ORCHESTRATOR_DOCKERFILE),
(self.image, INFRA_DOCKERFILE),
):
argv = ["docker", "build", "-t", tag,
"-f", str(self._repo_root / dockerfile),
str(self._repo_root)]
if os.environ.get("BOT_BOTTLE_NO_CACHE"):
argv.insert(2, "--no-cache")
proc = run_docker(argv)
if proc.returncode != 0:
raise GatewayError(f"{dockerfile} build failed: {proc.stderr.strip()}")
def _run_infra_container(self, current_hash: str) -> None:
"""Start the combined infra container (idempotent: clears a stale
fixed-name container first). Labels the container with `current_hash`
so a later `ensure_running` can detect a real code change."""
self._ensure_network()
run_docker(["docker", "rm", "--force", self._infra_name])
proc = run_docker([
"docker", "run", "--detach",
"--name", self._infra_name,
"--label", self._infra_label,
"--label", f"{INFRA_SOURCE_HASH_LABEL}={current_hash}",
"--network", self.network,
# Host CLI reaches the control plane here (loopback only).
# gateway_init always starts the orchestrator on DEFAULT_PORT (8099)
# inside the container; self.port is the host-side published port.
"--publish", f"127.0.0.1:{self.port}:{DEFAULT_PORT}",
# Persist the mitmproxy CA on the host so it survives container
# recreation AND docker volume pruning (issue #450): every agent
# trusts this one CA, so a fresh one would break all running bottles.
"--volume", f"{host_gateway_ca_dir()}:{MITMPROXY_HOME}",
# Shared supervise DB (same file the operator reads over HTTP).
"--volume", f"{_host_db_dir()}:{_SUPERVISE_DB_DIR_IN_CONTAINER}",
"--env", f"SUPERVISE_DB_PATH={DB_PATH_IN_CONTAINER}",
# Live control-plane source, mounted to a path that does not
# overlay the gateway's baked /app scripts.
"--volume", f"{self._repo_root}:{_SRC_IN_CONTAINER}:ro",
# PYTHONPATH lets the orchestrator (and other Python daemons)
# import the live source ahead of the installed package.
"--env", f"PYTHONPATH={_SRC_IN_CONTAINER}",
# Orchestrator registry DB on the host (sole writer: control plane).
"--volume", f"{self._host_root}:{_ROOT_IN_CONTAINER}",
"--env", f"BOT_BOTTLE_ROOT={_ROOT_IN_CONTAINER}",
# Control-plane secret: required by the orchestrator (to enforce)
# and by the gateway daemons (to present on /resolve calls).
"--env", CONTROL_PLANE_TOKEN_ENV,
# Gateway daemons reach the orchestrator over loopback at its
# fixed internal port (DEFAULT_PORT), independent of self.port.
"--env", f"BOT_BOTTLE_ORCHESTRATOR_URL=http://127.0.0.1:{DEFAULT_PORT}",
# Opt the orchestrator into gateway_init's supervise tree.
"--env", f"BOT_BOTTLE_GATEWAY_DAEMONS={_INFRA_DAEMONS}",
self.image,
], env={**os.environ, CONTROL_PLANE_TOKEN_ENV: host_control_plane_token()})
if proc.returncode != 0:
raise OrchestratorStartError(
f"infra container failed to start: {proc.stderr.strip()}"
)
def ensure_running(
self, *, startup_timeout: float = DEFAULT_STARTUP_TIMEOUT_SECONDS,
) -> str:
"""Ensure the control plane + shared gateway are up; return the host
control-plane URL. Idempotent a healthy control plane running
current code and a running gateway are left untouched. Raises
`OrchestratorStartError` on timeout."""
gateway = self._gateway()
gateway.ensure_built() # rebuild the bundle image on a source change
gateway.ensure_running() # creates the shared network + (re)starts gateway
"""Ensure the infra container (control plane + gateway) is up; return
the host control-plane URL. Idempotent a healthy container on current
source is left untouched. Raises `OrchestratorStartError` on timeout."""
self._build_images()
# Recreate the orchestrator container only when its bind-mounted
# source has actually changed since it started — its Python process
# loaded that code at startup and won't reload, so a stale container
# would keep running OLD control-plane code. Recreating on *every*
# launch (the prior behaviour) would drop every other active
# bottle's in-memory egress tokens each time a new bottle starts,
# since the orchestrator process holds them only in memory (#381).
current_hash = source_hash(self._repo_root)
if self.is_healthy() and self._orchestrator_source_current(current_hash):
if self.is_healthy() and self._infra_source_current(current_hash):
return self.url
self._ensure_orchestrator_image()
log.info(
"starting orchestrator container",
context={"name": self._orchestrator_name},
)
self._run_orchestrator_container(current_hash)
log.info("starting infra container", context={"name": self._infra_name})
self._run_infra_container(current_hash)
deadline = time.monotonic() + startup_timeout
while time.monotonic() < deadline:
if self.is_healthy():
log.info("orchestrator healthy", context={"url": self.url})
log.info("infra container healthy", context={"url": self.url})
return self.url
time.sleep(_HEALTH_POLL_SECONDS)
raise OrchestratorStartError(
f"orchestrator at {self.url} did not become healthy within {startup_timeout:g}s"
f"infra container at {self.url} did not become healthy within {startup_timeout:g}s"
)
def stop(self) -> None:
"""Remove the orchestrator + gateway containers (idempotent)."""
run_docker(["docker", "rm", "--force", self._orchestrator_name])
self._gateway().stop()
"""Remove the infra container (idempotent)."""
run_docker(["docker", "rm", "--force", self._infra_name])
__all__ = [
"OrchestratorService",
"OrchestratorStartError",
"ORCHESTRATOR_NAME",
"INFRA_NAME",
"INFRA_IMAGE",
"INFRA_SOURCE_HASH_LABEL",
"ORCHESTRATOR_IMAGE",
"DEFAULT_PORT",
"DEFAULT_STARTUP_TIMEOUT_SECONDS",
"source_hash",
]
+67
View File
@@ -113,6 +113,22 @@ _MIGRATIONS = TableMigrations(
# egress allowlist / routes / git config selected by source IP. The
# multi-tenant gateway resolves it per request via `attribute`.
"ALTER TABLE orchestrator_bottles ADD COLUMN policy TEXT NOT NULL DEFAULT ''",
# v4 — per-bottle encrypted egress secrets (PRD prd-new-secret-provider).
# One row per env-var: key (env-var name) is plaintext for auditing;
# value is the encrypted token string. The encryption key (ENV_VAR_SECRET)
# lives only in the agent's environment — a row alone cannot recover the
# credential.
"""
CREATE TABLE IF NOT EXISTS bottled_agent_secrets (
bottled_agent_id TEXT NOT NULL,
key TEXT NOT NULL,
value TEXT NOT NULL,
type TEXT NOT NULL DEFAULT 'injected_env_var'
)
""",
# v5 — index for fast per-bottle lookups and bulk DELETE on teardown.
"CREATE INDEX IF NOT EXISTS idx_bottled_agent_secrets_id "
"ON bottled_agent_secrets (bottled_agent_id, type)",
],
)
@@ -326,6 +342,57 @@ class RegistryStore(DbStore):
return None
return rec
# --- encrypted egress secret store ------------------------------------
def store_agent_secrets(
self,
bottle_id: str,
encrypted_values: dict[str, str],
secret_type: str = "injected_env_var",
) -> None:
"""Replace all stored secrets for *bottle_id* with *encrypted_values*
(env-var name encrypted ciphertext). Deletes then re-inserts so a
re-registration is always consistent with the current token set."""
with self._connection() as conn:
conn.execute(
"DELETE FROM bottled_agent_secrets "
"WHERE bottled_agent_id = ? AND type = ?",
(bottle_id, secret_type),
)
conn.executemany(
"INSERT INTO bottled_agent_secrets "
"(bottled_agent_id, key, value, type) VALUES (?, ?, ?, ?)",
[(bottle_id, k, v, secret_type) for k, v in encrypted_values.items()],
)
self._chmod()
def get_agent_secrets(
self,
bottle_id: str,
secret_type: str = "injected_env_var",
) -> dict[str, str]:
"""Return {env_var_name: encrypted_value} for *bottle_id*, or {} if none."""
with self._connection() as conn:
rows = conn.execute(
"SELECT key, value FROM bottled_agent_secrets "
"WHERE bottled_agent_id = ? AND type = ?",
(bottle_id, secret_type),
).fetchall()
return {row[0]: row[1] for row in rows}
def delete_agent_secrets(
self,
bottle_id: str,
secret_type: str = "injected_env_var",
) -> None:
"""Remove all stored secrets for *bottle_id* (e.g. on teardown)."""
with self._connection() as conn:
conn.execute(
"DELETE FROM bottled_agent_secrets "
"WHERE bottled_agent_id = ? AND type = ?",
(bottle_id, secret_type),
)
__all__ = [
"BottleRecord",
+35
View File
@@ -0,0 +1,35 @@
"""Shared host-side join for backend-discovered bottle encryption keys."""
from __future__ import annotations
from .client import OrchestratorClient, OrchestratorClientError
def reprovision_bottles(
client: OrchestratorClient,
secrets_by_source_ip: dict[str, str],
) -> int:
"""Restore tokens for registered bottles whose backend exposes a key.
Backends own discovery because containers and microVMs have different
enumeration primitives. This helper owns the shared registry join and
intentionally tolerates one bad/missing key without blocking a launch.
"""
restored = 0
for bottle in client.list_bottles():
bottle_id = bottle.get("bottle_id")
source_ip = bottle.get("source_ip")
if not isinstance(bottle_id, str) or not isinstance(source_ip, str):
continue
secret = secrets_by_source_ip.get(source_ip, "").strip()
if not secret:
continue
try:
if client.reprovision_gateway(bottle_id, secret):
restored += 1
except OrchestratorClientError:
continue
return restored
__all__ = ["reprovision_bottles"]
+62
View File
@@ -0,0 +1,62 @@
"""Rotate the shared gateway's mitmproxy CA (issue #450).
python -m bot_bottle.orchestrator.rotate_ca
A deliberate CA rollover has two halves: drop the *persisted* CA so a fresh one
is minted, and drop the *running* gateway so its mitmproxy (which holds the old
CA in memory) is replaced. This one-shot command does both:
1. Delete the persisted CA under the host gateway-CA dir the next gateway
start generates a new one (mitmproxy reuses an existing CA, generates only
when absent).
2. Force-remove the infra / standalone-gateway containers so the stale
in-memory CA is gone; the next bottle launch's idempotent `ensure_running`
brings the gateway back up and mints the fresh CA.
It does NOT re-provision the new CA into already-running bottles those must be
re-attached so they install the new trust anchor. Rotation is thus an explicit,
operator-driven action with a brief egress interruption, not an automatic one.
"""
from __future__ import annotations
import sys
from pathlib import Path
from ..docker_cmd import run_docker
from ..paths import host_gateway_ca_dir
from .gateway import GATEWAY_NAME, rotate_gateway_ca
from .lifecycle import INFRA_NAME
# The containers whose mitmproxy would still be serving the old CA from memory:
# the consolidated infra container and the standalone per-host gateway.
_GATEWAY_CONTAINERS = (INFRA_NAME, GATEWAY_NAME)
def _out(msg: str) -> None:
sys.stdout.write(f"rotate-ca: {msg}\n")
def main(argv: list[str] | None = None) -> int:
del argv # no flags — a single deliberate action
ca_dir: Path = host_gateway_ca_dir()
removed = rotate_gateway_ca(ca_dir)
if removed:
_out(f"removed {len(removed)} CA file(s) from {ca_dir}")
else:
_out(f"no persisted CA under {ca_dir}; a fresh one is minted on next start")
# Drop any running gateway so its in-memory (now-stale) CA is replaced on
# the next launch. `rm --force` on an absent name is a tolerated no-op.
for name in _GATEWAY_CONTAINERS:
proc = run_docker(["docker", "rm", "--force", name])
if proc.returncode == 0 and proc.stdout.strip():
_out(f"removed running container {name}")
_out("done — the next bottle launch remints the CA; re-attach bottles to "
"install the new trust anchor")
return 0
if __name__ == "__main__":
raise SystemExit(main())
+94
View File
@@ -0,0 +1,94 @@
"""Symmetric encryption for per-bottle egress secrets (PRD prd-new-secret-provider).
Each agent receives a random ENV_VAR_SECRET at startup passed as an env var,
never logged or persisted. The host uses this key to encrypt each egress auth
token value before writing it to the bottled_agent_secrets table; the DB rows
(ciphertext, plaintext env-var name) without the key are insufficient to
recover the credentials.
On orchestrator restart the in-memory token map is lost. The host-side
reattachment path reads ENV_VAR_SECRET from the running agent container via
``docker exec printenv ENV_VAR_SECRET`` and posts it to
``POST /bottles/<id>/reprovision_gateway``; the orchestrator decrypts the
stored rows and re-populates ``_tokens``.
Encryption scheme: HMAC-SHA256 used as a PRF in CTR mode (stdlib-only,
no external deps). Each value is encrypted independently. The output blob is
``nonce (16 bytes) || ciphertext`` encoded as URL-safe base64 (no padding).
keystream_block_i = HMAC-SHA256(key, nonce || i.to_bytes(4, "big"))
ciphertext_i = plaintext_i XOR keystream_block_i[:len(plaintext_i)]
"""
from __future__ import annotations
import base64
import hashlib
import hmac
import secrets
_KEY_BYTES = 32 # 256-bit key from ENV_VAR_SECRET
_NONCE_BYTES = 16 # 128-bit random nonce per encrypt call
_BLOCK = 32 # HMAC-SHA256 output width == one keystream block
# Env-var name the agent container receives at startup.
ENV_VAR_SECRET_NAME = "ENV_VAR_SECRET"
def new_env_var_secret() -> str:
"""Generate a fresh ENV_VAR_SECRET: 32 random bytes as URL-safe base64."""
return base64.urlsafe_b64encode(secrets.token_bytes(_KEY_BYTES)).rstrip(b"=").decode()
def _b64dec(s: str) -> bytes:
return base64.urlsafe_b64decode(s + "=" * (-len(s) % 4))
def _keystream(key: bytes, nonce: bytes, block_index: int) -> bytes:
return hmac.new(
key, nonce + block_index.to_bytes(4, "big"), hashlib.sha256
).digest()
def encrypt_value(secret_b64: str, plaintext: str) -> str:
"""Encrypt a single string value with *secret_b64* (the ENV_VAR_SECRET).
Returns a URL-safe base64 blob ``nonce || ciphertext`` suitable for
the ``bottled_agent_secrets.value`` column."""
key = _b64dec(secret_b64)
pt = plaintext.encode()
nonce = secrets.token_bytes(_NONCE_BYTES)
ct = bytearray()
for i in range(0, len(pt), _BLOCK):
chunk = pt[i : i + _BLOCK]
ks = _keystream(key, nonce, i)[: len(chunk)]
ct.extend(p ^ k for p, k in zip(chunk, ks))
return base64.urlsafe_b64encode(nonce + bytes(ct)).rstrip(b"=").decode()
def decrypt_value(secret_b64: str, blob_b64: str) -> str:
"""Decrypt a blob produced by :func:`encrypt_value`.
Returns the original plaintext string. Raises ``ValueError`` for malformed
input or a key mismatch (wrong key produces garbage, not an error, unless
the plaintext is non-UTF-8 treat all such failures as wrong key)."""
key = _b64dec(secret_b64)
try:
blob = _b64dec(blob_b64)
except Exception as exc:
raise ValueError(f"invalid ciphertext blob: {exc}") from exc
if len(blob) < _NONCE_BYTES:
raise ValueError("ciphertext blob too short")
nonce, ciphertext = blob[:_NONCE_BYTES], blob[_NONCE_BYTES:]
pt = bytearray()
for i in range(0, len(ciphertext), _BLOCK):
chunk = ciphertext[i : i + _BLOCK]
ks = _keystream(key, nonce, i)[: len(chunk)]
pt.extend(c ^ k for c, k in zip(chunk, ks))
try:
return bytes(pt).decode()
except UnicodeDecodeError as exc:
raise ValueError(f"decryption produced non-UTF-8 output (wrong key?): {exc}") from exc
__all__ = ["ENV_VAR_SECRET_NAME", "new_env_var_secret", "encrypt_value", "decrypt_value"]
+30 -1
View File
@@ -87,13 +87,22 @@ class Orchestrator:
metadata: str = "",
policy: str = "",
tokens: dict[str, str] | None = None,
env_var_secret: str = "",
) -> BottleRecord:
"""Register a bottle (with its gateway policy + in-memory egress auth
tokens) and broker its launch. Rolls the registry entry back if the
launch doesn't take, so a failure leaves no orphan."""
launch doesn't take, so a failure leaves no orphan.
When *env_var_secret* is provided alongside *tokens*, the token values
are also encrypted and written to ``bottled_agent_secrets`` so they can
survive an orchestrator restart (see ``reprovision_from_secret``)."""
rec = self.registry.register(source_ip, metadata=metadata, policy=policy)
if tokens:
self._tokens[rec.bottle_id] = dict(tokens)
if env_var_secret:
from .secret_store import encrypt_value
encrypted = {k: encrypt_value(env_var_secret, v) for k, v in tokens.items()}
self.registry.store_agent_secrets(rec.bottle_id, encrypted)
req = LaunchRequest(
op="launch",
bottle_id=rec.bottle_id,
@@ -284,6 +293,26 @@ class Orchestrator:
))
return True, ""
# --- secret reprovision -----------------------------------------------
def reprovision_from_secret(self, bottle_id: str, env_var_secret: str) -> bool:
"""Re-inject a bottle's egress tokens from its ENV_VAR_SECRET.
Reads the encrypted rows from ``bottled_agent_secrets``, decrypts each
value with *env_var_secret*, and restores ``_tokens[bottle_id]``.
Returns True on success, False when no stored secrets exist for this
bottle or decryption fails (wrong key / corrupt data)."""
from .secret_store import decrypt_value
encrypted = self.registry.get_agent_secrets(bottle_id)
if not encrypted:
return False
try:
self._tokens[bottle_id] = {k: decrypt_value(env_var_secret, v)
for k, v in encrypted.items()}
except ValueError:
return False
return True
# --- consolidated gateway ----------------------------------------------
def ensure_gateway(self) -> None:
+35 -3
View File
@@ -1,8 +1,14 @@
"""Foundational filesystem paths for bot-bottle.
`bot_bottle_root()` is the app data root state, queue, audit logs,
git-gate keys, and the shared DB all live under it. It defaults to
`~/.bot-bottle` and is overridable with the **`BOT_BOTTLE_ROOT`** env var.
`bot_bottle_root()` is the app data root per-bottle state, git-gate
keys, the gateway CA, and the shared SQLite DB all live under it. It
defaults to `~/.bot-bottle` and is overridable with the
**`BOT_BOTTLE_ROOT`** env var.
Note that the supervise queue and the audit log are *tables in the shared
DB*, not directories under the root see `queue_store.py` / `audit_store.py`.
The root held a `queue/` directory before the SQLite migration (PRD 0067);
nothing writes there now.
The env override is the single knob for redirecting the root: the test
suite points it at a throwaway dir instead of monkey-patching the function
@@ -33,6 +39,13 @@ HOST_DB_FILENAME = "bot-bottle.db"
CONTROL_PLANE_TOKEN_FILENAME = "control-plane-token"
CONTROL_PLANE_TOKEN_ENV = "BOT_BOTTLE_CONTROL_PLANE_TOKEN"
# The host directory holding the gateway's persistent mitmproxy CA. Bind-mounted
# into the infra/gateway container at mitmproxy's confdir so the self-generated
# CA survives container recreation — every agent installs this one CA to trust
# the shared gateway's TLS interception, so it must not rotate on restart. See
# host_gateway_ca_dir() for why this is a host bind-mount, not a named volume.
GATEWAY_CA_DIRNAME = "gateway-ca"
def bot_bottle_root() -> Path:
"""The app data root — `$BOT_BOTTLE_ROOT` if set, else `~/.bot-bottle`."""
@@ -59,6 +72,23 @@ def host_db_dir() -> Path:
return db_dir
def host_gateway_ca_dir() -> Path:
"""The directory holding the gateway's persistent mitmproxy CA, created if
missing. Backends bind-mount this into the infra/gateway container at
mitmproxy's confdir so the CA persists across container recreation.
A host bind-mount under the app-data root deliberately NOT a Docker
named volume. A named volume survives `docker rm` but is silently wiped by
`docker volume prune` / `docker system prune --volumes` during routine host
maintenance; the gateway then mints a fresh CA that every already-running
bottle distrusts, failing the TLS handshake even after it reconnects to the
moved gateway (issue #450). A path under the root docker never prunes it,
and it stays directly inspectable + rotatable from the host."""
ca_dir = bot_bottle_root() / GATEWAY_CA_DIRNAME
ca_dir.mkdir(parents=True, exist_ok=True)
return ca_dir
def host_control_plane_token() -> str:
"""The per-host control-plane secret, minted (256-bit, url-safe) and
persisted 0600 on first use, then reused.
@@ -94,8 +124,10 @@ __all__ = [
"HOST_DB_FILENAME",
"CONTROL_PLANE_TOKEN_FILENAME",
"CONTROL_PLANE_TOKEN_ENV",
"GATEWAY_CA_DIRNAME",
"bot_bottle_root",
"host_db_path",
"host_db_dir",
"host_gateway_ca_dir",
"host_control_plane_token",
]
+4
View File
@@ -6,9 +6,11 @@ from pathlib import Path
try:
from .audit_store import AuditStore
from .config_store import ConfigStore
from .queue_store import QueueStore
except ImportError:
from audit_store import AuditStore # type: ignore[import-not-found] # pylint: disable=import-error,no-name-in-module
from config_store import ConfigStore # type: ignore[import-not-found] # pylint: disable=import-error,no-name-in-module
from queue_store import QueueStore # type: ignore[import-not-found] # pylint: disable=import-error,no-name-in-module
_instance: StoreManager | None = None
@@ -47,11 +49,13 @@ class StoreManager:
return (
QueueStore("", self.db_path).is_migrated()
and AuditStore(self.db_path).is_migrated()
and ConfigStore(self.db_path).is_migrated()
)
def migrate(self) -> None:
QueueStore("", self.db_path).migrate()
AuditStore(self.db_path).migrate()
ConfigStore(self.db_path).migrate()
__all__ = ["StoreManager"]
+10 -9
View File
@@ -168,15 +168,16 @@ _ROUTES_YAML_DESCRIPTION = (
"Full proposed /etc/egress/routes.yaml content. "
"Each route entry accepts these keys:\n"
" host: <hostname> (required)\n"
" auth_scheme: Bearer|token (must pair with token_env)\n"
" token_env: <ENV_VAR_NAME> (must pair with auth_scheme)\n"
" matches: (optional list of match entries)\n"
" - paths: [{type: prefix|exact|regex, value: /...}]\n"
" methods: [GET, POST, ...]\n"
" headers: [{name: X-Hdr, value: val, type: exact|regex}]\n"
" git: (optional; omit to block git clone/fetch)\n"
" fetch: true\n"
" dlp: (optional DLP scanner overrides)\n"
" inspect: false (opaque whole-host TLS tunnel; no HTTP controls)\n"
" inspect: (omit for inspected defaults)\n"
" auth_scheme: Bearer|token (must pair with token_env)\n"
" token_env: <ENV_VAR_NAME> (must pair with auth_scheme)\n"
" matches: (optional list of match entries)\n"
" - paths: [{type: prefix|exact|regex, value: /...}]\n"
" methods: [GET, POST, ...]\n"
" headers: [{name: X-Hdr, value: val, type: exact|regex}]\n"
" git: (optional; omit to block git clone/fetch)\n"
" fetch: true\n"
" outbound_detectors: [token_patterns, known_secrets]\n"
" inbound_detectors: [naive_injection_detection]\n"
" outbound_on_match: block|redact|supervise (default supervise)\n"
+13 -3
View File
@@ -1,13 +1,23 @@
# CI
The test workflow lives at [`.gitea/workflows/test.yml`](../.gitea/workflows/test.yml).
It runs `tests/run_tests.py` (full suite — unit + integration) on:
It runs the unit suite plus one integration job per backend
(`integration-docker`, `integration-firecracker`) on:
- every push to a branch with an open pull request, and
- every push to `main`.
Integration tests need Docker on the runner; they skip cleanly via
`tests/_docker.skip_unless_docker` when no daemon is reachable.
Each integration job selects its backend via `BOT_BOTTLE_BACKEND` and
runs a **preflight** (`./cli.py backend status --backend=<name>`) that
prints a clear per-check readiness summary and fails the job when the
backend is missing — so absent infrastructure is visible at the job level
rather than hidden among per-test `unittest.skip` lines. The skip guards in
[`tests/_backend.py`](../tests/_backend.py) gate on the same readiness
check (`bot_bottle.backend.has_backend`): backend-agnostic tests use
`skip_unless_selected_backend_available()` and run through whichever
backend is selected (checking, e.g., Linux + `/dev/kvm` for Firecracker
rather than unrelated Docker availability); Docker-implementation tests use
`skip_unless_backend("docker")` and no-op under a non-Docker run.
A small subset of integration tests skip when running specifically
under Gitea Actions (`GITEA_ACTIONS=true`), because `act_runner` runs
+38
View File
@@ -312,6 +312,44 @@ reaches over the RPC rather than a shared mount into the VM. WAL on the
shared DB is therefore a deliberate, tested future change — not enabled ad
hoc. `sqlite3` itself is stdlib, so "the host needs SQLite" is a non-cost.
### Gateway CA: host-resident, like the DB
The shared gateway bumps TLS with a self-generated mitmproxy CA, and **every
bottle installs that CA** into its trust store to accept the bumped leaves. So
the CA is durable per-host state with the same rule as the DB: it must outlive
any single gateway container, or a restart mints a fresh CA that every
already-running bottle distrusts — the TLS handshake then fails even after the
bottle re-resolves and reconnects to the moved gateway (issue #450, a
re-attachment blocker distinct from #443/#445).
The CA lives on the **host filesystem** at `bot_bottle_root()/gateway-ca`
(`host_gateway_ca_dir()`), bind-mounted into the container at mitmproxy's
confdir. This is deliberately a host bind-mount, **not a container-runtime
named volume**: a named volume survives ordinary container removal but can be
silently wiped by Docker's or Apple Container's volume-prune commands during
routine host maintenance, which is exactly how the ephemeral-CA symptom shows
up in practice. A path under the app-data root is not managed or pruned by the
container runtime, and stays directly inspectable and rotatable from the host.
mitmproxy reuses an existing CA and generates one only on first run, so the
bind-mount alone gives
"adopt-existing, generate-on-first-run" for free.
The macOS backend uses the same host-resident CA directory and bind-mounts it
into the consolidated Apple infra container. Its `bot-bottle-mac-db` named
volume remains container-only because that prevents incoherent cross-kernel
SQLite locking, but the CA is deliberately not stored there: Apple Container
also has a `container volume prune` operation, and the named volume is
temporarily unreferenced while the infra container is recreated. Keeping the
CA on the host makes both ordinary recreation and volume pruning safe.
**Deliberate rollover** is the explicit inverse: `rotate_gateway_ca()` removes
the persisted CA material so the next start remints it, and the
`python -m bot_bottle.orchestrator.rotate_ca` one-shot wires that together with
dropping the running gateway container (whose mitmproxy still holds the old CA
in memory). Rotation does not auto-re-provision the new CA into running bottles
— those re-attach to install the new anchor — so it is an operator action with
a brief egress interruption, never an implicit one.
## Sequencing
Jump straight to the **virtualized** end state (not a host-daemon stepping
+110
View File
@@ -0,0 +1,110 @@
# PRD prd-new: CI artifact-based coverage and local Firecracker candidate flow
- **Status:** Active
- **Author:** Claude
- **Created:** 2026-07-21
- **Issue:** #446
## Summary
Restructure the CI test pipeline to run each test suite exactly once, upload
small `.coverage.*` artifacts, and combine them in a lightweight aggregation
job. Move the infra build onto the KVM runner so the ~194 MB rootfs never
crosses the network for PRs. On main-branch pushes, publish the byte-identical
rootfs that was tested.
## Motivation
The prior pipeline had two redundant costs:
1. **Duplicate artifact transfers.** `build-infra` (ubuntu-latest) built and
uploaded the ~194 MB rootfs; `integration-firecracker` downloaded it; the
`coverage` job downloaded it a second time. Combined download overhead: ~83
seconds per run, plus the ~70-second upload.
2. **Duplicate test execution.** `integration-firecracker` ran the Firecracker
integration suite; `coverage` ran the entire unit + integration suite again
on the same KVM runner to collect coverage data. Every line of Firecracker
code was tested twice per CI run.
## Goals
- Each test suite (unit, integration-docker, integration-firecracker) executes
exactly once per workflow run.
- PRs incur no large artifact transfers — the rootfs stays on the KVM runner.
- Main-branch pushes publish a byte-for-byte identical rootfs to the one that
passed the integration tests.
- Concurrent workflow runs cannot cross-publish candidates (naturally enforced
by Gitea Actions' per-run artifact scoping).
- Failed or cancelled runs block publication (enforced by the `needs:` chain on
`publish-infra`).
## Non-goals
- Changing test semantics or the coverage policy (ADR 0004).
- Removing the KVM runner guard on `integration-firecracker` and `coverage`.
- Changing how `publish_infra.py` builds or uploads the rootfs.
## Design
### Job graph
```
unit ──────────────────────────────────┐
integration-docker ────────────────────┤──► coverage ──► publish-infra (main only)
integration-firecracker (KVM) ─────────┘
```
### `unit`
Unchanged except: `coverage run` writes `--data-file=.coverage.unit`; the file
is uploaded as the `coverage-unit` artifact.
### `integration-docker`
Adds a `coverage` install step. `coverage run` writes `--data-file=.coverage.docker`;
the file is uploaded as `coverage-docker`.
### `integration-firecracker` (KVM runner)
Replaces the old `stage-firecracker-inputs``build-infra` → download chain:
1. Builds the infra candidate locally with
`BOT_BOTTLE_FC_DROPBEAR=/var/cache/bot-bottle-fc/dropbear`.
2. Boots the candidate and runs integration tests with coverage, writing
`.coverage.firecracker`.
3. Uploads the small `coverage-firecracker` artifact unconditionally.
4. On main-branch pushes only, uploads the rootfs as `infra-candidate` and the
dropbear as `firecracker-inputs` so `publish-infra` can verify and publish
the byte-identical artifact.
### `coverage`
Moves from a KVM runner to `ubuntu-latest`. No tests are re-executed:
1. Downloads `coverage-unit`, `coverage-docker`, and `coverage-firecracker`.
2. Runs `scripts/coverage.sh aggregate critical`, which calls
`coverage combine` then `coverage report`.
3. Runs the diff-coverage gate (`scripts/diff_coverage.py`).
Coverage files use `relative_files = True` (`.coveragerc`) so they combine
cleanly across runners with different absolute workspace paths.
### `publish-infra`
Depends on all four predecessor jobs (unchanged gate). Downloads `infra-candidate`
and `firecracker-inputs` that were uploaded by `integration-firecracker` on
main — the same byte sequence that passed the integration tests.
### Eliminated jobs
- `stage-firecracker-inputs`: existed only to copy the dropbear to ubuntu-latest
for `build-infra`. No longer needed.
- `build-infra`: the infra candidate is now built on the KVM runner in
`integration-firecracker`.
### Script changes
`scripts/coverage.sh` gains an `aggregate` mode (`coverage.sh aggregate [critical]`)
that combines pre-existing `.coverage.*` files instead of re-running tests.
The existing run mode (`coverage.sh [critical]`) is preserved for local dev.
@@ -0,0 +1,146 @@
# PRD prd-new: Claude forward_host_credentials
- **Status:** Draft
- **Author:** claude
- **Created:** 2026-07-01
- **Issue:** #325
## Summary
Add `agent_provider.forward_host_credentials: true` support for the
`claude` template, mirroring the existing Codex flow. When enabled,
bot-bottle reads the host's Claude OAuth session key from
`~/.claude/.credentials.json` at launch, forwards it only to the egress sidecar,
and injects a placeholder `CLAUDE_CODE_OAUTH_TOKEN` into the agent so
Claude Code starts without ever seeing the real credential.
## Problem
Running a Claude agent in a container today requires the operator to
manually extract a long-lived OAuth token (`claude setup-token`), export
it as `BOT_BOTTLE_CLAUDE_OAUTH_TOKEN`, and reference it explicitly in
the manifest with `agent_provider.auth_token:
"BOT_BOTTLE_CLAUDE_OAUTH_TOKEN"`. This is a two-step manual ceremony
that is easy to skip or do incorrectly.
The host already stores a valid Claude session in `~/.claude/.credentials.json`
after `claude login`. Codex already automates an
equivalent extraction from `~/.codex/auth.json`. There is no reason
Claude bottles cannot do the same.
## Goals / Success Criteria
- A Claude bottle with `forward_host_credentials: true` in the manifest
uses the host's `~/.claude/.credentials.json` session key at launch with no
additional operator steps.
- The agent container receives only `CLAUDE_CODE_OAUTH_TOKEN=egress-placeholder`
— never the real token.
- The real session key lives only in the egress sidecar's environment.
- Missing, malformed, or expired host Claude auth fails launch with a
clear operator-facing message.
- Existing `auth_token` behavior is unchanged.
- `forward_host_credentials: true` is rejected in the manifest when both
`auth_token` and `forward_host_credentials` are set, since they serve
the same purpose.
## Non-goals
- Refreshing Claude OAuth tokens in the sidecar.
- Writing a dummy `~/.claude.json` auth state to the agent (unlike the
Codex flow, Claude Code reads its credential from `CLAUDE_CODE_OAUTH_TOKEN`
in env, not from an auth file — no guest-side auth marker is needed).
- Supporting `forward_host_credentials` for providers other than `codex`
and `claude`.
## Design
### Manifest schema
```yaml
agent_provider:
template: claude
forward_host_credentials: true
```
Rejects in manifest validation when:
- Template is not `codex` or `claude`.
- Both `auth_token` and `forward_host_credentials` are set.
### Host auth extraction (`contrib/claude/claude_auth.py`)
Claude Code credential storage varies by platform:
- **Linux**: `~/.claude/.credentials.json`
- **macOS**: macOS Keychain, service `"Claude Code-credentials"`
(the file path is tried first; Keychain is the fallback when the file
is absent)
`~/.claude.json` contains only UI state and profile metadata — no token.
The credentials JSON schema (same whether from file or Keychain):
```json
{
"claudeAiOauth": {
"accessToken": "<access-token>",
"refreshToken": "<refresh-token>",
"expiresAt": 1748276587173,
"scopes": ["user:inference", "user:profile"]
}
}
```
`expiresAt` is in **milliseconds** (not seconds).
At prepare/launch time, when `forward_host_credentials: true`:
1. Try `~/.claude/.credentials.json`; on macOS, if absent, run
`security find-generic-password -s "Claude Code-credentials" -w`
and parse its stdout as JSON.
2. Require a `claudeAiOauth` dict.
3. Require a non-empty `claudeAiOauth.accessToken` string.
4. If `claudeAiOauth.expiresAt` is present, divide by 1000 and require
the result to be in the future.
5. Return only the access token to the launch path.
Errors name the missing or invalid condition and point the operator at
`claude login`, without printing token values.
### Egress route
When `forward_host_credentials: true`:
- Provision the session key in `provisioned_env` under
`BOT_BOTTLE_CLAUDE_HOST_ACCESS_TOKEN` (new constant in `egress.py`).
- Set up the `api.anthropic.com` egress route with `auth_scheme: Bearer`
and `token_ref: BOT_BOTTLE_CLAUDE_HOST_ACCESS_TOKEN`.
- Set `CLAUDE_CODE_OAUTH_TOKEN=egress-placeholder` in the agent env and
add it to `hidden_env_names`.
No dummy auth file and no `verify` step are needed — Claude Code reads
the credential from the env var, not from a file.
### Constants
- `CLAUDE_HOST_CREDENTIAL_TOKEN_REF = "BOT_BOTTLE_CLAUDE_HOST_ACCESS_TOKEN"`
in `egress.py` (alongside the existing `CODEX_HOST_CREDENTIAL_TOKEN_REF`).
- `CLAUDE_HOST_CREDENTIAL_HOSTS = ("api.anthropic.com",)` in
`agent_provider.py` (alongside the existing `CODEX_HOST_CREDENTIAL_HOSTS`).
### Data flow
```
Host ~/.claude/.credentials.json → bot-bottle launch
├──► egress sidecar env (real token only)
└──► agent env: CLAUDE_CODE_OAUTH_TOKEN=egress-placeholder
Agent → HTTPS to api.anthropic.com (via egress)
Egress → injects Authorization: Bearer <real token>
Egress → forwards to api.anthropic.com
```
## Open questions
None — the Codex precedent makes the design clear.
@@ -0,0 +1,156 @@
# PRD prd-new: Consolidate infra backend for Docker
- **Status:** Active
- **Author:** Claude
- **Created:** 2026-07-20
- **Issue:** #431
## Summary
The Docker backend runs two containers — `bot-bottle-orch-gateway` (gateway
data plane) and `bot-bottle-orchestrator` (control plane) — where the
macOS and Firecracker backends already run a single combined infra
unit. This PRD collapses Docker to the same model: one `bot-bottle-infra`
container running both processes under the `gateway_init` supervise tree, a
restructured `Dockerfile.infra` as the shared gateway+orchestrator base,
and a handful of extracted shared utilities (CA cert polling, teardown
sequence, launch skeleton) that are currently duplicated across all three
`consolidated_launch.py` files.
## Goals / success criteria
- Docker backend starts exactly one infra container instead of two.
- `Dockerfile.infra` is the shared base image (gateway + orchestrator, no
buildah); the Firecracker image layers buildah on top of it.
- The orchestrator process runs under the `gateway_init` supervise tree
inside the combined container (one PID-1, one restart/health surface).
- CA cert polling, the teardown sequence, and the shared launch skeleton
(ensure-infra → register → provision → return context) live in a single
shared module; all three backends import from it.
- No functional change to macOS or Firecracker launch paths.
## Non-goals
- Changing per-bottle isolation — agents stay one-VM/container-each.
- Consolidating transport implementations (`DockerGatewayTransport`,
`AppleGatewayTransport`, `SshGatewayTransport`) — these are already the
right abstraction boundary.
- macOS DHCP-inversion of registration order — irreducible backend
difference, stays as-is.
- Any changes to the orchestrator RPC protocol or the attribution model.
## Design
### Dockerfile restructuring
**Current shape:**
- `Dockerfile.gateway` — data plane (mitmproxy, gitleaks, git, openssh,
supervise daemons)
- `Dockerfile.orchestrator` — control plane (python:3.12-slim + bot_bottle
package; stdlib-only, no third-party deps)
- `Dockerfile.infra` — Firecracker only: `FROM bot-bottle-gateway` +
buildah + `COPY --from bot-bottle-orchestrator`
**New shape:**
- `Dockerfile.gateway` — unchanged
- `Dockerfile.orchestrator` — unchanged (single definition of orchestrator
content; both Docker infra and Firecracker infra `COPY --from` it)
- `Dockerfile.infra`**shared base**: `FROM bot-bottle-gateway` + `COPY
--from bot-bottle-orchestrator` (no buildah — Docker infra image)
- `Dockerfile.infra.fc` — Firecracker only: `FROM bot-bottle-infra` +
buildah/crun/netavark/aardvark-dns (layered on the shared base, same net
result as today)
The comment in `Dockerfile.infra` that says "the docker backend keeps
orchestrator + gateway as separate images; this combined image exists only
for the Firecracker single-VM cut" is removed.
### Orchestrator in the supervise tree
`gateway_init` already supervises the data-plane daemons (egress, git-http,
supervise-MCP). The orchestrator control plane is added as another supervised
process: `python3 -m bot_bottle.orchestrator --host 0.0.0.0 --port <port>
--broker stub`.
The orchestrator source is bind-mounted (`/app` → repo root, as today) so
dev live-reload still works. `source_hash`-based container recreation in
`OrchestratorService.ensure_running` continues to apply — a code change
recreates the combined infra container, which bounces both gateway and
orchestrator. This is acceptable: the docker backend is a dev/legacy target
where in-flight egress connections across a code deploy are not a hard
requirement.
### `OrchestratorService` changes
`OrchestratorService` currently starts two containers in sequence: gateway
first (`DockerGateway.ensure_running`), then orchestrator. After this PRD:
- Single `docker run` of `bot-bottle-infra:latest`
- Container name: `bot-bottle-infra` (replaces `bot-bottle-orch-gateway` +
`bot-bottle-orchestrator`)
- Published ports: `127.0.0.1:{host_port}:8099` for the control plane
(`gateway_init` listens on a fixed internal port 8099; the caller-chosen
host port maps to it)
- Bind mounts: repo root + host root (same as today)
- `DockerGateway` becomes an implementation detail of `OrchestratorService`
rather than a separately started container; the gateway image name
(`GATEWAY_IMAGE`) is no longer referenced at runtime, only at build time
for the `Dockerfile.infra` base
The `_gateway()` / `ensure_running` two-step in `OrchestratorService` is
replaced by a single `_run_infra_container()`.
### Shared backend utilities
Three items are duplicated across
`backend/docker/consolidated_launch.py`,
`backend/macos_container/consolidated_launch.py`, and
`backend/firecracker/consolidated_launch.py`:
1. **CA cert polling loop** — `deadline = time.monotonic() + timeout; while
...: try fetch CA; sleep` — extracted to
`backend/consolidated_util.py:poll_ca_cert(transport, *, timeout)`.
2. **Teardown sequence**`OrchestratorClient(url).teardown_bottle(id)` +
`deprovision_git_gate(transport, id)` — extracted to
`backend/consolidated_util.py:teardown_consolidated(url, transport,
bottle_id)`.
3. **Launch skeleton** — all three follow: ensure-infra → allocate/register
→ provision git-gate → fetch CA cert → return launch context. The macOS
inversion (agent starts before registration, source IP from DHCP) is the
only deviation. Extract a shared `_provision_bottle(transport, bottle_id,
plan, orchestrator_url)` helper covering the register → provision →
return-token steps; the backends keep their own `launch_consolidated`
wrappers for the before/after (infra-ensure + agent-start + IP
allocation), calling the shared helper.
The new `backend/consolidated_util.py` module holds only backend-neutral,
transport-agnostic logic. All three backends import from it.
## Implementation chunks
1. **(this PR)** Dockerfile restructuring: rename current `Dockerfile.infra`
content to `Dockerfile.infra.fc`; write new `Dockerfile.infra` as
gateway+orchestrator base. Update Firecracker image-build references from
`Dockerfile.infra``Dockerfile.infra.fc`.
2. Add orchestrator process to `gateway_init` supervise tree.
3. Collapse `OrchestratorService` to a single-container start; rename
container from `bot-bottle-orch-gateway`/`bot-bottle-orchestrator`
`bot-bottle-infra`; update image name constant.
4. Extract `backend/consolidated_util.py` with `poll_ca_cert`,
`teardown_consolidated`, and `_provision_bottle`; update all three
`consolidated_launch.py` files to import from it.
5. Update tests that reference the old container names or two-container
startup sequence.
## Open questions
None — the supervise-tree approach and shared Dockerfile layering were
confirmed in issue #431.
@@ -0,0 +1,47 @@
# PRD prd-new: Modernize built-in agent images
- **Status:** Draft
- **Author:** Codex
- **Created:** 2026-07-21
- **Issue:** #451
## Summary
Keep every built-in agent provider on Debian's current stable release and make
Podman available inside each image. This gives agents a consistent, modern
userspace and an OCI container tool without requiring per-project setup.
## Problem
The Claude, Codex, and Pi images inherit the generic `node:22-slim` tag. That
tag does not state which Debian release the project supports and currently
leaves the images on the older Bookworm release. None of the built-in images
installs Podman, so tasks that need to inspect or build OCI images must first
modify the bottle or cannot run at all.
## Goals / success criteria
- Every Dockerfile under `bot_bottle/contrib/*/Dockerfile` explicitly inherits
`node:22-trixie-slim`, based on Debian 13 (the current stable release).
- Every built-in agent image installs Podman from Debian stable.
- Every built-in agent image retains an SSH client for Git-over-SSH workflows.
- The non-root agent user owns a traversable XDG Git configuration directory,
so Git can load bot-bottle's global git-gate rewrites without permission
errors.
- A shared test enforces both requirements for current and future built-in
providers.
## Non-goals
- Configuring privileged or nested-container execution for bottles.
- Pinning Podman outside Debian's stable package repository.
- Changing the Node.js or agent CLI release policy.
## Design
Use the explicit `node:22-trixie-slim` base rather than the floating `slim`
variant. Install the `podman` package with each image's existing `apt-get`
dependency layer, so package metadata and caches are still removed in the same
layer. Treat Debian stable as the Podman stability and update channel; this
keeps the images stdlib/distribution-first and avoids adding a third-party
package repository.
+153
View File
@@ -0,0 +1,153 @@
# PRD prd-new: Containers inside a bottle
- **Status:** Draft
- **Author:** Claude
- **Created:** 2026-07-21
- **Issue:** #392
## Summary
Let an agent run `docker` and `docker compose` *inside* its bottle by starting
a guest-local rootless podman service that exposes a Docker-compatible API
socket. Gated per bottle by `nested_containers: true`. No host daemon socket is
mounted and no capability is added to the guest, on any backend.
## Problem
Agent tasks routinely involve containers: `docker compose up` to verify a
scaffolded stack, a throwaway database for a test run, building an image to
check a Dockerfile works. Bottles have no way to do any of that, so those tasks
either fail or get done outside the bottle — which defeats the point of the
bottle.
## Goals / success criteria
- A bottle with `nested_containers: true` on the `macos-container` backend can
run `docker compose up -d --wait` against a stock compose file, pulling from
a routed registry and serving from the workspace.
- The agent's habits do not change: `docker` and `docker compose`, not
`podman`.
- No host Docker socket is mounted, on any backend.
- No capability is added to the Apple Container guest.
- Backends without a guest-local engine reject the flag with a clear error
rather than ignoring it.
- Bottles that do not set the flag carry none of its cost.
## Non-goals
- Nested containers as an isolation layer. The single-UID mapping means `root`
inside a nested container is the agent user outside it. This is for build and
test workloads; the bottle stays the security boundary.
- The `docker` and `firecracker` backends. Both reject the flag for now.
- Reaching registries that are not routed through the bottle's egress proxy.
## Design
### Why podman, not rootless Docker
Apple Container's capability bounding set omits `CAP_SYS_ADMIN`, which the
kernel requires in order to write a multi-range `uid_map` via `newuidmap`.
Rootless Docker has no path that avoids that write, so it cannot run in a
bottle without granting `CAP_SYS_ADMIN` — which is close to root in practical
terms and gives up most of what the bottle is for.
Podman does have a path: with **no** subordinate UID range configured it falls
back to a single-UID self-mapping, which an unprivileged process may write
itself. The image build therefore *removes* the agent user's `/etc/subuid` and
`/etc/subgid` entries rather than adding them — their presence is exactly what
would send podman down the `newuidmap` path. Full negative result in
[`docs/research/rootless-docker-in-apple-container-spike.md`](../research/rootless-docker-in-apple-container-spike.md).
### The flag, and its name
The flag is kept because it gates real costs, not because podman needs a
privilege grant: ~100MB of derived image, a resident service per bottle, and
relaxed modes on `/dev/fuse` and `/dev/net/tun` that the majority of bottles
should not get.
It is named `nested_containers`, not `docker_access`. It implies no Docker and
grants access to nothing on the host — naming it after Docker access would
describe the one thing the design refuses to do.
### Shape
- `bot_bottle/backend/macos_container/nested_containers.py` — derived-image
build, guest env, device preparation, service start/readiness.
- `bot_bottle/backend/macos_container/nested-containers-init.sh` — the
unprivileged bootstrap that runs inside the bottle. Fails closed on every
prerequisite it needs (podman, the storage/network helpers, writable device
nodes, an *empty* subordinate range) rather than degrading.
- The derived image `…-nested-containers` layers the Docker CLI, the compose
plugin, and podman 5's networking stack — `passt` (pasta), `netavark`,
`nftables`, `aardvark-dns` — plus `fuse-overlayfs`, `slirp4netns`, and
`uidmap` onto the agent image. Podman itself already ships in every built-in
image (#451), but with `--no-install-recommends`, so none of the networking
pieces arrive with it. Each absence fails at a different and misleading
layer: no pasta and nothing starts; no `nft` and containers are created but
never start; no aardvark-dns and DNS inside a container fails while
everything else looks healthy. Image pulls keep working throughout, which is
what makes these read as compat-API bugs.
- `BottleBackend.supports_nested_containers` — false by default, so a backend
that cannot honor the flag fails in the shared `prepare` template.
Guest configuration the single-UID mapping forces:
`ignore_chown_errors` (no second UID for layers to be chowned to),
`cgroups="disabled"` and `cgroup_manager="cgroupfs"` (no cgroup delegation
reaches the guest), `events_logger="file"` (no journald socket).
### Registry reach
Image pulls egress through the bottle's proxy like everything else, so each
registry needs a route — **and so does the CDN it redirects blobs to**, which
is a separate host: `production.cloudfront.docker.com` for Docker Hub,
`pkg-containers.githubusercontent.com` for GHCR, `cdn0*.quay.io` for quay.
Without those the pull authenticates, fetches the manifest, and then 403s
partway through. Docker Hub and GHCR need `preserve_auth: true` — their
token dance uses a client-fetched per-scope bearer token that the proxy would
otherwise strip. Layer pulls should also set `dlp.outbound_detectors: false`
and `dlp.inbound_detectors: false`: body scanning on a multi-hundred-MB layer
is what triggers #455, and a registry route's scanned bodies are compressed
layer blobs rather than anything a detector can read.
### Reaching the network from a nested container
Public DNS inside a nested container fails by design: everything egresses
through the proxy. What was actually broken was reaching the proxy at all, and
it took four attempts to fix because podman applies proxy settings at several
layers and the last writer wins:
| Layer | Applies to |
|---|---|
| `~/.docker/config.json` proxies block | **every container the Docker CLI starts** — client-side, beats everything below |
| service environment | podman's own pulls, non-CLI API clients |
| `containers.conf` `env` | native `podman run` |
| `containers.conf` `hosts_file`, `http_proxy` | native `podman run` only — the compat API ignores both |
The gateway is named `bot-bottle-gateway`, which resolves only through the
bottle's `/etc/hosts`; a nested container gets its own. Since no config key
reaches the compat path, the name is resolved in the bottle and the *address*
is substituted into the proxy URL at each layer above. `NO_PROXY` keeps the
name, which is matched against what a client asks for.
The gateway TLS-intercepts, so the CA bundle is mounted read-only and
`SSL_CERT_FILE`, `CURL_CA_BUNDLE`, `REQUESTS_CA_BUNDLE`, and
`NODE_EXTRA_CA_CERTS` point at it — no distro-specific trust commands.
## Verified on macOS 26 / Apple Container 1.0.0 / podman 5.4.2
With plain `docker`, no extra flags: image pulls from Docker Hub, GHCR, and
quay; `docker run` with correct exit-code propagation; `docker compose` up,
logs, down, and a published port curl'd from the bottle; `curl https://quay.io`
and `https://pypi.org/simple/` returning 200 from inside a nested container;
and `https://example.com` returning **403** — the egress allowlist applies to
nested containers, not just the bottle.
Alpine's BusyBox `wget` drops the connection after TLS interception and
reports "error getting response". The proxy logs the decrypted request, and
`curl` on the same host succeeds, so this is a BusyBox client limitation
rather than anything in the egress path.
## Open questions
- Whether the `docker` and `firecracker` backends want an equivalent, or
whether guest-local containers stay macOS-only.
@@ -0,0 +1,131 @@
# PRD prd-new: Encrypted at-rest egress secrets (SecretProvider, interim slice)
- **Status:** Draft
- **Author:** didericis
- **Created:** 2026-07-21
- **Issue:** #355
## Summary
An interim step toward the generic `SecretProvider` (#355) that stops short
of per-request minting. Today the orchestrator holds each bottle's egress
auth tokens **in process memory only**, so any infra-container recreation
silently strips every already-running bottle of its upstream credentials.
This PRD makes those secrets survive a gateway restart by persisting them
**encrypted**, under a key that is not itself sitting next to the
ciphertext.
The end state in #355 — short-lived, scoped credentials minted per request
— removes the need to store anything durable at all. That is a larger
change gated on per-upstream minting support. This slice buys back
restart-survivability now without regressing to plaintext secrets at rest.
## Problem
`Orchestrator._tokens` (`bot_bottle/orchestrator/service.py:74-79`) is a
plain in-memory dict, deliberately never written to the registry DB:
> Held **in memory only** — never written to the registry DB — so the
> gateway can inject each bottle's upstream credential without secrets at
> rest. Lost on restart (re-launch re-registers them); the future
> SecretProvider (#355) replaces this with per-request minting.
The registry itself *is* durable (SQLite on a container-only volume), and
so is the gateway CA since #450 / `2cd44cf7`. The tokens are now the only
piece of gateway state that does not survive a restart, which makes the
failure mode both silent and confusing.
### Observed failure
Checking out a branch that touches `bot_bottle/**/*.py` changes
`source_hash()` (`bot_bottle/orchestrator/lifecycle.py:88-99`).
`MacosInfraService._source_current()`
(`bot_bottle/backend/macos_container/infra.py:159-169`) sees the mismatch
and `ensure_running()` force-removes and recreates the infra container
(`infra.py:198-208`). The registry rows survive on the DB volume; the CA
survives on its host bind-mount; `_tokens` comes back empty.
Every already-running bottle then fails closed, mid-session, on its next
outbound request:
- `/resolve` succeeds — the bottle is still `active` in
`orchestrator_bottles` and its policy blob is served intact, including
`- host: "api.anthropic.com"` with `auth_scheme: Bearer` /
`token_env: EGRESS_TOKEN_0`.
- `tokens_for()` returns `{}`, so the resolved env overlay has no
`EGRESS_TOKEN_0`.
- `decide()` (`bot_bottle/egress_addon_core.py:644-652`) blocks with
`egress: route for 'api.anthropic.com' declared auth but env var
'EGRESS_TOKEN_0' is unset` — an 89-byte `403` on every request.
Confirmed live on the macOS backend on 2026-07-21: two bottles running
since 20:29/20:30 were still registered `active` with valid policy after
the 23:05 infra recreation, and both took 89-byte `403`s from then on,
while a bottle launched *after* the recreation egressed normally. The
recovery today is to relaunch every affected bottle.
Note this is a re-attachment blocker distinct from #443/#445 and from #450
— the CA and the gateway address were both fine. It is specifically the
credential wipe.
## Goals / Success criteria
1. A bottle's egress auth tokens survive infra-container recreation: an
already-running bottle keeps egressing across a gateway restart with no
relaunch and no operator action.
2. Secrets are **never** at rest in plaintext, and never at rest next to a
key that trivially decrypts them.
3. Compromise of the registry DB file alone does not yield usable
upstream credentials.
4. The stored form is revocable and rotatable without relaunching bottles
that are not affected.
5. When the retained bottle record reaches the lifecycle status `removed`,
its stored secrets are destroyed. Status/event persistence and the removal
transition land separately; this interim slice intentionally retains the
ciphertext across today's teardown/reconcile calls until that lifecycle is
available.
6. Migration is transparent: existing bottles keep working, no manifest
changes required.
## Non-goals
- **Per-request minting** of short-lived scoped credentials. That is the
#355 end state; this PRD is explicitly the interim slice and should not
foreclose it.
- Generalizing `DeployKeyProvisioner` into the full `SecretProvider` ABC,
or the manifest-level `{ provider: <name> }` reference surface.
- User-extensible provider discovery (`~/.bot-bottle/contrib/<name>/`).
- Changing the `/resolve` contract's shape (it already carries `tokens`).
- Fixing the *trigger*`source_hash` churn on branch switch. Recreating
infra is legitimate; it just must not cost running bottles their
credentials. A separate guard that refuses recreation while bottles are
active is complementary and out of scope here.
## Design
> **TODO (didericis):** the encryption flow goes here — key custody, where
> the key material lives relative to the ciphertext, the wrap/unwrap path
> at register and at `/resolve`, and what an attacker who holds only the
> DB (or only the host, or only the infra container) can recover.
Constraints the design has to satisfy, for reference while drafting:
- The gateway's `PolicyResolver` needs the cleartext at request time, on
the data-plane path, so unwrap has to be cheap enough to sit in a
per-flow `/resolve` (or be cached in memory after first unwrap).
- The infra container is recreated routinely and unattended. Anything
requiring an interactive unlock on every recreation defeats the goal.
- The DB lives on a container-only volume that the host does not mount, so
host-side and guest-side components see different filesystems — that
asymmetry is available as a place to split custody.
- The agent must never be able to reach the key material. It is a separate
container with no control-plane token, which is the existing boundary.
## Open questions
- Where does the unwrap key live, and what recreates/re-derives it when the
infra container is rebuilt?
- Is the cleartext cached in memory after first unwrap, or unwrapped per
request? (Latency vs. exposure window.)
- What is the rotation story — re-wrap in place, or force re-registration?
- Does this land behind a flag, or replace `_tokens` outright?
@@ -0,0 +1,86 @@
# Egress proxy OOMs on large downloads
Found on 2026-07-21 while running the rootless-podman spike
(`docs/research/rootless-docker-in-apple-container-spike.md`). Recorded
rather than fixed — the fix is a security-relevant decision, not a
mechanical patch.
## Summary
A single large HTTPS download through the gateway kills the egress
proxy. `mitmdump` buffers whole response bodies so the DLP detectors can
scan them, grows past the gateway container's memory limit, and is
OOM-killed by the cgroup. Nothing restarts it.
Two properties make this worse than a failed download:
- **The gateway is a per-host singleton.** Every bottle shares it, so
one bottle's download takes egress away from all of them.
- **There is no restart on death.** The gateway supervisor is
`while : ; do wait ; done`; a killed daemon stays dead until the infra
container is recreated.
So ordinary agent activity — pulling a container image, downloading a
model or dataset, fetching a large tarball — is a denial of service
against every other bottle on the host. No malice required, though it is
trivially reachable on purpose.
## Evidence
Triggered by `docker compose up` pulling `quay.io/fedora/python-312`
(two layers, ~82MB and ~83MB) inside a bottle. The pull itself
succeeded; the *next* request failed:
```
initializing source docker://quay.io/fedora/python-312:latest:
pinging container registry quay.io: Get "https://quay.io/v2/":
proxyconnect tcp: dial tcp 192.168.128.39:9099: connect: connection refused
```
From the gateway's `dmesg`:
```
python3 invoked oom-killer: gfp_mask=0x100cca(GFP_HIGHUSER_MOVABLE), order=0
oom-kill:constraint=CONSTRAINT_MEMCG,
oom_memcg=/container/bot-bottle-mac-infra,
task_memcg=/container/bot-bottle-mac-infra,task=mitmdump,pid=118
Memory cgroup out of memory: Killed process 118 (mitmdump)
total-vm:1391936kB, anon-rss:997768kB
```
~1GB RSS against a 1024MB container. Note the amplification: ~165MB of
layers produced ~1GB of resident memory, so the buffering is several
copies deep (encoded body, decoded body, and the text conversion the
regex detectors scan).
Afterwards the gateway container was still running and healthy-looking —
orchestrator, supervise, and git-http all alive — with no `mitmdump`
process at all, and it stayed that way until the container was
recreated. A liveness check on the container would not have caught this.
## Reproduction
1. Launch any bottle with an egress route to a host serving a large file.
2. Download >~150MB over HTTPS through the proxy.
3. `dmesg | grep -i oom` inside `bot-bottle-mac-infra`, and note that no
`mitmdump` process remains.
Beware a false negative when checking: truncating the process listing
(`cut -c1-45`) cuts before the binary name, because `mitmdump` runs as
`/usr/local/bin/python3.12 /usr/local/bin/mitmdump …`.
## Fix options, not yet chosen
1. **Restart dead daemons.** Smallest change and strictly an
improvement: an OOM then degrades one download instead of removing
egress for every bottle. Does not stop the OOM.
2. **Cap the scanned body size.** Above a threshold, stop buffering —
either skip the scan or stream it. This is the root-cause fix and a
security decision: a size threshold is exactly the hole an exfiltrator
would aim for, so "skip above N" trades a DoS for a covert channel.
Streaming with a bounded window keeps coverage, at more complexity.
3. **Raise the gateway's memory limit.** Moves the threshold; does not
remove it.
Worth noting that (1) and (2) are complementary — the restart gap is
worth closing regardless of how the memory behaviour is resolved.
@@ -0,0 +1,178 @@
# Firecracker Image Remote Store
**Date:** 2026-07-22
**Context:** PR #459 (run-dir leak fix) surfaced that committed Firecracker snapshots
currently live only on the host machine. Once a host is wiped or a run-dir is
evicted, a user's preserved bottle is gone. This note investigates a secure remote
store so committed images survive host turnover and can be restored without the
user having to pre-flag which sessions to keep.
## Verdict
Backblaze B2 + Cloudflare CDN is the cost-optimal choice for most deployments.
Cloudflare R2 is the simpler zero-config option at slightly higher storage cost.
Self-hosted MinIO is the right call for air-gapped or on-premises installs.
On the image-size front, zstd-compressing the committed tar before upload
produces roughly a 6070% reduction with negligible impact on restore latency.
OverlayFS (for in-flight working rootfs, not for the committed artifact) cuts
per-instance disk use to ~1050 MB per extra bottle sharing the same base.
---
## What Gets Stored
The Firecracker backend produces two artifact types:
| Artifact | Created by | Size | Lifetime |
|----------|-----------|------|---------|
| `rootfs/agent-<digest>/` (dir) | `image_builder.py``mke2fs -d` | ~1 GB as ext4 | Cached per Dockerfile hash; evictable |
| `committed/<slug>/rootfs.tar` | `FirecrackerFreezer._freeze` via SSH tar | 500 MB1 GB | User-preserved; must survive host wipe |
The committed artifact is a tar of the guest's live filesystem streamed out over
SSH (`freezer.py:5890`). At resume time `launch.py` calls `mke2fs -d` to
rebuild a fresh ext4 from this tar. The tar — not the ext4 — is what needs to be
pushed to remote storage and pulled back at restore time.
The base image cache (`rootfs/agent-<digest>/`) is derivable from the Dockerfile
and can be rebuilt on demand; it is lower priority for remote storage.
---
## Storage Candidates
### Object storage
| Provider | Storage | Egress | Notes |
|----------|---------|--------|-------|
| **Backblaze B2** | $0.006/GB | Free (via Cloudflare Bandwidth Alliance) | Cheapest storage; pairs with Cloudflare CDN to eliminate egress |
| **Cloudflare R2** | $0.015/GB | $0 always | Zero-config egress; no lifecycle transitions (limitation) |
| **Wasabi** | $0.0069/GB | Free (1:1 ratio) | 90-day minimum retention; good for archival; lifecycle evaluated daily |
| **AWS S3** | $0.023/GB | $0.09/GB | Richest lifecycle support; expensive at scale; avoid unless already in AWS |
| **MinIO** (self-hosted) | Host cost only | None | S3-compatible; best for private/on-prem deployments |
**B2 + Cloudflare CDN** is effectively $0.006/GB with zero egress — about 18×
cheaper than S3 for restore-heavy workloads. **R2** is the zero-config choice
($0 egress by default, no Bandwidth Alliance pairing needed) at a slightly
higher storage rate.
**Cloudflare R2's missing lifecycle support** is the main caveat: auto-eviction
rules (evict images older than N days) cannot currently be expressed natively in
R2. Wasabi and S3 both support declarative lifecycle policies.
### Retention policy recommendation
The comment proposes:
- Retain images for ~1 week by default
- Warn when approaching a capacity threshold
- Auto-evict oldest images once threshold is exceeded
This maps cleanly to an application-level policy (not a provider lifecycle rule),
which avoids the R2 limitation and works consistently across providers:
1. On `commit`: upload tar, record `(slug, size_bytes, uploaded_at)` in a local
or remote manifest file.
2. On startup / on `list`: scan the manifest, warn if total stored size exceeds
e.g. 80% of the configured threshold.
3. On eviction run (CLI or cron): delete objects older than `retention_days`
(default 7) that push total over `max_capacity`; oldest-first.
This keeps the policy logic in bot-bottle and the storage provider as a dumb
object store — no vendor-specific lifecycle API required.
---
## Image Size Reduction
### Current artifact sizes
A typical committed tar for a Claude Code agent image is 500 MB1 GB uncompressed.
The per-run ext4 (copy of the base, written at `start`) adds another ~1 GB of
local disk. The leak fix in this PR addresses the ext4 copies; the remote store
addresses the committed tars.
### Compression
zstd compression of the committed tar before upload is the highest-leverage
single change:
| Codec | Typical size (1 GB rootfs) | Compress speed | Decompress speed |
|-------|---------------------------|---------------|-----------------|
| gzip | 350430 MB | ~100 MB/s | ~500 MB/s |
| **zstd (default)** | **330360 MB** | **~400 MB/s** | **~2 GB/s** |
| xz | 290320 MB | ~20 MB/s | ~200 MB/s |
**zstd is the best trade-off**: 6567% size reduction, near-instantaneous
decompression. The `tar` call in `freezer.py` could pipe through `zstd` before
writing to disk and to the remote; `resume` decompresses on the way back. A
`.tar.zst` suffix marks compressed artifacts so old tars remain restorable
without the codec.
### SquashFS for the base image cache
The `rootfs/agent-<digest>/` directory (the buildah-exported tree) is rebuilt by
`image_builder.py` and turned into per-run ext4 by `mke2fs -d`. Storing the
base as a SquashFS image instead of a flat directory tree would reduce it from
~1 GB to ~330360 MB and make the cache remote-friendly. Firecracker does not
directly boot SquashFS, but the existing `mke2fs -d` path reads a directory tree
— a SquashFS mount could serve as the source. This is a larger change and lower
priority than tar compression.
### OverlayFS for per-run rootfs
Multiple simultaneous bottles sharing the same agent image today each get a full
`mke2fs -d` copy (~1 GB). OverlayFS (read-only base + writable sparse overlay)
would reduce this to ~1050 MB per instance beyond the first:
- Mount the base image directory as read-only lower layer
- Attach a sparse ext4 or tmpfs writable layer per bottle
- Pass the merged overlay to Firecracker as the block device
E2B's public write-up on Firecracker + OverlayFS confirms this approach works
at scale. The `launch.py` changes would be non-trivial (device mapper or
`fuse-overlayfs` plumbing), so this is a follow-up rather than a prerequisite
for the remote store.
---
## Recommended Approach
**Phase 1 — remote store with zstd (tight scope, actionable now)**
1. Add `--zstd` to the `tar` call in `FirecrackerFreezer._freeze`; name the
artifact `rootfs.tar.zst`. Keep uncompressed restore path for legacy tars.
2. Add a `bb firecracker upload <slug>` / `bb firecracker pull <slug>` pair that
pushes/fetches the compressed tar to the configured object store (S3-compatible
API, so B2, R2, MinIO, and Wasabi all work with the same client).
3. Store a `manifest.json` in the bucket (or a local mirror) tracking slug →
`{size, uploaded_at}`. Use it for threshold warnings and eviction.
4. Default retention: 7 days, configurable via `firecracker.image_retention_days`
in `~/.config/bot-bottle/config.toml` (or equivalent).
5. Warn at 80% of `max_capacity` (default e.g. 50 GB); evict oldest on commit
once at 100%.
**Storage recommendation:** Cloudflare R2 for hosted deployments (zero egress,
zero config), MinIO for private/on-premises.
**Phase 2 — base image cache compression**
Compress the `agent-<digest>` cache dir as a `.tar.zst` to save ~65% on repeated
image uploads. Low urgency since the base image is rebuildable.
**Phase 3 — OverlayFS per-run disk**
Replace the full per-run ext4 copy with an OverlayFS sparse layer. Largest disk
impact (~9095% savings per concurrent bottle) but highest implementation
complexity. Track as a separate PRD.
---
## Open Questions
- Does the host have a configured object-store credential path, or should the
remote store be an opt-in with an explicit `bb config set image-store.url ...`?
- Should `commit` automatically upload, or should upload be an explicit step to
avoid surprise egress?
- What is the acceptable cold-start latency for a restore from remote? A 330 MB
zstd tar at 100 Mbit/s takes ~26 s; at 1 Gbit/s, ~2.6 s. This bounds the
retention strategy (evict from local after successful upload vs keep local copy).
@@ -0,0 +1,359 @@
# Rootless Docker inside Apple Container bottles
Spike branch: `spike/rootless-docker-macos` (`a4d8461`)
**Outcome:** the podman recommendation below shipped as the
`nested_containers` bottle flag — see
[`docs/prds/prd-new-nested-containers.md`](../prds/prd-new-nested-containers.md).
The `docker_access` name used throughout the spike text was renamed on the
way in; it granted no access to anything on the host.
## Summary
**Negative result.** Rootless Docker cannot run inside an Apple
Container bottle without granting the bottle `CAP_SYS_ADMIN`. This is a
kernel constraint on writing multi-range `uid_map`, not a packaging gap
we can close with a better init script, a different base image, or more
careful `/etc/subuid` handling.
The spike was built on the premise — stated in
`bot_bottle/backend/macos_container/rootless_docker.py` — that it would
*"deliberately refuse to compensate for missing prerequisites with outer
capabilities, a privileged container, or a host Docker socket."* That
premise is exactly what the experiment falsified. The two ways forward
are to abandon the premise (add `CAP_SYS_ADMIN` to the bottle, and with
it most of the isolation the bottle exists to provide) or to abandon
rootless Docker.
Recommendation: abandon rootless Docker. Podman does not have this
problem — see [Podman is not blocked by
this](#podman-is-not-blocked-by-this) below.
## Local environment
Tested on 2026-07-21:
```console
$ sw_vers
ProductName: macOS
ProductVersion: 26.5.1
BuildVersion: 25F80
$ container --version
container CLI version 1.0.0 (build: release, commit: ee848e3)
$ uname -a # inside the bottle
Linux ... 6.18.15 #1 SMP Tue Mar 17 01:36:53 UTC 2026 aarch64 GNU/Linux
```
## The failure
`tests/integration/test_macos_rootless_docker_spike.py` builds the
image, launches the bottle, and dies in `rootless_docker.start`:
```
+ exec rootlesskit --net=slirp4netns --mtu=65520 ... dockerd-rootless.sh
[rootlesskit:parent] error: failed to setup UID/GID map:
newuidmap 1100 [0 1000 1 1 100000 65536] failed:
newuidmap: write to uid_map failed: Operation not permitted
```
## Why it fails
Every prerequisite you would normally suspect is present and correct in
the guest:
| Check | Result |
| --- | --- |
| `/usr/bin/newuidmap` | `-rwsr-xr-x root root` — setuid bit intact, survived the OCI export |
| `/` mount options | `rw,relatime`**not** `nosuid` |
| `NoNewPrivs` | `0` |
| `Seccomp` | `0`, no filters |
| `/etc/subuid`, `/etc/subgid` | `node:100000:65536` in both |
| user namespace | `user:[4026531837]`, identical to pid 1 — the *initial* userns |
| `unshare -U -r true` | succeeds |
| `/proc/sys/user/max_user_namespaces` | `4505` |
The one thing that is missing is in the capability bounding set that
Apple Container gives the container:
```
CapBnd: 00000000a80425fb
= chown, dac_override, fowner, fsetid, kill, setgid, setuid, setpcap,
net_bind_service, net_raw, sys_chroot, mknod, audit_write, setfcap
```
No `CAP_SYS_ADMIN`. That is the whole story, and the chain is:
1. The kernel's `map_write()` gates writing a `uid_map` on
`file_ns_capable(file, ns, CAP_SYS_ADMIN)` — capability over the
**new** user namespace, evaluated against the credentials that opened
`/proc/<pid>/uid_map`.
2. `newuidmap` is setuid-root, so it runs with euid 0 — but its
capability sets are clamped by the bounding set, which has no
`CAP_SYS_ADMIN`.
3. `cap_capable()` has a shortcut that grants *all* capabilities when
the caller's userns is the new namespace's parent **and**
`ns->owner == cred->euid`. It does not apply: the namespace was
created by `node` (uid 1000) while `newuidmap` runs as euid 0.
4. So the check falls through to the effective-set test in the initial
userns, which fails. `EPERM`.
Note that the single-line unprivileged path (`unshare -U -r`) works
precisely because it does not go through `newuidmap` and does not need
`CAP_SYS_ADMIN`. Only the multi-range subuid mapping that rootless
Docker requires does.
This is the same constraint that makes upstream's `dind-rootless` image
require `--privileged`. It is not specific to Apple Container, except
that Apple Container gives us no bounding set that includes
`CAP_SYS_ADMIN` by default.
## It does work with the capability — which is the point
Adding the capability clears the failure immediately, and exposes one
further, much smaller blocker: `/dev/net/tun` exists (the kernel has
tun; `/proc/misc` lists `200 tun`) but Apple Container creates it
`crw------- root root`, so uid 1000 cannot open it and `slirp4netns`
fails with `open: Permission denied`. A `chmod 0666 /dev/net/tun` as
root inside the bottle fixes that, and needs no capability beyond what
the bottle already has.
With both applied by hand, the daemon comes up completely:
```console
$ container run --rm -u root --cap-add CAP_SYS_ADMIN \
bot-bottle-claude:latest-rootless-docker sh -c '...'
Server Version: 20.10.24+dfsg1
Storage Driver: fuse-overlayfs
Cgroup Driver: none
Cgroup Version: 2
API listen on /tmp/rt/docker.sock
```
So `rootless-docker-init.sh` and `rootless_docker.py` are *correct*.
The spike did not fail on a bug. It failed on its own premise.
Two secondary findings from that run, relevant if anyone revisits this:
- Debian's `docker.io` package pins Docker **20.10** (EOL), not the 28.x
implied by the `docker:28-cli` compose plugin the image copies in.
- `Cgroup Driver: none` — no resource limits on nested containers.
## Why we should not just add the capability
`CAP_SYS_ADMIN` is close to a superset of "root" in practical terms —
mount, `pivot_root`, namespace manipulation, and a long tail of
subsystem-specific powers. Granting it to the agent bottle would
undercut the containment argument the rest of the backend is built
around, including the deliberately narrow choices immediately adjacent
to it in `launch.py` (`--cap-drop CAP_NET_RAW`, no `NET_ADMIN`, a
host-only agent network). Trading all of that for nested `docker
compose` is a bad exchange.
## Podman is not blocked by this
Sanity-checked on the same host, same kernel, same runtime, so the
comparison is apples to apples:
| Scenario | Result |
| --- | --- |
| Podman rootless, `/etc/subuid` populated | **Fails identically**`newuidmap: write to uid_map failed: Operation not permitted` |
| Podman rootless, no subuid ranges, `--network=host` | **Works**, no added capabilities |
| Podman rootless, no subuid ranges, default netns, `/dev/net/tun` at `0600` | Fails — `slirp4netns: open("/dev/net/tun"): Permission denied` |
| Podman rootless, no subuid ranges, default netns, `/dev/net/tun` at `0666` | **Works**, no added capabilities |
The difference is that podman degrades gracefully when no subuid range
is available: it falls back to a single-UID self-mapping, which an
unprivileged process may write itself, so `newuidmap` is never invoked
and `CAP_SYS_ADMIN` is never needed. Docker's rootless mode has no
equivalent fallback.
The cost of that fallback is real and should be weighed before building
on it: with a single-UID mapping, every UID inside a nested container
collapses onto the bottle's own uid 1000. There is no UID separation
between the agent and anything it runs — `root` in a nested container is
the agent user outside it. It also requires `ignore_chown_errors` on the
storage driver. Whether that is acceptable depends on whether the bottle
boundary (which is unchanged) or the nested-container boundary (which is
effectively nil) is the one we are relying on.
## What the podman spike then needed
The podman implementation that replaced the Docker one on this branch
turned up two more device-node blockers of the same shape as
`/dev/net/tun` — Apple Container creates the node, but 0600 root:root:
- **`/dev/fuse`** — blocks the `fuse-overlayfs` storage driver
(`fuse: failed to open /dev/fuse: Permission denied`). Without it the
only working driver is `vfs`, which copies whole layers per container.
- **`/dev/net/tun`** — blocks `slirp4netns`, which rootless podman uses
for the default bridge network.
Both are fixed by `chmod 0666` as root inside the bottle, which needs no
capability the bottle does not already hold. This is categorically
different from the `CAP_SYS_ADMIN` requirement: it is a permission on a
node that already exists, not an outer privilege grant.
One design note worth recording: the agent-facing surface stays `docker`
and `docker compose`, pointed at podman's Docker-compatible API socket
via `DOCKER_HOST`. Setting `netns="host"` in `containers.conf` does *not*
propagate through that compat API — stock `docker run` and compose files
request bridge networking explicitly — so slirp4netns (and therefore the
`/dev/net/tun` chmod) is required for ordinary compose files to work at
all. Host networking remains available per-workload via
`--network=host`.
Verified working in a bottle with zero added capabilities: fuse-overlayfs
storage, the compat API socket, `docker run` on both bridge and host
networking, and published ports.
### Nested pulls collide with our own egress DLP
The first live run got podman up and `docker compose` running, then
failed on the image pull:
```
web Pulling
initializing source docker://python:3.12-alpine: reading manifest ...
StatusCode: 403, egress DLP: Generic Bearer JWT found in body
```
This is bot-bottle's own egress scanner, not a podman problem. The
Docker registry auth flow carries a bearer JWT *by protocol*, and the
`token_patterns` detector's `Generic Bearer JWT` rule
(`Bearer\s+[A-Za-z0-9._\-]{50,}`) matches it on every pull. Any bottle
that pulls images will hit this.
The fix is per-route detector scoping, which the egress config already
supports — drop `token_patterns` on the registry hosts and keep
`known_secrets`:
```json
{"host": "registry-1.docker.io",
"dlp": {"outbound_detectors": ["known_secrets"]}}
```
That is the right trade rather than a grudging one: `known_secrets`
matches the bottle's *actual* credential values, so real exfil through a
registry host is still caught. `token_patterns` on a registry route only
ever produces protocol noise.
Worth generalising later: any manifest enabling `nested_containers` needs
this on its registry routes, so it probably belongs in a shared
registry-route snippet rather than being copy-pasted per bottle.
### And then registry auth collides with the Authorization strip
With DLP scoped, the pull failed differently: `unauthorized:
authentication required`. This one is architectural.
`egress_addon.py` strips agent-set `Authorization` unconditionally
before forwarding — deliberately, so an agent cannot smuggle a
credential out in a header the DLP detectors don't recognise. A route
may carry gateway-injected auth instead, but only from a *static* token
in an env var (`auth_scheme` + `token_env`).
Docker registry auth doesn't fit that shape. The client fetches a
short-lived, per-repository-scope bearer token from `auth.docker.io` and
presents it to `registry-1.docker.io`. There is no static token to
inject, and the token the client legitimately obtained is stripped.
Measured inside a bottle, by hand:
| Step | Result |
| --- | --- |
| Fetch token from `auth.docker.io` | 200, 5409-byte token body |
| Manifest request **with** that valid token | 401 |
| Manifest request with **no** Authorization | 401 — identical |
A valid token behaves exactly like sending none, which is direct
evidence the header never arrives. Any nested-container workflow that
pulls from a registry is blocked on this, so it is not a detail that can
be deferred: pulling base images is most of what nested containers are
for.
### Registries that skip the token dance work today
Not every registry needs the stripped header. Measured directly:
| Registry | Manifest request with no `Authorization` |
| --- | --- |
| `quay.io` | 200 |
| `mcr.microsoft.com` | 200 |
| `registry.k8s.io` | 307 (redirect, no auth) |
| `ghcr.io` | 401 |
| `registry-1.docker.io` | 401 |
So "just add the registry to the bottle config" genuinely works — for
quay, MCR, registry.k8s.io, or any unauthenticated internal registry.
Docker Hub and GHCR are the ones that need the strip resolved. The
acceptance test uses quay for exactly this reason.
Resolving it for Docker Hub means picking one of:
1. **Per-route opt-in to preserve client Authorization.** Smallest
change. Note the compounding effect on exactly these routes: the DLP
scoping above already removed `token_patterns` there, so a
preserved-auth registry route is one where the agent may send bearer
tokens that neither the strip nor the pattern detector inspects.
`known_secrets` still applies, so the bottle's real credentials are
still caught.
2. **A registry-aware gateway** that performs the token dance itself and
injects the result. Preserves the invariant fully; materially more
work, and it makes the gateway speak a specific registry protocol.
3. **Pre-seed images at provision time** (host-side `container image
save` into podman storage), so bottles never pull at runtime.
Preserves the invariant, and limits nested containers to
pre-approved images — which fits the custody positioning, at the cost
of no ad-hoc `docker pull`.
4. **Stop.** Nested containers are not supported on this backend.
### Podman 4.3.1 silently swallows container exit codes
Debian bookworm — which the current agent base image is built on —
ships podman 4.3.1. Through its Docker-compatible API, `docker run`
returns 0 no matter what the container did:
| Command | podman 4.3.1 | podman 5.4.2 |
| --- | --- | --- |
| `docker run … sh -c 'exit 7'` (compat API) | **0** | 7 |
| `docker run … sh -c 'exit 0'` (compat API) | 0 | 0 |
| `podman run … sh -c 'exit 7'` (native) | 7 | 7 |
This is worse than a broken feature: every failing command an agent runs
via `docker run` reports success. A test suite, a build step, or a CI
script inside a bottle would pass while failing. It also silently
defeated the acceptance test's egress-containment assertion, which is
why that assertion now checks an in-band marker rather than an exit
code.
Podman 5.4.2 (Debian trixie) fixes it, but needs two packages that
bookworm's podman does not: `passt` (podman 5's default network tool)
and `nftables` (netavark shells out to `nft`; without it every run fails
with `unable to upgrade to tcp, received 500`). With both installed,
exit codes propagate correctly and the compat API behaves.
The open question this leaves is where podman 5 comes from, since the
agent base is bookworm-based:
1. **Move the agent images to Debian trixie.** Trixie is current stable.
Correct, and the blast radius is every bottle, not just this feature.
2. **Drop the compat socket and use podman natively** (`podman-docker`
provides a `docker` shim; compose comes from `podman-compose`).
Native podman propagates exit codes correctly even on 4.3.1. Contained
to this feature, at the cost of `docker compose` becoming
`docker-compose`/`podman-compose`.
3. **Ship bookworm's podman 4.3.1 with the compat socket** — not viable.
Silent false success is a correctness bug agents cannot see.
## Recommendation
1. Do not revive rootless Docker on this backend. This document is the
record of why.
2. Nested containers, if wanted, come from podman under the
single-mapping constraint — with the explicit understanding that the
nested-container boundary carries no security weight. `root` in a
nested container is the agent user outside it.
3. Nested containers are therefore a build/test convenience. The bottle
remains the security boundary, exactly as it was.
+28 -8
View File
@@ -1,15 +1,19 @@
#!/usr/bin/env bash
# Combined unit + integration coverage (see docs/decisions/0004-coverage-policy.md).
#
# Runs the unit suite, then appends the integration suite (which skips
# cleanly when Docker / the backend CLIs are unavailable), and prints one
# combined report. The integration suite is what scores the subprocess /
# backend orchestration modules, so the number here is the policy's
# yardstick — not the unit-only badge.
# Two modes:
#
# Usage:
# scripts/coverage.sh # combined report
# scripts/coverage.sh critical # also report just the critical modules
# scripts/coverage.sh [critical]
# Run mode (default, for local dev): executes the unit suite then the
# integration suite under coverage and prints a combined report.
#
# scripts/coverage.sh aggregate [critical]
# Aggregate mode (used by CI): combines pre-existing .coverage.* files
# produced by individual test jobs and prints a combined report. No tests
# are re-executed; no KVM or Docker dependency.
#
# Pass "critical" as the last argument in either mode to also report just the
# critical modules (ADR 0004 target: 90%).
set -euo pipefail
cd "$(dirname "$0")/.."
@@ -21,6 +25,22 @@ PY="${PYTHON:-python3}"
# README "core coverage" badge can't drift; comma-join it for --include.
CRITICAL=$(grep -vE '^[[:space:]]*(#|$)' scripts/critical-modules.txt | paste -sd, -)
if [ "${1:-}" = "aggregate" ]; then
# Aggregate mode: combine .coverage.* artifacts already in the workspace.
echo "== combining coverage artifacts ==" >&2
"$PY" -m coverage combine
echo "== combined report ==" >&2
"$PY" -m coverage report -m
if [ "${2:-}" = "critical" ]; then
echo "== critical modules (ADR 0004 target: 90%) ==" >&2
"$PY" -m coverage report --include="$CRITICAL"
fi
exit 0
fi
# Run mode (default): execute both suites under coverage in this process.
rm -f .coverage
echo "== unit ==" >&2
+20 -5
View File
@@ -2,14 +2,15 @@
Plain-Python test suite using stdlib `unittest`. No external
dependencies. Unit tests run anywhere Python 3 is present; integration
tests need Docker and skip cleanly otherwise.
tests run through the backend named by `BOT_BOTTLE_BACKEND` (default
`docker`) and skip cleanly when that backend isn't available on the host.
## Layout
```
tests/
fixtures.py # JSON manifest builders (shared)
_docker.py # docker-availability skip helper (shared)
_backend.py # backend selection + skip guards (shared)
unit/
test_egress.py
test_egress_addon_core.py
@@ -73,7 +74,7 @@ BOT_BOTTLE_RUN_CANARIES=1 python -m unittest discover -t . -s tests/canaries -v
## Adding a test
1. Pick the directory: `tests/unit/` for a pure unit test,
`tests/integration/` for one that needs Docker.
`tests/integration/` for one that needs a backend.
2. Filename: `test_<topic>.py`.
3. Boilerplate:
```python
@@ -88,5 +89,19 @@ BOT_BOTTLE_RUN_CANARIES=1 python -m unittest discover -t . -s tests/canaries -v
if __name__ == "__main__":
unittest.main()
```
4. For Docker-dependent tests, decorate the class with
`@skip_unless_docker()` from `tests._docker`.
4. Skip guards live in `tests._backend` and gate on the backend's own
readiness check, `bot_bottle.backend.has_backend` — the same probe
behind `./cli.py backend status`:
- Backend-agnostic tests (go through `get_bottle_backend()`) decorate
the class with `@skip_unless_selected_backend_available()` — the test
runs against whichever backend `BOT_BOTTLE_BACKEND` selects and skips
unless that backend is available (checking, e.g., Linux + `/dev/kvm`
for Firecracker rather than unrelated Docker availability).
- Backend-specific tests (exercise `DockerBroker`, `DockerGateway`,
`backend.docker.*`, …) decorate with `@skip_unless_backend("docker")`
so they no-op under a run targeting a different backend.
Each CI integration job runs `./cli.py backend status --backend=<name>`
as a preflight, which prints a clear per-check summary and exits non-zero
when the backend is missing — so absent infrastructure fails the job
instead of hiding among per-test `unittest.skip` lines.
+71
View File
@@ -0,0 +1,71 @@
"""Backend selection + readiness-aware skip guards for the integration suite.
Each integration test targets the backend named by ``BOT_BOTTLE_BACKEND``
(default ``docker``) and gates on that backend's full readiness check —
``is_backend_ready()`` (equivalent to ``./cli.py backend status``), not just
a binary-on-PATH probe. When the backend is not ready, diagnostic output is
printed during test discovery so the operator sees a concrete reason for each
skip.
"""
from __future__ import annotations
import os
import unittest
from bot_bottle.backend import is_backend_ready
# Default when ``BOT_BOTTLE_BACKEND`` is unset. Docker preserves the historical
# Docker-backed CI path (and mirrors the pin in ``test_sandbox_escape``).
DEFAULT_BACKEND = "docker"
def selected_backend() -> str:
"""The backend this test run targets, from ``BOT_BOTTLE_BACKEND``.
Mirrors the CLI's env selector; unset means ``docker`` so an
unconfigured run behaves exactly as the suite did before backends were
pluggable.
"""
return os.environ.get("BOT_BOTTLE_BACKEND") or DEFAULT_BACKEND
def skip_unless_backend(backend: str):
"""Skip a backend-specific test unless the selected backend matches AND
that backend is fully ready.
Docker-implementation tests (``DockerBroker``, ``DockerGateway``,
``backend.docker.*``) use ``skip_unless_backend("docker")`` so they no-op
under a run targeting a different backend instead of testing Docker
internals that run doesn't exercise — the guard reads
``BOT_BOTTLE_BACKEND`` rather than "is Docker installed".
When the backend is not ready, ``status()`` output is printed so the
operator sees a concrete diagnostic for each skipped test module.
"""
sel = selected_backend()
if sel != backend:
return unittest.skip(
f"backend {backend!r} not selected (BOT_BOTTLE_BACKEND={sel})"
)
return unittest.skipUnless(
is_backend_ready(backend, quiet=False),
f"{backend} backend not ready",
)
def skip_unless_selected_backend_available():
"""Skip a backend-agnostic test unless the *selected* backend is fully ready.
The test then runs through whichever backend ``BOT_BOTTLE_BACKEND`` names,
gated on that backend's full status() check (e.g. daemon reachable, TAP
pool present for Firecracker) rather than just a binary-on-PATH probe.
When the backend is not ready, ``status()`` output is printed so the
operator sees a concrete diagnostic for each skipped test module.
"""
backend = selected_backend()
return unittest.skipUnless(
is_backend_ready(backend, quiet=False),
f"selected backend {backend!r} not ready",
)
-45
View File
@@ -1,45 +0,0 @@
"""Docker availability check used by integration tests."""
from __future__ import annotations
import os
import shutil
import subprocess
import unittest
def docker_available() -> bool:
if os.environ.get("SKIP_DOCKER_TESTS"):
return False
if shutil.which("docker") is None:
return False
try:
return (
subprocess.run(
["docker", "info"],
stdout=subprocess.DEVNULL,
stderr=subprocess.DEVNULL,
check=False,
timeout=5,
).returncode
== 0
)
except subprocess.TimeoutExpired:
return False
def skip_unless_docker(reason: str = "docker unreachable"):
return unittest.skipUnless(docker_available(), reason)
def skip_unless_docker_or_firecracker(
reason: str = "neither Docker nor Firecracker selected",
):
"""Skip a backend-agnostic test unless one supported backend can run.
Firecracker does not require the host Docker daemon. The KVM coverage job
deliberately sets ``SKIP_DOCKER_TESTS`` to exclude Docker-only integration
classes while still exercising this path.
"""
firecracker_selected = os.environ.get("BOT_BOTTLE_BACKEND") == "firecracker"
return unittest.skipUnless(firecracker_selected or docker_available(), reason)
+2 -2
View File
@@ -25,14 +25,14 @@ import os
import subprocess
import unittest
from tests._docker import skip_unless_docker
from tests._backend import skip_unless_backend
_IMAGE = "bot-bottle-gateway-test:chunk1"
_DOCKERFILE = "Dockerfile.gateway"
@skip_unless_docker()
@skip_unless_backend("docker")
@unittest.skipIf(
os.environ.get("GITEA_ACTIONS") == "true",
"skipped under act_runner: multi-stage build pulls a 200+MB "
@@ -31,7 +31,7 @@ from bot_bottle.backend.docker.gateway_net import next_free_ip
from bot_bottle.orchestrator.client import OrchestratorClient
from bot_bottle.orchestrator.gateway import GATEWAY_IMAGE, GATEWAY_NAME, GATEWAY_NETWORK
from bot_bottle.orchestrator.lifecycle import OrchestratorService
from tests._docker import skip_unless_docker
from tests._backend import skip_unless_backend
# One upstream reachable under two names; reflects the Authorization header so
# the probe can see exactly what the gateway injected (or didn't).
@@ -72,7 +72,7 @@ _PROBE_SRC = (
)
@skip_unless_docker()
@skip_unless_backend("docker")
@unittest.skipIf(
os.environ.get("GITEA_ACTIONS") == "true",
"skipped under act_runner: the orchestrator container bind-mounts the repo "
@@ -13,12 +13,12 @@ import unittest
from bot_bottle.orchestrator.broker import LaunchRequest, sign_request
from bot_bottle.orchestrator.docker_broker import DockerBroker, container_name
from tests._docker import skip_unless_docker
from tests._backend import skip_unless_backend
IMAGE = "busybox"
@skip_unless_docker()
@skip_unless_backend("docker")
class TestDockerBrokerIntegration(unittest.TestCase):
def setUp(self) -> None:
self.secret = secrets.token_bytes(16)
@@ -28,15 +28,16 @@ from pathlib import Path
from bot_bottle.orchestrator.client import OrchestratorClient
from bot_bottle.orchestrator.lifecycle import OrchestratorService
from bot_bottle.paths import host_control_plane_token
from tests._docker import skip_unless_docker
from tests._backend import skip_unless_backend
# Fixed (not per-run-suffixed) so repeated runs reuse the same layer-cached
# image instead of leaking a new dangling tag on every invocation.
_TEST_ORCHESTRATOR_IMAGE = "bot-bottle-orchestrator:itest"
_TEST_GATEWAY_IMAGE = "bot-bottle-gateway:itest"
_TEST_INFRA_IMAGE = "bot-bottle-infra:itest"
@skip_unless_docker()
@skip_unless_backend("docker")
@unittest.skipIf(
os.environ.get("GITEA_ACTIONS") == "true",
"skipped under act_runner: the orchestrator container bind-mounts the repo "
@@ -69,20 +70,17 @@ class TestDockerControlPlaneAuthIntegration(unittest.TestCase):
os.environ["BOT_BOTTLE_ROOT"] = cls._tmp.name
cls.addClassCleanup(_restore_root)
orchestrator_name = f"bot-bottle-orch-itest-{suffix}"
gateway_name = f"bot-bottle-gw-itest-{suffix}"
infra_name = f"bot-bottle-infra-itest-{suffix}"
network = f"bot-bottle-net-itest-{suffix}"
host_root = Path(cls._tmp.name)
cls.addClassCleanup(
cls._teardown_docker, orchestrator_name, gateway_name, network, host_root
cls._teardown_docker, infra_name, network, host_root
)
cls.svc = OrchestratorService(
orchestrator_name=orchestrator_name,
gateway_name=gateway_name,
infra_name=infra_name,
network=network,
image=_TEST_ORCHESTRATOR_IMAGE,
gateway_image=_TEST_GATEWAY_IMAGE,
image=_TEST_INFRA_IMAGE,
port=20000 + secrets.randbelow(10000),
host_root=host_root,
)
@@ -91,23 +89,23 @@ class TestDockerControlPlaneAuthIntegration(unittest.TestCase):
@staticmethod
def _teardown_docker(
orchestrator_name: str, gateway_name: str, network: str, host_root: Path
infra_name: str, network: str, host_root: Path
) -> None:
subprocess.run(
["docker", "rm", "--force", orchestrator_name, gateway_name],
["docker", "rm", "--force", infra_name],
stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL, check=False,
)
subprocess.run(
["docker", "network", "rm", network],
stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL, check=False,
)
# The orchestrator container (no USER directive) wrote the registry
# The infra container (no USER directive) wrote the registry
# DB as root into the throwaway host_root; chown it back so the
# (non-root) tempdir cleanup can remove it. Same workaround
# test_multitenant_isolation.py uses for the identical bind mount.
subprocess.run(
["docker", "run", "--rm", "-v", f"{host_root}:/r",
"--entrypoint", "chown", _TEST_GATEWAY_IMAGE, "-R",
"--entrypoint", "chown", _TEST_INFRA_IMAGE, "-R",
f"{os.getuid()}:{os.getgid()}", "/r"],
stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL, check=False,
)
@@ -11,12 +11,12 @@ import subprocess
import unittest
from bot_bottle.orchestrator.gateway import DockerGateway
from tests._docker import skip_unless_docker
from tests._backend import skip_unless_backend
IMAGE = "busybox"
@skip_unless_docker()
@skip_unless_backend("docker")
class TestDockerGatewayIntegration(unittest.TestCase):
def setUp(self) -> None:
self.name = "bot-bottle-orch-gateway-itest-" + secrets.token_hex(4)
@@ -12,12 +12,12 @@ import subprocess
import unittest
from bot_bottle.orchestrator.gateway import DockerGateway
from tests._docker import skip_unless_docker
from tests._backend import skip_unless_backend
IMAGE = "busybox"
@skip_unless_docker()
@skip_unless_backend("docker")
class TestDockerGatewayImageExists(unittest.TestCase):
def test_image_exists_true_for_present_false_for_absent(self) -> None:
# Ensure the tiny image is present (build_if_missing is disabled here
+2 -2
View File
@@ -18,10 +18,10 @@ from bot_bottle.backend.docker.network import (
network_create_internal,
network_remove,
)
from tests._docker import skip_unless_docker
from tests._backend import skip_unless_backend
@skip_unless_docker()
@skip_unless_backend("docker")
class TestOrphanCleanup(unittest.TestCase):
def setUp(self):
self.slug = f"cb-test-orphan-{os.getpid()}"
+3 -3
View File
@@ -31,7 +31,7 @@ from pathlib import Path
from bot_bottle.backend import BottleSpec, get_bottle_backend
from bot_bottle.bottle_state import cleanup_state
from bot_bottle.manifest import ManifestIndex
from tests._docker import skip_unless_docker_or_firecracker
from tests._backend import skip_unless_selected_backend_available
# Secrets planted in the bottle env as literals (agents substitute via
@@ -67,7 +67,7 @@ _DUMMY_HOST_KEY = (
)
@skip_unless_docker_or_firecracker()
@skip_unless_selected_backend_available()
@unittest.skipIf(
os.environ.get("GITEA_ACTIONS") == "true"
and os.environ.get("BOT_BOTTLE_BACKEND") != "firecracker",
@@ -172,7 +172,7 @@ class TestSandboxEscape(unittest.TestCase):
# base image without producing five confusing
# command-not-found failures down the suite.
missing: list[str] = []
for tool in ("curl", "git", "dig"):
for tool in ("curl", "git", "dig", "ssh"):
r = cls._bottle.exec(f"command -v {tool} >/dev/null 2>&1")
if r.returncode != 0:
missing.append(tool)
+3 -1
View File
@@ -46,7 +46,9 @@ def _manifest(*, supervise: bool, with_git: bool, with_egress: bool) -> Manifest
bottle["egress"] = {
"routes": [{
"host": "api.example",
"auth": {"scheme": "Bearer", "token_ref": "TOK"},
"inspect": {
"auth": {"scheme": "Bearer", "token_ref": "TOK"},
},
}],
}
return ManifestIndex.from_json_obj({
+66 -1
View File
@@ -9,11 +9,15 @@ import unittest
from pathlib import Path
from bot_bottle.agent_provider import (
CLAUDE_HOST_CREDENTIAL_HOSTS,
CODEX_HOST_CREDENTIAL_HOSTS,
build_agent_provision_plan,
prompt_args,
)
from bot_bottle.egress import CODEX_HOST_CREDENTIAL_TOKEN_REF
from bot_bottle.egress import (
CLAUDE_HOST_CREDENTIAL_TOKEN_REF,
CODEX_HOST_CREDENTIAL_TOKEN_REF,
)
def _jwt(exp: int) -> str:
@@ -292,6 +296,67 @@ class TestAgentProviderRuntime(unittest.TestCase):
)
self.assertEqual({}, plan.provisioned_env)
def test_claude_forward_host_credentials_populates_egress_route(self):
access_token = "sk-ant-oat01-test-key" # gitleaks:allow
with tempfile.TemporaryDirectory(prefix="bb-provider.") as tmp:
home = Path(tmp) / "host-claude"
cred_dir = home / ".claude"
cred_dir.mkdir(parents=True)
(cred_dir / ".credentials.json").write_text(json.dumps({
"claudeAiOauth": {"accessToken": access_token},
}))
plan = build_agent_provision_plan(
template="claude",
dockerfile="",
state_dir=Path(tmp),
instance_name="bot-bottle-test",
prompt_file=Path(tmp) / "prompt.txt",
forward_host_credentials=True,
host_env={"HOME": str(home)},
)
self.assertEqual(1, len(plan.egress_routes))
route = plan.egress_routes[0]
self.assertIn(route.host, CLAUDE_HOST_CREDENTIAL_HOSTS)
self.assertEqual("Bearer", route.auth_scheme)
self.assertEqual(CLAUDE_HOST_CREDENTIAL_TOKEN_REF, route.token_ref)
self.assertEqual("egress-placeholder", plan.env_vars["CLAUDE_CODE_OAUTH_TOKEN"])
self.assertEqual(frozenset({"CLAUDE_CODE_OAUTH_TOKEN"}), plan.hidden_env_names)
def test_claude_forward_host_credentials_populates_provisioned_env(self):
access_token = "sk-ant-oat01-test-key" # gitleaks:allow
with tempfile.TemporaryDirectory(prefix="bb-provider.") as tmp:
home = Path(tmp) / "host-claude"
cred_dir = home / ".claude"
cred_dir.mkdir(parents=True)
(cred_dir / ".credentials.json").write_text(json.dumps({
"claudeAiOauth": {"accessToken": access_token},
}))
plan = build_agent_provision_plan(
template="claude",
dockerfile="",
state_dir=Path(tmp),
instance_name="bot-bottle-test",
prompt_file=Path(tmp) / "prompt.txt",
forward_host_credentials=True,
host_env={"HOME": str(home)},
)
self.assertEqual(
{CLAUDE_HOST_CREDENTIAL_TOKEN_REF: access_token},
plan.provisioned_env,
)
def test_claude_without_forward_host_credentials_has_empty_provisioned_env(self):
with tempfile.TemporaryDirectory(prefix="bb-provider.") as tmp:
plan = build_agent_provision_plan(
template="claude",
dockerfile="",
state_dir=Path(tmp),
instance_name="bot-bottle-test",
prompt_file=Path(tmp) / "prompt.txt",
forward_host_credentials=False,
)
self.assertEqual({}, plan.provisioned_env)
def test_pi_plan_writes_default_ollama_models(self):
with tempfile.TemporaryDirectory(prefix="bb-provider.") as tmp:
plan = build_agent_provision_plan(
+42 -3
View File
@@ -21,7 +21,7 @@ from bot_bottle.backend.firecracker import FirecrackerBottleBackend
from bot_bottle.manifest import ManifestIndex
def _manifest() -> ManifestIndex:
def _manifest(*, nested_containers: bool = False) -> ManifestIndex:
return ManifestIndex.from_json_obj({
"bottles": {
"dev": {
@@ -29,6 +29,7 @@ def _manifest() -> ManifestIndex:
"LITERAL_ENV": "literal-value",
"FORWARDED_ENV": "${HOST_SECRET_ENV}",
},
"nested_containers": nested_containers,
},
},
"agents": {
@@ -41,9 +42,11 @@ def _manifest() -> ManifestIndex:
})
def _spec(tmp: Path, *, identity: str) -> BottleSpec:
def _spec(
tmp: Path, *, identity: str, nested_containers: bool = False,
) -> BottleSpec:
return BottleSpec(
manifest=_manifest(),
manifest=_manifest(nested_containers=nested_containers),
agent_name="demo",
copy_cwd=False,
user_cwd=str(tmp),
@@ -113,6 +116,42 @@ class TestFirecrackerPrepare(_FakeStateMixin, unittest.TestCase):
self.assertEqual({"FORWARDED_ENV": "secret-value"}, plan.forwarded_env)
class TestNestedContainersRejection(_FakeStateMixin, unittest.TestCase):
"""A backend with no guest-local engine must refuse the flag outright.
Ignoring it would leave the agent without `docker`, and the only ways to
fake it here (a host daemon socket, a privileged container) are what
issue #392 rules out.
"""
def test_docker_backend_refuses_the_flag(self) -> None:
backend = DockerBottleBackend()
spec = _spec(
Path(self.tmp.name), identity="demo-docker", nested_containers=True,
)
with (
patch(
"bot_bottle.backend.resolve_common.die", side_effect=RuntimeError,
) as die,
self.assertRaises(RuntimeError),
):
backend.prepare(spec, Path(self.tmp.name) / "stage")
self.assertIn("nested_containers", die.call_args.args[0])
self.assertIn("docker", die.call_args.args[0])
def test_firecracker_backend_refuses_the_flag(self) -> None:
backend = FirecrackerBottleBackend()
spec = _spec(Path(self.tmp.name), identity="demo-fc", nested_containers=True)
with (
patch(
"bot_bottle.backend.resolve_common.die", side_effect=RuntimeError,
) as die,
self.assertRaises(RuntimeError),
):
backend.prepare(spec, Path(self.tmp.name) / "stage")
self.assertIn("firecracker", die.call_args.args[0])
class TestMintSlug(unittest.TestCase):
def _spec(self, *, label: str = "", identity: str = "") -> BottleSpec:
manifest = _manifest()

Some files were not shown because too many files have changed in this diff Show More