Compare commits

..

3 Commits

Author SHA1 Message Date
didericis ec3b85ac1b test(egress): set orchestrator URL for entrypoint tests; cover fail-closed guard
lint / lint (push) Successful in 2m21s
test / unit (pull_request) Failing after 1m9s
test / integration (pull_request) Successful in 27s
test / coverage (pull_request) Failing after 1m9s
The egress_entrypoint.sh fail-closed guard (this branch) exits 1 when
BOT_BOTTLE_ORCHESTRATOR_URL is unset, which broke the argv-construction
tests that ran the script without it. Set the URL in the shared
_run_entrypoint helper (a precondition for reaching mitmdump now, like
PATH) and add a test asserting the guard fails closed when it's absent.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UoEZHDjv84ChoZbozQERhJ
2026-07-17 22:03:35 -04:00
didericis 3fed570da6 refactor(gateway): fail closed without an orchestrator URL; drop stale single-tenant comments
Follow-ups from the #402 review of the single-tenant data-plane teardown.

- egress_entrypoint.sh: refuse to launch mitmdump when
  BOT_BOTTLE_ORCHESTRATOR_URL is unset, so the fail-closed guarantee no
  longer rests solely on mitmproxy's errorcheck addon exiting on the
  addon's load-time raise. A misconfigured gateway can never come up as
  a bare TLS-bumping open proxy with no policy.
- orchestrator/gateway.py: ensure_running() raises GatewayError on an
  empty orchestrator URL — a URL-less launch would only crash-loop the
  now-resolver-only daemons (egress raises, git-http exits 1, supervise
  exits 2). The env-injection branch is now unconditional.
- Drop stale "single-tenant" / "reads routes.yaml" comments in
  gateway.py and egress_entrypoint.sh, and the /etc/egress/routes.yaml
  layout line in Dockerfile.gateway.
- Tests: gateway fixtures supply an orchestrator URL; add a
  refuse-without-URL test and assert the URL env is injected.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UoEZHDjv84ChoZbozQERhJ
2026-07-17 22:03:35 -04:00
didericis 4e7df6206e refactor(gateway): remove the single-tenant data-plane paths (audit #400 finding 3)
All three backends (docker, firecracker, macos-container) now launch through
the consolidated orchestrator, and every production gateway sets
BOT_BOTTLE_ORCHESTRATOR_URL — so the legacy single-tenant (`resolver is None`)
branches in the shared gateway's data plane were unreachable dead code, a second
security-relevant path to keep correct in parallel with the live one. Make the
orchestrator resolver mandatory and delete the single-tenant paths from the
three data-plane modules.

egress_addon.py: drop the static routes file entirely — EGRESS_ROUTES, _reload,
the SIGHUP handler, self.config, and the SUPERVISE_BOTTLE_SLUG env slug. The
per-request /resolve is the only policy source; __init__ fail-closes if
BOT_BOTTLE_ORCHESTRATOR_URL is unset. Introspection (`_egress.local/allowlist`)
now reports the calling bottle's *resolved* routes. The block/redact log gates
and _req_ctx redaction now read the per-flow config/env from the request-time
stash, so they use each bottle's log level and token set (they silently used the
empty static config before). Nothing sends `docker kill --signal HUP` to the
gateway in the consolidated model (the egress applicators fail closed), so
removing the SIGHUP reload is safe.

git_http_backend.py: resolver mandatory; no flat-root fallback. main() refuses
to start without an orchestrator URL; a request whose source resolves to no
bottle 404s.

supervise_server.py: resolver mandatory; every proposal is attributed to the
source-IP-resolved bottle. Remove handle_list_egress_routes (the proxy-fetch
introspection that only worked when the proxy carried one bottle's identity) —
list-egress-routes is answered from the resolved policy. main() refuses to start
without an orchestrator URL.

Tests: a host-side fake resolver serves each test's Config through the real
parse path (a small YAML-subset emitter round-trips route_to_yaml_dict); the
response/websocket hooks stash it as request() would. Deletes the tests for the
removed static-config, SIGHUP-reload, and single-tenant-passthrough paths; adds
fail-closed-without-orchestrator coverage.

Follow-up: gateway_init still forwards SIGHUP to the egress child (now dormant —
no one sends it); the README still describes the docker backend's per-bottle
topology. Both are outside the data-plane teardown.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UoEZHDjv84ChoZbozQERhJ
2026-07-17 22:03:35 -04:00
308 changed files with 6522 additions and 21591 deletions
-4
View File
@@ -1,10 +1,6 @@
[run]
branch = True
source = .
# Store paths relative to the project root so .coverage.* files produced on
# different runners (ubuntu-latest vs self-hosted KVM) can be combined by the
# coverage job without a [paths] remapping section.
relative_files = True
[report]
# Coverage policy: see docs/decisions/0004-coverage-policy.md.
+5 -2
View File
@@ -22,7 +22,10 @@ jobs:
- name: Checkout
uses: actions/checkout@v4
# No actions/setup-python: canaries are stdlib unittest on the image's
# system Python 3.12 (older act_runner mishandles setup-python's PATH).
- name: Set up Python
uses: actions/setup-python@v5
with:
python-version: "3.12"
- name: Run canaries
run: python3 -m unittest discover -t . -s tests/canaries -v
+10 -20
View File
@@ -13,30 +13,20 @@ jobs:
steps:
- uses: actions/checkout@v3
# No actions/setup-python: the runner image already ships Python 3.12,
# and older act_runner engines mishandle setup-python's PATH. Install
# into the ephemeral job container's system Python — the pylint/pyright
# console scripts land on /usr/local/bin (on PATH) so the steps below
# still resolve. --break-system-packages is safe: the container is
# disposable.
- name: Set up Python
uses: actions/setup-python@v4
with:
python-version: "3.12"
- name: Install dev dependencies
run: python3 -m pip install --break-system-packages -r requirements-dev.txt
run: |
python -m pip install --upgrade pip
pip install -r requirements-dev.txt
- name: Run pylint
run: |
# Pylint's normal exit code is nonzero for any emitted finding,
# regardless of --fail-under. Preserve the full report but enforce
# the aggregate score this workflow promises.
set +e
find . -name '*.py' -not -path './.venv/*' -not -path './.git/*' \
| xargs pylint --fail-under=8.0 \
| tee /tmp/pylint-output.txt
set -e
SCORE=$(sed -n \
's/^Your code has been rated at \([-0-9.]*\)\/10.*/\1/p' \
/tmp/pylint-output.txt | tail -1)
test -n "$SCORE"
awk -v score="$SCORE" 'BEGIN { exit !(score >= 8.0) }'
# Run pylint on all Python files in the repo
find . -name '*.py' -not -path './.venv/*' -not -path './.git/*' | xargs pylint --fail-under=8.0
- name: Run pyright
run: |
+5 -2
View File
@@ -37,8 +37,11 @@ jobs:
fetch-depth: 0
token: ${{ secrets.GITHUB_TOKEN }}
# No actions/setup-python: the inline script is stdlib-only on the
# image's system Python 3.12 (older act_runner mishandles its PATH).
- name: Set up Python
uses: actions/setup-python@v5
with:
python-version: "3.12"
- name: Configure git
run: |
git config user.name "github-actions[bot]"
+46 -252
View File
@@ -1,21 +1,19 @@
# Run the project's test suite when package or runtime inputs change on a PR
# or on push to main.
# Run the project's test suite on every PR push and on push to main.
#
# The suite uses stdlib `unittest` discovery — no external Python
# dependencies are required to execute it. Tests are split by directory:
#
# tests/unit/ — pure unit tests; always run
# tests/integration/ — need a reachable backend; skip cleanly when
# the backend isn't available on the runner
# tests/integration/ — need a reachable Docker daemon; skip cleanly
# (via tests/_docker.py:skip_unless_docker) when
# Docker isn't available on the runner
# tests/canaries/ — upstream regression canaries; run on a separate
# schedule (see canaries.yml), not here
#
# Each test job runs once under coverage and uploads a small .coverage.*
# artifact. The `coverage` job combines them — no test reruns, no KVM
# dependency on that job. For main-branch pushes only, the tested rootfs
# and matching dropbear are uploaded so `publish-infra` can publish the
# byte-identical artifact that was tested. PRs avoid the ~194 MB rootfs
# transfer entirely.
# This workflow assumes the Gitea Actions runner exposes the host Docker
# socket to the job container so `docker` commands inside the job can
# reach the daemon. If that's not yet configured on the runner the
# integration tests will skip rather than fail.
name: test
@@ -24,35 +22,10 @@ on:
branches:
- main
paths:
- 'bot_bottle/**'
- 'tests/**/*.py'
- 'cli.py'
- 'scripts/coverage.sh'
- 'scripts/critical-modules.txt'
- 'scripts/diff_coverage.py'
- 'scripts/tracker_policy.py'
- 'scripts/firecracker-netpool.sh'
- 'Dockerfile*'
- 'pyproject.toml'
- 'requirements-dev.txt'
- '.coveragerc'
- '.dockerignore'
- '**.py'
pull_request:
paths:
- 'bot_bottle/**'
- 'tests/**/*.py'
- 'cli.py'
- 'scripts/coverage.sh'
- 'scripts/critical-modules.txt'
- 'scripts/diff_coverage.py'
- 'scripts/tracker_policy.py'
- 'scripts/firecracker-netpool.sh'
- 'Dockerfile*'
- 'pyproject.toml'
- 'requirements-dev.txt'
- '.coveragerc'
- '.dockerignore'
workflow_dispatch:
- '**.py'
jobs:
unit:
@@ -61,246 +34,67 @@ jobs:
- name: Checkout
uses: actions/checkout@v4
# No actions/setup-python: the runner image already ships Python 3.12,
# and older act_runner engines mishandle setup-python's PATH (coverage
# lands in one interpreter, `python3` resolves to another). Install
# straight into the ephemeral job container's system Python —
# --break-system-packages is safe because the container is disposable.
- name: Install dev requirements
run: python3 -m pip install --break-system-packages -r requirements-dev.txt
- name: Set up Python
uses: actions/setup-python@v5
with:
python-version: "3.12"
- name: Run unit tests with coverage
env:
COVERAGE_FILE: ${{ github.workspace }}/.coverage.unit
- name: Install dev requirements
run: python3 -m pip install -r requirements-dev.txt
- name: Run unit tests
run: python3 -m coverage run -m unittest discover -t . -s tests/unit -v
- name: Report unit coverage
env:
COVERAGE_FILE: ${{ github.workspace }}/.coverage.unit
run: python3 -m coverage report -m
# upload-artifact@v3's glob skips dotfiles, so a bare `.coverage.unit`
# silently uploads nothing ("No files were found"). Stage it under a
# non-dot name; the coverage job renames it back before `coverage
# combine`. `cp` also fails loudly if coverage never wrote the file.
- name: Stage unit coverage for upload
run: cp .coverage.unit coverage-unit.dat
- name: Upload unit coverage artifact
uses: actions/upload-artifact@v3
with:
name: coverage-unit
path: coverage-unit.dat
integration-docker:
integration:
runs-on: ubuntu-latest
steps:
- name: Checkout
uses: actions/checkout@v4
# No actions/setup-python (see the note in the `unit` job); the
# container's system Python 3.12 runs the stdlib test suite directly.
- name: Install coverage
run: python3 -m pip install --break-system-packages coverage
- name: Set up Python
uses: actions/setup-python@v5
with:
python-version: "3.12"
# Fail loudly if the backend this job promises isn't actually usable,
# rather than letting every test silently `unittest.skip` and the job
# go green on zero coverage. `backend status` prints a clear per-check
# summary (docker on PATH, daemon reachable) and exits non-zero when a
# prerequisite is missing — the same readiness check the skip guards
# gate on via `has_backend`.
- name: Preflight — Docker backend is ready
- name: Show environment
run: |
python3 --version
python3 cli.py backend status --backend=docker
if command -v docker >/dev/null 2>&1; then
docker version || true
else
echo "docker not on PATH — integration tests will skip"
fi
- name: Run integration tests (docker) with coverage
env:
BOT_BOTTLE_BACKEND: docker
COVERAGE_FILE: ${{ github.workspace }}/.coverage.docker
run: python3 -m coverage run -m unittest discover -t . -s tests/integration -v
- name: Run integration tests
run: python3 -m unittest discover -t . -s tests/integration -v
# Non-dot name so upload-artifact's dotfile-skipping glob picks it up.
- name: Stage docker coverage for upload
run: cp .coverage.docker coverage-docker.dat
- name: Upload docker coverage artifact
uses: actions/upload-artifact@v3
with:
name: coverage-docker
path: coverage-docker.dat
# Integration tests against the Firecracker backend. Runs on a self-hosted
# KVM runner (label `kvm`) where /dev/kvm and the TAP/nft pool are available.
# Combined unit+integration coverage report (informational). See
# docs/decisions/0004-coverage-policy.md.
#
# Restricted to same-repo PRs, push to main, and workflow_dispatch — fork
# PRs don't execute untrusted code on the privileged runner.
#
# Runner prerequisites (provision once; see README "Firecracker on Linux"):
# `firecracker` on PATH, `/dev/kvm` accessible, cached kernel +
# static dropbear at /var/cache/bot-bottle-fc/dropbear, and the pool as a
# persistent systemd unit.
#
# The infra candidate is built here directly (no artifact download) to
# eliminate the ~70 s ubuntu-latest upload + ~83 s combined download that
# the old build-infra → integration-firecracker + coverage chain incurred.
# For main-branch pushes the tested rootfs and matching dropbear are
# uploaded so publish-infra can publish the byte-identical artifact; PRs
# skip those uploads entirely.
integration-firecracker:
runs-on: [self-hosted, kvm]
if: >-
github.event_name == 'push' ||
github.event_name == 'workflow_dispatch' ||
(github.event_name == 'pull_request' &&
github.event.pull_request.head.repo.full_name == github.repository)
steps:
- name: Checkout
uses: actions/checkout@v4
- name: Preflight — Firecracker host is ready
run: |
command -v firecracker >/dev/null || {
echo "firecracker not on PATH — provision the runner (README: Firecracker on Linux)"; exit 1; }
test -e /dev/kvm || { echo "/dev/kvm missing — KVM not available on this runner"; exit 1; }
# `backend status` exits non-zero unless the TAP pool is up + no
# range overlap; it prints the exact `backend setup` fix.
python3 cli.py backend status --backend=firecracker
- name: Build infra candidate from this checkout
env:
BOT_BOTTLE_FC_DROPBEAR: /var/cache/bot-bottle-fc/dropbear
run: python3 -m bot_bottle.backend.firecracker.publish_infra --output infra-candidate --reuse-published
- name: Replace the persistent infra VM with the candidate
run: python3 -c 'from bot_bottle.backend.firecracker import infra_vm; infra_vm.stop()'
# No dev-requirements install: `coverage` is already provided by the
# self-hosted runner's Nix python env, and that env has no `pip`
# module to install into anyway.
- name: Run integration tests (firecracker) with coverage
env:
BOT_BOTTLE_BACKEND: firecracker
BOT_BOTTLE_INFRA_ARTIFACT_DIR: ${{ github.workspace }}/infra-candidate
COVERAGE_FILE: ${{ github.workspace }}/.coverage.firecracker
run: python3 -m coverage run -m unittest discover -t . -s tests/integration -v
# Non-dot name so upload-artifact's dotfile-skipping glob picks it up.
- name: Stage firecracker coverage for upload
run: cp .coverage.firecracker coverage-firecracker.dat
- name: Upload firecracker coverage artifact
uses: actions/upload-artifact@v3
with:
name: coverage-firecracker
path: coverage-firecracker.dat
# Only upload the large rootfs artifact on main-branch pushes;
# PRs avoid the ~194 MB transfer. publish-infra only runs on main
# and downloads these to publish the byte-identical tested rootfs.
- name: Upload tested rootfs (main branch only)
if: github.event_name == 'push' && github.ref == 'refs/heads/main'
uses: actions/upload-artifact@v3
with:
name: infra-candidate
path: infra-candidate/
- name: Upload dropbear for publish verification (main branch only)
if: github.event_name == 'push' && github.ref == 'refs/heads/main'
uses: actions/upload-artifact@v3
with:
name: firecracker-inputs
path: /var/cache/bot-bottle-fc/dropbear
# Combined coverage gate: aggregates .coverage.* artifacts uploaded by each
# test job, then runs the diff-coverage gate (new/changed lines >= 90%).
#
# Runs on ubuntu-latest — no KVM needed, no test reruns. Coverage files use
# relative_files = True (.coveragerc) so they combine cleanly across runners.
# Each test job sets COVERAGE_FILE to an absolute path so coverage.py writes
# to a known location that upload-artifact can find regardless of runner env.
#
# Restricted to the same events as integration-firecracker: it depends on
# that job's coverage artifact and skips for fork PRs alongside it.
# The hard diff-coverage gate (changed lines >= 90%) is DEFERRED: the
# Firecracker backend's VM/SSH orchestration is covered by the integration
# suite, which needs /dev/kvm + the provisioned TAP/nft pool — a
# container-based runner skips it and those lines read uncovered, so the
# gate can't pass here. Re-enabling it on a self-hosted KVM runner is
# tracked separately (see PRD 0069 / #348 and the ci-runner branch).
coverage:
needs: [unit, integration-docker, integration-firecracker]
timeout-minutes: 15
runs-on: ubuntu-latest
if: >-
github.event_name == 'push' ||
github.event_name == 'workflow_dispatch' ||
(github.event_name == 'pull_request' &&
github.event.pull_request.head.repo.full_name == github.repository)
steps:
- name: Checkout
uses: actions/checkout@v4
with:
fetch-depth: 0
- name: Install coverage
run: python3 -m pip install --break-system-packages coverage
- name: Download unit coverage artifact
uses: actions/download-artifact@v3
- name: Set up Python
uses: actions/setup-python@v5
with:
name: coverage-unit
path: ${{ github.workspace }}
python-version: "3.12"
- name: Download docker coverage artifact
uses: actions/download-artifact@v3
with:
name: coverage-docker
path: ${{ github.workspace }}
- name: Install dev requirements
run: python3 -m pip install -r requirements-dev.txt
- name: Download firecracker coverage artifact
uses: actions/download-artifact@v3
with:
name: coverage-firecracker
path: ${{ github.workspace }}
# Rename the non-dot upload names back to the .coverage.* files that
# `coverage combine` discovers (see the staging steps in each test job).
- name: Reassemble coverage data files
run: |
mv coverage-unit.dat .coverage.unit
mv coverage-docker.dat .coverage.docker
mv coverage-firecracker.dat .coverage.firecracker
- name: Combined coverage (unit + integration, incl. firecracker)
run: PYTHON=python3 bash scripts/coverage.sh aggregate critical
- name: Diff-coverage gate (changed lines >= 90%)
run: |
git fetch --no-tags origin main:refs/remotes/origin/main
python3 scripts/diff_coverage.py --base origin/main --min 90
publish-infra:
needs: [unit, integration-docker, integration-firecracker, coverage]
runs-on: ubuntu-latest
if: github.event_name == 'push' && github.ref == 'refs/heads/main'
steps:
- name: Checkout the tested revision
uses: actions/checkout@v4
- name: Download the tested rootfs
uses: actions/download-artifact@v3
with:
name: infra-candidate
path: infra-candidate
# publish_infra re-derives the version from the checkout to confirm the
# bundle matches before uploading, and the version hashes the dropbear
# bytes. Download the SAME dropbear integration-firecracker used, or
# the recheck computes a "<missing>"-dropbear version and rejects the
# candidate.
- name: Download the staged dropbear (matches build's version)
uses: actions/download-artifact@v3
with:
name: firecracker-inputs
path: firecracker-inputs
- name: Publish the tested candidate
env:
BOT_BOTTLE_INFRA_ARTIFACT_TOKEN: ${{ secrets.BOT_BOTTLE_INFRA_ARTIFACT_TOKEN }}
BOT_BOTTLE_FC_DROPBEAR: ${{ github.workspace }}/firecracker-inputs/dropbear
run: python3 -m bot_bottle.backend.firecracker.publish_infra --publish-dir infra-candidate
- name: Combined coverage report (unit + integration)
run: PYTHON=python3 bash scripts/coverage.sh critical
@@ -1,17 +0,0 @@
name: tracker-policy-issues
on:
issues:
types: [opened, unlabeled]
jobs:
label-issue:
runs-on: ubuntu-latest
permissions:
issues: write
steps:
- uses: actions/checkout@v4
- name: Ensure the issue has a label
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
run: python3 scripts/tracker_policy.py label-issue
-18
View File
@@ -1,18 +0,0 @@
name: tracker-policy-pr
on:
pull_request:
types: [opened, edited, reopened, synchronize, labeled, unlabeled]
jobs:
check-pr:
runs-on: ubuntu-latest
permissions:
issues: read
pull-requests: read
steps:
- uses: actions/checkout@v4
- name: Require an unlabeled PR linked to an issue
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
run: python3 scripts/tracker_policy.py check-pr
+12 -12
View File
@@ -14,27 +14,27 @@ on:
jobs:
update-badges:
runs-on: ubuntu-latest
permissions:
contents: write
steps:
- uses: actions/checkout@v3
with:
fetch-depth: 0
token: ${{ secrets.BADGE_PUSH_TOKEN }}
token: ${{ secrets.GITHUB_TOKEN }}
- name: Set up Python
uses: actions/setup-python@v4
with:
python-version: '3.12'
# No actions/setup-python: the runner image ships Python 3.12 and older
# act_runner engines mishandle setup-python's PATH. Install into the
# ephemeral job container's system Python (--break-system-packages is
# safe because the container is disposable).
- name: Install dev dependencies
run: python3 -m pip install --break-system-packages -r requirements-dev.txt
run: |
python -m pip install --upgrade pip
pip install -r requirements-dev.txt
- name: Run coverage and extract percentage
id: coverage
run: |
python3 -m coverage run -m unittest discover -t . -s tests/unit > /dev/null 2>&1 || true
PERCENT=$(python3 -m coverage report 2>/dev/null | grep '^TOTAL' | grep -oP '\d+(?=%)' | tail -1)
python -m coverage run -m unittest discover -t . -s tests/unit > /dev/null 2>&1 || true
PERCENT=$(python -m coverage report 2>/dev/null | grep '^TOTAL' | grep -oP '\d+(?=%)' | tail -1)
echo "percent=$PERCENT" >> $GITHUB_OUTPUT
echo "Coverage: $PERCENT%"
@@ -45,7 +45,7 @@ jobs:
# the single source of truth in scripts/critical-modules.txt; every
# core module is unit-tested, so the unit-only run is accurate for it.
INCLUDE=$(grep -vE '^[[:space:]]*(#|$)' scripts/critical-modules.txt | paste -sd, -)
PERCENT=$(python3 -m coverage report --include="$INCLUDE" 2>/dev/null | grep '^TOTAL' | grep -oP '\d+(?=%)' | tail -1)
PERCENT=$(python -m coverage report --include="$INCLUDE" 2>/dev/null | grep '^TOTAL' | grep -oP '\d+(?=%)' | tail -1)
echo "percent=$PERCENT" >> $GITHUB_OUTPUT
echo "Core coverage: $PERCENT%"
-1
View File
@@ -35,7 +35,6 @@ backend remains available with `BOT_BOTTLE_BACKEND=docker` or
- `.bot-bottle/` — per-repo agent and bottle manifests (YAML markdown format).
- `examples/` — example bottles and agents showing the manifest format.
- `docs/README.md` — docs overview; when to write which document.
- `docs/glossary.md` — canonical term definitions (Agent Provider, Bottle, Sealed Bottle, etc.).
- `docs/prds/` — product requirement docs (see `docs/prds/README.md` for format).
- `docs/research/` — research notes (see `docs/research/README.md`).
- `docs/decisions/` — decision records (ADR-lite).
+42 -44
View File
@@ -8,27 +8,24 @@
# this image's mitmproxy / git / gitleaks payload.
#
# Collapses the prior per-daemon images (egress, git-gate,
# supervise) into one. A small stdlib-Python init supervisor
# (`bot_bottle.gateway.bootstrap`) spawns all daemons, forwards SIGTERM, and
# supervise) into one. A small stdlib-Python init supervisor at
# /app/gateway_init.py spawns all daemons, forwards SIGTERM, and
# propagates per-daemon stdout/stderr to the container log with a
# `[name]` prefix. See PRD 0024 for the rationale.
#
# Layout:
#
# /usr/bin/gitleaks gitleaks binary
# /app/egress_addon.py mitmproxy addon entry point
# /app/egress_addon.py + siblings mitmproxy addon (egress)
# /app/egress-entrypoint.sh mitmdump launcher
# /usr/local/lib/python*/bot_bottle/ installed package (all daemons + shared modules)
# /app/egress_addon.py one-line shim: re-exports addons from package
# (mitmdump -s requires a file path, not a module)
# /etc/egress/routes.yaml bind-mounted at run time
# /app/supervise_server.py + .py supervise MCP server
# /app/gateway_init.py PID 1 supervisor
# /etc/git-gate/pre-receive docker-cp'd at start time
# /git-gate-entrypoint.sh docker-cp'd at start time
# /git-gate/creds/* docker-cp'd at start time
# /git/* bare repos, populated at runtime
# /run/supervise/bot-bottle.db bind-mounted at run time
# /home/mitmproxy/.mitmproxy/ mitmproxy CA dir
# (No bot-bottle.db mount: the data plane reaches the supervise queue over the
# control-plane RPC and never opens the DB — PRD 0070 / issue #469.)
#
# Exposed ports inside the container:
# 9099 egress (mitmproxy, agent-facing HTTPS proxy)
@@ -37,9 +34,12 @@
# 9100 supervise (MCP HTTP)
# Based on `python:3.12-slim` (Debian trixie) rather than the
# `mitmproxy/mitmproxy` image (Debian bookworm), matching the trixie base the
# orchestrator image needs for buildah (Dockerfile.orchestrator.fc). mitmproxy
# is pip-installed to the same effect as the upstream image.
# `mitmproxy/mitmproxy` image (Debian bookworm) so the whole stack —
# gateway here, and the firecracker infra image that builds FROM this —
# lands on trixie, whose buildah (1.39) can build agent Dockerfiles that
# use heredocs. mitmproxy is pip-installed to the same effect as the
# upstream image. (bookworm's buildah is 1.28, which can't parse
# `RUN ... <<EOF`; see the infra image + PR discussion.)
FROM python:3.12-slim
# Runtime system deps:
@@ -65,42 +65,36 @@ RUN pip install --no-cache-dir mitmproxy==11.1.3
# would pin us to that image's cadence). python (already present) does the
# download so we add no curl/wget. trixie apt also ships gitleaks, but an
# older 8.16; the pinned download keeps the verified 8.30.1.
#
# Arch-aware: the asset + SHA are picked from the build's target
# architecture so an arm64 host (Apple Silicon) gets the arm64 binary
# rather than an x86_64 one that dies with "Exec format error" the first
# time the pre-receive hook runs it. TARGETARCH is auto-populated by
# BuildKit; the dpkg fallback keeps it correct under a legacy builder.
ARG GITLEAKS_VERSION=8.30.1
ARG GITLEAKS_SHA256_AMD64=551f6fc83ea457d62a0d98237cbad105af8d557003051f41f3e7ca7b3f2470eb
ARG GITLEAKS_SHA256_ARM64=e4a487ee7ccd7d3a7f7ec08657610aa3606637dab924210b3aee62570fb4b080
ARG TARGETARCH
RUN arch="${TARGETARCH:-$(dpkg --print-architecture)}" \
&& case "$arch" in \
amd64) asset="linux_x64"; sha="${GITLEAKS_SHA256_AMD64}" ;; \
arm64) asset="linux_arm64"; sha="${GITLEAKS_SHA256_ARM64}" ;; \
*) echo "unsupported gitleaks target arch: $arch" >&2; exit 1 ;; \
esac \
&& url="https://github.com/gitleaks/gitleaks/releases/download/v${GITLEAKS_VERSION}/gitleaks_${GITLEAKS_VERSION}_${asset}.tar.gz" \
ARG GITLEAKS_SHA256=551f6fc83ea457d62a0d98237cbad105af8d557003051f41f3e7ca7b3f2470eb
RUN url="https://github.com/gitleaks/gitleaks/releases/download/v${GITLEAKS_VERSION}/gitleaks_${GITLEAKS_VERSION}_linux_x64.tar.gz" \
&& python3 -c "import sys,urllib.request; urllib.request.urlretrieve(sys.argv[1], '/tmp/gitleaks.tar.gz')" "$url" \
&& echo "${sha} /tmp/gitleaks.tar.gz" | sha256sum -c - \
&& echo "${GITLEAKS_SHA256} /tmp/gitleaks.tar.gz" | sha256sum -c - \
&& tar -xzf /tmp/gitleaks.tar.gz -C /usr/bin gitleaks \
&& rm /tmp/gitleaks.tar.gz
# Install bot_bottle as a proper package so entry-point scripts can use
# `from bot_bottle.X import Y` absolute imports. A rename or a missing
# module is caught at pip-install time — not at container runtime.
COPY pyproject.toml /src/
COPY bot_bottle/ /src/bot_bottle/
RUN pip install --no-cache-dir /src/
# mitmdump -s requires a file path, not a module. Write a one-line shim that
# re-exports `addons` from the installed package; mitmdump finds it there.
# WORKDIR here also creates /app so the shim + COPYs below can write into it
# (nothing created /app before this point).
WORKDIR /app
RUN printf 'from bot_bottle.gateway.egress.addon import addons\n' > /app/egress_addon.py
COPY bot_bottle/gateway/egress/entrypoint.sh /app/egress-entrypoint.sh
# Project Python: addon + server modules + the init supervisor.
# Kept flat under /app/ so mitmdump's loader resolves them as
# top-level siblings (absolute imports), matching the prior
# Dockerfile.egress / Dockerfile.supervise layout.
COPY bot_bottle/egress_addon_core.py /app/egress_addon_core.py
COPY bot_bottle/egress_dlp_config.py /app/egress_dlp_config.py
COPY bot_bottle/egress_addon.py /app/egress_addon.py
COPY bot_bottle/policy_resolver.py /app/policy_resolver.py
COPY bot_bottle/dlp_detectors.py /app/dlp_detectors.py
COPY bot_bottle/yaml_subset.py /app/yaml_subset.py
COPY bot_bottle/paths.py /app/paths.py
COPY bot_bottle/migrations.py /app/migrations.py
COPY bot_bottle/db_store.py /app/db_store.py
COPY bot_bottle/supervise_types.py /app/supervise_types.py
COPY bot_bottle/queue_store.py /app/queue_store.py
COPY bot_bottle/audit_store.py /app/audit_store.py
COPY bot_bottle/store_manager.py /app/store_manager.py
COPY bot_bottle/supervise.py /app/supervise.py
COPY bot_bottle/supervise_server.py /app/supervise_server.py
COPY bot_bottle/gateway_init.py /app/gateway_init.py
COPY bot_bottle/git_http_backend.py /app/git_http_backend.py
COPY bot_bottle/egress_entrypoint.sh /app/egress-entrypoint.sh
RUN chmod +x /app/egress-entrypoint.sh
# Pre-create runtime directories the compose renderer + start
@@ -118,6 +112,10 @@ RUN mkdir -p \
# subset the bottle uses.
EXPOSE 8888 9099 9418 9420 9100
# WORKDIR matches Dockerfile.supervise's prior layout so the
# in-app same-dir import in supervise_server.py stays deterministic.
WORKDIR /app
# PID 1 is the supervisor. It owns signal handling and exit-code
# propagation; no `exec` chain in the entrypoint itself.
ENTRYPOINT ["python3", "-m", "bot_bottle.gateway.bootstrap"]
ENTRYPOINT ["python3", "/app/gateway_init.py"]
+45
View File
@@ -0,0 +1,45 @@
# Firecracker single infra-VM image (PRD 0070 Stage B).
#
# The per-host infra VM runs the orchestrator control plane, the gateway
# data plane, AND builds agent images (buildah) — all in one microVM (see
# backend/firecracker/infra_vm.py). It composes:
# * FROM the gateway image (mitmproxy / git / gitleaks / supervise + the
# flat daemon modules) — now trixie-based, so buildah 1.39 is available;
# * `COPY --from` the orchestrator image's content (the single definition
# of the control-plane payload — see Dockerfile.orchestrator), so this
# VM and the docker backend share one orchestrator definition; and
# * buildah, installed HERE only (the docker orchestrator/gateway images
# never carry it).
#
# multi-`FROM` can't union two bases (that's multi-stage, not multiple
# inheritance), so the orchestrator content is pulled in via `COPY --from`
# rather than a second base. Both images share the trixie `python:3.12-slim`
# base, so the copy is clean (same python; future installed deps copy too).
#
# The docker backend keeps orchestrator + gateway as separate images; this
# combined image exists only for the Firecracker single-VM cut. Splitting a
# service back into its own VM later is a routing change, not a repackaging
# (PRD 0070's "secret concentration"; a disposable builder can boot from
# this same image on its own TAP).
FROM bot-bottle-gateway:latest
# --- in-VM agent-image builder (PRD 0069 Stage 3) -------------------
# The Firecracker backend builds users' agent Dockerfiles *inside this VM*
# with buildah (rootless, daemonless) instead of on the host — no host
# Docker daemon, no root-equivalent `docker` group. `crun` is the OCI
# runtime; `netavark` + `aardvark-dns` are the network backend for `FROM`
# pulls + `RUN` egress. Requires the trixie base (buildah 1.39: bookworm's
# 1.28 can't parse Dockerfile heredocs that agent images use).
RUN apt-get update \
&& apt-get install -y --no-install-recommends \
buildah crun netavark aardvark-dns \
&& rm -rf /var/lib/apt/lists/*
# vfs + chroot: buildah works as root in the bare microVM (no
# fuse-overlayfs / overlay module / subuid maps). Matches image_builder.
ENV STORAGE_DRIVER=vfs \
BUILDAH_ISOLATION=chroot
# The orchestrator content, pulled from its single definition. The gateway
# image already has the flat daemon modules under /app; this adds the full
# `bot_bottle` package so `python3 -m bot_bottle.orchestrator` resolves.
COPY --from=bot-bottle-orchestrator:latest /app/bot_bottle /app/bot_bottle
+10 -8
View File
@@ -4,19 +4,21 @@
# `bot_bottle` package baked onto a Python runtime — referenced by BOTH:
# * the docker backend, which runs this image directly as the lean
# control-plane container; and
# * the firecracker orchestrator VM image (Dockerfile.orchestrator.fc),
# which is `FROM` this image and adds buildah for in-VM agent builds.
# Keeping the content in one place means future orchestrator deps (e.g.
# iroh) are added here once, not duplicated per backend.
# * the firecracker infra image (Dockerfile.infra), which `COPY --from`s
# this image's `/app/bot_bottle` so the single infra VM runs the same
# control plane. Keeping it in one place means future orchestrator deps
# (e.g. iroh) are added here once, not duplicated per backend.
#
# It stays deliberately lean: the control plane is **stdlib-only** today, so
# no third-party payload — none of the gateway's mitmproxy/git/gitleaks
# (that's Dockerfile.gateway) and no buildah (that's the firecracker
# builder, and lives only in Dockerfile.orchestrator.fc). Keeping the
# secret-dense control plane on a minimal dependency surface is the point
# (PRD 0070's "secret concentration").
# builder, and lives only in Dockerfile.infra). Keeping the secret-dense
# control plane on a minimal dependency surface is the point (PRD 0070's
# "secret concentration").
#
# Shares the trixie `python:3.12-slim` base with the gateway image.
# Shares the trixie `python:3.12-slim` base with the gateway image, so when
# the orchestrator grows real deps they can be `COPY --from`'d into the
# infra image cleanly (same base/python — installed packages copy safely).
FROM python:3.12-slim
-23
View File
@@ -1,23 +0,0 @@
# Firecracker orchestrator-VM image (PRD 0070).
#
# The control-plane rootfs the orchestrator microVM boots: the lean orchestrator
# image + the in-VM agent-image builder. The Firecracker backend builds users'
# agent Dockerfiles *inside this VM* with buildah (rootless, daemonless) instead
# of on the host — no host Docker daemon, no root-equivalent `docker` group. The
# gateway VM boots a *separate*, slimmer rootfs (bot-bottle-gateway:latest) that
# carries none of this build tooling — the exposed data plane stays minimal.
#
# `crun` is the OCI runtime; `netavark` + `aardvark-dns` are the network backend
# for `FROM` pulls + `RUN` egress. `vfs` + `chroot`: buildah works as root in the
# bare microVM (no fuse-overlayfs / overlay module / subuid maps). The trixie
# base (from Dockerfile.orchestrator's python:3.12-slim) carries buildah 1.39,
# which parses the Dockerfile heredocs agent images use (bookworm's 1.28 can't).
# Matches image_builder.
FROM bot-bottle-orchestrator:latest
RUN apt-get update \
&& apt-get install -y --no-install-recommends \
buildah crun netavark aardvark-dns \
&& rm -rf /var/lib/apt/lists/*
ENV STORAGE_DRIVER=vfs \
BUILDAH_ISOLATION=chroot
+10 -103
View File
@@ -5,8 +5,8 @@
# bot-bottle
[![test](https://gitea.dideric.is/didericis/bot-bottle/actions/workflows/test.yml/badge.svg?branch=main)](https://gitea.dideric.is/didericis/bot-bottle/actions?workflow=test.yml)
[![coverage](https://img.shields.io/badge/coverage-83%25-brightgreen)](https://coverage.readthedocs.io/)
[![core coverage](https://img.shields.io/badge/core%20coverage-94%25-brightgreen)](https://gitea.dideric.is/didericis/bot-bottle/src/branch/main/docs/decisions/0004-coverage-policy.md)
[![coverage](https://img.shields.io/badge/coverage-82%25-brightgreen)](https://coverage.readthedocs.io/)
[![core coverage](https://img.shields.io/badge/core%20coverage-95%25-brightgreen)](https://gitea.dideric.is/didericis/bot-bottle/src/branch/main/docs/decisions/0004-coverage-policy.md)
**Problem:** Developer wants to run a coding agent without supervision, but they don't want a prompt injected or misbehaving agent wrecking their environment or exfiltrating sensitive data.
@@ -75,88 +75,6 @@ On compatible macOS hosts, the default backend requires Apple's `container` CLI
Use `BOT_BOTTLE_BACKEND=docker ./cli.py start <agent>` on hosts where neither Apple Container nor KVM is available and Docker is the desired backend.
### Containers inside a bottle
A bottle may set `nested_containers: true`. On the macOS backend this starts a
guest-local, rootless **podman** service after the bottle is registered and
exposes its Docker-compatible API socket, so the agent still runs `docker` and
`docker compose`. Nothing is mounted from the host: Docker Desktop's socket
stays out of the bottle and the guest gains no outer VM capabilities. Backends
that cannot do this (`docker`, `firecracker`) reject the flag rather than
silently ignore it.
Rootless Docker was tried first and does not work here at all: Apple
Container's capability bounding set omits `CAP_SYS_ADMIN`, which the kernel
requires to write a multi-range `uid_map`. See
[`docs/research/rootless-docker-in-apple-container-spike.md`](docs/research/rootless-docker-in-apple-container-spike.md).
The tradeoff to understand before enabling it: podman avoids that requirement
by falling back to a single-UID mapping, so nested containers provide **no
isolation from the agent itself** — `root` inside a nested container is the
agent user outside it. Nested containers are a build/test convenience, not a
security boundary. The bottle remains the boundary.
Pulling images goes through the bottle's egress proxy like every other
request, so each registry needs a route — **and so does the CDN it redirects
layer blobs to**, which is a different host. Without the CDN route the pull
authenticates, fetches the manifest, then 403s partway through.
Docker Hub and GHCR additionally need `preserve_auth: true`: their token dance
uses a client-fetched per-scope bearer token that the proxy would otherwise
strip. Turn DLP off on every registry and CDN route — the bodies are
compressed layer blobs that no detector can read, and buffering them is what
triggers the shared-proxy OOM in #455.
```yaml
nested_containers: true
egress:
routes:
# Docker Hub: registry, token endpoint, blob CDN.
- host: registry-1.docker.io
preserve_auth: true
dlp: { outbound_detectors: false, inbound_detectors: false }
- host: auth.docker.io
preserve_auth: true
dlp: { outbound_detectors: false, inbound_detectors: false }
- host: production.cloudfront.docker.com
dlp: { outbound_detectors: false, inbound_detectors: false }
# GHCR: registry + blob CDN.
- host: ghcr.io
preserve_auth: true
dlp: { outbound_detectors: false, inbound_detectors: false }
- host: pkg-containers.githubusercontent.com
dlp: { outbound_detectors: false, inbound_detectors: false }
# quay.io: registry + blob CDNs. No preserve_auth needed for public pulls.
- host: quay.io
dlp: { outbound_detectors: false, inbound_detectors: false }
- host: cdn01.quay.io
dlp: { outbound_detectors: false, inbound_detectors: false }
- host: cdn02.quay.io
dlp: { outbound_detectors: false, inbound_detectors: false }
- host: cdn03.quay.io
dlp: { outbound_detectors: false, inbound_detectors: false }
```
`mcr.microsoft.com` and `registry.k8s.io` follow the same shape and also
redirect blobs elsewhere (`*.data.mcr.microsoft.com` and
`us-*-docker.pkg.dev` respectively); route whichever host the 403 names.
Inside a nested container the same allowlist applies: an allowlisted host
returns 200 and anything else gets a 403 straight from the proxy. The
gateway's CA bundle and proxy settings are wired in automatically, so
`docker run … curl https://…` works with no extra flags — no `--add-host`,
`-e`, or `-v`.
Two things worth knowing when testing that:
- Public DNS inside a nested container fails **by design**. Everything
egresses through the proxy, so `nslookup` failing is expected and is not
evidence of a problem.
- Alpine's BusyBox `wget` drops the connection after the proxy's TLS
interception and reports `error getting response`, even though the proxy
logs the decrypted request and returns a response. Use `curl` to test
egress; BusyBox `wget` will lie to you.
### Firecracker on Linux
On Linux, a KVM-capable host defaults to the Firecracker backend. It needs:
@@ -172,8 +90,6 @@ BOT_BOTTLE_BACKEND=firecracker ./cli.py start <agent>
> **NixOS:** enable `virtualisation.docker`, ensure the KVM module is loaded (`boot.kernelModules = [ "kvm-intel" ];` or `kvm-amd`), and add your user to the `kvm` and `docker` groups. For the network pool, consume the flake module — `imports = [ inputs.bot-bottle.nixosModules.firecracker-netpool ]; services.bot-bottle-firecracker = { enable = true; owner = "you"; };` — then `nixos-rebuild switch` (imperative nft/TAP rules don't survive a rebuild; channel users can `imports = [ <bot-bottle>/nix/firecracker-netpool.nix ]`). `firecracker` isn't in nixpkgs by default as a user binary — install the release binary (pin the version) and put it on `PATH`.
> **CI:** the coverage gate (`.gitea/workflows/test.yml` → `coverage` job) runs on a self-hosted runner labelled `kvm`, because the Firecracker backend's VM/SSH orchestration is exercised only by the integration suite, which needs `/dev/kvm` + the provisioned pool (a container runner would skip it and read as uncovered). Provision that runner exactly like a normal Firecracker host — `firecracker` on `PATH`, `/dev/kvm`, the cached guest kernel + static dropbear, and the pool installed as the persistent systemd unit — then register it with the `kvm` label. A Docker-capable hosted job builds the candidate once; KVM tests boot those exact bytes, and a successful main run publishes them. The unit/lint jobs still run on `ubuntu-latest`.
```sh
./cli.py start <agent> # builds the image on first run, drops you into claude
```
@@ -205,14 +121,14 @@ git:
egress:
routes:
- host: gitea.dideric.is
inspect:
auth:
scheme: token # Bearer | token
token_ref: BOT_BOTTLE_GITEA_TOKEN
matches: # optional — restrict to specific paths/methods/headers
- paths:
- {type: prefix, value: /api/v1/}
methods: [GET, POST, PATCH, DELETE]
auth:
scheme: token # Bearer | token
token_ref: BOT_BOTTLE_GITEA_TOKEN
matches: # optional — restrict to specific paths/methods/headers
- paths:
- {type: prefix, value: /api/v1/}
methods: [GET, POST, PATCH, DELETE]
dlp: # optional — per-route detector overrides (default: all on)
outbound_detectors: [token_patterns, known_secrets]
inbound_detectors: false # disable response scanning for this host
---
@@ -255,15 +171,6 @@ When an outbound DLP detector matches a token, the route's `dlp.outbound_on_matc
More examples in `examples/`. Full design lives under `docs/prds/`; the trust-boundary rationale is in `docs/prds/0011-per-file-md-manifest.md`.
## Tracker policy
Issues are the canonical work items and own all tracker labels; every issue
must have at least one. Pull requests stay unlabeled and deliberately reference
an issue with `Closes #…`, `Part of #…`, or another form defined in
[`ADR 0005`](docs/decisions/0005-issues-own-tracker-metadata.md). Gitea Actions
enforces the convention for new work from 2026-07-18 onward. Earlier closed
PRs are grandfathered rather than given artificial retrospective issues.
## Trademarks
bot-bottle is an independent project and is not affiliated with, endorsed by, or sponsored by Anthropic, PBC. "Claude" and "Claude Code" are trademarks of Anthropic, PBC; the project name uses "claude" descriptively to indicate that the tool runs Claude Code inside a sandbox.
+2 -43
View File
@@ -45,10 +45,6 @@ PROVIDER_TEMPLATES = frozenset({PROVIDER_CLAUDE, PROVIDER_CODEX, PROVIDER_PI})
# forward_host_credentials is enabled. Pipelock must pass these through
# (no TLS MITM) or its header DLP blocks the injected JWT.
CODEX_HOST_CREDENTIAL_HOSTS = ("api.openai.com", "chatgpt.com")
# Host that egress injects the host Claude bearer on when Claude
# forward_host_credentials is enabled.
CLAUDE_HOST_CREDENTIAL_HOSTS = ("api.anthropic.com",)
PromptMode = Literal[
"append_file",
"read_prompt_file",
@@ -261,28 +257,7 @@ class AgentProvider(ABC):
Default: Debian/node — writes the git-gate insteadOf gitconfig
and sets user.name/email as node. Workspace copy runs through
BottleBackend.provision_workspace against the running bottle."""
from .log import die, info
# Firecracker exports image rootfs files through an unprivileged host
# tar extraction, so image-time ownership of XDG directories is not
# preserved. Git consults ~/.config/git even when the actual config
# is ~/.gitconfig; an unreadable directory there can prevent the
# git-gate insteadOf rules below from taking effect. Repair this at
# runtime, after every backend's copy/export path has completed.
git_xdg_dir = f"{plan.guest_home}/.config/git"
repair = bottle.exec(
f"chown node:node {shlex.quote(plan.guest_home)} && "
f"chmod 755 {shlex.quote(plan.guest_home)} && "
f"mkdir -p {shlex.quote(git_xdg_dir)} && "
f"chown -R node:node {shlex.quote(f'{plan.guest_home}/.config')} && "
f"chmod -R u+rwX,go+rX {shlex.quote(f'{plan.guest_home}/.config')}",
user="root",
)
if repair.returncode != 0:
die(
"git provisioning: could not make the runtime Git config "
f"directory readable: {(repair.stderr or repair.stdout).strip()}"
)
from .log import info
manifest_bottle = plan.manifest.bottle
if manifest_bottle.git:
@@ -305,27 +280,11 @@ class AgentProvider(ABC):
f"{len(manifest_bottle.git)} insteadOf rule(s)"
)
bottle.cp_in(str(config_file), guest_gitconfig)
permissions = bottle.exec(
bottle.exec(
f"chown node:node {shlex.quote(guest_gitconfig)} && "
f"chmod 644 {shlex.quote(guest_gitconfig)}",
user="root",
)
if permissions.returncode != 0:
die(
"git provisioning: could not set ownership on "
f"{guest_gitconfig}: "
f"{(permissions.stderr or permissions.stdout).strip()}"
)
configured = bottle.exec(
"git config --global --get-regexp '^url\\..*\\.insteadof$'",
user="node",
)
if configured.returncode != 0:
die(
"git provisioning: the runtime user cannot read the "
f"git-gate insteadOf rules from {guest_gitconfig}: "
f"{(configured.stderr or configured.stdout).strip()}"
)
gu = manifest_bottle.git_user
if not gu.is_empty():
@@ -5,10 +5,16 @@ from __future__ import annotations
import sqlite3
from pathlib import Path
from ..supervisor.types import AuditEntry
from ..paths import host_db_path
from .db_store import DbStore
from .migrations import TableMigrations
try:
from .supervise_types import AuditEntry
from .paths import host_db_path
from .db_store import DbStore
from .migrations import TableMigrations
except ImportError:
from supervise_types import AuditEntry # type: ignore[import-not-found] # pylint: disable=import-error,no-name-in-module
from paths import host_db_path # type: ignore[import-not-found] # pylint: disable=import-error,no-name-in-module
from db_store import DbStore # type: ignore[import-not-found] # pylint: disable=import-error,no-name-in-module
from migrations import TableMigrations # type: ignore[import-not-found] # pylint: disable=import-error,no-name-in-module
class AuditStore(DbStore):
@@ -36,7 +42,7 @@ class AuditStore(DbStore):
super().__init__(db_path or host_db_path(), migrations)
def write_audit_entry(self, entry: AuditEntry) -> Path:
with self._connection() as conn:
with self._connect() as conn:
conn.execute(
"""
INSERT INTO supervise_audit_entries (
@@ -60,7 +66,7 @@ class AuditStore(DbStore):
def read_audit_entries(self, component: str, slug: str) -> list[AuditEntry]:
if not self.db_path.is_file():
return []
with self._connection() as conn:
with self._connect() as conn:
rows = conn.execute(
"""
SELECT * FROM supervise_audit_entries
+671 -85
View File
@@ -1,115 +1,701 @@
"""The bottle-backend package: abstract contract + backend selection.
"""Per-backend bottle factories.
Thin by design — nothing framework-heavy is imported at package init, so
importing any `backend.*` submodule (e.g. `backend.docker.util`) doesn't drag
the manifest / egress / git-gate framework into memory. The public names are
re-exported lazily:
A bottle is a running, isolated environment with claude inside. Each
backend exposes five methods:
* the abstract contract (`BottleBackend`, `BottleSpec`, `Bottle`, …) lives in
`backend.base`;
* backend selection / enumeration (`get_bottle_backend`,
`enumerate_active_agents`, …) in `backend.selection`;
* the concrete backends and the freeze helpers in their own submodules.
prepare(spec, stage_dir=...) -> BottlePlan
Resolves names, validates host-side prerequisites, and writes
scratch files. No remote/runtime resources are created yet.
Safe to call before the y/N preflight.
`from bot_bottle.backend import X` resolves X on first access via `__getattr__`
and caches it at package level, so existing call-sites (and
`patch.object(backend_mod, X, …)`) keep working.
launch(plan) -> ContextManager[Bottle]
Brings up the container (or VM, or remote machine), provisions
it, yields a Bottle handle, and tears everything down on exit.
prepare_cleanup() -> BottleCleanupPlan
Enumerates orphaned resources left behind by previous bottles
(containers, networks, ...). Idempotent; no side effects.
cleanup(plan) -> None
Actually removes everything described by the cleanup plan.
enumerate_active() -> Sequence[ActiveAgent]
Return every currently-running bottle on this backend, with
enough metadata for callers (CLI `list active`, dashboard
agents pane) to render a row.
Selection is driven by `--backend` on `start` or BOT_BOTTLE_BACKEND
(env var). When neither is set, compatible macOS hosts default to
`macos-container`; Linux hosts with KVM default to `firecracker`;
otherwise `docker`. Per PRD 0003 the manifest does not carry a
backend field; the host picks.
"""
from __future__ import annotations
from typing import TYPE_CHECKING, Any
import os
import shlex
import sys
from abc import ABC, abstractmethod
from contextlib import AbstractContextManager
from dataclasses import dataclass
from pathlib import Path
from typing import Any, Generic, Sequence, TypeVar
if TYPE_CHECKING:
from .base import (
ActiveAgent,
BackendStatus,
Bottle,
BottleBackend,
BottleCleanupPlan,
BottleImages,
BottlePlan,
BottleSpec,
ExecResult,
)
from .selection import (
enumerate_active_agents,
get_bottle_backend,
has_backend,
is_backend_available,
is_backend_ready,
known_backend_names,
)
from .docker import DockerBottleBackend
from .firecracker import FirecrackerBottleBackend
from .macos_container import MacosContainerBottleBackend
from .freeze import CommitCancelled, Freezer, get_freezer
from ..agent_provider import AgentProvisionPlan, get_provider, build_agent_provision_plan
from ..egress import EgressPlan
from ..git_gate import GitGatePlan
from ..log import die, info
from ..manifest import Manifest, ManifestIndex
from ..supervise import SupervisePlan
from ..util import expand_tilde
from ..env import resolve_env, ResolvedEnv
from ..workspace import WorkspacePlan, workspace_plan
from .print_util import print_multi, visible_agent_env_names
from .util import host_skill_dir
# Public name -> submodule that defines it. Contract types resolve from `base`,
# selection/enumeration from `selection`, the concrete backends + freeze helpers
# from their own subpackages.
_LAZY_MODULES: dict[str, str] = {
"BottleSpec": "base",
"BottlePlan": "base",
"BottleCleanupPlan": "base",
"ExecResult": "base",
"ActiveAgent": "base",
"Bottle": "base",
"BottleImages": "base",
"BottleBackend": "base",
"BackendStatus": "base",
"get_bottle_backend": "selection",
"known_backend_names": "selection",
"has_backend": "selection",
"is_backend_available": "selection",
"is_backend_ready": "selection",
"enumerate_active_agents": "selection",
"_print_vm_install_instructions": "selection",
"DockerBottleBackend": "docker",
"FirecrackerBottleBackend": "firecracker",
"MacosContainerBottleBackend": "macos_container",
"CommitCancelled": "freeze",
"Freezer": "freeze",
"get_freezer": "freeze",
@dataclass(frozen=True)
class BottleSpec:
"""CLI-supplied intent. Backend-agnostic — each backend's prepare
step consumes it and produces its own backend-specific plan.
Resolved values (image names, container name, scratch paths, runsc
availability) live on the plan, not the spec."""
manifest: ManifestIndex
agent_name: str
copy_cwd: bool
user_cwd: str
# PRD 0016 follow-up: when set, the backend's prepare step uses
# this identity instead of minting a fresh one — the resume path
# (`cli.py resume <identity>`) sets this to continue an existing
# bottle's state. Empty string for a fresh `start`.
identity: str = ""
label: str = ""
color: str = ""
# Ordered bottle names selected at launch (issue #269). When non-empty
# they are merged in order and replace the agent's `bottle:` field.
bottle_names: tuple[str, ...] = ()
# True when launched via --headless (no TTY, no interactive prompts).
# The git-gate host-key preflight uses this to error rather than prompt.
headless: bool = False
@dataclass(frozen=True)
class BottlePlan(ABC):
"""Base output of a backend's prepare step. Concrete subclasses
(e.g. DockerBottlePlan) add backend-specific resolved fields."""
spec: BottleSpec
manifest: Manifest
stage_dir: Path
git_gate_plan: GitGatePlan
@property
def guest_home(self) -> str:
return self.agent_provision.guest_home
@property
def git_gate_insteadof_host(self) -> str:
"""Host (and optional port) used in git-gate insteadOf URLs.
Docker uses the compose-network DNS alias; VM backends may
override with an IP:port when the guest has no DNS."""
return "git-gate"
@property
def git_gate_insteadof_scheme(self) -> str:
"""URL scheme for git-gate insteadOf rewrites. 'git' for
Docker (git daemon); VM backends may override (e.g. 'http'
over a published host port)."""
return "git"
egress_plan: EgressPlan
supervise_plan: SupervisePlan | None
agent_provision: AgentProvisionPlan
@property
def workspace_plan(self) -> WorkspacePlan:
return workspace_plan(self.spec, guest_home=self.guest_home)
def print(self) -> None:
"""Render the y/N preflight summary to stderr."""
spec = self.spec
manifest = self.manifest
agent = manifest.agent
bottle = manifest.bottle
env_names = visible_agent_env_names(
sorted(
set(bottle.env.keys())
| set(self.agent_provision.guest_env.keys())
),
hidden_env_names=self.agent_provision.hidden_env_names,
)
print(file=sys.stderr)
info(f"agent : {spec.agent_name}")
info(f"provider : {self.agent_provision.template}")
print_multi("env ", env_names)
print_multi("skills ", list(agent.skills))
effective_bottles = (
list(spec.bottle_names) if spec.bottle_names
else ([agent.bottle] if agent.bottle else [])
)
print_multi("bottle ", effective_bottles)
identity = manifest.git_identity_summary()
if identity:
info(f" git identity : {identity}")
git_lines = [
f"{u.name}{u.upstream_host}:{u.upstream_port}"
for u in self.git_gate_plan.upstreams
]
if git_lines:
print_multi(" git gate ", git_lines)
if self.egress_plan.routes:
egress_lines = []
for r in self.egress_plan.routes:
auth = f" [auth:{r.auth_scheme}]" if r.auth_scheme else ""
egress_lines.append(f"{r.host}{auth}")
print_multi(" egress ", egress_lines)
print(file=sys.stderr)
@dataclass(frozen=True)
class BottleCleanupPlan(ABC):
"""Base output of a backend's prepare_cleanup step. Concrete
subclasses (e.g. DockerBottleCleanupPlan) carry backend-specific
lists of resources to be removed and implement `print` + `empty`."""
@abstractmethod
def print(self) -> None:
"""Render the cleanup y/N summary to stderr."""
@property
@abstractmethod
def empty(self) -> bool:
"""True iff there is nothing to clean up; the CLI uses this to
short-circuit before showing the y/N."""
@dataclass(frozen=True)
class ExecResult:
"""Captured result of `Bottle.exec`. Backend-neutral: the Docker
impl populates it from a `subprocess.CompletedProcess`, but a
VM backend could populate it from any source that produces a
returncode + captured streams."""
returncode: int
stdout: str
stderr: str
@dataclass(frozen=True)
class ActiveAgent:
"""One currently-running agent, as the CLI `list active` and
dashboard agents pane render it. ("Agent" is the project's
consistent name for the thing running inside a bottle — the
bottle is the container, the agent is what runs in it.)
Fields are deliberately backend-neutral. `services` is the set
of gateway daemons currently up for this bottle (`egress`,
`git-gate`, `supervise`); the dashboard uses it to
gate edit verbs. `backend_name` is the matching key in
`_BACKENDS` (`docker` / `firecracker` / `macos-container`) — used by the active-
list rendering to disambiguate and by the dashboard's
re-attach path."""
backend_name: str
slug: str
agent_name: str # from metadata.json; "?" if missing
started_at: str # ISO 8601 from metadata.json; "" if missing
services: tuple[str, ...] # alphabetical
label: str = ""
color: str = ""
class Bottle(ABC):
"""Handle to a running bottle. Yielded by a backend's launch step.
`exec_agent` runs the selected agent CLI inside the bottle and
blocks until the session ends. `exec` runs a POSIX shell script inside the bottle
and returns the captured result. `cp_in` copies a host path into
the bottle. `close` is an idempotent alias for context-manager
teardown.
"""
name: str
@abstractmethod
def agent_argv(
self, argv: list[str], *, tty: bool = True,
) -> list[str]:
"""Return the host-side argv that runs the selected agent
inside the bottle. Used by `exec_agent` for foreground
handoffs and by the dashboard's tmux `respawn-pane` flow,
which needs the argv up front (it spawns claude in a tmux
pane rather than as a child of the current process).
Implementations transparently inject
`--append-system-prompt-file` when the bottle was launched
with a provisioned prompt path."""
...
@abstractmethod
def exec_agent(self, argv: list[str], *, tty: bool = True) -> int: ...
@abstractmethod
def exec(self, script: str, *, user: str = "node") -> ExecResult:
"""Run `script` as a POSIX shell script inside the bottle as
`user` (default `node`, matching the agent image's USER
directive) and return the captured stdout/stderr/returncode.
The bottle's environment (including HTTPS_PROXY pointing at
the egress daemon) is inherited by the child. Non-zero
exit does not raise — callers inspect `returncode`
themselves.
Pass `user="root"` for shell-outs that need privileged file
writes / package install — provisioning calls that need root
bypass `Bottle.exec` and use the backend-specific raw
machine-exec helper, but the tests have a legitimate use
case for arbitrary-user runs."""
@abstractmethod
def cp_in(self, host_path: str, container_path: str) -> None: ...
@abstractmethod
def close(self) -> None: ...
PlanT = TypeVar("PlanT", bound=BottlePlan)
CleanupT = TypeVar("CleanupT", bound=BottleCleanupPlan)
class BottleBackend(ABC, Generic[PlanT, CleanupT]):
"""Abstract base for selectable bottle backends. Concrete subclasses
(e.g. DockerBottleBackend) own their own prepare/launch impls.
Parameterized over the backend's concrete plan + cleanup-plan types
so subclass methods get the narrow type without isinstance
boilerplate."""
name: str
def prepare(self, spec: BottleSpec, stage_dir: Path) -> PlanT:
"""Template method: run cross-backend host-side validation, then
delegate to the subclass's `_resolve_plan` for the
backend-specific resolution (names, scratch files, etc.). The
validation step is enforced here so a future backend cannot
accidentally skip it. No remote/runtime resources are created."""
from .resolve_common import (
merge_provision_env_vars,
mint_slug,
prepare_agent_state_dir,
prepare_egress,
prepare_git_gate,
prepare_supervise,
resolve_manifest_dockerfile,
write_launch_metadata,
)
manifest = self._validate(spec)
self._preflight()
from ..git_gate_host_key import preflight_host_keys
manifest = preflight_host_keys(
manifest,
headless=spec.headless,
home_md=spec.manifest.home_md,
)
manifest_bottle = manifest.bottle
manifest_agent_provider = manifest_bottle.agent_provider
agent_provider = get_provider(manifest_agent_provider.template)
resolved_env = resolve_env(manifest)
workspace = workspace_plan(spec, guest_home=agent_provider.guest_home)
slug = mint_slug(spec)
write_launch_metadata(slug, spec, compose_project="", backend=self.name)
# Manifest may override the Dockerfile per-bottle; otherwise fall
# back to the provider plugin's bundled Dockerfile (next to its
# agent_provider.py module).
if manifest_agent_provider.dockerfile:
agent_dockerfile_path = resolve_manifest_dockerfile(
manifest_agent_provider.dockerfile, spec,
)
else:
agent_dockerfile_path = str(agent_provider.dockerfile)
agent_dir, prompt_file = prepare_agent_state_dir(slug, manifest)
agent_provision_plan = build_agent_provision_plan(
template=manifest_agent_provider.template,
dockerfile=agent_dockerfile_path,
state_dir=agent_dir,
instance_name=f"bot-bottle-{slug}",
prompt_file=prompt_file,
guest_env=self._build_guest_env(resolved_env),
forward_host_credentials=manifest_agent_provider.forward_host_credentials,
auth_token=manifest_agent_provider.auth_token,
host_env=dict(os.environ),
trusted_project_path=workspace.workdir,
label=spec.label,
color=spec.color,
provider_settings=manifest_agent_provider.settings,
)
agent_provision_plan = merge_provision_env_vars(agent_provision_plan)
egress_plan = prepare_egress(manifest_bottle, slug, agent_provision_plan)
supervise_plan = prepare_supervise(manifest_bottle, slug)
git_gate_plan = prepare_git_gate(manifest_bottle, slug)
return self._resolve_plan(
spec,
manifest=manifest,
slug=slug,
resolved_env=resolved_env,
agent_provision_plan=agent_provision_plan,
egress_plan=egress_plan,
supervise_plan=supervise_plan,
git_gate_plan=git_gate_plan,
stage_dir=stage_dir,
)
def _build_guest_env(self, resolved_env: ResolvedEnv) -> dict[str, str]:
return {}
def _preflight(self) -> None:
"""
tasks to do before resolving a plan
"""
pass
def _validate(self, spec: BottleSpec) -> Manifest:
"""Cross-backend pre-launch checks. Parses the selected agent and
its bottle (raising ManifestError on invalid content), confirms
skills are present on the host, and every git IdentityFile resolves.
Returns the loaded Manifest for the selected agent. Subclasses with
additional preconditions should override and call
`super()._validate(spec)` first."""
manifest = spec.manifest.load_for_agent(spec.agent_name, spec.bottle_names)
self._validate_skills(manifest.agent.skills)
self._validate_agent_provider_dockerfile(spec, manifest)
return manifest
def _validate_skills(self, skills: Sequence[str]) -> None:
"""Each named skill must be a directory under the host's
`~/.claude/skills/`. The check is purely host-side, so the
default impl covers every backend."""
for name in skills:
path = host_skill_dir(name)
if not os.path.isdir(path):
die(
f"skill '{name}' not found on host at {path}. "
f"Create it under ~/.claude/skills/, then re-run."
)
def _validate_agent_provider_dockerfile(self, spec: BottleSpec, manifest: Manifest) -> None:
bottle = manifest.bottle
dockerfile = bottle.agent_provider.dockerfile
if not dockerfile:
return
path = Path(expand_tilde(dockerfile))
if not path.is_absolute():
path = Path(spec.user_cwd) / path
if not path.is_file():
effective = (
", ".join(spec.bottle_names) if spec.bottle_names else manifest.agent.bottle
)
die(
f"agent_provider.dockerfile for bottle "
f"'{effective}' not found: {path}"
)
@abstractmethod
def _resolve_plan(self,
spec: BottleSpec,
*,
manifest: Manifest,
slug: str,
resolved_env: ResolvedEnv,
agent_provision_plan: AgentProvisionPlan,
egress_plan: EgressPlan,
git_gate_plan: GitGatePlan,
supervise_plan: SupervisePlan | None,
stage_dir: Path) -> PlanT:
"""Backend-specific plan resolution: image/container names,
env-file, prompt-file, proxy plan, runtime detection. Called by
`prepare` after `_validate` succeeds. Instance name, image,
prompt file, Dockerfile path, and guest home all live on
`agent_provision_plan` — the source of truth."""
@abstractmethod
def launch(self, plan: PlanT) -> AbstractContextManager[Bottle]:
"""Build/run the bottle and yield a handle; tear down on exit."""
def provision(self, plan: PlanT, bottle: "Bottle") -> str | None:
"""Copy host-side files (CA cert, prompt, skills, .git) into
the running bottle. Called from `launch` after the container
/ machine is up. Returns the in-container prompt path if a
prompt was provisioned, else None — the Bottle handle uses it
to decide whether to add provider-specific prompt args to the
agent's argv.
Default orchestration: ca → prompt → provider apply → skills
→ workspace → git → supervise-mcp. CA install runs first so
the agent's trust store is rebuilt before anything inside the
agent makes a TLS call.
Per PRD 0050 the per-provider steps (prompt, skills,
declarative provision-plan apply, supervise MCP registration)
live on the `AgentProvider` plugin. The backend only owns the
steps that are about backend infrastructure (CA, workspace,
git) and surfaces the supervise daemon URL its launch step
knows about via `supervise_mcp_url`.
PRD 0017: cred-proxy's agent-side dotfile rewrites (~/.npmrc,
~/.gitconfig insteadOf, tea config) are gone. Egress-proxy is
on the agent's HTTP_PROXY path so every tool that respects
HTTPS_PROXY (claude-code, git over HTTPS, npm, curl) is
intercepted without per-tool reconfiguration."""
provider = get_provider(plan.agent_provision.template)
provider.provision_ca(bottle, plan)
prompt_path = provider.provision_prompt(plan, bottle)
provider.provision(plan, bottle)
provider.provision_skills(plan, bottle)
self.provision_workspace(plan, bottle)
provider.provision_git(bottle, plan)
provider.provision_supervise_mcp(
plan, bottle, self.supervise_mcp_url(plan),
)
return prompt_path
def provision_workspace(self, plan: PlanT, bottle: "Bottle") -> None:
"""Copy the operator workspace into the running bottle.
This is the only supported workspace-provisioning path: Docker
does not build a derived image containing the current
workspace."""
workspace = plan.workspace_plan
if not (workspace.enabled and workspace.copy_contents):
return
guest_parent = workspace.guest_path.rsplit("/", 1)[0] or "/"
guest_path = shlex.quote(workspace.guest_path)
guest_parent = shlex.quote(guest_parent)
owner = shlex.quote(workspace.owner)
mode = shlex.quote(workspace.mode)
info(f"copying {workspace.host_path} -> {bottle.name}:{workspace.guest_path}")
bottle.exec(
f"rm -rf {guest_path} && mkdir -p {guest_parent}",
user="root",
)
bottle.cp_in(str(workspace.host_path), workspace.guest_path)
bottle.exec(
f"chown -R {owner} {guest_path} && chmod {mode} {guest_path}",
user="root",
)
def supervise_mcp_url(self, plan: PlanT) -> str:
"""Return the agent-side URL of the per-bottle supervise
gateway, or "" when this bottle has no gateway. The provider
plugin's `provision_supervise_mcp` uses it to register the
MCP entry inside the guest.
Default returns "" so backends without supervise support
don't have to implement it. Docker and firecracker override."""
del plan
return ""
def ensure_orchestrator(self) -> str:
"""Bring up this backend's per-host orchestrator + shared gateway
(idempotent) and return the host-reachable control-plane URL.
This is the backend-agnostic bring-up entry point: `launch` calls
it as part of starting a bottle, and operator tools (`supervise`)
call it to start the control plane on demand when none is running
yet. Docker starts the orchestrator + gateway containers;
firecracker boots the infra VM. Backends with no orchestrator
(macos-container) die with a pointer — the default here."""
die(f"backend {self.name!r} has no orchestrator control plane")
@abstractmethod
def prepare_cleanup(self) -> CleanupT:
"""Enumerate orphaned resources from previous bottles. No side
effects; safe to call before the y/N."""
@abstractmethod
def cleanup(self, plan: CleanupT) -> None:
"""Remove everything described by the cleanup plan."""
@abstractmethod
def enumerate_active(self) -> Sequence[ActiveAgent]:
"""Return every currently-running agent on this backend.
Empty when none. Backend-specific: docker queries `docker
compose ls`; firecracker cross-references its running gateway
containers against per-bottle metadata."""
@classmethod
@abstractmethod
def is_available(cls) -> bool:
"""Whether this backend's runtime prerequisites are satisfied
on the current host. Docker → `docker` on PATH; firecracker →
Linux + KVM. Used by the cross-backend
`enumerate_active_agents` / `cmd_cleanup` to skip backends
the operator hasn't installed, so a docker-only host
doesn't fail when `cli.py list active` walks past
firecracker."""
@classmethod
@abstractmethod
def setup(cls) -> int:
"""Emit this backend's one-time host setup — privileged network
pool, daemon bring-up, install pointers, etc. — as
host-appropriate config or commands. Prints to stdout/stderr and
returns a shell exit code (0 = nothing to report / success). A
backend that needs no host setup prints a short note and returns
0. Invoked generically by `./cli.py backend setup [--backend=…]`
so operators can provision any backend without a
backend-specific command. Classmethod (like `is_available`) —
it's a host query, not per-bottle state."""
@classmethod
@abstractmethod
def status(cls) -> int:
"""Report whether this backend's prerequisites are satisfied on
the host — binaries, daemon reachability, network pool, range
conflicts, etc. Prints a human-readable summary; returns 0 when
the backend is ready to launch and non-zero when something is
missing. Invoked by `./cli.py backend status [--backend=…]`."""
@classmethod
@abstractmethod
def teardown(cls) -> int:
"""Undo `setup()` — the inverse operation, surfaced as
`./cli.py backend teardown [--backend=…]` (uninstall). Symmetric
with setup: where setup is advisory (prints the privileged
commands / declarative config to apply), teardown prints the
commands / config change to remove the host prerequisites. A
backend with no host setup prints a short note and returns 0.
Not called by the launch path or the test suite."""
# Import concrete backend classes AFTER the base types are defined, so
# each backend module can pull BottleSpec / BottlePlan / BottleBackend
# via `from . import ...` without hitting a partially-initialized module.
from .docker import DockerBottleBackend # noqa: E402 # pylint: disable=wrong-import-position
from .firecracker import FirecrackerBottleBackend # noqa: E402 # pylint: disable=wrong-import-position
from .macos_container import MacosContainerBottleBackend # noqa: E402 # pylint: disable=wrong-import-position
# Freezer is imported after the backend classes for the same reason:
# Freezer.commit_slug constructs ActiveAgent, which must be fully
# defined first.
from .freeze import CommitCancelled, Freezer, get_freezer # noqa: E402 # pylint: disable=wrong-import-position
# The dict is heterogeneous: each value is a BottleBackend specialized
# over its own plan type. Concrete plan types are erased here because
# the registry is selected at runtime and the CLI only needs the
# unparameterized methods (prepare → plan → launch(plan), cleanup, etc.).
_BACKENDS: dict[str, BottleBackend[Any, Any]] = {
"docker": DockerBottleBackend(),
"firecracker": FirecrackerBottleBackend(),
"macos-container": MacosContainerBottleBackend(),
}
def __getattr__(name: str) -> Any:
"""Lazily surface the package's public names from their submodules and
cache them at package level — so `from bot_bottle.backend import X` and
`patch.object(backend_mod, X, …)` keep working without importing the
framework (or every backend) at package-init time."""
mod = _LAZY_MODULES.get(name)
if mod is None:
raise AttributeError(f"module {__name__!r} has no attribute {name!r}")
from importlib import import_module
def get_bottle_backend(
name: str | None = None,
) -> BottleBackend[Any, Any]:
"""Resolve the bottle backend.
value = getattr(import_module(f"{__name__}.{mod}"), name)
globals()[name] = value
return value
`name` precedence:
1. explicit arg (CLI `--backend=<name>` passes through here)
2. BOT_BOTTLE_BACKEND env var
3. `macos-container` on compatible macOS hosts
4. `firecracker` on KVM-capable Linux hosts
5. default `docker`
Dies with a pointer at the known backends if the chosen name
isn't implemented."""
resolved = name or os.environ.get("BOT_BOTTLE_BACKEND") or _default_backend_name()
if resolved not in _BACKENDS:
known = ", ".join(sorted(_BACKENDS))
die(f"unknown backend {resolved!r}; known backends: {known}")
return _BACKENDS[resolved]
def _default_backend_name() -> str:
if has_backend("macos-container"):
return "macos-container"
# A KVM-capable Linux host defaults to firecracker even when the
# `firecracker` binary isn't installed yet: selecting it here routes
# start through firecracker's preflight, which prints an install
# pointer, instead of silently falling back to docker.
if FirecrackerBottleBackend.is_host_capable():
return "firecracker"
return "docker"
def known_backend_names() -> tuple[str, ...]:
"""Sorted tuple of all backend keys in `_BACKENDS`. Used by
argparse (`--backend` choices) and the dashboard's backend
picker."""
return tuple(sorted(_BACKENDS))
def has_backend(name: str) -> bool:
"""Whether the named backend's runtime prerequisites are
available on the current host. Cross-backend callers (list,
cleanup) skip unavailable backends so a docker-only host
doesn't fail when the firecracker backend isn't usable,
and vice versa.
Returns False for unknown names so callers can pass
arbitrary input without separate validation."""
if name not in _BACKENDS:
return False
return _BACKENDS[name].is_available()
def enumerate_active_agents() -> list[ActiveAgent]:
"""All currently-running agents, across every available
backend. Used by CLI `list active` and the dashboard's agents
pane so neither has to know which backends exist. Skips
backends whose `is_available()` reports False.
Sorted by `(started_at, slug)` so the list is stable across
dashboard refresh ticks — agents don't shift position while
the operator navigates with arrow keys. ISO 8601 timestamps
sort lexicographically in chronological order; `slug` is the
deterministic tiebreaker. Agents with missing metadata
(`started_at == ""`) sort first."""
out: list[ActiveAgent] = []
for name in known_backend_names():
if not has_backend(name):
continue
out.extend(_BACKENDS[name].enumerate_active())
out.sort(key=lambda a: (a.started_at, a.slug))
return out
__all__ = [
"ActiveAgent",
"BackendStatus",
"Bottle",
"BottleBackend",
"BottleCleanupPlan",
"BottleImages",
"BottlePlan",
"BottleSpec",
"ExecResult",
"CommitCancelled",
"ExecResult",
"Freezer",
"get_freezer",
"DockerBottleBackend",
"FirecrackerBottleBackend",
"MacosContainerBottleBackend",
"enumerate_active_agents",
"get_bottle_backend",
"get_freezer",
"has_backend",
"is_backend_available",
"is_backend_ready",
"known_backend_names",
]
-619
View File
@@ -1,619 +0,0 @@
"""The abstract backend contract (PRD 0018 / 0070).
The backend-neutral types every bottle backend implements: the launch
`BottleSpec`, the `BottlePlan` / `BottleCleanupPlan` ABCs, the running-`Bottle`
+ `ExecResult` shapes, `BottleImages`, and the `BottleBackend` ABC itself.
This carries the framework imports (manifest, egress, git-gate, env, workspace,
agent-provider) the contract's signatures and helpers need — which is why it
lives here rather than in `backend/__init__.py`: touching an unrelated
`backend.*` module then doesn't drag the whole framework into memory. The
thin package `__init__` re-exports these names lazily.
"""
from __future__ import annotations
import enum
import os
import shlex
import sys
from abc import ABC, abstractmethod
from contextlib import AbstractContextManager, contextmanager
from dataclasses import dataclass
from pathlib import Path
from typing import Generator, Generic, Sequence, TypeVar
from ..agent_provider import AgentProvisionPlan, get_provider, build_agent_provision_plan
from ..egress import EgressPlan
from ..git_gate import GitGatePlan
from ..log import die, info
from ..util import expand_tilde
from ..manifest import Manifest, ManifestIndex
from ..supervisor.plan import SupervisePlan
from ..env import resolve_env, ResolvedEnv
from ..workspace import WorkspacePlan, workspace_plan
from .print_util import print_multi, visible_agent_env_names
from .util import host_skill_dir
class BackendStatus(enum.IntEnum):
"""Return codes for BottleBackend.status(). READY == 0 so callsites
can compare against 0 or the named constant interchangeably."""
READY = 0
@dataclass(frozen=True)
class BottleSpec:
"""CLI-supplied intent. Backend-agnostic — each backend's prepare
step consumes it and produces its own backend-specific plan.
Resolved values (image names, container name, scratch paths, runsc
availability) live on the plan, not the spec."""
manifest: ManifestIndex
agent_name: str
copy_cwd: bool
user_cwd: str
# PRD 0016 follow-up: when set, the backend's prepare step uses
# this identity instead of minting a fresh one — the resume path
# (`cli.py resume <identity>`) sets this to continue an existing
# bottle's state. Empty string for a fresh `start`.
identity: str = ""
label: str = ""
color: str = ""
# Ordered bottle names selected at launch (issue #269). When non-empty
# they are merged in order and replace the agent's `bottle:` field.
bottle_names: tuple[str, ...] = ()
# True when launched via --headless (no TTY, no interactive prompts).
# The git-gate host-key preflight uses this to error rather than prompt.
headless: bool = False
# Image startup policy. "fresh" preserves the normal build path;
# "cached" reuses the current local image/artifact without rebuilding.
image_policy: str = "fresh"
@dataclass(frozen=True)
class BottlePlan(ABC):
"""Base output of a backend's prepare step. Concrete subclasses
(e.g. DockerBottlePlan) add backend-specific resolved fields."""
spec: BottleSpec
manifest: Manifest
stage_dir: Path
git_gate_plan: GitGatePlan
@property
def guest_home(self) -> str:
return self.agent_provision.guest_home
@property
def git_gate_insteadof_host(self) -> str:
"""Host (and optional port) used in git-gate insteadOf URLs.
Docker uses the compose-network DNS alias; VM backends may
override with an IP:port when the guest has no DNS."""
return "git-gate"
@property
def git_gate_insteadof_scheme(self) -> str:
"""URL scheme for git-gate insteadOf rewrites. 'git' for
Docker (git daemon); VM backends may override (e.g. 'http'
over a published host port)."""
return "git"
egress_plan: EgressPlan
supervise_plan: SupervisePlan | None
agent_provision: AgentProvisionPlan
@property
def workspace_plan(self) -> WorkspacePlan:
return workspace_plan(self.spec, guest_home=self.guest_home)
def print(self) -> None:
"""Render the y/N preflight summary to stderr."""
spec = self.spec
manifest = self.manifest
agent = manifest.agent
bottle = manifest.bottle
env_names = visible_agent_env_names(
sorted(
set(bottle.env.keys())
| set(self.agent_provision.guest_env.keys())
),
hidden_env_names=self.agent_provision.hidden_env_names,
)
print(file=sys.stderr)
info(f"agent : {spec.agent_name}")
info(f"provider : {self.agent_provision.template}")
print_multi("env ", env_names)
print_multi("skills ", list(agent.skills))
effective_bottles = (
list(spec.bottle_names) if spec.bottle_names
else ([agent.bottle] if agent.bottle else [])
)
print_multi("bottle ", effective_bottles)
identity = manifest.git_identity_summary()
if identity:
info(f" git identity : {identity}")
git_lines = [
f"{u.name}{u.upstream_host}:{u.upstream_port}"
for u in self.git_gate_plan.upstreams
]
if git_lines:
print_multi(" git gate ", git_lines)
if self.egress_plan.routes:
egress_lines = []
for r in self.egress_plan.routes:
auth = f" [auth:{r.auth_scheme}]" if r.auth_scheme else ""
egress_lines.append(f"{r.host}{auth}")
print_multi(" egress ", egress_lines)
print(file=sys.stderr)
@dataclass(frozen=True)
class BottleCleanupPlan(ABC):
"""Base output of a backend's prepare_cleanup step. Concrete
subclasses (e.g. DockerBottleCleanupPlan) carry backend-specific
lists of resources to be removed and implement `print` + `empty`."""
@abstractmethod
def print(self) -> None:
"""Render the cleanup y/N summary to stderr."""
@property
@abstractmethod
def empty(self) -> bool:
"""True iff there is nothing to clean up; the CLI uses this to
short-circuit before showing the y/N."""
@dataclass(frozen=True)
class ExecResult:
"""Captured result of `Bottle.exec`. Backend-neutral: the Docker
impl populates it from a `subprocess.CompletedProcess`, but a
VM backend could populate it from any source that produces a
returncode + captured streams."""
returncode: int
stdout: str
stderr: str
@dataclass(frozen=True)
class ActiveAgent:
"""One currently-running agent, as the CLI `active` and
dashboard agents pane render it. ("Agent" is the project's
consistent name for the thing running inside a bottle — the
bottle is the container, the agent is what runs in it.)
Fields are deliberately backend-neutral. `services` is the set
of gateway daemons currently up for this bottle (`egress`,
`git-gate`, `supervise`); the dashboard uses it to
gate edit verbs. `backend_name` is the matching key in
`_BACKENDS` (`docker` / `firecracker` / `macos-container`) — used by the active-
list rendering to disambiguate and by the dashboard's
re-attach path."""
backend_name: str
slug: str
agent_name: str # from metadata.json; "?" if missing
started_at: str # ISO 8601 from metadata.json; "" if missing
services: tuple[str, ...] # alphabetical
label: str = ""
color: str = ""
class Bottle(ABC):
"""Handle to a running bottle. Yielded by a backend's launch step.
`exec_agent` runs the selected agent CLI inside the bottle and
blocks until the session ends. `exec` runs a POSIX shell script inside the bottle
and returns the captured result. `cp_in` copies a host path into
the bottle. `close` is an idempotent alias for context-manager
teardown.
"""
name: str
@abstractmethod
def agent_argv(
self, argv: list[str], *, tty: bool = True,
) -> list[str]:
"""Return the host-side argv that runs the selected agent
inside the bottle. Used by `exec_agent` for foreground
handoffs and by the dashboard's tmux `respawn-pane` flow,
which needs the argv up front (it spawns claude in a tmux
pane rather than as a child of the current process).
Implementations transparently inject
`--append-system-prompt-file` when the bottle was launched
with a provisioned prompt path."""
...
@abstractmethod
def exec_agent(self, argv: list[str], *, tty: bool = True) -> int: ...
@abstractmethod
def exec(self, script: str, *, user: str = "node") -> ExecResult:
"""Run `script` as a POSIX shell script inside the bottle as
`user` (default `node`, matching the agent image's USER
directive) and return the captured stdout/stderr/returncode.
The bottle's environment (including HTTPS_PROXY pointing at
the egress daemon) is inherited by the child. Non-zero
exit does not raise — callers inspect `returncode`
themselves.
Pass `user="root"` for shell-outs that need privileged file
writes / package install — provisioning calls that need root
bypass `Bottle.exec` and use the backend-specific raw
machine-exec helper, but the tests have a legitimate use
case for arbitrary-user runs."""
@abstractmethod
def cp_in(self, host_path: str, container_path: str) -> None: ...
@abstractmethod
def close(self) -> None: ...
PlanT = TypeVar("PlanT", bound=BottlePlan)
CleanupT = TypeVar("CleanupT", bound=BottleCleanupPlan)
@dataclass(frozen=True)
class BottleImages:
"""Resolved image references (or artifact paths) for a bottle launch.
For Docker/macOS-container backends, `agent` and `sidecar` are string
image refs. For the smolmachines backend they are Path objects pointing
to pre-built `.smolmachine` artifacts."""
agent: str | Path
sidecar: str | Path = ""
class BottleBackend(ABC, Generic[PlanT, CleanupT]):
"""Abstract base for selectable bottle backends. Concrete subclasses
(e.g. DockerBottleBackend) own their own prepare/launch impls.
Parameterized over the backend's concrete plan + cleanup-plan types
so subclass methods get the narrow type without isinstance
boilerplate."""
name: str
# Whether this backend can run a container engine *inside* the bottle.
# Backends that cannot must reject `nested_containers: true` rather than
# reach for a host daemon socket (issue #392).
supports_nested_containers: bool = False
def prepare(self, spec: BottleSpec, stage_dir: Path) -> PlanT:
"""Template method: run cross-backend host-side validation, then
delegate to the subclass's `_resolve_plan` for the
backend-specific resolution (names, scratch files, etc.). The
validation step is enforced here so a future backend cannot
accidentally skip it. No remote/runtime resources are created."""
from .resolve_common import (
merge_provision_env_vars,
mint_slug,
prepare_agent_state_dir,
prepare_egress,
prepare_git_gate,
prepare_supervise,
reject_nested_containers,
resolve_manifest_dockerfile,
write_launch_metadata,
)
manifest = self._validate(spec)
if not self.supports_nested_containers:
reject_nested_containers(self.name, manifest)
self._preflight()
from ..git_gate import GitGate
manifest = GitGate().preflight_host_keys(
manifest,
headless=spec.headless,
home_md=spec.manifest.home_md,
)
manifest_bottle = manifest.bottle
manifest_agent_provider = manifest_bottle.agent_provider
agent_provider = get_provider(manifest_agent_provider.template)
resolved_env = resolve_env(manifest)
workspace = workspace_plan(spec, guest_home=agent_provider.guest_home)
slug = mint_slug(spec)
write_launch_metadata(slug, spec, compose_project="", backend=self.name)
# Manifest may override the Dockerfile per-bottle; otherwise fall
# back to the provider plugin's bundled Dockerfile (next to its
# agent_provider.py module).
if manifest_agent_provider.dockerfile:
agent_dockerfile_path = resolve_manifest_dockerfile(
manifest_agent_provider.dockerfile, spec,
)
else:
agent_dockerfile_path = str(agent_provider.dockerfile)
agent_dir, prompt_file = prepare_agent_state_dir(slug, manifest)
agent_provision_plan = build_agent_provision_plan(
template=manifest_agent_provider.template,
dockerfile=agent_dockerfile_path,
state_dir=agent_dir,
instance_name=f"bot-bottle-{slug}",
prompt_file=prompt_file,
guest_env=self._build_guest_env(resolved_env),
forward_host_credentials=manifest_agent_provider.forward_host_credentials,
auth_token=manifest_agent_provider.auth_token,
host_env=dict(os.environ),
trusted_project_path=workspace.workdir,
label=spec.label,
color=spec.color,
provider_settings=manifest_agent_provider.settings,
)
agent_provision_plan = merge_provision_env_vars(agent_provision_plan)
egress_plan = prepare_egress(manifest_bottle, slug, agent_provision_plan)
supervise_plan = prepare_supervise(manifest_bottle, slug)
git_gate_plan = prepare_git_gate(manifest_bottle, slug)
return self._resolve_plan(
spec,
manifest=manifest,
slug=slug,
resolved_env=resolved_env,
agent_provision_plan=agent_provision_plan,
egress_plan=egress_plan,
supervise_plan=supervise_plan,
git_gate_plan=git_gate_plan,
stage_dir=stage_dir,
)
def _build_guest_env(self, resolved_env: ResolvedEnv) -> dict[str, str]:
return {}
def _preflight(self) -> None:
"""
tasks to do before resolving a plan
"""
pass
def _validate(self, spec: BottleSpec) -> Manifest:
"""Cross-backend pre-launch checks. Parses the selected agent and
its bottle (raising ManifestError on invalid content), confirms
skills are present on the host, and every git IdentityFile resolves.
Returns the loaded Manifest for the selected agent. Subclasses with
additional preconditions should override and call
`super()._validate(spec)` first."""
manifest = spec.manifest.load_for_agent(spec.agent_name, spec.bottle_names)
self._validate_skills(manifest.agent.skills)
self._validate_agent_provider_dockerfile(spec, manifest)
return manifest
def _validate_skills(self, skills: Sequence[str]) -> None:
"""Each named skill must be a directory under the host's
`~/.claude/skills/`. The check is purely host-side, so the
default impl covers every backend."""
for name in skills:
path = host_skill_dir(name)
if not os.path.isdir(path):
die(
f"skill '{name}' not found on host at {path}. "
f"Create it under ~/.claude/skills/, then re-run."
)
def _validate_agent_provider_dockerfile(self, spec: BottleSpec, manifest: Manifest) -> None:
bottle = manifest.bottle
dockerfile = bottle.agent_provider.dockerfile
if not dockerfile:
return
path = Path(expand_tilde(dockerfile))
if not path.is_absolute():
path = Path(spec.user_cwd) / path
if not path.is_file():
effective = (
", ".join(spec.bottle_names) if spec.bottle_names else manifest.agent.bottle
)
die(
f"agent_provider.dockerfile for bottle "
f"'{effective}' not found: {path}"
)
@abstractmethod
def _resolve_plan(self,
spec: BottleSpec,
*,
manifest: Manifest,
slug: str,
resolved_env: ResolvedEnv,
agent_provision_plan: AgentProvisionPlan,
egress_plan: EgressPlan,
git_gate_plan: GitGatePlan,
supervise_plan: SupervisePlan | None,
stage_dir: Path) -> PlanT:
"""Backend-specific plan resolution: image/container names,
env-file, prompt-file, proxy plan, runtime detection. Called by
`prepare` after `_validate` succeeds. Instance name, image,
prompt file, Dockerfile path, and guest home all live on
`agent_provision_plan` — the source of truth."""
def prelaunch_checks(self, plan: PlanT) -> None:
"""Raise StaleImageError if any cached image used by this plan is stale.
No-op default; backends override to call the shared check_stale*
helpers on their image/artifact timestamps. Called by the CLI before
launch so the operator can be prompted outside the launch context."""
@contextmanager
def launch(self, plan: PlanT) -> Generator[Bottle, None, None]:
"""Template: build or load images, then delegate to _launch_impl."""
images = self._build_or_load_images(plan)
with self._launch_impl(plan, images) as bottle:
yield bottle
@abstractmethod
def _build_or_load_images(self, plan: PlanT) -> BottleImages:
"""Return the agent and sidecar image references (or artifact paths)
for this plan, building fresh images when the policy requires it."""
@abstractmethod
def _launch_impl(self, plan: PlanT, images: BottleImages) -> AbstractContextManager[Bottle]:
"""Bring up the bottle using pre-resolved images; yield a handle; tear down on exit."""
def provision(self, plan: PlanT, bottle: "Bottle") -> str | None:
"""Copy host-side files (CA cert, prompt, skills, .git) into
the running bottle. Called from `launch` after the container
/ machine is up. Returns the in-container prompt path if a
prompt was provisioned, else None — the Bottle handle uses it
to decide whether to add provider-specific prompt args to the
agent's argv.
Default orchestration: ca → prompt → provider apply → skills
→ workspace → git → supervise-mcp. CA install runs first so
the agent's trust store is rebuilt before anything inside the
agent makes a TLS call.
Per PRD 0050 the per-provider steps (prompt, skills,
declarative provision-plan apply, supervise MCP registration)
live on the `AgentProvider` plugin. The backend only owns the
steps that are about backend infrastructure (CA, workspace,
git) and surfaces the supervise daemon URL its launch step
knows about via `supervise_mcp_url`.
PRD 0017: cred-proxy's agent-side dotfile rewrites (~/.npmrc,
~/.gitconfig insteadOf, tea config) are gone. Egress-proxy is
on the agent's HTTP_PROXY path so every tool that respects
HTTPS_PROXY (claude-code, git over HTTPS, npm, curl) is
intercepted without per-tool reconfiguration."""
provider = get_provider(plan.agent_provision.template)
provider.provision_ca(bottle, plan)
prompt_path = provider.provision_prompt(plan, bottle)
provider.provision(plan, bottle)
provider.provision_skills(plan, bottle)
self.provision_workspace(plan, bottle)
provider.provision_git(bottle, plan)
provider.provision_supervise_mcp(
plan, bottle, self.supervise_mcp_url(plan),
)
return prompt_path
def provision_workspace(self, plan: PlanT, bottle: "Bottle") -> None:
"""Copy the operator workspace into the running bottle.
This is the only supported workspace-provisioning path: Docker
does not build a derived image containing the current
workspace."""
workspace = plan.workspace_plan
if not (workspace.enabled and workspace.copy_contents):
return
guest_parent = workspace.guest_path.rsplit("/", 1)[0] or "/"
guest_path = shlex.quote(workspace.guest_path)
guest_parent = shlex.quote(guest_parent)
owner = shlex.quote(workspace.owner)
mode = shlex.quote(workspace.mode)
info(f"copying {workspace.host_path} -> {bottle.name}:{workspace.guest_path}")
bottle.exec(
f"rm -rf {guest_path} && mkdir -p {guest_parent}",
user="root",
)
bottle.cp_in(str(workspace.host_path), workspace.guest_path)
bottle.exec(
f"chown -R {owner} {guest_path} && chmod {mode} {guest_path}",
user="root",
)
def supervise_mcp_url(self, plan: PlanT) -> str:
"""Return the agent-side URL of the per-bottle supervise
gateway, or "" when this bottle has no gateway. The provider
plugin's `provision_supervise_mcp` uses it to register the
MCP entry inside the guest.
Default returns "" so backends without supervise support
don't have to implement it. Docker and firecracker override."""
del plan
return ""
def ensure_orchestrator(self) -> str:
"""Bring up this backend's per-host orchestrator + shared gateway
(idempotent) and return the host-reachable control-plane URL.
This is the backend-agnostic bring-up entry point: `launch` calls
it as part of starting a bottle, and operator tools (`supervise`)
call it to start the control plane on demand when none is running
yet. Docker starts the orchestrator + gateway containers;
firecracker boots the infra VM. Backends with no orchestrator
(macos-container) die with a pointer — the default here."""
die(f"backend {self.name!r} has no orchestrator control plane")
@abstractmethod
def prepare_cleanup(self) -> CleanupT:
"""Enumerate orphaned resources from previous bottles. No side
effects; safe to call before the y/N."""
@abstractmethod
def cleanup(self, plan: CleanupT) -> None:
"""Remove everything described by the cleanup plan."""
@abstractmethod
def enumerate_active(self) -> Sequence[ActiveAgent]:
"""Return every currently-running agent on this backend.
Empty when none. Backend-specific: docker queries `docker
compose ls`; firecracker cross-references its running gateway
containers against per-bottle metadata."""
@classmethod
@abstractmethod
def is_available(cls) -> bool:
"""Whether this backend's runtime prerequisites are satisfied
on the current host. Docker → `docker` on PATH; firecracker →
Linux + KVM. Used by the cross-backend
`enumerate_active_agents` / `cmd_cleanup` to skip backends
the operator hasn't installed, so a docker-only host
doesn't fail when `cli.py active` walks past
firecracker."""
@classmethod
@abstractmethod
def setup(cls) -> int:
"""Emit this backend's one-time host setup — privileged network
pool, daemon bring-up, install pointers, etc. — as
host-appropriate config or commands. Prints to stdout/stderr and
returns a shell exit code (0 = nothing to report / success). A
backend that needs no host setup prints a short note and returns
0. Invoked generically by `./cli.py backend setup [--backend=…]`
so operators can provision any backend without a
backend-specific command. Classmethod (like `is_available`) —
it's a host query, not per-bottle state."""
@classmethod
@abstractmethod
def status(cls, *, quiet: bool = False) -> int:
"""Report whether this backend's prerequisites are satisfied on
the host — binaries, daemon reachability, network pool, range
conflicts, etc. Returns BackendStatus.READY (0) when the backend
is ready to launch and non-zero when something is missing.
When quiet=False (default) prints a human-readable summary to
stderr. When quiet=True returns the status code silently —
useful for cheap programmatic checks.
Invoked by `./cli.py backend status [--backend=…]` (quiet=False)
and by is_backend_ready() (caller-controlled)."""
@classmethod
@abstractmethod
def teardown(cls) -> int:
"""Undo `setup()` — the inverse operation, surfaced as
`./cli.py backend teardown [--backend=…]` (uninstall). Symmetric
with setup: where setup is advisory (prints the privileged
commands / declarative config to apply), teardown prints the
commands / config change to remove the host prerequisites. A
backend with no host setup prints a short note and returns 0.
Not called by the launch path or the test suite."""
+7 -31
View File
@@ -11,42 +11,18 @@ The bulk of the implementation lives in sibling modules:
- launch: bring-up + teardown context manager
- cleanup: orphan enumeration, removal, active listing
- backend: DockerBottleBackend façade wiring the above
- infra: DockerInfraService (the per-host orchestrator + gateway pair)
Thin by design: the public names are re-exported lazily via `__getattr__`, so
importing a leaf like `backend.docker.util` doesn't drag `DockerBottleBackend`
(and the whole framework it pulls) into memory.
This file only re-exports the public names so
`from bot_bottle.backend.docker import DockerBottleBackend` keeps
working.
"""
from __future__ import annotations
from typing import TYPE_CHECKING, Any
if TYPE_CHECKING:
from .backend import DockerBottleBackend
from .bottle import DockerBottle
from .bottle_cleanup_plan import DockerBottleCleanupPlan
from .bottle_plan import DockerBottlePlan
_LAZY_MODULES: dict[str, str] = {
"DockerBottleBackend": "backend",
"DockerBottle": "bottle",
"DockerBottleCleanupPlan": "bottle_cleanup_plan",
"DockerBottlePlan": "bottle_plan",
}
def __getattr__(name: str) -> Any:
mod = _LAZY_MODULES.get(name)
if mod is None:
raise AttributeError(f"module {__name__!r} has no attribute {name!r}")
from importlib import import_module
value = getattr(import_module(f"{__name__}.{mod}"), name)
globals()[name] = value
return value
from .backend import DockerBottleBackend
from .bottle import DockerBottle
from .bottle_cleanup_plan import DockerBottleCleanupPlan
from .bottle_plan import DockerBottlePlan
__all__ = [
"DockerBottle",
+9 -19
View File
@@ -20,19 +20,18 @@ infrastructure: CA install and git copy-in.
from __future__ import annotations
import shutil
import io
from contextlib import contextmanager, redirect_stderr
from contextlib import contextmanager
from pathlib import Path
from typing import Generator, Sequence
from ...supervisor.types import SUPERVISE_HOSTNAME, SUPERVISE_PORT
from ...supervise import SUPERVISE_HOSTNAME, SUPERVISE_PORT
from ...agent_provider import AgentProvisionPlan
from ...egress import EgressPlan
from ...env import ResolvedEnv
from ...git_gate import GitGatePlan
from ...supervisor.plan import SupervisePlan
from ...supervise import SupervisePlan
from ...manifest import Manifest
from .. import ActiveAgent, BottleBackend, BottleImages, BottleSpec
from .. import ActiveAgent, BottleBackend, BottleSpec
from . import cleanup as _cleanup
from . import enumerate as _enumerate
from . import launch as _launch
@@ -61,11 +60,8 @@ class DockerBottleBackend(BottleBackend["DockerBottlePlan", "DockerBottleCleanup
return _setup.setup()
@classmethod
def status(cls, *, quiet: bool = False) -> int:
def status(cls) -> int:
from . import setup as _setup
if quiet:
with redirect_stderr(io.StringIO()):
return _setup.status()
return _setup.status()
@classmethod
@@ -104,20 +100,14 @@ class DockerBottleBackend(BottleBackend["DockerBottlePlan", "DockerBottleCleanup
stage_dir=stage_dir,
)
def prelaunch_checks(self, plan: DockerBottlePlan) -> None:
_launch.stale_checks(plan)
def _build_or_load_images(self, plan: DockerBottlePlan) -> BottleImages:
return _launch.build_or_load_images(plan)
@contextmanager
def _launch_impl(self, plan: DockerBottlePlan, images: BottleImages) -> Generator[DockerBottle, None, None]:
with _launch.launch(plan, images, provision=self.provision) as bottle:
def launch(self, plan: DockerBottlePlan) -> Generator[DockerBottle, None, None]:
with _launch.launch(plan, provision=self.provision) as bottle:
yield bottle
def ensure_orchestrator(self) -> str:
from .infra import DockerInfraService
return DockerInfraService().ensure_running()
from ...orchestrator.lifecycle import OrchestratorService
return OrchestratorService().ensure_running()
def supervise_mcp_url(self, plan: DockerBottlePlan) -> str:
"""Docker bottles reach the supervise daemon via the
-4
View File
@@ -39,10 +39,6 @@ class DockerBottlePlan(BottlePlan):
# (egress proxy credentials, git-gate/supervise headers); set by launch
# from the orchestrator registration. Empty pre-registration.
identity_token: str = ""
# Encryption key for the agent's stored egress secrets; injected into the
# agent container as ENV_VAR_SECRET via the compose subprocess env (bare
# name — value never written to the compose file). Empty pre-registration.
env_var_secret: str = ""
@property
def container_name(self) -> str:
@@ -16,8 +16,7 @@ from __future__ import annotations
from typing import Any
from ...egress import Egress
from ...orchestrator.store.secret_store import ENV_VAR_SECRET_NAME
from ...egress import egress_agent_env_entries
from ..util import AGENT_CA_BUNDLE, AGENT_CA_PATH
from .bottle_plan import DockerBottlePlan
from .egress import EGRESS_PORT
@@ -59,11 +58,7 @@ def consolidated_agent_compose(
# the secret value never lands on argv or in the compose file.
for name in sorted(plan.forwarded_env.keys()):
env.append(name)
# ENV_VAR_SECRET: bare name so the value comes from the compose subprocess
# env (set in launch.py) and is never written to the compose file on disk.
if getattr(plan, "env_var_secret", ""):
env.append(ENV_VAR_SECRET_NAME)
env.extend(Egress().agent_env_entries(plan.egress_plan))
env.extend(egress_agent_env_entries(plan.egress_plan))
service: dict[str, Any] = {
"image": plan.image,
@@ -1,13 +1,19 @@
"""Consolidated bottle launch sequence for the docker backend (PRD 0070).
Composes the orchestrator primitives into the register/teardown sequence:
Composes the orchestrator primitives into the register/teardown sequence that
replaces the per-bottle gateway:
1. ensure the per-host pair (orchestrator + gateway containers) is up;
2. allocate the bottle a pinned source IP on the gateway network;
3. register it and provision its git-gate repos/creds into the gateway.
1. ensure the orchestrator control plane + shared gateway are up;
2. allocate the bottle a pinned source IP on the gateway network (the
attribution key), skipping the gateway's own address + live bottles;
3. register it (egress policy blob + slug metadata) → bottle id + identity
token;
4. provision its git-gate repos/creds into the running gateway.
Returns a `LaunchContext` with everything the agent container needs to
attach. The agent `docker run` itself is the backend's job; this owns the
It returns a `LaunchContext` with everything the agent container needs to
attach — network, pinned IP, the gateway's address (its proxy target), the
orchestrator URL, and the identity token. The agent `docker run` itself is
the backend's job (it owns provider provisioning); this owns the
orchestrator-facing wiring so that sequence stays testable in isolation.
"""
@@ -15,18 +21,19 @@ from __future__ import annotations
from dataclasses import dataclass
from ... import log
from .util import run_docker
from ...docker_cmd import run_docker
from ...egress import EgressPlan
from ...git_gate import GitGatePlan
from ...orchestrator.client import OrchestratorClient
from ...gateway import GATEWAY_NETWORK
from .infra import INFRA_NAME, DockerInfraService
from ...orchestrator.store.secret_store import ENV_VAR_SECRET_NAME
from ...orchestrator.reprovision import reprovision_bottles
from ..provision_bottle import deprovision_bottle, provision_bottle
from .gateway_transport import DockerGatewayTransport
from ...orchestrator.gateway import GATEWAY_NAME, GATEWAY_NETWORK
from ...orchestrator.lifecycle import OrchestratorService
from ...orchestrator.registration import registration_inputs
from .gateway_net import next_free_ip
from .gateway_provision import (
DockerGatewayTransport,
deprovision_git_gate,
provision_git_gate,
)
class ConsolidatedLaunchError(RuntimeError):
@@ -43,7 +50,6 @@ class LaunchContext:
network: str # the shared gateway network to attach to
gateway_ip: str # the gateway's address — the agent's proxy target
orchestrator_url: str
env_var_secret: str = "" # encryption key injected into the agent's env
def _network_cidr(network: str) -> str:
@@ -60,103 +66,71 @@ def _network_cidr(network: str) -> str:
return cidr
def _container_ip(name: str, network: str) -> str:
"""A container's IPv4 address on `network`, or raise."""
proc = run_docker([
"docker", "inspect", "--format",
f'{{{{(index .NetworkSettings.Networks "{network}").IPAddress}}}}', name,
])
ip = proc.stdout.strip()
if proc.returncode != 0 or not ip:
raise ConsolidatedLaunchError(
f"gateway {name} has no address on {network}: {proc.stderr.strip()}"
)
return ip
def _network_container_ips(network: str) -> list[str]:
"""Every address currently assigned on the gateway network — the ground
truth for "in use": the gateway container and every live agent. Read from
the network so a new bottle can't collide with anything actually attached."""
truth for "in use": the gateway + orchestrator infrastructure containers
and every live agent. Read from the network so a new bottle can't collide
with anything actually attached (a registry-only view would miss the
orchestrator/gateway containers)."""
proc = run_docker([
"docker", "network", "inspect", "--format",
"{{range .Containers}}{{.IPv4Address}} {{end}}", network,
])
ips: list[str] = []
for entry in proc.stdout.split():
# entries look like "172.20.0.2/16" — keep the address.
ips.append(entry.split("/", 1)[0])
return ips
def _reprovision_running_bottles(
orchestrator_url: str,
network: str = GATEWAY_NETWORK,
infra_name: str = INFRA_NAME,
) -> None:
"""Re-inject egress tokens for any registered bottles that lost their
in-memory tokens (e.g., after an orchestrator restart).
For each registered bottle whose source IP maps to a live container on the
gateway network, reads ENV_VAR_SECRET via ``docker exec … printenv`` and
calls ``POST /bottles/<id>/reprovision_gateway``. Idempotent — a no-op
when the orchestrator already has all tokens loaded. Best-effort: a single
container exec failure never blocks a new bottle launch."""
client = OrchestratorClient(orchestrator_url)
# Build {source_ip: container_name} from live containers on the gateway
# network, excluding the gateway container itself.
try:
proc = run_docker([
"docker", "network", "inspect",
"--format", "{{range .Containers}}{{.Name}} {{.IPv4Address}}\n{{end}}",
network,
])
except OSError as exc:
log.info(f"egress token reprovision skipped: {exc}")
return
ip_to_container: dict[str, str] = {}
for line in proc.stdout.splitlines():
parts = line.strip().split()
if len(parts) >= 2 and parts[0] != infra_name:
ip = parts[1].split("/", 1)[0]
if ip:
ip_to_container[ip] = parts[0]
secrets_by_ip: dict[str, str] = {}
for source_ip, container_name in ip_to_container.items():
proc = run_docker(
["docker", "exec", container_name, "printenv", ENV_VAR_SECRET_NAME]
)
if proc.returncode == 0 and proc.stdout.strip():
secrets_by_ip[source_ip] = proc.stdout.strip()
reprovisioned = reprovision_bottles(client, secrets_by_ip)
if reprovisioned:
log.info(
"reprovisioned egress tokens",
context={"count": reprovisioned},
)
def launch_consolidated(
egress_plan: EgressPlan,
git_gate_plan: GitGatePlan,
*,
image_ref: str = "",
tokens: dict[str, str] | None = None,
service: DockerInfraService | None = None,
infra_name: str = INFRA_NAME,
service: OrchestratorService | None = None,
gateway_name: str = GATEWAY_NAME,
network: str = GATEWAY_NETWORK,
) -> LaunchContext:
"""Ensure the orchestrator + gateway pair is up, allocate + register the bottle, and
provision its git-gate state. Returns the agent's attach context.
Also reprovisiones egress tokens for any already-running bottles that lost
their in-memory credentials (e.g. after an orchestrator restart), so
they regain egress access before the new bottle is registered."""
service = service or DockerInfraService()
"""Ensure the orchestrator + gateway are up, allocate + register the
bottle, and provision its git-gate state. Returns the agent's attach
context. Raises `ConsolidatedLaunchError` (or the primitives' own errors)
if any step fails — the caller tears down on failure."""
service = service or OrchestratorService()
url = service.ensure_running()
# Agents attribute against the *gateway* container (data plane), not the
# orchestrator — the two planes are now separate containers. Read its
# agent-facing address + provisioning transport off the Gateway service.
gateway = service.gateway()
_reprovision_running_bottles(url, network=network, infra_name=gateway.name)
client = OrchestratorClient(url)
cidr = _network_cidr(network)
gateway_ip = gateway.address()
gateway_ip = _container_ip(gateway_name, network)
source_ip = next_free_ip(cidr, _network_container_ips(network))
transport = gateway.provisioning_transport()
reg = provision_bottle(
client, source_ip, egress_plan, git_gate_plan, transport,
image_ref=image_ref, tokens=tokens,
inputs = registration_inputs(egress_plan)
reg = client.register_bottle(
source_ip, image_ref=image_ref, policy=inputs.policy,
metadata=inputs.metadata, tokens=tokens,
)
try:
provision_git_gate(
DockerGatewayTransport(gateway_name), reg.bottle_id, git_gate_plan)
except Exception:
# Roll the registration back so a provisioning failure leaves no orphan.
client.teardown_bottle(reg.bottle_id)
raise
return LaunchContext(
bottle_id=reg.bottle_id,
identity_token=reg.identity_token,
@@ -164,22 +138,21 @@ def launch_consolidated(
network=network,
gateway_ip=gateway_ip,
orchestrator_url=url,
env_var_secret=reg.env_var_secret,
)
def deprovision_consolidated(
bottle_id: str, *, orchestrator_url: str, infra_name: str = INFRA_NAME,
timeout: float | None = None,
def teardown_consolidated(
bottle_id: str, *, orchestrator_url: str, gateway_name: str = GATEWAY_NAME,
) -> None:
"""Deregister the bottle and remove its git-gate state. Idempotent."""
deprovision_bottle(bottle_id, DockerGatewayTransport(infra_name),
orchestrator_url=orchestrator_url, timeout=timeout)
"""Deregister the bottle and remove its git-gate state from the gateway.
Both steps are idempotent so this is safe from a cleanup trap."""
OrchestratorClient(orchestrator_url).teardown_bottle(bottle_id)
deprovision_git_gate(DockerGatewayTransport(gateway_name), bottle_id)
__all__ = [
"LaunchContext",
"launch_consolidated",
"deprovision_consolidated",
"teardown_consolidated",
"ConsolidatedLaunchError",
]
+2 -2
View File
@@ -1,6 +1,6 @@
"""Active-agent enumeration for the docker backend.
Returns `ActiveAgent` records the CLI `active` command and the
Returns `ActiveAgent` records the CLI `list active` command and the
dashboard agents pane consume. Empty when docker isn't reachable
— gated by `has_backend('docker')` at the cross-backend caller
so this module trusts that docker is available when called.
@@ -60,7 +60,7 @@ def _parse_services_by_project(stdout: str) -> dict[str, set[str]]:
def _query_services_by_project() -> dict[str, set[str]]:
"""One `docker ps` call → `{project: {service, ...}}`. Used
by the CLI's `active` and the dashboard's agents pane —
by the CLI's `list active` and the dashboard's agents pane —
one subprocess per refresh tick, not one per bottle."""
try:
r = subprocess.run(
@@ -1,14 +1,12 @@
"""Provision one bottle's git-gate state into the running shared gateway
(PRD 0070). Backend-neutral: it drives any `GatewayTransport` (docker/apple
exec+cp, firecracker SSH), and the guest-side paths it writes are identical
inside every backend's gateway because they all run the same gateway image.
(PRD 0070, docker slice).
The consolidated gateway serves every bottle's repos under `/git/<bottle_id>/`
with per-repo credentials under `/git-gate/creds/<bottle_id>/`. When a bottle
is registered the launcher must place *its* deploy keys + known_hosts into
that per-bottle creds dir and init its bare repos there so this copies the
credential files into the live gateway and runs the (namespaced, init-only)
provisioning script produced by `git_gate_render_provision`.
credential files into the live gateway container and runs the (namespaced,
init-only) provisioning script produced by `git_gate_render_provision`.
Isolating each bottle's creds dir + repo root by id is what keeps one
bottle's push credentials out of another's repos on the shared gateway.
@@ -17,9 +15,10 @@ bottle's push credentials out of another's repos on the shared gateway.
from __future__ import annotations
import re
from typing import Protocol
from ..git_gate import GitGatePlan, git_gate_render_provision
from ..gateway import GatewayProvisionError, GatewayTransport
from ...docker_cmd import run_docker
from ...git_gate import GitGatePlan, git_gate_render_provision
# bottle ids index the gateway's per-bottle repo + creds dirs; they land in
# exec/cp path arguments, so validate before any path is built (a traversal
@@ -28,6 +27,46 @@ from ..gateway import GatewayProvisionError, GatewayTransport
_SAFE_BOTTLE_ID = re.compile(r"[A-Za-z0-9_-]+")
class GatewayProvisionError(RuntimeError):
"""A git-gate provisioning step against the running gateway failed."""
class GatewayTransport(Protocol):
"""How the launcher stages files + runs commands in the running gateway.
Backend-neutral so the same provisioning logic serves the docker gateway
(exec/cp over the docker socket) and the firecracker gateway VM (over
SSH)."""
def exec(self, argv: list[str]) -> None:
"""Run `argv` in the gateway, raising `GatewayProvisionError` on
failure."""
def cp_into(self, src: str, dest: str) -> None:
"""Copy host file `src` to `dest` in the gateway, raising on
failure."""
class DockerGatewayTransport:
"""`GatewayTransport` for the docker gateway container (exec/cp)."""
def __init__(self, gateway: str) -> None:
self.gateway = gateway
def exec(self, argv: list[str]) -> None:
proc = run_docker(["docker", "exec", self.gateway, *argv])
if proc.returncode != 0:
raise GatewayProvisionError(
f"gateway exec {argv!r} failed: {proc.stderr.strip()}"
)
def cp_into(self, src: str, dest: str) -> None:
proc = run_docker(["docker", "cp", src, f"{self.gateway}:{dest}"])
if proc.returncode != 0:
raise GatewayProvisionError(
f"gateway cp {src} -> {dest} failed: {proc.stderr.strip()}"
)
def _require_safe(bottle_id: str) -> None:
if not _SAFE_BOTTLE_ID.fullmatch(bottle_id):
raise GatewayProvisionError(f"unsafe bottle id {bottle_id!r}")
@@ -89,5 +128,5 @@ def deprovision_git_gate(transport: GatewayTransport, bottle_id: str) -> None:
__all__ = [
"provision_git_gate", "deprovision_git_gate",
"GatewayProvisionError", "GatewayTransport",
"GatewayProvisionError", "GatewayTransport", "DockerGatewayTransport",
]
@@ -1,35 +0,0 @@
"""The `GatewayTransport` for the docker gateway container (PRD 0070).
How the launcher stages files + runs commands in the running gateway container:
`docker exec` / `docker cp` over the docker socket. The backend-neutral
provisioning logic that drives it lives in `backend.provision_gateway`.
"""
from __future__ import annotations
from .util import run_docker
from ...gateway import GatewayProvisionError
class DockerGatewayTransport:
"""`GatewayTransport` for the docker gateway container (exec/cp)."""
def __init__(self, gateway: str) -> None:
self.gateway = gateway
def exec(self, argv: list[str]) -> None:
proc = run_docker(["docker", "exec", self.gateway, *argv])
if proc.returncode != 0:
raise GatewayProvisionError(
f"gateway exec {argv!r} failed: {proc.stderr.strip()}"
)
def cp_into(self, src: str, dest: str) -> None:
proc = run_docker(["docker", "cp", src, f"{self.gateway}:{dest}"])
if proc.returncode != 0:
raise GatewayProvisionError(
f"gateway cp {src} -> {dest} failed: {proc.stderr.strip()}"
)
__all__ = ["DockerGatewayTransport"]
-152
View File
@@ -1,152 +0,0 @@
"""The per-host control plane + gateway for the docker backend (PRD 0070).
Runs the orchestrator (control plane) and the gateway (data plane) as **two
separate containers**, split now that #469 got the DB off the data plane:
* `bot-bottle-orchestrator` — the lean control-plane container
(`DockerOrchestrator`). Joins the `bot-bottle-orchestrator` **control
network** (`--internal`) only, plus a host-loopback publish for the CLI.
Sole opener of `bot-bottle.db`; holds the signing key.
* `bot-bottle-gateway` — the data-plane container (`DockerGateway`).
**Dual-homed** on the agent-facing `bot-bottle-gateway` network *and* the
control network, so it reaches the orchestrator by name over docker DNS
(`http://bot-bottle-orchestrator:8099`) while agents — which are never on
the control network — cannot reach the control plane at all (the L3 block
Firecracker's nft already has). Holds the `gateway` JWT + the mitmproxy CA.
`DockerInfraService` composes the two services (`orchestrator()` + `gateway()`)
and brings them up as an idempotent per-host pair. Callers use `ensure_running()`
(returns the host control-plane URL) + `gateway_name` (the container the launch
flow attributes agents against).
"""
from __future__ import annotations
from pathlib import Path
from .util import run_docker
from .gateway import DockerGateway
from .orchestrator import (
DockerOrchestrator,
ORCHESTRATOR_IMAGE,
ORCHESTRATOR_LABEL,
ORCHESTRATOR_NAME,
ORCHESTRATOR_NETWORK,
)
from ...paths import bot_bottle_root
from ...gateway import (
GATEWAY_IMAGE,
GATEWAY_NAME,
GATEWAY_NETWORK,
)
from ..infra_service import InfraService
from ...orchestrator.lifecycle import (
DEFAULT_PORT,
DEFAULT_STARTUP_TIMEOUT_SECONDS,
)
# Back-compat aliases: the split retired the combined `bot-bottle-infra`
# container, but the fixed-name constants some callers/tests import still map to
# the pair's public identity.
INFRA_NAME = GATEWAY_NAME # the container agents attribute against is the gateway
_REPO_ROOT = Path(__file__).resolve().parents[3]
class DockerInfraService(InfraService):
"""Composes the per-host control plane + gateway as two containers.
`orchestrator_name` / `gateway_name` let callers run independent pairs on
one host without name collisions (e.g. isolated integration tests that
can't share the production singletons)."""
def __init__(
self,
*,
port: int = DEFAULT_PORT,
network: str = GATEWAY_NETWORK,
control_network: str = ORCHESTRATOR_NETWORK,
orchestrator_image: str = ORCHESTRATOR_IMAGE,
gateway_image: str = GATEWAY_IMAGE,
repo_root: Path = _REPO_ROOT,
host_root: Path | None = None,
orchestrator_name: str = ORCHESTRATOR_NAME,
orchestrator_label: str = ORCHESTRATOR_LABEL,
gateway_name: str = GATEWAY_NAME,
) -> None:
self.port = port
self.network = network
self.control_network = control_network
self.orchestrator_image = orchestrator_image
self.gateway_image = gateway_image
self._repo_root = repo_root
self._host_root = host_root or bot_bottle_root()
self._orchestrator_name = orchestrator_name
self._orchestrator_label = orchestrator_label
self._gateway_name = gateway_name
def orchestrator(self) -> DockerOrchestrator:
"""The control-plane service. Cheap to reconstruct — `ensure_built`
builds its image, `ensure_running` brings it up, and the launch flow
reads its `url()` / `gateway_url()` / `mint_gateway_token()` off it."""
return DockerOrchestrator(
self.orchestrator_image,
name=self._orchestrator_name,
label=self._orchestrator_label,
port=self.port,
control_network=self.control_network,
repo_root=self._repo_root,
host_root=self._host_root,
)
def gateway(self) -> DockerGateway:
"""The data-plane gateway, dual-homed on the agent network + the control
network, resolving policy against the orchestrator by name. Cheap to
reconstruct — the launch flow reads its `address()` / provisioning
transport off it, and `ensure_running` connects it to the control
plane."""
return DockerGateway(
self.gateway_image,
name=self._gateway_name,
network=self.network,
control_network=self.control_network,
build_context=self._repo_root,
)
def ensure_running(
self, *, startup_timeout: float = DEFAULT_STARTUP_TIMEOUT_SECONDS,
) -> str:
"""Ensure the orchestrator + gateway containers are up; return the host
control-plane URL. Idempotent — a healthy orchestrator on current source
(and a current-image gateway) is left untouched. Raises
`OrchestratorStartError` on control-plane startup timeout."""
orchestrator = self.orchestrator()
gateway = self.gateway()
# Build both images (cache-aware; a no-op when nothing changed) before
# bringing either plane up.
orchestrator.ensure_built()
gateway.ensure_built()
orchestrator.ensure_running(startup_timeout=startup_timeout)
# Bring up (or refresh) the gateway once the control plane it resolves
# against is healthy. The orchestrator (which holds the signing key)
# mints the role-scoped `gateway` JWT here and hands it to the gateway,
# which never sees the key (#469). `connect_to_orchestrator` is
# idempotent.
gateway.connect_to_orchestrator(
orchestrator.gateway_url(), orchestrator.mint_gateway_token(),
)
return orchestrator.url()
def stop(self) -> None:
"""Remove both containers (idempotent)."""
run_docker(["docker", "rm", "--force", self._gateway_name])
run_docker(["docker", "rm", "--force", self._orchestrator_name])
__all__ = [
"DockerInfraService",
"ORCHESTRATOR_NAME",
"INFRA_NAME",
]
+35 -77
View File
@@ -37,11 +37,12 @@ from pathlib import Path
from typing import Callable, Generator
from ...agent_provider import runtime_for
from ...egress import Egress
from ...git_gate import GitGate
from ...image_cache import check_stale
from ...log import die, info, warn
from .. import BottleImages
from ...egress import egress_resolve_token_values
from ...git_gate import (
provision_git_gate_dynamic_keys,
revoke_git_gate_provisioned_keys,
)
from ...log import info, warn
from . import util as docker_mod
from .bottle import DockerBottle
from .bottle_plan import DockerBottlePlan
@@ -61,57 +62,24 @@ from .compose import (
write_compose_file,
)
from .consolidated_compose import consolidated_agent_compose
from ...orchestrator.store.config_store import resolve_teardown_timeout
from .consolidated_launch import launch_consolidated, deprovision_consolidated
from .infra import INFRA_NAME
from .gateway import DockerGateway
from .consolidated_launch import launch_consolidated, teardown_consolidated
from ...orchestrator.gateway import DockerGateway
# Where the repo root lives, for `docker build` context. Computed once.
_REPO_DIR = str(Path(__file__).resolve().parent.parent.parent.parent)
def build_or_load_images(plan: DockerBottlePlan) -> BottleImages:
"""Resolve the agent image ref for this plan.
Returns the committed snapshot if one exists, the cached image when the
policy is 'cached', or builds a fresh image and returns that."""
committed = read_committed_image(plan.slug)
if committed and docker_mod.image_exists(committed):
info(f"using committed image {committed!r}")
return BottleImages(agent=committed)
if plan.spec.image_policy == "cached":
if not docker_mod.image_exists(plan.image):
die(
f"cached agent image {plan.image!r} not found; "
"run without --cached-images to build it"
)
info(f"using cached agent image {plan.image!r}")
return BottleImages(agent=plan.image)
docker_mod.build_image(plan.image, _REPO_DIR, dockerfile=plan.dockerfile_path)
docker_mod.verify_agent_image(
plan.image, runtime_for(plan.agent_provider_template).smoke_test,
)
return BottleImages(agent=plan.image)
@contextmanager
def launch(
plan: DockerBottlePlan,
images: BottleImages,
*,
provision: Callable[[DockerBottlePlan, "DockerBottle"], str | None],
) -> Generator[DockerBottle, None, None]:
"""Launch and provision a Docker bottle via compose. Teardown on exit."""
"""Build, launch, and provision a Docker bottle via compose.
Teardown on exit."""
stack = ExitStack()
# Stamp the resolved agent image ref into the plan so compose rendering
# picks up the right image (may be a committed snapshot or cached ref).
plan = dataclasses.replace(
plan,
agent_provision=dataclasses.replace(plan.agent_provision, image=str(images.agent)),
)
_bottle_for_revoke = plan.manifest.bottle
_git_gate_dir_for_revoke = git_gate_state_dir(plan.slug)
@@ -123,16 +91,35 @@ def launch(
f"teardown failed for container {plan.container_name}"
f" (compose-down): {exc!r}"
)
GitGate().revoke_provisioned_keys(
revoke_git_gate_provisioned_keys(
_bottle_for_revoke, _git_gate_dir_for_revoke
)
try:
# Step 1: agent image. Use a committed snapshot when one exists
# and is present in the local daemon; otherwise build from the
# Dockerfile. (The gateway image is built by the orchestrator.)
committed = read_committed_image(plan.slug)
if committed and docker_mod.image_exists(committed):
info(f"using committed image {committed!r}")
plan = dataclasses.replace(
plan,
agent_provision=dataclasses.replace(plan.agent_provision, image=committed),
)
else:
docker_mod.build_image(
plan.image, _REPO_DIR,
dockerfile=plan.dockerfile_path,
)
docker_mod.verify_agent_image(
plan.image, runtime_for(plan.agent_provider_template).smoke_test,
)
# Step 2: mint the git-gate dynamic (gitea) deploy keys, if any, before
# provisioning the bottle's repos into the shared gateway.
git_gate_plan = plan.git_gate_plan
if git_gate_plan.upstreams:
git_gate_plan = GitGate().provision_dynamic_keys(
git_gate_plan = provision_git_gate_dynamic_keys(
plan.manifest.bottle, git_gate_plan, git_gate_state_dir(plan.slug),
)
@@ -143,27 +130,22 @@ def launch(
# handed to the orchestrator (in memory) for the gateway to inject —
# the agent never sees them.
effective_env = {**os.environ, **plan.agent_provision.provisioned_env}
token_values = Egress().resolve_token_values(
token_values = egress_resolve_token_values(
plan.egress_plan.token_env_map, effective_env,
)
teardown_timeout = resolve_teardown_timeout()
ctx = launch_consolidated(
plan.egress_plan, git_gate_plan, image_ref=plan.image, tokens=token_values,
)
stack.callback(
deprovision_consolidated, ctx.bottle_id,
orchestrator_url=ctx.orchestrator_url,
timeout=teardown_timeout,
teardown_consolidated, ctx.bottle_id, orchestrator_url=ctx.orchestrator_url,
)
# Step 4: install the SHARED gateway CA into the agent (replaces the
# per-bottle CA) — read it out of the running gateway container
# (INFRA_NAME now aliases the gateway container name; the orchestrator,
# split into its own container, holds no CA).
# per-bottle CA) — read it out of the running gateway.
ca_dir = egress_state_dir(plan.slug) / "gateway-ca"
ca_dir.mkdir(parents=True, exist_ok=True)
ca_file = ca_dir / "gateway-ca.pem"
ca_file.write_text(DockerGateway(name=INFRA_NAME).ca_cert_pem())
ca_file.write_text(DockerGateway(network=ctx.network).ca_cert_pem())
egress_plan = dataclasses.replace(
plan.egress_plan,
mitmproxy_ca_host_path=ca_file,
@@ -186,7 +168,6 @@ def launch(
agent_git_gate_url=git_gate_url,
agent_supervise_url=supervise_url,
identity_token=ctx.identity_token,
env_var_secret=ctx.env_var_secret,
)
# Step 5: render + up the agent-only compose, pinned on the shared
@@ -199,12 +180,7 @@ def launch(
project = compose_project_name(plan.slug)
# Forwarded vars (OAuth token, host interpolations) flow through the
# subprocess env as bare names so values never land in the file.
# ENV_VAR_SECRET follows the same pattern: bare name in the compose
# spec, value only in the subprocess env so it is never written to disk.
compose_env: dict[str, str] = {**os.environ, **plan.forwarded_env}
if plan.env_var_secret:
from ...orchestrator.store.secret_store import ENV_VAR_SECRET_NAME
compose_env[ENV_VAR_SECRET_NAME] = plan.env_var_secret
info(
f"docker compose up -d (project {project}, agent on shared "
f"gateway {ctx.gateway_ip}, ip {ctx.source_ip})"
@@ -231,21 +207,3 @@ def launch(
yield bottle
finally:
teardown()
def stale_checks(plan: DockerBottlePlan) -> None:
"""Raise StaleImageError if a cached image is older than the configured
threshold. Only runs when image_policy is 'cached'. Called by the backend
class's _image_stale_checks before _launch_impl starts any resources."""
if plan.spec.image_policy != "cached":
return
committed = read_committed_image(plan.slug)
if committed and docker_mod.image_exists(committed):
ts = docker_mod.image_created_at(committed)
if ts is not None:
check_stale(f"agent image {committed!r}", ts)
return
if docker_mod.image_exists(plan.image):
ts = docker_mod.image_created_at(plan.image)
if ts is not None:
check_stale(f"agent image {plan.image!r}", ts)
-221
View File
@@ -1,221 +0,0 @@
"""The docker orchestrator (control plane) as a single, fixed-name container
(PRD 0070).
`DockerOrchestrator` is the docker implementation of the backend-neutral
`Orchestrator` service. The lean control-plane container joins the `--internal`
control network only (agents are never on it, so they have no L3 route to it)
plus a host-loopback publish for the CLI. It is the sole opener of
`bot-bottle.db` and the sole holder of the signing key (#469).
"""
from __future__ import annotations
import os
import time
from pathlib import Path
from ... import log
from .util import run_docker
from ...paths import (
ORCHESTRATOR_TOKEN_ENV,
bot_bottle_root,
)
from ...gateway import GatewayError
from ...orchestrator.lifecycle import (
DEFAULT_PORT,
DEFAULT_STARTUP_TIMEOUT_SECONDS,
Orchestrator,
OrchestratorStartError,
source_hash,
)
# The control plane's own container + the dedicated `--internal` control network
# the gateway reaches it over. The orchestrator + gateway join it, agents never
# do, so agents have no route to the control plane (PRD 0070 "Separating the
# planes"). Same name across docker + macOS.
ORCHESTRATOR_NAME = "bot-bottle-orchestrator"
ORCHESTRATOR_LABEL = "bot-bottle-orchestrator=1"
ORCHESTRATOR_NETWORK = "bot-bottle-orchestrator"
ORCHESTRATOR_IMAGE = os.environ.get(
"BOT_BOTTLE_ORCHESTRATOR_IMAGE", "bot-bottle-orchestrator:latest"
)
ORCHESTRATOR_DOCKERFILE = "Dockerfile.orchestrator"
# Baked as a container label so `ensure_running` can detect whether the running
# orchestrator is executing the current bind-mounted source.
ORCHESTRATOR_SOURCE_HASH_LABEL = "bot-bottle-orchestrator-source-hash"
# The bind-mount path for the live control-plane source inside the container.
# PYTHONPATH points here so a code change takes effect on the next launch
# without an image rebuild.
_SRC_IN_CONTAINER = "/bot-bottle-src"
# Bot-bottle host-root bind-mount (DB + state) inside the orchestrator. The
# control plane opens bot-bottle.db under here (via BOT_BOTTLE_ROOT ->
# host_db_path()); it is the ONLY container with a handle on it (issue #469).
_ROOT_IN_CONTAINER = "/bot-bottle-root"
_HEALTH_POLL_SECONDS = 0.25
_REPO_ROOT = Path(__file__).resolve().parents[3]
class DockerOrchestrator(Orchestrator):
"""The control plane as a single fixed-name container. `ensure_built` builds
it from `Dockerfile.orchestrator`; `ensure_running` starts it on the control
network + host loopback and blocks until `/health` answers."""
def __init__(
self,
image_ref: str = ORCHESTRATOR_IMAGE,
*,
name: str = ORCHESTRATOR_NAME,
label: str = ORCHESTRATOR_LABEL,
port: int = DEFAULT_PORT,
control_network: str = ORCHESTRATOR_NETWORK,
repo_root: Path = _REPO_ROOT,
host_root: Path | None = None,
dockerfile: str | None = ORCHESTRATOR_DOCKERFILE,
) -> None:
self.image_ref = image_ref
self.name = name
self.label = label
self.port = port
self.control_network = control_network
self._repo_root = repo_root
self._host_root = host_root or bot_bottle_root()
self._dockerfile = dockerfile
def url(self) -> str:
"""Host-side control-plane URL — the orchestrator's published loopback,
which the CLI reaches."""
return f"http://127.0.0.1:{self.port}"
def gateway_url(self) -> str:
"""The URL the gateway's data plane resolves policy against — the
orchestrator container by name on the control network (docker DNS,
container↔container, no host firewall)."""
return f"http://{self.name}:{DEFAULT_PORT}"
def ensure_built(self) -> None:
"""Build the control-plane image from its Dockerfile, cache-aware (a
no-op when nothing changed). No-op when no dockerfile is configured (a
pre-pulled image). BOT_BOTTLE_NO_CACHE forces a full rebuild."""
if self._dockerfile is None:
return
argv = ["docker", "build", "-t", self.image_ref,
"-f", str(self._repo_root / self._dockerfile),
str(self._repo_root)]
if os.environ.get("BOT_BOTTLE_NO_CACHE"):
argv.insert(2, "--no-cache")
proc = run_docker(argv)
if proc.returncode != 0:
raise GatewayError(
f"{self._dockerfile} build failed: {proc.stderr.strip()}")
def is_running(self) -> bool:
proc = run_docker([
"docker", "ps", "--filter", f"name=^/{self.name}$", "--format", "{{.Names}}",
])
return self.name in proc.stdout.split()
def _source_current(self, current_hash: str) -> bool:
"""True iff the running orchestrator was started from the current
bind-mounted source."""
if not self.is_running():
return False
proc = run_docker([
"docker", "inspect", "--format",
"{{ index .Config.Labels \"" + ORCHESTRATOR_SOURCE_HASH_LABEL + "\" }}",
self.name,
])
if proc.returncode != 0:
return True # can't compare → don't churn a working container
return proc.stdout.strip() == current_hash
def _ensure_control_network(self) -> None:
if run_docker(["docker", "network", "inspect", self.control_network]).returncode == 0:
return
proc = run_docker(["docker", "network", "create", "--internal", self.control_network])
if proc.returncode != 0 and "already exists" not in proc.stderr:
raise GatewayError(
f"control network {self.control_network} failed to create: "
f"{proc.stderr.strip()}"
)
def ensure_running(
self, *, startup_timeout: float = DEFAULT_STARTUP_TIMEOUT_SECONDS,
) -> None:
"""Ensure the control-plane container is up on current source; block
until healthy. Idempotent — a healthy orchestrator on current source is
left untouched (its in-memory egress tokens survive). Raises
`OrchestratorStartError` on startup timeout."""
current_hash = source_hash(self._repo_root)
if self.is_healthy() and self._source_current(current_hash):
return
log.info("starting orchestrator container", context={"name": self.name})
self._run_container(current_hash)
deadline = time.monotonic() + startup_timeout
while time.monotonic() < deadline:
if self.is_healthy():
log.info("orchestrator healthy", context={"url": self.url()})
return
time.sleep(_HEALTH_POLL_SECONDS)
raise OrchestratorStartError(
f"orchestrator at {self.url()} did not become healthy "
f"within {startup_timeout:g}s"
)
def _run_container(self, current_hash: str) -> None:
"""Start the lean control-plane container on the control network only,
published to host loopback for the CLI. Idempotent (clears a stale
fixed-name container first)."""
self._ensure_control_network()
run_docker(["docker", "rm", "--force", self.name])
# The signing key comes through the shared provisioning contract (#476),
# which fail-closes rather than yield an empty key that would run OPEN.
_signing_key = self.control_plane_key()
proc = run_docker([
"docker", "run", "--detach",
"--name", self.name,
"--label", self.label,
"--label", f"{ORCHESTRATOR_SOURCE_HASH_LABEL}={current_hash}",
# Control network only — agents are never on it, so they have no
# route to the control plane (the L3 block, not just the JWT).
"--network", self.control_network,
# Host CLI reaches the control plane here (loopback only). The
# orchestrator listens on the fixed DEFAULT_PORT inside the
# container; self.port is the host-side published port.
"--publish", f"127.0.0.1:{self.port}:{DEFAULT_PORT}",
# Live control-plane source (code changes without an image rebuild).
"--volume", f"{self._repo_root}:{_SRC_IN_CONTAINER}:ro",
"--env", f"PYTHONPATH={_SRC_IN_CONTAINER}",
# Orchestrator registry DB on the host (sole writer: control plane).
"--volume", f"{self._host_root}:{_ROOT_IN_CONTAINER}",
"--env", f"BOT_BOTTLE_ROOT={_ROOT_IN_CONTAINER}",
# The signing key — held ONLY by the orchestrator (it verifies
# tokens); the gateway gets the pre-minted `gateway` JWT, never the
# key (issue #469). Bare `--env NAME` keeps the value off argv.
"--env", ORCHESTRATOR_TOKEN_ENV,
self.image_ref,
# Dockerfile.orchestrator ENTRYPOINT is `python3 -m bot_bottle.orchestrator`;
# these are its args.
"--host", "0.0.0.0", "--port", str(DEFAULT_PORT), "--broker", "stub",
], env={**os.environ, ORCHESTRATOR_TOKEN_ENV: _signing_key})
if proc.returncode != 0:
raise OrchestratorStartError(
f"orchestrator container failed to start: {proc.stderr.strip()}"
)
def stop(self) -> None:
"""Remove the control-plane container (idempotent)."""
run_docker(["docker", "rm", "--force", self.name])
__all__ = [
"DockerOrchestrator",
"ORCHESTRATOR_NAME",
"ORCHESTRATOR_LABEL",
"ORCHESTRATOR_NETWORK",
"ORCHESTRATOR_IMAGE",
"ORCHESTRATOR_DOCKERFILE",
"ORCHESTRATOR_SOURCE_HASH_LABEL",
]
@@ -0,0 +1,12 @@
"""Backend-infrastructure provisioners for the Docker backend.
Per PRD 0050 the per-provider provisioning steps (prompt, skills,
declarative provision-plan apply, supervise MCP registration) live on
the `AgentProvider` plugin under `bot_bottle/contrib/`. CA and git
provisioning also moved to the AgentProvider ABC (with Debian/node
defaults); user plugins override them for non-standard images.
No modules remain in this subpackage — the directory is kept so that
existing imports of `from .provision import ...` don't need updating
if new backend-specific provisioners are added later.
"""
+1 -1
View File
@@ -19,7 +19,7 @@ from ...env import ResolvedEnv
from ...agent_provider import AgentProvisionPlan
from ...egress import EgressPlan
from ...manifest import Manifest
from ...supervisor.plan import SupervisePlan
from ...supervise import SupervisePlan
from ...git_gate import GitGatePlan
def preflight() -> None:
+4 -8
View File
@@ -27,14 +27,10 @@ def _docker_on_path() -> bool:
def _daemon_reachable() -> bool:
if not _docker_on_path():
return False
try:
return subprocess.run(
["docker", "info"],
stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL,
check=False, timeout=5,
).returncode == 0
except subprocess.TimeoutExpired:
return False
return subprocess.run(
["docker", "info"],
stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL, check=False,
).returncode == 0
def _print_install_pointer() -> None:
+36 -68
View File
@@ -1,34 +1,18 @@
"""Docker host-side primitives used by DockerBottleBackend: the lean
`run_docker` subprocess wrapper, probing for docker on PATH, slugifying agent
names, checking image/container existence, and building images."""
"""Docker host-side primitives used by DockerBottleBackend: probing
for docker on PATH, slugifying agent names, checking image/container
existence, and building images."""
from __future__ import annotations
import os
from datetime import datetime, timezone
import re
import shutil
import subprocess
from typing import Iterator
from typing import Iterable, Iterator
from ...docker_cmd import run_docker
from ...log import die, info
def run_docker(
argv: list[str], *, env: dict[str, str] | None = None,
) -> subprocess.CompletedProcess[str]:
"""Run a `docker` command, capturing stdout/stderr as text. Never raises on
a non-zero exit — callers inspect `returncode` / `stderr` so they can stay
fail-closed or tolerate idempotent no-ops (e.g. removing an already-absent
container).
`env` sets the child process environment — used to hand a secret to a bare
`--env NAME` flag (docker inherits its value from this process) so the value
never lands on argv or in `docker inspect`'s recorded command line."""
return subprocess.run(
argv, stdout=subprocess.PIPE, stderr=subprocess.PIPE, text=True,
check=False, env=env,
)
# from ...workspace import WorkspacePlan
# Cap on the suffix the container-name conflict logic will try before
@@ -48,7 +32,12 @@ def container_name_candidates(base: str) -> Iterator[str]:
def runsc_available() -> bool:
"""Return True if the Docker daemon has the gVisor (`runsc`) runtime
registered. Called once per prepare; the result lives on the plan."""
r = run_docker(["docker", "info", "--format", "{{json .Runtimes}}"])
r = subprocess.run(
["docker", "info", "--format", "{{json .Runtimes}}"],
capture_output=True,
text=True,
check=False,
)
return r.returncode == 0 and "runsc" in r.stdout
@@ -62,15 +51,20 @@ def require_docker() -> None:
def image_exists(ref: str) -> bool:
return run_docker(["docker", "image", "inspect", ref]).returncode == 0
return _silent_run(["docker", "image", "inspect", ref]) == 0
def container_exists(name: str) -> bool:
"""Returns True if a container (running or stopped) with the given
name exists. Uses `docker ps -a -q -f name=^<name>$` so substring
matches don't false-positive."""
result = run_docker(["docker", "ps", "-a", "-q", "-f", f"name=^{name}$"])
return result.returncode == 0 and bool(result.stdout.strip())
result = subprocess.run(
["docker", "ps", "-a", "-q", "-f", f"name=^{name}$"],
capture_output=True,
text=True,
check=True,
)
return bool(result.stdout.strip())
def force_remove_container(name: str) -> None:
@@ -78,7 +72,12 @@ def force_remove_container(name: str) -> None:
doesn't — and the rm itself is best-effort (errors swallowed) so
this is safe to register as a teardown callback."""
if container_exists(name):
run_docker(["docker", "rm", "-f", name])
subprocess.run(
["docker", "rm", "-f", name],
stdout=subprocess.DEVNULL,
stderr=subprocess.DEVNULL,
check=False,
)
def docker_exec_root(container: str, argv: list[str]) -> None:
@@ -206,7 +205,10 @@ def verify_agent_image(image: str, argv: tuple[str, ...]) -> None:
def commit_container(container_name: str, image_tag: str) -> None:
"""Run `docker commit <container_name> <image_tag>` to snapshot the
running container's filesystem state as a local Docker image."""
result = run_docker(["docker", "commit", container_name, image_tag])
result = subprocess.run(
["docker", "commit", container_name, image_tag],
capture_output=True, text=True, check=False,
)
if result.returncode != 0:
die(
f"docker commit {container_name!r}{image_tag!r} failed: "
@@ -215,44 +217,10 @@ def commit_container(container_name: str, image_tag: str) -> None:
info(f"committed {container_name!r}{image_tag!r}")
def image_created_at(ref: str) -> datetime | None:
"""Return Docker's image Created timestamp as an aware UTC datetime, or
None when the field is absent or unparseable. Callers should skip the
stale check when None is returned."""
r = subprocess.run(
["docker", "image", "inspect", "--format", "{{.Created}}", ref],
capture_output=True,
text=True,
def _silent_run(cmd: Iterable[str]) -> int:
return subprocess.run(
list(cmd),
stdout=subprocess.DEVNULL,
stderr=subprocess.DEVNULL,
check=False,
)
if r.returncode != 0:
die(
f"docker image inspect for {ref!r} failed: "
f"{(r.stderr or '').strip() or '<no stderr>'}"
)
raw = r.stdout.strip()
if not raw:
return None
try:
return _parse_docker_timestamp(raw)
except ValueError:
return None
def _parse_docker_timestamp(raw: str) -> datetime:
text = raw.strip()
if text.endswith("Z"):
text = text[:-1] + "+00:00"
dot = text.find(".")
if dot != -1:
tz_plus = text.find("+", dot)
tz_minus = text.find("-", dot)
tz_candidates = [pos for pos in (tz_plus, tz_minus) if pos != -1]
if tz_candidates:
tz_pos = min(tz_candidates)
frac = text[dot + 1:tz_pos]
text = text[:dot + 1] + frac[:6].ljust(6, "0") + text[tz_pos:]
dt = datetime.fromisoformat(text)
if dt.tzinfo is None:
dt = dt.replace(tzinfo=timezone.utc)
return dt.astimezone(timezone.utc)
).returncode
+1 -1
View File
@@ -11,7 +11,7 @@ from pathlib import Path
from ..bottle_state import egress_state_dir
from ..egress import EGRESS_ROUTES_FILENAME
from ..gateway.egress.addon_core import LOG_OFF, load_config
from ..egress_addon_core import LOG_OFF, load_config
class EgressApplyError(RuntimeError):
+2 -20
View File
@@ -1,25 +1,7 @@
"""Firecracker backend: Linux KVM microVM isolation (issue #342).
Thin by design: `FirecrackerBottleBackend` is re-exported lazily via
`__getattr__`, so importing a leaf module under this package doesn't drag the
backend (and the framework it pulls) into memory.
"""
"""Firecracker backend: Linux KVM microVM isolation (issue #342)."""
from __future__ import annotations
from typing import TYPE_CHECKING, Any
if TYPE_CHECKING:
from .backend import FirecrackerBottleBackend
def __getattr__(name: str) -> Any:
if name == "FirecrackerBottleBackend":
from .backend import FirecrackerBottleBackend
globals()[name] = FirecrackerBottleBackend
return FirecrackerBottleBackend
raise AttributeError(f"module {__name__!r} has no attribute {name!r}")
from .backend import FirecrackerBottleBackend
__all__ = ["FirecrackerBottleBackend"]
+9 -20
View File
@@ -8,8 +8,7 @@ fail-closed nftables egress boundary. Selected by
from __future__ import annotations
import io
from contextlib import contextmanager, redirect_stderr
from contextlib import contextmanager
from pathlib import Path
from typing import Generator, Sequence
@@ -18,8 +17,8 @@ from ...egress import EgressPlan
from ...env import ResolvedEnv
from ...git_gate import GitGatePlan
from ...manifest import Manifest
from ...supervisor.plan import SupervisePlan
from .. import ActiveAgent, BottleBackend, BottleImages, BottleSpec
from ...supervise import SupervisePlan
from .. import ActiveAgent, BottleBackend, BottleSpec
from . import cleanup as _cleanup
from . import enumerate as _enumerate
from . import launch as _launch
@@ -53,11 +52,8 @@ class FirecrackerBottleBackend(
return _setup.setup()
@classmethod
def status(cls, *, quiet: bool = False) -> int:
def status(cls) -> int:
from . import setup as _setup
if quiet:
with redirect_stderr(io.StringIO()):
return _setup.status()
return _setup.status()
@classmethod
@@ -96,18 +92,11 @@ class FirecrackerBottleBackend(
stage_dir=stage_dir,
)
def _build_or_load_images(self, plan: FirecrackerBottlePlan) -> BottleImages:
return BottleImages(agent=_launch.build_or_load_agent_base(plan))
def prelaunch_checks(self, plan: FirecrackerBottlePlan) -> None:
_launch.stale_checks(plan)
@contextmanager
def _launch_impl(
self, plan: FirecrackerBottlePlan, images: BottleImages,
def launch(
self, plan: FirecrackerBottlePlan
) -> Generator[FirecrackerBottle, None, None]:
assert isinstance(images.agent, Path)
with _launch.launch(plan, images.agent, provision=self.provision) as bottle:
with _launch.launch(plan, provision=self.provision) as bottle:
yield bottle
def prepare_cleanup(self) -> FirecrackerBottleCleanupPlan:
@@ -123,5 +112,5 @@ class FirecrackerBottleBackend(
return plan.agent_supervise_url
def ensure_orchestrator(self) -> str:
from .infra import FirecrackerInfraService
return FirecrackerInfraService().ensure_running()
from . import infra_vm
return infra_vm.ensure_running().control_plane_url
@@ -22,9 +22,6 @@ class FirecrackerBottlePlan(BottlePlan):
# (egress proxy credentials, git-gate/supervise headers); set by launch
# from the orchestrator registration. Empty pre-registration.
identity_token: str = ""
# Applied to every agent SSH exec and mirrored into /run inside the VM so
# the host can recover it after the infra VM restarts.
env_var_secret: str = ""
@property
def container_name(self) -> str:
+16 -72
View File
@@ -1,23 +1,8 @@
"""Cleanup for the Firecracker backend.
Reaps *orphans* only — resources with no live VM behind them:
* orphan run dirs: a per-bottle run dir (holding the ~1G rootfs.ext4)
whose firecracker process has exited. These leak when a launch is
hard-killed before its teardown runs (host OOM/crash, a cancelled CI
job, `kill -9`); the clean-exit path already removes its own dir in
launch.py.
* orphan VM pids: a firecracker process whose run dir is already gone
— a VMM left lingering after its dir was removed.
A run dir with a *live* firecracker process is a running bottle and is
left strictly alone: it is neither killed nor removed. (The backend's
`enumerate_active` registry is still a stub — #354 — so a live process
is the only reliable "this bottle is in use" signal we have. Once the
registry lands, registry-orphaned-but-running VMs can be reaped too.)
TAP slots free themselves (the flock drops when the launcher exits), so
there is nothing to reclaim there.
Orphans are: firecracker VMM processes whose config lives under our run
dir, and the per-bottle run dirs. TAP slots free themselves (the flock
drops when the launcher exits), so there is nothing to reclaim there.
"""
from __future__ import annotations
@@ -37,79 +22,38 @@ def _run_root() -> Path:
return util.cache_dir() / "run"
def _run_dir_of(cmd: str, run_root: Path) -> Path | None:
"""The bottle run dir a firecracker cmdline belongs to, or None.
A bottle VM is launched with `--config-file <run_root>/<slug>/config.json`,
so the run dir is the config file's parent when it sits directly under
the run root. Anything else (a builder VM, the infra VM elsewhere) is
not ours to reap here.
"""
toks = cmd.split()
for i, tok in enumerate(toks):
if tok == "--config-file" and i + 1 < len(toks):
parent = Path(toks[i + 1]).parent
if parent.parent == run_root:
return parent
return None
def _scan_processes(run_root: Path) -> tuple[set[str], list[int]]:
"""Inspect running firecracker VMs under ``run_root``.
Returns ``(live_run_dirs, orphan_pids)``:
* ``live_run_dirs`` — run dirs backed by a running VM (never reaped);
* ``orphan_pids`` — firecracker pids whose run dir no longer exists
(a lingering VMM to kill).
"""
def _orphan_vm_pids() -> list[int]:
"""firecracker processes whose --config-file is under our run dir."""
run_root = str(_run_root())
result = subprocess.run(
["pgrep", "-a", "firecracker"],
capture_output=True, text=True, check=False,
)
if result.returncode != 0:
return set(), []
live: set[str] = set()
orphan_pids: list[int] = []
return []
pids: list[int] = []
for line in result.stdout.splitlines():
parts = line.split(None, 1)
if len(parts) != 2:
if len(parts) != 2 or run_root not in parts[1]:
continue
try:
pid = int(parts[0])
pids.append(int(parts[0]))
except ValueError:
continue
run_dir = _run_dir_of(parts[1], run_root)
if run_dir is None:
continue
if run_dir.is_dir():
live.add(str(run_dir))
else:
orphan_pids.append(pid)
return live, orphan_pids
return pids
def live_run_dirs() -> tuple[Path, ...]:
"""Run directories backed by currently running agent microVMs."""
live, _ = _scan_processes(_run_root())
return tuple(Path(path) for path in sorted(live))
def _orphan_run_dirs(run_root: Path, live: set[str]) -> list[str]:
"""Run dirs with no live VM behind them — the leaked ones to remove."""
def _run_dirs() -> list[str]:
run_root = _run_root()
if not run_root.is_dir():
return []
return sorted(
str(p) for p in run_root.iterdir()
if p.is_dir() and str(p) not in live
)
return sorted(str(p) for p in run_root.iterdir() if p.is_dir())
def prepare_cleanup() -> FirecrackerBottleCleanupPlan:
run_root = _run_root()
live, orphan_pids = _scan_processes(run_root)
return FirecrackerBottleCleanupPlan(
vm_pids=tuple(orphan_pids),
run_dirs=tuple(_orphan_run_dirs(run_root, live)),
vm_pids=tuple(_orphan_vm_pids()),
run_dirs=tuple(_run_dirs()),
)
@@ -1,23 +1,22 @@
"""Consolidated bottle launch sequence for the Firecracker backend
(PRD 0070, Stage B).
The orchestrator control plane and the shared gateway run in **two** persistent
per-host **infra VMs** (`infra_vm.py`), split per PRD 0070 — not Docker
containers. Agent VMs reach the gateway's egress / supervise / git-http ports
via a PREROUTING DNAT on their own host-side TAP IP that redirects to the
**gateway** VM (see `scripts/firecracker-netpool.sh`) — never the orchestrator,
so a breached agent has no L3 route to the control plane. The host CLI reaches
the control plane over HTTP at the orchestrator VM's guest IP.
The shared gateway + orchestrator control plane run in a single persistent
per-host **infra VM** (`infra_vm.py`), not Docker containers. Agent VMs reach
the gateway's egress / supervise / git-http ports at the infra VM via a
PREROUTING DNAT on their own host-side TAP IP (see
`scripts/firecracker-netpool.sh`), and the host CLI reaches the control plane
over HTTP at the infra VM's guest IP.
Attribution is by the agent VM's guest IP, unspoofable by construction: the
/31 point-to-point TAP + the `bot_bottle_fc` nft table ensure only the
expected VM can source-IP that address.
Sequence:
1. ensure the infra VMs (orchestrator control plane + gateway data plane)
are up (a singleton pair — a prior launcher may already have booted them);
2. register the bottle on the orchestrator by its guest IP (attribution key)
→ bottle id + identity token;
1. ensure the infra VM (control plane + gateway) is up (a singleton — a
prior launcher may already have booted it);
2. register the bottle by its guest IP (attribution key) → bottle id +
identity token;
3. provision its git-gate repos/creds into the gateway VM (over SSH);
4. fetch the shared gateway CA for the provisioner to install in the rootfs.
@@ -26,26 +25,17 @@ The TAP slot allocation, rootfs build, and VM boot are the caller's job.
from __future__ import annotations
import json
import subprocess
from dataclasses import dataclass
from pathlib import Path
from ...egress import EgressPlan
from ...git_gate import GitGatePlan
from ...log import info
from ...orchestrator.client import OrchestratorClient, OrchestratorClientError
from ...orchestrator.client import OrchestratorClient
from ...orchestrator.lifecycle import (
OrchestratorStartError, # re-exported so callers can catch it
)
from ...orchestrator.reprovision import reprovision_bottles
from ...orchestrator.store.secret_store import ENV_VAR_SECRET_NAME
from ..provision_bottle import deprovision_bottle, provision_bottle
from . import cleanup, util
from .gateway import FirecrackerGateway
from .infra import FirecrackerInfraService
_ENV_VAR_SECRET_PATH = "/run/bot-bottle/env-var-secret"
from ...orchestrator.registration import registration_inputs
from ..docker.gateway_provision import deprovision_git_gate, provision_git_gate
from . import infra_vm
class ConsolidatedLaunchError(RuntimeError):
@@ -61,55 +51,6 @@ class LaunchContext:
source_ip: str # the VM's guest IP — the attribution key
gateway_ca_pem: str # the shared gateway CA the provisioner installs
orchestrator_url: str
env_var_secret: str = "" # encryption key injected into the agent's env
def _guest_ip_from_config(config_path: Path) -> str:
"""Read the kernel's configured guest IP from a Firecracker config."""
try:
config = json.loads(config_path.read_text())
args = config["boot-source"]["boot_args"]
ip_arg = next(part for part in args.split() if part.startswith("ip="))
return ip_arg.removeprefix("ip=").split(":", 1)[0]
except (OSError, ValueError, KeyError, TypeError, StopIteration):
return ""
def persist_env_var_secret(private_key: Path, guest_ip: str, secret: str) -> None:
"""Mirror the exec-time key into guest tmpfs for restart recovery."""
proc = subprocess.run(
util.ssh_base_argv(private_key, guest_ip)
+ [f"umask 077; mkdir -p /run/bot-bottle; cat > {_ENV_VAR_SECRET_PATH}"],
input=secret, capture_output=True, text=True, check=False,
)
if proc.returncode != 0:
raise ConsolidatedLaunchError(
f"failed to persist {ENV_VAR_SECRET_NAME} in agent VM: "
f"{proc.stderr.strip() or '<no stderr>'}"
)
def _reprovision_running_bottles(client: OrchestratorClient) -> None:
"""Read keys from live agent VMs and restore the restarted gateway."""
try:
secrets_by_ip: dict[str, str] = {}
for run_dir in cleanup.live_run_dirs():
guest_ip = _guest_ip_from_config(run_dir / "config.json")
private_key = run_dir / "bottle_id_ed25519"
if not guest_ip or not private_key.is_file():
continue
proc = subprocess.run(
util.ssh_base_argv(private_key, guest_ip)
+ [f"cat {_ENV_VAR_SECRET_PATH}"],
capture_output=True, text=True, check=False,
)
if proc.returncode == 0 and proc.stdout.strip():
secrets_by_ip[guest_ip] = proc.stdout.strip()
count = reprovision_bottles(client, secrets_by_ip)
if count:
info(f"reprovisioned egress tokens for {count} Firecracker bottle(s)")
except (OSError, OrchestratorClientError) as exc:
info(f"egress token reprovision skipped: {exc}")
def launch_consolidated(
@@ -123,45 +64,46 @@ def launch_consolidated(
"""Ensure the infra VM is up, register the bottle by its guest IP, and
provision its git-gate state into the gateway VM. Returns the context the
agent-VM launch needs. Raises on failure — the caller tears down."""
service = FirecrackerInfraService()
url = service.ensure_running()
infra = infra_vm.ensure_running()
url = infra.control_plane_url
client = OrchestratorClient(url)
_reprovision_running_bottles(client)
# Read the gateway's provisioning transport + CA off the Gateway service.
gateway = service.gateway()
transport = gateway.provisioning_transport()
reg = provision_bottle(
client, guest_ip, egress_plan, git_gate_plan, transport,
image_ref=image_ref, tokens=tokens,
inputs = registration_inputs(egress_plan)
reg = client.register_bottle(
guest_ip, image_ref=image_ref, policy=inputs.policy,
metadata=inputs.metadata, tokens=tokens,
)
try:
provision_git_gate(
infra_vm.gateway_transport(), reg.bottle_id, git_gate_plan)
except Exception:
client.teardown_bottle(reg.bottle_id)
raise
# The shared gateway CA every agent on this host trusts for TLS
# interception — fetched from the gateway VM over SSH.
# interception — fetched from the infra VM over SSH.
return LaunchContext(
bottle_id=reg.bottle_id,
identity_token=reg.identity_token,
source_ip=guest_ip,
gateway_ca_pem=gateway.ca_cert_pem(),
gateway_ca_pem=infra.gateway_ca_pem(),
orchestrator_url=url,
env_var_secret=reg.env_var_secret,
)
def deprovision_consolidated(
bottle_id: str, *, orchestrator_url: str, timeout: float | None = None,
) -> None:
def teardown_consolidated(bottle_id: str, *, orchestrator_url: str) -> None:
"""Deregister the bottle and remove its git-gate state from the gateway
VM. Both steps are idempotent so this is safe from a cleanup trap. Does
NOT stop the infra VMs — they're persistent per-host singletons shared by
NOT stop the infra VM — it's a persistent per-host singleton shared by
every bottle."""
deprovision_bottle(bottle_id, FirecrackerGateway().provisioning_transport(),
orchestrator_url=orchestrator_url, timeout=timeout)
OrchestratorClient(orchestrator_url).teardown_bottle(bottle_id)
deprovision_git_gate(infra_vm.gateway_transport(), bottle_id)
__all__ = [
"LaunchContext",
"launch_consolidated",
"deprovision_consolidated",
"teardown_consolidated",
"ConsolidatedLaunchError",
"OrchestratorStartError",
]
@@ -60,18 +60,12 @@ class VmHandle:
self.process.wait(timeout=5)
def _boot_args(
guest_ip: str, host_ip: str, pubkey: str, extra: str = "",
) -> str:
def _boot_args(guest_ip: str, host_ip: str, pubkey: str) -> str:
# ip=<client>::<gw>:<netmask>::<dev>:off — /31 point-to-point link,
# so netmask is 255.255.255.254 and the gateway is the host TAP IP.
ip_arg = f"ip={guest_ip}::{host_ip}:255.255.255.254::eth0:off"
pub_b64 = base64.b64encode(pubkey.encode()).decode()
args = f"{_BASE_BOOT_ARGS} {ip_arg} bb_pubkey={pub_b64}"
# `extra` carries caller-supplied cmdline params the guest init reads
# (e.g. the gateway VM's `bb_orch=<orchestrator guest IP>`). Agent VMs pass
# nothing.
return f"{args} {extra}".rstrip() if extra else args
return f"{_BASE_BOOT_ARGS} {ip_arg} bb_pubkey={pub_b64}"
def _config(
@@ -85,7 +79,6 @@ def _config(
mem_mib: int,
guest_mac: str,
data_drive: Path | None = None,
extra_boot_args: str = "",
) -> dict[str, object]:
drives: list[dict[str, object]] = [
{
@@ -108,7 +101,7 @@ def _config(
return {
"boot-source": {
"kernel_image_path": str(util.kernel_path()),
"boot_args": _boot_args(guest_ip, host_ip, pubkey, extra_boot_args),
"boot_args": _boot_args(guest_ip, host_ip, pubkey),
},
"drives": drives,
"network-interfaces": [
@@ -139,7 +132,6 @@ def boot(
guest_mac: str = "06:00:AC:10:00:02",
detached: bool = False,
data_drive: Path | None = None,
extra_boot_args: str = "",
) -> VmHandle:
"""Write the config and launch the VMM. Returns once the process is
spawned; callers wait for SSH readiness separately.
@@ -155,7 +147,7 @@ def boot(
_config(
rootfs=rootfs, tap=tap, guest_ip=guest_ip, host_ip=host_ip,
pubkey=pubkey, vcpus=vcpus, mem_mib=mem_mib, guest_mac=guest_mac,
data_drive=data_drive, extra_boot_args=extra_boot_args,
data_drive=data_drive,
),
indent=2,
))
+1 -1
View File
@@ -5,7 +5,7 @@ backend — we stream the guest root filesystem out over the control
channel (SSH here). Unlike the other backends this needs no Docker: the
tar *is* the resumable artifact. `resume` extracts it and rebuilds a
fresh per-bottle ext4 with `mke2fs -d` (see `util.build_committed_rootfs_dir`
and `launch.build_or_load_agent_base`). The bottle keeps running after the
and `launch._build_agent_base`). The bottle keeps running after the
snapshot.
"""
-156
View File
@@ -1,156 +0,0 @@
"""The Firecracker gateway data plane as a microVM (PRD 0070).
`FirecrackerGateway` is the Firecracker implementation of the backend-neutral
`Gateway` service, and owns the gateway's host-side logic directly: booting the
gateway microVM on its NAT'd link, seeding the pre-minted `gateway` token,
fetching the mitmproxy CA, and the SSH exec/cp provisioning transport. The
gateway VM never opens `bot-bottle.db` (#469), so it holds no signing key —
only the token the host hands it via `connect_to_orchestrator`.
The plane-agnostic VM substrate the orchestrator VM also uses stays in
`infra_vm` — booting a VM from a per-plane rootfs (`boot_vm`), the stable SSH
keypair, the secret-push retry loop, the PID lifecycle, and the
adoption/version helpers. The guest-side gateway daemon startup lives in the
gateway rootfs's guest init (`_gateway_init` in `infra_vm`), so it can't live in
a host-side method either. The pair coordinator (orchestrator-first, under a
singleton flock) is `FirecrackerInfraService` (`infra.py`).
"""
from __future__ import annotations
import subprocess
from urllib.parse import urlparse
from ...gateway import (
DEFAULT_CA_TIMEOUT_SECONDS,
Gateway,
GatewayError,
GatewayTransport,
)
from .. import util as backend_util
from . import infra_vm, netpool, util
from .gateway_transport import FirecrackerGatewayTransport
# The gateway microVM's name (its run dir). Fixed per host — one gateway VM,
# shared by every agent VM.
GATEWAY_NAME = "bot-bottle-gateway"
# The gateway VM's slim memory ceiling — the data plane carries no build
# tooling (buildah lives only on the orchestrator rootfs) — PRD 0070
# "Memory: fixed ceilings".
_GW_MEM_MIB = 2048
# The pre-minted `gateway` JWT path in the guest. The gateway init (in
# `infra_vm`) waits for it before starting the data plane, so its canonical
# definition lives with that init; imported here for the push so the
# load-bearing path isn't duplicated.
_GUEST_GATEWAY_JWT_PATH = infra_vm._GUEST_GATEWAY_JWT_PATH
# mitmproxy writes its CA here a beat after start; agents install it to trust
# the gateway's TLS interception. Host-side only (SSH cat), so it lives here.
_GATEWAY_CA_PATH = "/home/mitmproxy/.mitmproxy/mitmproxy-ca-cert.pem"
_CA_FETCH_TIMEOUT_SECONDS = 15.0
class FirecrackerGateway(Gateway):
"""The consolidated gateway as a Firecracker microVM on the gateway link.
The gateway rootfs is built/downloaded by `infra_vm.ensure_built` (the ABC's
`ensure_built` no-op here); `connect_to_orchestrator` boots the VM resolving
policy against the orchestrator and seeds the pre-minted `gateway` token."""
name = GATEWAY_NAME
def __init__(self, vm: infra_vm.InfraVm | None = None) -> None:
# The live VM handle when this process booted it; None when adapting an
# already-running gateway (CA fetch / provisioning go through the stable
# key + fixed link, so no live handle is needed).
self._vm = vm
self._orchestrator_url = ""
self._gateway_token = ""
def connect_to_orchestrator(self, orchestrator_url: str, gateway_token: str) -> None:
"""Boot the gateway VM on its link resolving policy against
`orchestrator_url`, then seed `gateway_token` over SSH. The orchestrator's
guest IP is parsed from the URL and passed on the cmdline (`bb_orch=`), so
the baked init stays IP-independent. Boot the orchestrator first — the
gateway daemons reach it at startup. The gateway never mints, so it holds
no signing key, only this token (#469)."""
self._orchestrator_url = orchestrator_url
self._gateway_token = gateway_token
if not self._orchestrator_url:
raise GatewayError(
"gateway requires an orchestrator URL to run "
"(resolver-only data plane; no single-tenant fallback)"
)
if not self._gateway_token:
raise GatewayError(
"gateway requires a pre-minted `gateway` token to run "
"(the orchestrator mints it; the gateway never holds the key)"
)
orchestrator_guest_ip = urlparse(self._orchestrator_url).hostname or ""
if not orchestrator_guest_ip:
raise GatewayError(
f"cannot resolve orchestrator guest IP from {self._orchestrator_url!r}"
)
# Boot on the gateway link from the gateway rootfs, then push the token
# the init waits for before starting the data plane.
vm = infra_vm.boot_vm(
name=GATEWAY_NAME, slot=netpool.gw_slot(), run_dir=infra_vm._gw_dir(),
role="gateway", mem_mib=_GW_MEM_MIB,
extra_boot_args=f"bb_orch={orchestrator_guest_ip}",
)
infra_vm.push_secret(
vm, self._gateway_token, _GUEST_GATEWAY_JWT_PATH,
"the gateway JWT to the gateway VM (its data plane will not start)",
)
self._vm = vm
def is_running(self) -> bool:
return infra_vm._pidfile_alive(infra_vm._gw_dir())
def stop(self) -> None:
"""Stop only the gateway VM (idempotent — absent is success). A dead
gateway already fails `infra_vm.adoptable`, so no version marker to
clear here."""
infra_vm._kill_pidfile(infra_vm._gw_dir())
infra_vm._pid_file(infra_vm._gw_dir()).unlink(missing_ok=True)
def address(self) -> str:
"""The gateway VM's guest IP — the agent-facing target agent VMs'
gateway-port traffic is DNAT'd to."""
return netpool.gw_slot().guest_ip
def ca_cert_pem(self, *, timeout: float = DEFAULT_CA_TIMEOUT_SECONDS) -> str:
"""The gateway's mitmproxy CA (PEM) agents install to trust its TLS
interception. Fetched from the gateway VM over SSH; polls because
mitmproxy writes it a beat after boot. Works from any later launcher —
falls back to the stable key + fixed link when this process didn't boot
the VM."""
key = self._vm.private_key if self._vm else infra_vm._infra_dir() / "id_ed25519"
ip = self._vm.guest_ip if self._vm else netpool.gw_slot().guest_ip
def _fetch() -> str | None:
proc = subprocess.run(
util.ssh_base_argv(key, ip) + [f"cat {_GATEWAY_CA_PATH}"],
capture_output=True, text=True,
timeout=_CA_FETCH_TIMEOUT_SECONDS, check=False,
)
ok = proc.returncode == 0 and "BEGIN CERTIFICATE" in proc.stdout
return proc.stdout if ok else None
try:
return backend_util.poll_ca_cert(_fetch, timeout=timeout)
except TimeoutError as exc:
raise GatewayError(str(exc)) from exc
def provisioning_transport(self) -> GatewayTransport:
"""The exec/cp transport git-gate provisioning stages per-bottle repos +
deploy keys through (over SSH to the gateway VM). Needs no live handle —
built from the stable key + the gateway link's guest IP, so teardown can
use it too."""
return FirecrackerGatewayTransport(
infra_vm._infra_dir() / "id_ed25519", netpool.gw_slot().guest_ip)
__all__ = ["FirecrackerGateway", "FirecrackerGatewayTransport", "GATEWAY_NAME"]
@@ -1,56 +0,0 @@
"""The `GatewayTransport` for the gateway running in the gateway microVM
(PRD 0070).
How the launcher stages files + runs commands in the running gateway: the
docker exec/cp equivalents over SSH (dropbear + the stable infra key). The
backend-neutral provisioning logic that drives it lives in
`backend.provision_gateway`.
"""
from __future__ import annotations
import os
import shlex
import stat
import subprocess
from pathlib import Path
from ...gateway import GatewayProvisionError
from . import util
class FirecrackerGatewayTransport:
"""`GatewayTransport` for the gateway running in the gateway VM — the docker
exec/cp equivalents over SSH (dropbear + the stable infra key)."""
def __init__(self, private_key: Path, guest_ip: str) -> None:
self._key = private_key
self._ip = guest_ip
def exec(self, argv: list[str]) -> None:
proc = subprocess.run(
util.ssh_base_argv(self._key, self._ip) + [shlex.join(argv)],
capture_output=True, text=True, timeout=60, check=False,
)
if proc.returncode != 0:
raise GatewayProvisionError(
f"infra gateway exec {argv!r} failed: {proc.stderr.strip()}")
def cp_into(self, src: str, dest: str) -> None:
# Preserve the source mode (docker cp does): the access-hook is staged
# 0700 and git-http execs it directly — a plain `cat >` would land it
# 0644 and the exec fails with EACCES; keys stay 0600.
mode = stat.S_IMODE(os.stat(src).st_mode)
q = shlex.quote(dest)
proc = subprocess.run(
util.ssh_base_argv(self._key, self._ip)
+ [f"cat > {q} && chmod {mode:o} {q}"],
input=Path(src).read_bytes(), capture_output=True, timeout=30, check=False,
)
if proc.returncode != 0:
raise GatewayProvisionError(
f"infra gateway cp {src} -> {dest} failed: "
f"{proc.stderr.decode(errors='replace').strip()}")
__all__ = ["FirecrackerGatewayTransport"]
+21 -44
View File
@@ -1,17 +1,17 @@
"""Docker-free agent-image builds for the Firecracker backend (PRD 0069 Stage 3).
Agent Dockerfiles build **inside the persistent per-host orchestrator VM**
Agent Dockerfiles build **inside the persistent per-host infra VM**
(`infra_vm.py`), which carries buildah (rootless, daemonless): no host Docker
daemon, no root-equivalent `docker` group. The build runs over SSH against the
orchestrator VM and its rootfs streams back to the host, where the existing
infra VM and its rootfs streams back to the host, where the existing
`mke2fs -d` path (`util.build_rootfs_ext4`) turns it into a bootable ext4.
Builds run on the control-plane (orchestrator) side — not the data-plane
gateway — per PRD 0070 v1 ("builds stay in the orchestrator"): it owns
launches and already carries buildah. Tradeoff: an untrusted Dockerfile's
`RUN` steps share the VM with the control plane (buildah `--isolation chroot`
isn't a hard boundary) — the accepted blast-radius tradeoff, re-splittable
into a disposable builder (booted from this same image on its own TAP) later.
Building in the infra VM — rather than a throwaway builder VM — means there is
one buildah image (`bot-bottle-infra`) and no contention for the orchestrator
TAP. Tradeoff: an untrusted Dockerfile's `RUN` steps share the VM with the
control plane + gateway (buildah `--isolation chroot` isn't a hard boundary) —
the accepted single-VM blast-radius tradeoff, re-splittable into a disposable
builder (booted from this same image on its own TAP) later.
"""
from __future__ import annotations
@@ -26,8 +26,7 @@ from pathlib import Path
from typing import Generator
from ...log import die, info
from . import util
from .infra import FirecrackerInfraService
from . import infra_vm, util
# vfs + chroot: buildah works as root in the microVM (no fuse-overlayfs /
# overlay module / subuid maps). `--isolation` is a build/run-only flag;
@@ -39,47 +38,27 @@ _BUILD_TIMEOUT_SECONDS = 900.0
def _dockerfile_hash(dockerfile: Path) -> str:
"""The Dockerfile's content hash. The shipped agent Dockerfiles COPY
nothing from the build context (see .dockerignore), so their content fully
determines the built image; a Dockerfile that adds COPY will want the
"""Cache key: the Dockerfile's content. The shipped agent Dockerfiles
COPY nothing from the build context (see .dockerignore), so their content
fully determines the image; a Dockerfile that adds COPY will want the
context folded in here too."""
return hashlib.sha256(dockerfile.read_bytes()).hexdigest()[:16]
def _rootfs_digest(dockerfile: Path) -> str:
"""Cache key for the built AND boot-injected agent rootfs. Two inputs
determine the on-disk rootfs: the Dockerfile (the image) and the guest init
injected into it (`util._GUEST_INIT`). Folding the init in means a fix to
it — e.g. making /tmp world-writable — busts the cache instead of silently
reusing a stale rootfs built with the old init."""
h = hashlib.sha256()
h.update(_dockerfile_hash(dockerfile).encode())
h.update(b"\0")
h.update(util._GUEST_INIT.encode())
return h.hexdigest()[:16]
def cached_agent_rootfs_dir(dockerfile: Path) -> Path | None:
"""Return the ready cached rootfs for ``dockerfile``, if one exists."""
base = util.cache_dir() / "rootfs" / f"agent-{_rootfs_digest(dockerfile)}"
return base if (base / ".bb-ready").is_file() else None
def build_agent_rootfs_dir(
dockerfile: Path, *, image_tag: str, smoke_test: tuple[str, ...] = (),
) -> Path:
"""Build `dockerfile` in the infra VM (buildah, no host docker), export its
rootfs, inject the guest boot bits, and return the cached base dir — the
same shape `util.build_rootfs_ext4` consumes. Cached by Dockerfile content
+ injected guest init, so a repeat launch skips the rebuild but an init or
Dockerfile change rebuilds.
same shape `util.build_rootfs_ext4` consumes. Cached by Dockerfile content,
so a repeat launch skips the rebuild.
`smoke_test` (the provider's declared argv, e.g. `("claude","--version")`)
is run in the freshly built image before export, catching an npm
silent-failure image at build time rather than at first agent use."""
digest = _rootfs_digest(dockerfile)
digest = _dockerfile_hash(dockerfile)
base = util.cache_dir() / "rootfs" / f"agent-{digest}"
if cached_agent_rootfs_dir(dockerfile) is not None:
if (base / ".bb-ready").is_file():
info(f"using cached agent rootfs {base.name}")
return base
@@ -122,13 +101,11 @@ def _build_lock() -> Generator[None, None, None]:
def _build_in_infra(
dockerfile: Path, base: Path, smoke_test: tuple[str, ...], digest: str,
) -> None:
"""Ensure the infra VMs are up, `buildah build` the Dockerfile in the
orchestrator VM (the build-capable control plane), smoke test the image,
and stream its rootfs into `base`. The infra VMs persist; only the
per-build container/image/context are cleaned up."""
service = FirecrackerInfraService()
service.ensure_running()
key, ip = service.orchestrator().ssh_target()
"""Ensure the infra VM is up, `buildah build` the Dockerfile in it, smoke
test the image, and stream its rootfs into `base`. The infra VM persists;
only the per-build container/image/context are cleaned up."""
infra = infra_vm.ensure_running()
key, ip = infra.private_key, infra.guest_ip
tag = f"bot-bottle-agent-build-{digest}"
ctx = f"/tmp/agent-build-{digest}"
smoke_ctr, export_ctr = f"{tag}-smoke", f"{tag}-export"
-80
View File
@@ -1,80 +0,0 @@
"""The per-host infra service for the Firecracker backend (PRD 0070).
`FirecrackerInfraService` is the Firecracker `InfraService`: it composes the
orchestrator + gateway microVMs as an idempotent per-host singleton pair over
the shared `infra_vm` substrate (boot primitives, the singleton flock, the
adopt/version machinery). Unlike the docker/macOS services it takes no
per-instance config — the firecracker pair is a **hard** per-host singleton,
pinned to the fixed netpool links (`orch_slot()` / `gw_slot()`), host cache
paths, and a `booted-version` marker, so two pairs can't coexist on one host.
"""
from __future__ import annotations
from ...log import info
from ..infra_service import InfraService
from ...orchestrator.lifecycle import DEFAULT_STARTUP_TIMEOUT_SECONDS
from . import infra_vm
from .gateway import FirecrackerGateway
from .orchestrator import FirecrackerOrchestrator
class FirecrackerInfraService(InfraService):
"""Bring up the orchestrator + gateway microVM pair as a per-host singleton."""
def orchestrator(self) -> FirecrackerOrchestrator:
"""The control-plane service (adopts the running orchestrator VM by its
fixed link; no live handle needed for URL / SSH / health)."""
return FirecrackerOrchestrator()
def gateway(self) -> FirecrackerGateway:
"""The data-plane service (adopts the running gateway VM by its fixed
link; CA fetch / provisioning go through the stable key)."""
return FirecrackerGateway()
def ensure_running(
self, *, startup_timeout: float = DEFAULT_STARTUP_TIMEOUT_SECONDS,
) -> str:
"""Idempotent per-host singleton pair. Adopt the running VMs if the
orchestrator's control plane is healthy AND the gateway VM is alive AND
both booted the CURRENT code version (a prior launcher booted them — they
outlive short-lived `start` processes); otherwise clear any stale VMs and
boot a fresh pair (orchestrator first, then the gateway that resolves
policy against it). Returns the host control-plane URL.
Concurrency-safe: the cold stop/build/boot path is serialized by a host
flock, so two simultaneous first launches don't both boot on the same
links/PIDs. The healthy fast-path takes no lock."""
key = infra_vm._infra_dir() / "id_ed25519"
orchestrator = self.orchestrator()
url = orchestrator.url()
want = infra_vm.expected_version()
if infra_vm.adoptable(key, url, want):
info(f"adopting running infra VMs (orchestrator at {url})")
return url
with infra_vm.singleton_lock():
# Re-check under the lock: another launcher may have booted them while
# we waited (double-checked, so we adopt not re-boot).
if infra_vm.adoptable(key, url, want):
info(f"adopting running infra VMs (orchestrator at {url})")
return url
# Clear stale/hung/OUTDATED VMs holding either link before booting fresh.
infra_vm.stop()
infra_vm.ensure_built()
# Orchestrator first — the gateway daemons reach the control plane at
# startup. Each service owns its own boot; the orchestrator (which
# holds the signing key) mints the role-scoped `gateway` JWT for the
# gateway, which never sees the key (#469).
orchestrator.ensure_running(startup_timeout=startup_timeout)
self.gateway().connect_to_orchestrator(
orchestrator.gateway_url(), orchestrator.mint_gateway_token())
infra_vm.record_booted_version(want)
return url
def stop(self) -> None:
"""Stop both infra VMs (idempotent) and drop the version marker."""
infra_vm.stop()
__all__ = ["FirecrackerInfraService"]
+48 -104
View File
@@ -1,26 +1,22 @@
"""Prebuilt infra-VM rootfs images, pulled as artifacts (PRD 0069 Stage 2).
"""Prebuilt infra-VM rootfs, pulled as an artifact (PRD 0069 Stage 2).
The two Firecracker infra VMs each boot a fixed per-plane rootfs that does not
vary per launch — the per-boot bits (authorized_keys, guest IP) ride the kernel
cmdline, so one rootfs boots on any host:
* `orchestrator` — the control plane + buildah (in-VM agent-image builds);
* `gateway` — the slim data plane (no build tooling on the exposed VM).
Instead of building them on the launch host with Docker, we build them
**off-host** and publish each as a versioned, ready-to-boot ext4 (gzip-
compressed) to a per-role Gitea **generic package**; the launch host downloads +
verifies + boots them. No Docker, no image tooling on the launch host — just an
HTTP fetch and gunzip.
The Firecracker infra VM boots a fixed rootfs (orchestrator control plane +
gateway + buildah, control-plane init as PID 1) that does not vary per launch —
the per-boot bits (authorized_keys, guest IP) ride the kernel cmdline, so one
rootfs boots on any host. Instead of building that rootfs on the launch host
with Docker, we build it **off-host** and publish it as a versioned, ready-to-
boot ext4 (gzip-compressed) to a Gitea **generic package**; the launch host
downloads + verifies + boots it. No Docker, no image tooling on the launch
host — just an HTTP fetch and gunzip.
publish (off-host, see publish_infra.py):
docker build -> rootfs dir -> mke2fs -> gzip -> PUT generic package
pull (this module, launch host):
GET .../rootfs.ext4.gz (+ .sha256) -> verify -> gunzip -> boot
Each artifact **version** is a content hash of everything baked into that rootfs
(the shipped bot_bottle package, the role's Dockerfiles, and the role init), so
a launch host always pulls the artifact matching its code and a content change
The artifact **version** is a content hash of everything baked into the rootfs
(the shipped bot_bottle package, the three Dockerfiles, and the init), so a
launch host always pulls the artifact matching its code and a content change
can't silently boot a stale rootfs. A checksum mismatch fails closed.
Set `BOT_BOTTLE_INFRA_BUILD=local` to skip the pull and build the rootfs
@@ -45,23 +41,11 @@ from . import util
_ARTIFACT_FORMAT = "1"
_REPO_ROOT = Path(__file__).resolve().parents[3]
# The two per-plane infra VM roles. Each publishes/pulls its own rootfs artifact
# from its own generic package; the Dockerfiles baked into each differ (only the
# orchestrator rootfs carries buildah), so the versions are hashed separately.
ROLES = ("orchestrator", "gateway")
_DOCKERFILES = {
"orchestrator": ("Dockerfile.orchestrator", "Dockerfile.orchestrator.fc"),
"gateway": ("Dockerfile.gateway",),
}
_DOCKERFILES = ("Dockerfile.orchestrator", "Dockerfile.gateway", "Dockerfile.infra")
_DEFAULT_BASE = "https://gitea.dideric.is"
_DEFAULT_OWNER = "didericis"
def _package(role: str) -> str:
return f"bot-bottle-firecracker-{role}"
_PACKAGE = "bot-bottle-firecracker-infra"
# Streaming copy chunk for the (hundreds-of-MB) download.
_CHUNK = 1 << 20
@@ -73,24 +57,21 @@ def local_build_requested() -> bool:
return os.environ.get("BOT_BOTTLE_INFRA_BUILD", "").strip().lower() == "local"
def infra_artifact_version(
init_script: str, role: str, *, repo_root: Path = _REPO_ROOT,
) -> str:
"""Content hash (16 hex) of everything baked into `role`'s infra rootfs: the
whole shipped `bot_bottle` package, that role's Dockerfiles, and its guest
init. Deterministic across the publish host and the launch host when both run
the same checkout, so the tag the launch host pulls is exactly the tag
publish produced.
def infra_artifact_version(init_script: str, *, repo_root: Path = _REPO_ROOT) -> str:
"""Content hash (16 hex) of everything baked into the infra rootfs: the
whole shipped `bot_bottle` package, the three fixed Dockerfiles, and the
guest init. Deterministic across the publish host and the launch host when
both run the same checkout, so the tag the launch host pulls is exactly the
tag publish produced.
The package is baked into both images wholesale (orchestrator `COPY`s it,
gateway `pip install`s it), so hash *every* regular file under it — not just
`*.py`. Non-Python inputs (e.g. `gateway/egress/entrypoint.sh`,
`netpool.defaults.env`) are baked in too, and a change to one must bump the
version or a launch host could boot a stale rootfs whose code differs from
its checkout. `__pycache__`/`.pyc` are the only exclusions — build artifacts,
never copied."""
The package is `COPY bot_bottle /app/bot_bottle`'d wholesale into the image,
so hash *every* regular file under it — not just `*.py`. Non-Python inputs
(e.g. `egress_entrypoint.sh`, `netpool.defaults.env`) are baked in too, and
a change to one must bump the version or a launch host could boot a stale
rootfs whose code differs from its checkout. `__pycache__`/`.pyc` are the
only exclusions — build artifacts, never copied."""
h = hashlib.sha256()
h.update(f"format={_ARTIFACT_FORMAT}\nrole={role}\n".encode())
h.update(f"format={_ARTIFACT_FORMAT}\n".encode())
pkg = repo_root / "bot_bottle"
for path in sorted(pkg.rglob("*")):
if not path.is_file():
@@ -100,15 +81,10 @@ def infra_artifact_version(
h.update(str(path.relative_to(repo_root)).encode())
h.update(b"\0")
h.update(path.read_bytes())
for name in _DOCKERFILES[role]:
for name in _DOCKERFILES:
h.update(name.encode())
h.update(b"\0")
h.update((repo_root / name).read_bytes())
h.update(b"pyproject.toml\0")
h.update((repo_root / "pyproject.toml").read_bytes())
h.update(b"dropbear\0")
dropbear = util.dropbear_path()
h.update(dropbear.read_bytes() if dropbear.is_file() else b"<missing>")
h.update(b"init\0")
h.update(init_script.encode())
return h.hexdigest()[:16]
@@ -126,20 +102,19 @@ def _config() -> tuple[str, str, str]:
return base, owner, token
def artifact_url(version: str, filename: str, *, role: str) -> str:
"""The generic-package download URL for one file of `role`'s artifact at
`version` (`rootfs.ext4.gz` / `rootfs.ext4.gz.sha256`)."""
def artifact_url(version: str, filename: str) -> str:
"""The generic-package download URL for one file of this version's
artifact (`rootfs.ext4.gz` / `rootfs.ext4.gz.sha256`)."""
base, owner, _ = _config()
return f"{base}/api/packages/{owner}/generic/{_package(role)}/{version}/{filename}"
return f"{base}/api/packages/{owner}/generic/{_PACKAGE}/{version}/{filename}"
_GZ_NAME = "rootfs.ext4.gz"
_SHA_NAME = "rootfs.ext4.gz.sha256"
_CANDIDATE_DIR_ENV = "BOT_BOTTLE_INFRA_ARTIFACT_DIR"
def _cache_root(version: str, role: str) -> Path:
return util.cache_dir() / "infra-artifact" / role / version
def _cache_root(version: str) -> Path:
return util.cache_dir() / "infra-artifact" / version
def _open(url: str) -> urllib.request.Request:
@@ -181,52 +156,21 @@ def _sha256_file(path: Path) -> str:
return h.hexdigest()
def ensure_artifact_gz(version: str, *, role: str) -> Path:
"""The verified, cached `rootfs.ext4.gz` for `role` at `version` —
downloading it (and its `.sha256`) once, then reusing it. Fail-closed on a
checksum mismatch: the partial is removed and we die rather than boot an
unverified rootfs.
A pre-staged candidate bundle (`BOT_BOTTLE_INFRA_ARTIFACT_DIR`) holds each
role under its own `<dir>/<role>/` subdir."""
candidate_dir = os.environ.get(_CANDIDATE_DIR_ENV, "").strip()
if candidate_dir:
root = Path(candidate_dir) / role
version_file = root / "version.txt"
# Guard the read so a missing version.txt is a clean error, not a raw
# FileNotFoundError.
if not version_file.is_file():
die(f"infra candidate bundle is incomplete: {root}")
declared = version_file.read_text(encoding="utf-8").strip()
if declared != version:
die(
f"infra candidate version mismatch ({role}): expected {version}, "
f"bundle contains {declared or '<empty>'}"
)
gz = root / _GZ_NAME
sha = root / _SHA_NAME
if not gz.is_file() or not sha.is_file():
die(f"infra candidate bundle is incomplete: {root}")
expected = sha.read_text().split()[0].strip().lower()
actual = _sha256_file(gz)
if actual != expected:
die(
f"infra candidate checksum mismatch ({role}) for {version}:\n"
f" expected {expected}\n actual {actual}"
)
return gz
root = _cache_root(version, role)
def ensure_artifact_gz(version: str) -> Path:
"""The verified, cached `rootfs.ext4.gz` for `version` — downloading it (and
its `.sha256`) once, then reusing it. Fail-closed on a checksum mismatch:
the partial is removed and we die rather than boot an unverified rootfs."""
root = _cache_root(version)
root.mkdir(parents=True, exist_ok=True)
gz = root / _GZ_NAME
ok = root / ".verified"
if gz.is_file() and ok.is_file():
return gz
info(f"pulling infra rootfs artifact {_package(role)}/{version}")
_download(artifact_url(version, _GZ_NAME, role=role), gz)
info(f"pulling infra rootfs artifact {_PACKAGE}/{version}")
_download(artifact_url(version, _GZ_NAME), gz)
sha = root / _SHA_NAME
_download(artifact_url(version, _SHA_NAME, role=role), sha)
_download(artifact_url(version, _SHA_NAME), sha)
expected = sha.read_text().split()[0].strip().lower()
actual = _sha256_file(gz)
@@ -234,7 +178,7 @@ def ensure_artifact_gz(version: str, *, role: str) -> Path:
gz.unlink(missing_ok=True)
sha.unlink(missing_ok=True)
die(
f"infra artifact checksum mismatch ({role}) for {version}:\n"
f"infra artifact checksum mismatch for {version}:\n"
f" expected {expected}\n"
f" actual {actual}\n"
f" refusing to boot an unverified rootfs."
@@ -243,13 +187,13 @@ def ensure_artifact_gz(version: str, *, role: str) -> Path:
return gz
def materialize_ext4(version: str, dest: Path, *, role: str) -> None:
"""Ensure the verified `role` artifact is cached, then gunzip it to `dest` —
a fresh, writable per-boot rootfs (the VM mutates it; the cached `.gz` stays
def materialize_ext4(version: str, dest: Path) -> None:
"""Ensure the verified artifact is cached, then gunzip it to `dest` — a
fresh, writable per-boot rootfs (the VM mutates it; the cached `.gz` stays
pristine). Atomic via a `.part` sibling."""
gz = ensure_artifact_gz(version, role=role)
gz = ensure_artifact_gz(version)
tmp = dest.with_suffix(dest.suffix + ".part")
info(f"expanding {role} infra rootfs -> {dest}")
info(f"expanding infra rootfs -> {dest}")
with gzip.open(gz, "rb") as src, open(tmp, "wb") as out:
shutil.copyfileobj(src, out, _CHUNK)
tmp.replace(dest)
+253 -309
View File
@@ -1,30 +1,18 @@
"""The per-host infra VMs for the Firecracker backend (PRD 0070).
"""The per-host infra VM for the Firecracker backend (PRD 0070 Stage B).
Two persistent microVMs, split now that #469 got the DB off the data plane
(PRD 0070 "Separating the planes"):
A single persistent microVM that runs the orchestrator **control plane** (and,
in a following step, the gateway **data plane**) — the trusted per-host service
the docker backend runs as containers. It boots on the NAT'd orchestrator link
(`netpool.orch_slot()`): the host CLI reaches its control plane over HTTP at the
guest IP, and agent VMs reach its gateway ports over VM-to-VM routing.
* **orchestrator VM** — the control plane. Boots on the NAT'd orchestrator
link (`netpool.orch_slot()`); the host CLI reaches its `/health` +
operator routes over HTTP at the guest IP. Sole opener of `bot-bottle.db`
(on its persistent /dev/vdb registry volume); holds the host-canonical
signing key (pushed post-boot). Also carries buildah, so in-VM agent-image
builds run here (PRD 0070 v1: builds stay with the control plane).
* **gateway VM** — the data plane. Boots on its own NAT'd link
(`netpool.gw_slot()`); runs the egress / git-http / supervise daemons that
agent VMs reach (their gateway-port traffic is DNAT'd here — never to the
orchestrator, so a breached agent has no L3 route to the control plane).
Holds the mitmproxy CA + a pre-minted `gateway` JWT (never the signing
key); reaches the orchestrator's control plane at `orch_guest:8099` over
the one nft forward rule that link allows.
Build-from-source (the default while the design churns): the rootfs is exported
from the locally built orchestrator image, which bakes the stdlib-only
control-plane source. A pull-from-registry mode (Gitea's OCI registry) becomes
the default later.
Each VM boots its **own** per-plane rootfs — the orchestrator rootfs carries the
control plane + buildah (in-VM agent builds), the gateway rootfs is the slim
data plane with no build tooling on the exposed VM. Two artifacts, built/pulled
per role (`infra_artifact`); each rootfs bakes only its own role init as PID 1.
The gateway VM also runs a slimmer memory ceiling.
SSH is left enabled for debugging + provisioning; the control plane is the
load-bearing surface.
SSH is left enabled for debugging; the control plane is the load-bearing
surface.
"""
from __future__ import annotations
@@ -32,7 +20,9 @@ from __future__ import annotations
import fcntl
import hashlib
import os
import shlex
import signal
import stat
import subprocess
import time
import urllib.error
@@ -44,52 +34,41 @@ from typing import Generator
from ...log import die, info
from ..docker import util as docker_mod
from ..docker.gateway_provision import GatewayProvisionError
from . import firecracker_vm, infra_artifact, netpool, util
# The orchestrator VM's signing-key path on its persistent /dev/vdb volume
# (mounted at BOT_BOTTLE_ROOT=/var/lib/bot-bottle). The launcher seeds this
# file with the host-canonical key AFTER boot (over SSH), so the VM verifies
# tokens with the same key the host CLI signs with — the host token file stays
# the single source of truth, never clobbered per-backend (issue #469 review).
_GUEST_SIGNING_KEY_PATH = "/var/lib/bot-bottle/orchestrator-token"
# The gateway VM's pre-minted `gateway` JWT path (rootfs, not /dev/vdb — the
# data plane has no registry volume and never opens the DB). Pushed post-boot;
# the gateway daemons present it to the orchestrator, and never see the key.
_GUEST_GATEWAY_JWT_PATH = "/var/lib/bot-bottle/gateway-jwt"
# The two per-plane rootfs source images. The orchestrator VM boots a control
# plane + buildah rootfs (Dockerfile.orchestrator.fc, FROM orchestrator); the
# gateway VM boots the slim data-plane image directly (no build tooling on the
# exposed VM). Built from source by default; the launch host pulls prebuilt
# artifacts instead (`infra_artifact`).
# The single infra-VM image: gateway data plane + baked control-plane source
# (Dockerfile.infra FROM the gateway image). Built from source by default;
# a pull-from-registry mode lands later.
_INFRA_IMAGE = "bot-bottle-infra:latest"
_GATEWAY_IMAGE = "bot-bottle-gateway:latest"
_ORCHESTRATOR_IMAGE = "bot-bottle-orchestrator:latest"
_ORCHESTRATOR_FC_IMAGE = "bot-bottle-orchestrator-fc:latest"
_REPO_ROOT = Path(__file__).resolve().parents[3]
# Per-role rootfs source image + the extra free space `mke2fs` leaves for the
# guest to grow into. The orchestrator keeps buildah's large build slack; the
# gateway carries no build tooling, so its rootfs is much smaller.
_ROOTFS_IMAGE = {"orchestrator": _ORCHESTRATOR_FC_IMAGE, "gateway": _GATEWAY_IMAGE}
_ROOTFS_SLACK_MIB = {"orchestrator": 8192, "gateway": 1024}
CONTROL_PLANE_PORT = 8099
# Gateway data-plane ports (agent-facing): egress proxy, supervise MCP,
# git-http. Reached by agent VMs over VM-to-VM routing (added next).
EGRESS_PORT = 9099
SUPERVISE_PORT = 9100
GIT_HTTP_PORT = 9420
# mitmproxy writes its CA here a beat after start; agents install it to trust
# the gateway's TLS interception.
_GATEWAY_CA_PATH = "/home/mitmproxy/.mitmproxy/mitmproxy-ca-cert.pem"
ORCHESTRATOR_PORT = 8099
# The infra VMs make direct upstream connections (gateway egress, and buildah
# The infra VM makes direct upstream connections (gateway egress, and buildah
# during builds), and the kernel `ip=` cmdline sets no resolver. Public for
# now; routing DNS through a filtered path is a later refinement.
_INFRA_RESOLVER = "1.1.1.1"
# How long the launcher retries pushing a secret while the guest's SSH comes
# up. Below the init's own wait window, so a failed push dies here first.
_SECRET_PUSH_TIMEOUT_SECONDS = 30.0
_SECRET_PUSH_POLL_SECONDS = 0.5
_HEALTH_TIMEOUT_SECONDS = 45.0
_HEALTH_POLL_SECONDS = 0.5
_CA_TIMEOUT_SECONDS = 30.0
@dataclass
class InfraVm:
"""A handle to one infra VM: its guest IP and the stable SSH key used to
seed secrets / fetch the CA / provision git-gate. `vm` is the live VMM
"""A handle to the per-host infra VM: its guest IP and the stable SSH key
used to fetch the gateway CA / provision git-gate. `vm` is the live VMM
handle when this process booted it, and None when adopting a singleton a
prior launcher started (teardown then goes through the PID file)."""
@@ -97,61 +76,111 @@ class InfraVm:
private_key: Path
vm: firecracker_vm.VmHandle | None = None
@property
def control_plane_url(self) -> str:
return f"http://{self.guest_ip}:{CONTROL_PLANE_PORT}"
def role_init(role: str) -> str:
"""The guest PID-1 init for `role` (each per-plane rootfs bakes only its
own — no `bb_role` branch, since the rootfs *is* the role)."""
return _orchestrator_init() if role == "orchestrator" else _gateway_init()
def terminate(self) -> None:
"""Stop the infra VM — via the live handle if we booted it, else the
PID file (adopting-process case)."""
if self.vm is not None:
self.vm.terminate()
else:
_kill_pidfile()
_pid_file().unlink(missing_ok=True)
def _role_version(role: str) -> str:
return infra_artifact.infra_artifact_version(role_init(role), role)
def gateway_ca_pem(self, *, timeout: float = _CA_TIMEOUT_SECONDS) -> str:
"""The gateway's mitmproxy CA (PEM) that agents install to trust its
TLS interception. Generated a moment after boot, so this polls over
SSH until it appears (mirrors DockerGateway.ca_cert_pem)."""
deadline = time.monotonic() + timeout
while True:
proc = subprocess.run(
util.ssh_base_argv(self.private_key, self.guest_ip)
+ [f"cat {_GATEWAY_CA_PATH}"],
capture_output=True, text=True, timeout=15, check=False,
)
if proc.returncode == 0 and "BEGIN CERTIFICATE" in proc.stdout:
return proc.stdout
if time.monotonic() >= deadline:
die(f"gateway CA not available after {timeout:g}s: "
f"{proc.stderr.strip() or 'empty'}")
time.sleep(_HEALTH_POLL_SECONDS)
def ensure_built() -> None:
"""Ensure both infra rootfs artifacts are available before boot.
"""Ensure the infra rootfs is available before boot.
Default (docker-free, PRD 0069 Stage 2): download + verify the prebuilt
orchestrator + gateway rootfs artifacts matching this code version (see
`infra_artifact`); the launch host needs no Docker.
`BOT_BOTTLE_INFRA_BUILD=local` instead builds the images from source with
host Docker (the orchestrator-fc image is `FROM` the orchestrator image, so
it must exist first) — for iterating on the Dockerfiles."""
rootfs artifact matching this code version (see `infra_artifact`); the
launch host needs no Docker. `BOT_BOTTLE_INFRA_BUILD=local` instead builds
the three fixed images from source with host Docker — the infra image
`COPY --from`s the orchestrator image and is `FROM` the gateway image, so
both must exist first — for iterating on the Dockerfiles."""
if infra_artifact.local_build_requested():
build_infra_images_with_docker()
return
for role in infra_artifact.ROLES:
infra_artifact.ensure_artifact_gz(_role_version(role), role=role)
infra_artifact.ensure_artifact_gz(
infra_artifact.infra_artifact_version(_infra_init()))
def build_infra_images_with_docker() -> None:
"""Build the fixed images from source with host Docker: orchestrator,
gateway, then the orchestrator-fc image (Dockerfile.orchestrator.fc: FROM
orchestrator + buildah). The gateway VM boots the gateway image directly.
The launch host uses this only in `BOT_BOTTLE_INFRA_BUILD=local` mode;
`publish_infra` uses it off-host to produce the published artifacts."""
"""Build the three fixed images from source with host Docker: orchestrator,
gateway, then the combined infra image (`COPY --from` orchestrator, `FROM`
gateway). The launch host uses this only in `BOT_BOTTLE_INFRA_BUILD=local`
mode; `publish_infra` uses it off-host to produce the published artifact."""
docker_mod.build_image(
_ORCHESTRATOR_IMAGE, str(_REPO_ROOT), dockerfile="Dockerfile.orchestrator")
docker_mod.build_image(
_GATEWAY_IMAGE, str(_REPO_ROOT), dockerfile="Dockerfile.gateway")
docker_mod.build_image(
_ORCHESTRATOR_FC_IMAGE, str(_REPO_ROOT), dockerfile="Dockerfile.orchestrator.fc")
_INFRA_IMAGE, str(_REPO_ROOT), dockerfile="Dockerfile.infra")
def build_rootfs_dir(role: str) -> Path:
"""`role`'s base rootfs dir: its source image prepared with the role init as
PID 1. The init's content is folded into the cache key so an init change
rebuilds the rootfs (the base image digest alone wouldn't catch it)."""
init = role_init(role)
def build_infra_rootfs_dir() -> Path:
"""The infra VM's base rootfs: the infra image prepared with the
control-plane + gateway init as PID 1. The init's content is folded into
the cache key so an init change rebuilds the rootfs (the base image digest
alone wouldn't catch it)."""
init = _infra_init()
tag = hashlib.sha256(init.encode()).hexdigest()[:8]
return util.build_base_rootfs_dir(
_ROOTFS_IMAGE[role], variant=f"-{role}-{tag}", init_script=init,
_INFRA_IMAGE, variant=f"-infra-{tag}", init_script=init,
)
def ensure_running() -> InfraVm:
"""Idempotent per-host singleton. Adopt the infra VM if its control plane
is already healthy (a prior launcher booted it — it outlives short-lived
`start` processes); otherwise clear any stale VM and boot a fresh one.
Returns a handle usable for CA fetch / git-gate provisioning.
Concurrency-safe: the cold stop/build/boot path is serialized by a host
flock, so two simultaneous first launches don't both boot on the same
rootfs/PID. The healthy fast-path takes no lock."""
slot = netpool.orch_slot()
url = f"http://{slot.guest_ip}:{CONTROL_PLANE_PORT}"
key = _infra_dir() / "id_ed25519"
if key.exists() and _health_ok(url):
info(f"adopting running infra VM at {url}")
return InfraVm(guest_ip=slot.guest_ip, private_key=key)
with _singleton_lock():
# Re-check under the lock: another launcher may have booted it while
# we waited for the lock (double-checked, so we adopt not re-boot).
if key.exists() and _health_ok(url):
info(f"adopting running infra VM at {url}")
return InfraVm(guest_ip=slot.guest_ip, private_key=key)
stop() # clear a stale/hung VM holding the link before booting fresh
ensure_built()
infra = boot()
wait_for_health(infra)
return infra
@contextmanager
def singleton_lock() -> Generator[None, None, None]:
"""Host-level exclusive lock serializing the infra pair's cold create path
def _singleton_lock() -> Generator[None, None, None]:
"""Host-level exclusive lock serializing the infra VM's cold create path
(`stop`/`ensure_built`/`boot`). flock auto-releases if the launcher
crashes, so the lock is never leaked."""
lock_path = _infra_dir() / "singleton.lock"
@@ -164,62 +193,38 @@ def singleton_lock() -> Generator[None, None, None]:
def stop() -> None:
"""Stop BOTH infra VMs (idempotent — absent is success). Reaps the
recorded VMMs AND any orphaned firecracker still bound to either infra
config — the PID files drift after crashes / out-of-band kills, and a
survivor would hold a link's TAP so the next boot dies with "tap …
Resource busy". Also reaps a surviving *legacy* single combined-VM (the
pre-split layout booted from `<infra>/config.json` on the orchestrator
link): the two-VM `stop` otherwise wouldn't know about it, and it would
hold the orchestrator TAP so the first split boot fails — so the cutover
is self-healing, no manual host teardown. Drops the version marker so a
stopped pair is never treated as adoptable."""
_kill_pidfile(_orch_dir())
_kill_pidfile(_gw_dir())
_kill_pidfile(_infra_dir()) # legacy pre-split combined VM (migration)
_kill_infra_firecrackers()
_pid_file(_orch_dir()).unlink(missing_ok=True)
_pid_file(_gw_dir()).unlink(missing_ok=True)
_pid_file(_infra_dir()).unlink(missing_ok=True) # legacy
_version_file().unlink(missing_ok=True)
"""Stop the infra VM singleton (idempotent — absent is success)."""
_kill_pidfile()
_pid_file().unlink(missing_ok=True)
def boot_vm(
*,
name: str,
slot: netpool.Slot,
run_dir: Path,
role: str,
mem_mib: int,
data_drive: Path | None = None,
extra_boot_args: str = "",
) -> InfraVm:
"""Boot the `role` infra VM from its per-plane rootfs on `slot`'s link.
Records the PID."""
def boot() -> InfraVm:
"""Boot the infra VM (detached, so it outlives the launcher) on the
orchestrator link, recording its PID. Prefer `ensure_running`."""
slot = netpool.orch_slot()
if not netpool.tap_present(slot.iface):
die(f"infra link {slot.iface} not present.\n"
die(f"orchestrator link {slot.iface} not present.\n"
f" ./cli.py backend setup --backend=firecracker")
run_dir.mkdir(parents=True, exist_ok=True)
run_dir = _infra_dir()
rootfs = run_dir / "rootfs.ext4"
if infra_artifact.local_build_requested():
util.build_rootfs_ext4(
build_rootfs_dir(role), rootfs, slack_mib=_ROOTFS_SLACK_MIB[role])
util.build_rootfs_ext4(build_infra_rootfs_dir(), rootfs, slack_mib=8192)
else:
# Prebuilt artifact already carries the role's build slack; expand it to
# a fresh writable rootfs for this boot.
infra_artifact.materialize_ext4(_role_version(role), rootfs, role=role)
# Prebuilt artifact already carries the buildah build slack; expand it
# to a fresh writable rootfs for this boot.
infra_artifact.materialize_ext4(
infra_artifact.infra_artifact_version(_infra_init()), rootfs)
private_key, pubkey = _stable_keypair()
info(f"booting {role} VM on {slot.iface} (guest {slot.guest_ip})")
boot_args = extra_boot_args
info(f"booting infra VM on {slot.iface} (guest {slot.guest_ip})")
vm = firecracker_vm.boot(
name=name, rootfs=rootfs, tap=slot.iface,
name="bot-bottle-infra", rootfs=rootfs, tap=slot.iface,
guest_ip=slot.guest_ip, host_ip=slot.host_ip, pubkey=pubkey,
run_dir=run_dir, mem_mib=mem_mib, detached=True,
data_drive=data_drive, extra_boot_args=boot_args,
run_dir=run_dir, mem_mib=4096, detached=True,
data_drive=_ensure_registry_volume(),
)
_pid_file(run_dir).write_text(str(vm.process.pid))
_pid_file().write_text(str(vm.process.pid))
return InfraVm(guest_ip=slot.guest_ip, private_key=private_key, vm=vm)
@@ -229,85 +234,43 @@ def _infra_dir() -> Path:
return d
def _orch_dir() -> Path:
d = _infra_dir() / "orchestrator"
d.mkdir(parents=True, exist_ok=True)
return d
def _pid_file() -> Path:
return _infra_dir() / "vm.pid"
def _gw_dir() -> Path:
d = _infra_dir() / "gateway"
d.mkdir(parents=True, exist_ok=True)
return d
# The registry "volume": a host-side ext4 file attached to the infra VM as a
# second virtio-block device (guest /dev/vdb), mounted at the control plane's
# DB dir. It outlives the ephemeral rootfs, so the bottle registry survives an
# infra-VM restart — the firecracker analogue of a docker volume. It is a
# plain ext4 file: `sudo mount -o loop <path>` on the host (with the VM
# stopped) to inspect bot-bottle.db directly.
_REGISTRY_SIZE = "512M"
def _pid_file(run_dir: Path) -> Path:
return run_dir / "vm.pid"
def registry_volume_path() -> Path:
return _infra_dir() / "registry.ext4"
def _version_file() -> Path:
"""Records the infra-artifact version the *running* pair booted from, so a
later launcher can tell whether the singletons it found are the current
code. Without it, a healthy pair built from an older image gets adopted
forever and the new code never boots — every infra change would need an
out-of-band kill to dislodge the stale VMs (and races whatever launched
next). Both VMs boot the same artifact, so one marker covers the pair."""
return _infra_dir() / "booted-version"
def expected_version() -> str:
"""The combined marker for the running pair: both per-plane artifact
versions, so a change to either rootfs dislodges the adopted pair."""
return " ".join(f"{role}={_role_version(role)}" for role in infra_artifact.ROLES)
def adoptable(key: Path, url: str, want: str) -> bool:
"""Adopt the running pair only if it booted from the CURRENT version, the
orchestrator's control plane is healthy, and the gateway VM is still alive.
A missing/mismatched marker means a prior launcher booted an older infra
image; a dead gateway means the pair is half-down — reboot both rather than
reuse stale or partial state."""
if not key.exists():
return False
try:
booted = _version_file().read_text(encoding="utf-8").strip()
except OSError:
return False
if booted != want:
return False
if not _health_ok(url):
return False
return _pidfile_alive(_gw_dir())
def record_booted_version(version: str) -> None:
_version_file().write_text(version + "\n", encoding="utf-8")
def push_secret(infra: InfraVm, secret: str, dest: str, what: str) -> None:
"""Pipe `secret` to an atomic write of `dest` in the guest over SSH,
retrying while the guest's SSH comes up; die (naming `what`) if it never
succeeds. Bare-pipe input keeps the value off argv."""
push = f"umask 077; cat > {dest}.tmp && mv {dest}.tmp {dest}"
deadline = time.monotonic() + _SECRET_PUSH_TIMEOUT_SECONDS
last = ""
while time.monotonic() < deadline:
proc = subprocess.run(
util.ssh_base_argv(infra.private_key, infra.guest_ip) + [push],
input=secret, capture_output=True, text=True, check=False,
)
if proc.returncode == 0:
return
last = proc.stderr.strip()
time.sleep(_SECRET_PUSH_POLL_SECONDS)
die(f"could not push {what}: {last or '<no stderr>'}")
def _ensure_registry_volume() -> Path:
"""Create the empty ext4 registry volume on first use; reuse it after."""
vol = registry_volume_path()
if vol.exists():
return vol
info(f"creating infra registry volume {vol} ({_REGISTRY_SIZE})")
proc = subprocess.run(
["mke2fs", "-q", "-t", "ext4", "-F", str(vol), _REGISTRY_SIZE],
capture_output=True, text=True, check=False,
)
if proc.returncode != 0:
vol.unlink(missing_ok=True)
die(f"creating registry volume failed: {proc.stderr.strip()}")
return vol
def _stable_keypair() -> tuple[Path, str]:
"""The infra VMs' shared SSH keypair — generated once and reused, so any
later launcher can SSH in (seed secrets / fetch CA / provision) even though
a different process booted the VMs. Both VMs get the same pubkey re-injected
on every boot via the cmdline."""
"""The infra VM's SSH keypair — generated once and reused, so any later
launcher can SSH in (fetch CA / provision) even though a different process
booted the VM. The pubkey is re-injected on every boot via the cmdline."""
d = _infra_dir()
key, pub = d / "id_ed25519", d / "id_ed25519.pub"
if key.exists() and pub.exists():
@@ -322,24 +285,11 @@ def _stable_keypair() -> tuple[Path, str]:
return key, pub.read_text().strip()
def _pidfile_alive(run_dir: Path) -> bool:
"""True iff `run_dir`'s recorded VMM is still a live firecracker (guards a
recycled PID by checking `comm`)."""
try:
pid = int(_pid_file(run_dir).read_text().strip())
except (OSError, ValueError):
return False
try:
return Path(f"/proc/{pid}/comm").read_text().strip() == "firecracker"
except OSError:
return False
def _kill_pidfile(run_dir: Path) -> None:
"""SIGTERM (then SIGKILL) the VMM recorded in `run_dir`, if it's still ours.
def _kill_pidfile() -> None:
"""SIGTERM (then SIGKILL) the recorded infra VMM, if it's still ours.
Guards against a recycled PID by checking the process is firecracker."""
try:
pid = int(_pid_file(run_dir).read_text().strip())
pid = int(_pid_file().read_text().strip())
except (OSError, ValueError):
return
try:
@@ -359,35 +309,6 @@ def _kill_pidfile(run_dir: Path) -> None:
pass
def _kill_infra_firecrackers(proc_root: Path = Path("/proc")) -> None:
"""SIGKILL any firecracker VMM whose `--config-file` is one of this host's
infra configs (orchestrator or gateway), independent of the PID files —
reaps orphans it lost track of so both links' TAPs are free to rebind.
Also matches the *legacy* pre-split combined-VM config (`<infra>/config.json`)
so a surviving old singleton is cleared off the orchestrator link during the
cutover. Scoped to these infra config paths, so the pool's agent VMs (other
config paths) are untouched."""
cfgs = {
str(_orch_dir() / "config.json"),
str(_gw_dir() / "config.json"),
str(_infra_dir() / "config.json"), # legacy pre-split combined VM
}
for entry in proc_root.iterdir():
if not entry.name.isdigit():
continue
try:
if (entry / "comm").read_text().strip() != "firecracker":
continue
args = (entry / "cmdline").read_bytes().split(b"\0")
except OSError:
continue # process vanished / not ours
if any(a.decode("utf-8", "replace") in cfgs for a in args):
try:
os.kill(int(entry.name), signal.SIGKILL)
except (OSError, ValueError):
pass
def _health_ok(url: str) -> bool:
try:
with urllib.request.urlopen(f"{url}/health", timeout=1.0) as resp:
@@ -396,12 +317,77 @@ def _health_ok(url: str) -> bool:
return False
def _init_head() -> str:
"""The shared PID-1 preamble both role inits open with: mount the pseudo-
filesystems, export a real PATH (a bare-init shell's built-in exec path
isn't in the *environment*, so backgrounded `python3 ...` children would
find no PATH), set the direct upstream resolver, install the per-boot SSH
pubkey from the cmdline, and start dropbear for debug/provisioning."""
class SshGatewayTransport:
"""`GatewayTransport` for the gateway running in the infra VM — the docker
exec/cp equivalents over SSH (dropbear + the stable infra key)."""
def __init__(self, private_key: Path, guest_ip: str) -> None:
self._key = private_key
self._ip = guest_ip
def exec(self, argv: list[str]) -> None:
proc = subprocess.run(
util.ssh_base_argv(self._key, self._ip) + [shlex.join(argv)],
capture_output=True, text=True, timeout=60, check=False,
)
if proc.returncode != 0:
raise GatewayProvisionError(
f"infra gateway exec {argv!r} failed: {proc.stderr.strip()}")
def cp_into(self, src: str, dest: str) -> None:
# Preserve the source mode (docker cp does): the access-hook is staged
# 0700 and git-http execs it directly — a plain `cat >` would land it
# 0644 and the exec fails with EACCES; keys stay 0600.
mode = stat.S_IMODE(os.stat(src).st_mode)
q = shlex.quote(dest)
proc = subprocess.run(
util.ssh_base_argv(self._key, self._ip)
+ [f"cat > {q} && chmod {mode:o} {q}"],
input=Path(src).read_bytes(), capture_output=True, timeout=30, check=False,
)
if proc.returncode != 0:
raise GatewayProvisionError(
f"infra gateway cp {src} -> {dest} failed: "
f"{proc.stderr.decode(errors='replace').strip()}")
def gateway_transport() -> SshGatewayTransport:
"""git-gate provisioning transport for the gateway in the infra VM, built
from the stable key + the orchestrator link's guest IP. Needs no live VM
handle, so teardown can use it too."""
return SshGatewayTransport(
_infra_dir() / "id_ed25519", netpool.orch_slot().guest_ip)
def wait_for_health(
infra: InfraVm, *, timeout: float = _HEALTH_TIMEOUT_SECONDS,
) -> None:
"""Poll the control plane's /health until it answers 200 or the deadline
passes. Dies (with the console tail) if the VMM exits early."""
url = f"{infra.control_plane_url}/health"
deadline = time.monotonic() + timeout
while time.monotonic() < deadline:
if infra.vm is not None and not infra.vm.is_alive():
die(f"infra VM exited during boot (rc={infra.vm.process.returncode}).\n"
f"{firecracker_vm._console_tail(infra.vm.console_log)}")
try:
with urllib.request.urlopen(url, timeout=1.0) as resp:
if resp.status == 200:
info(f"infra control plane healthy at {infra.control_plane_url}")
return
except (urllib.error.URLError, TimeoutError, OSError):
pass
time.sleep(_HEALTH_POLL_SECONDS)
tail = (firecracker_vm._console_tail(infra.vm.console_log)
if infra.vm is not None else "")
die(f"infra control plane at {url} did not become healthy within "
f"{timeout:.0f}s.\n{tail}")
def _infra_init() -> str:
"""PID-1 init for the infra VM: mount the pseudo-filesystems, wire a
resolver, start dropbear (debug SSH), then launch the control plane and
the gateway data plane (multi-tenant against the local control plane)."""
return f"""#!/bin/sh
# bot-bottle Firecracker infra VM init (PID 1).
mount -t proc proc /proc 2>/dev/null
@@ -410,6 +396,10 @@ mount -t devtmpfs dev /dev 2>/dev/null
mkdir -p /dev/pts && mount -t devpts devpts /dev/pts 2>/dev/null
mount -o remount,rw / 2>/dev/null
# Export a real PATH: a bare-init shell resolves its own execs via a
# built-in default path, but that isn't in the *environment*, so
# gateway_init's subprocess daemons (spawned as `python3 ...`) would
# inherit no PATH and fail to find python3. Export it for all children.
export PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
# Direct upstream resolver (control-plane / gateway egress + buildah).
@@ -425,72 +415,26 @@ fi
chown -R 0:0 /root 2>/dev/null || true
mkdir -p /etc/dropbear /run /var/lib/bot-bottle
# Persistent registry volume (second virtio-block device, /dev/vdb) mounted
# at the control plane's DB dir, so bot-bottle.db survives infra-VM restarts.
mount -t ext4 /dev/vdb /var/lib/bot-bottle 2>/dev/null || true
/bb-dropbear -R -E -p 22 &
# Control plane. Source is baked at /app; the package is stdlib-only.
cd /app
"""
BOT_BOTTLE_ROOT=/var/lib/bot-bottle python3 -m bot_bottle.orchestrator \\
--host 0.0.0.0 --port {CONTROL_PLANE_PORT} --broker stub &
# Gateway data plane, multi-tenant: each request resolves source-IP ->
# policy against the local control plane. The VM backend reaches git over
# git-http (9420), so the git:// daemon (git-gate, needs a per-bottle
# entrypoint the consolidated model doesn't use) is left out.
BOT_BOTTLE_GATEWAY_DAEMONS=egress,git-http,supervise \\
BOT_BOTTLE_ORCHESTRATOR_URL=http://127.0.0.1:{CONTROL_PLANE_PORT} \\
SUPERVISE_DB_PATH=/var/lib/bot-bottle/db/bot-bottle.db \\
python3 /app/gateway_init.py &
_INIT_TAIL = """
# Reap as PID 1; children are backgrounded, so `wait` blocks.
while : ; do wait ; done
"""
def _gateway_init() -> str:
"""PID-1 init for the gateway (data-plane) VM. Waits for the host-seeded
`gateway` JWT, then starts ONLY the data-plane daemons, multi-tenant against
the orchestrator at the `bb_orch` cmdline address. The VM backend reaches git
over git-http (9420), so the git:// daemon (a per-bottle entrypoint the
consolidated model doesn't use) is left out. No SUPERVISE_DB_PATH: the data
plane reaches the supervise queue over the control-plane RPC and never opens
bot-bottle.db (PRD 0070 / #469). If the JWT never arrives, REFUSE to start
rather than run without auth."""
return _init_head() + f"""
ORCH=$(sed -n 's/.*bb_orch=\\([^ ]*\\).*/\\1/p' /proc/cmdline)
GW_JWT=""
i=0
while [ "$i" -lt 600 ]; do
GW_JWT=$(cat {_GUEST_GATEWAY_JWT_PATH} 2>/dev/null)
[ -n "$GW_JWT" ] && break
i=$((i + 1))
sleep 0.1
done
if [ -z "$GW_JWT" ]; then
echo "infra gateway: gateway JWT never arrived; refusing to start the data plane" >&2
else
chmod 600 {_GUEST_GATEWAY_JWT_PATH} 2>/dev/null || true
BOT_BOTTLE_GATEWAY_DAEMONS=egress,git-http,supervise \\
BOT_BOTTLE_ORCHESTRATOR_URL=http://$ORCH:{ORCHESTRATOR_PORT} \\
BOT_BOTTLE_ORCHESTRATOR_AUTH_JWT="$GW_JWT" \\
python3 -m bot_bottle.gateway.bootstrap &
fi
""" + _INIT_TAIL
def _orchestrator_init() -> str:
"""PID-1 init for the orchestrator (control-plane) VM. Mounts the persistent
registry volume (/dev/vdb — bot-bottle.db survives a VM restart), waits for
the host-seeded signing key, then starts ONLY the control plane. If the key
never arrives, REFUSE to start rather than run OPEN — open mode would grant
every unauthenticated caller the `cli` role (#469)."""
return _init_head() + f"""
# Persistent registry volume (second virtio-block device, /dev/vdb) mounted at
# the DB dir, so bot-bottle.db survives orchestrator-VM restarts.
mount -t ext4 /dev/vdb /var/lib/bot-bottle 2>/dev/null || true
CP_KEY=""
i=0
while [ "$i" -lt 600 ]; do
CP_KEY=$(cat {_GUEST_SIGNING_KEY_PATH} 2>/dev/null)
[ -n "$CP_KEY" ] && break
i=$((i + 1))
sleep 0.1
done
if [ -z "$CP_KEY" ]; then
echo "infra: control-plane signing key never arrived; refusing to start the control plane (would run OPEN)" >&2
else
chmod 600 {_GUEST_SIGNING_KEY_PATH} 2>/dev/null || true
BOT_BOTTLE_ROOT=/var/lib/bot-bottle BOT_BOTTLE_ORCHESTRATOR_TOKEN="$CP_KEY" python3 -m bot_bottle.orchestrator \\
--host 0.0.0.0 --port {ORCHESTRATOR_PORT} --broker stub &
fi
""" + _INIT_TAIL
+36 -71
View File
@@ -11,23 +11,21 @@ Per bottle:
6. provision (shared gateway CA, prompt, skills, workspace, git, supervise)
over SSH.
The per-bottle Docker sidecar bundle is gone. The shared gateway (its own
data-plane VM) handles egress / git-gate / supervise for every VM; the nft
netpool's PREROUTING DNAT routes the VMs' gateway-port traffic to that gateway
VM, and the nft table's `ct status dnat accept` rule in the forward chain lets
it pass. The VM still sends to `host_tap_ip:PORT` — the address its world is,
by nft design, limited to.
The per-bottle Docker sidecar bundle is gone. The shared gateway handles
egress / git-gate / supervise for every VM; Docker's PREROUTING DNAT routes
the VMs' traffic to it, and the nft table's `ct status dnat accept` rule
in the forward chain lets it pass. The VM still sends to `host_tap_ip:PORT`
— the address its world is, by nft design, limited to.
Isolation is enforced by the operator-provisioned nft table (checked
fail-closed in preflight): a VM reaches only the gateway (DNAT'd from the host
TAP IP) and nothing else — not even the orchestrator control plane.
fail-closed in preflight): a VM reaches only the sidecar (DNAT'd from
the host TAP IP) and nothing else.
"""
from __future__ import annotations
import dataclasses
import os
import shutil
from contextlib import ExitStack, contextmanager
from pathlib import Path
from typing import Callable, Generator
@@ -39,22 +37,24 @@ from ...bottle_state import (
git_gate_state_dir,
read_committed_image,
)
from ...egress import Egress
from ...git_gate import GitGate
from ...image_cache import check_stale_path
from ...log import die, info, warn
from ...supervisor.types import SUPERVISE_PORT
from ...egress import (
egress_agent_env_entries,
egress_resolve_token_values,
)
from ...git_gate import (
provision_git_gate_dynamic_keys,
revoke_git_gate_provisioned_keys,
)
from ...log import info, warn
from ...supervise import SUPERVISE_PORT
from ..docker.egress import EGRESS_PORT
from ..util import AGENT_CA_BUNDLE, AGENT_CA_PATH
from . import firecracker_vm, image_builder, isolation_probe, netpool, util
from .bottle import FirecrackerBottle
from .bottle_plan import FirecrackerBottlePlan
from ...orchestrator.store.config_store import resolve_teardown_timeout
from ...orchestrator.store.secret_store import ENV_VAR_SECRET_NAME
from .consolidated_launch import (
launch_consolidated,
persist_env_var_secret,
deprovision_consolidated,
teardown_consolidated,
)
@@ -64,7 +64,6 @@ _GIT_HTTP_PORT = 9420
@contextmanager
def launch(
plan: FirecrackerBottlePlan,
agent_base: Path,
*,
provision: Callable[[FirecrackerBottlePlan, "FirecrackerBottle"], str | None],
) -> Generator[FirecrackerBottle, None, None]:
@@ -81,18 +80,20 @@ def launch(
except BaseException as exc: # noqa: W0718 - teardown must continue
teardown_exc = exc
warn(f"firecracker teardown failed: {exc!r}")
GitGate().revoke_provisioned_keys(bottle_for_revoke, git_gate_dir_for_revoke)
revoke_git_gate_provisioned_keys(bottle_for_revoke, git_gate_dir_for_revoke)
if teardown_exc is not None:
raise teardown_exc
try:
# Step 1 (rootfs resolution/build) runs in BottleBackend.launch before
# this context starts resources. ``agent_base`` is the selected cache,
# fresh build, or committed snapshot.
# Step 1: agent rootfs. Built from the Dockerfile inside a Firecracker
# builder VM (buildah, no host docker); a committed snapshot is reused
# when present. Returns the base dir the per-bottle ext4 is made from.
plan, agent_base = _build_agent_base(plan)
# Step 2: mint the git-gate dynamic (gitea) deploy keys, if any.
git_gate_plan = plan.git_gate_plan
if git_gate_plan.upstreams:
git_gate_plan = GitGate().provision_dynamic_keys(
git_gate_plan = provision_git_gate_dynamic_keys(
plan.manifest.bottle, git_gate_plan, git_gate_state_dir(plan.slug),
)
@@ -108,10 +109,9 @@ def launch(
# (in memory) for the gateway to inject — the agent never sees them.
# Attribution is by the VM's guest IP (unspoofable via /31 TAP + nft).
effective_env = {**os.environ, **plan.agent_provision.provisioned_env}
token_values = Egress().resolve_token_values(
token_values = egress_resolve_token_values(
plan.egress_plan.token_env_map, effective_env,
)
teardown_timeout = resolve_teardown_timeout()
ctx = launch_consolidated(
plan.egress_plan, git_gate_plan,
guest_ip=slot.guest_ip,
@@ -119,9 +119,8 @@ def launch(
tokens=token_values,
)
stack.callback(
deprovision_consolidated, ctx.bottle_id,
teardown_consolidated, ctx.bottle_id,
orchestrator_url=ctx.orchestrator_url,
timeout=teardown_timeout,
)
# Step 5: install the SHARED gateway CA (replaces the per-bottle CA).
@@ -137,7 +136,7 @@ def launch(
)
# Point the agent's git-gate insteadOf rewrites and supervise MCP URL
# at the shared gateway (reached at the slot's host TAP IP — the VM
# sends there and the nft netpool DNAT routes to the gateway VM).
# sends there and Docker DNAT routes to the gateway container).
git_gate_url = (
f"http://{slot.host_ip}:{_GIT_HTTP_PORT}" if git_gate_plan.upstreams else ""
)
@@ -150,7 +149,6 @@ def launch(
git_gate_plan=git_gate_plan,
egress_plan=egress_plan,
identity_token=ctx.identity_token,
env_var_secret=ctx.env_var_secret,
# Deliver the identity token as egress proxy credentials — clients
# honor `HTTPS_PROXY=http://id:token@gw` without app changes; the
# gateway reads Proxy-Authorization, validates the (source_ip,
@@ -166,10 +164,6 @@ def launch(
# Step 6: build the per-bottle rootfs + SSH key, then boot.
run_dir = util.cache_dir() / "run" / plan.slug
run_dir.mkdir(parents=True, exist_ok=True)
# Remove the run dir on teardown so the per-bottle rootfs.ext4 (~1G)
# doesn't leak. Registered before vm.terminate below so it runs *after*
# it (ExitStack is LIFO): the VM is gone before we rm its rootfs.
stack.callback(lambda: shutil.rmtree(run_dir, ignore_errors=True))
rootfs = run_dir / "rootfs.ext4"
util.build_rootfs_ext4(agent_base, rootfs)
private_key, pubkey = util.generate_keypair(run_dir)
@@ -185,7 +179,6 @@ def launch(
)
stack.callback(vm.terminate)
firecracker_vm.wait_for_ssh(vm, private_key)
persist_env_var_secret(private_key, slot.guest_ip, ctx.env_var_secret)
# Authoritative fail-closed egress-boundary check, before the agent
# runs: prove the VM cannot reach the host directly.
@@ -213,7 +206,9 @@ def launch(
teardown()
def build_or_load_agent_base(plan: FirecrackerBottlePlan) -> Path:
def _build_agent_base(
plan: FirecrackerBottlePlan,
) -> tuple[FirecrackerBottlePlan, Path]:
"""Produce the agent's base rootfs dir. Primary path: build the Dockerfile
inside a Firecracker builder VM (buildah, no host docker), smoke-testing
the image before export. A committed snapshot (freeze/migrate) is resumed
@@ -222,36 +217,13 @@ def build_or_load_agent_base(plan: FirecrackerBottlePlan) -> Path:
committed_tar = committed_rootfs_path(plan.slug)
if committed and committed_tar.is_file():
info(f"resuming from committed rootfs {committed_tar}")
return util.build_committed_rootfs_dir(committed_tar)
dockerfile = Path(plan.dockerfile_path)
if plan.spec.image_policy == "cached":
cached = image_builder.cached_agent_rootfs_dir(dockerfile)
if cached is None:
die(
f"cached agent rootfs for {plan.image!r} not found; "
"run without --cached-images to build it"
)
info(f"using cached agent rootfs {cached.name}")
return cached
return image_builder.build_agent_rootfs_dir(
dockerfile,
return plan, util.build_committed_rootfs_dir(committed_tar)
base = image_builder.build_agent_rootfs_dir(
Path(plan.dockerfile_path),
image_tag=plan.image,
smoke_test=runtime_for(plan.agent_provider_template).smoke_test,
)
def stale_checks(plan: FirecrackerBottlePlan) -> None:
"""Raise when the cached rootfs selected by this plan is stale."""
if plan.spec.image_policy != "cached":
return
committed = read_committed_image(plan.slug)
committed_tar = committed_rootfs_path(plan.slug)
if committed and committed_tar.is_file():
check_stale_path(f"agent rootfs {committed_tar}", committed_tar)
return
cached = image_builder.cached_agent_rootfs_dir(Path(plan.dockerfile_path))
if cached is not None:
check_stale_path(f"agent rootfs {cached}", cached / ".bb-ready")
return plan, base
# --- agent guest env -------------------------------------------------
@@ -267,11 +239,6 @@ def _agent_guest_env(plan: FirecrackerBottlePlan, host_ip: str) -> dict[str, str
"HTTPS_PROXY": proxy_url, "HTTP_PROXY": proxy_url,
"https_proxy": proxy_url, "http_proxy": proxy_url,
"NO_PROXY": no_proxy, "no_proxy": no_proxy,
# Rootfs export can leave Git's implicit XDG paths unreadable even
# after the runtime repair. Bypass that discovery and name the
# provisioned global config explicitly so insteadOf can never fall
# through to the credential-bearing upstream URL.
"GIT_CONFIG_GLOBAL": f"{plan.guest_home}/.gitconfig",
"NODE_EXTRA_CA_CERTS": AGENT_CA_PATH,
"SSL_CERT_FILE": AGENT_CA_BUNDLE,
"REQUESTS_CA_BUNDLE": AGENT_CA_BUNDLE,
@@ -280,9 +247,7 @@ def _agent_guest_env(plan: FirecrackerBottlePlan, host_ip: str) -> dict[str, str
env["GIT_GATE_URL"] = plan.agent_git_gate_url
if plan.agent_supervise_url:
env["MCP_SUPERVISE_URL"] = plan.agent_supervise_url
if plan.env_var_secret:
env[ENV_VAR_SECRET_NAME] = plan.env_var_secret
for entry in Egress().agent_env_entries(plan.egress_plan):
for entry in egress_agent_env_entries(plan.egress_plan):
key, _, value = entry.partition("=")
env[key] = value
env.update(plan.agent_provision.guest_env)
@@ -15,19 +15,11 @@ BOT_BOTTLE_FC_POOL_SIZE=8
BOT_BOTTLE_FC_IP_BASE=10.243.0.0
BOT_BOTTLE_FC_IFACE_PREFIX=bbfc
BOT_BOTTLE_FC_NFT_TABLE=bot_bottle_fc
# The orchestrator VM's own TAP — a dedicated link OUTSIDE the bbfc*
# agent pool. Unlike agent VMs (which reach only their gateway), the
# orchestrator is trusted infra that needs real NAT'd internet egress:
# to FROM-pull + apt/npm during in-VM agent-image builds (buildah). Its
# The orchestrator/gateway VM's own TAP — a dedicated link OUTSIDE the
# bbfc* agent pool. Unlike agent VMs (which reach only their gateway),
# the orchestrator is trusted infra that needs real NAT'd internet
# egress: to FROM-pull + apt/npm during in-VM agent-image builds
# (buildah) and to forward agent egress upstream (Stage B gateway). Its
# /31 is the top of the IP_BASE /16 (host x.y.255.0, guest x.y.255.1),
# clear of the pool near the bottom of the block.
BOT_BOTTLE_FC_ORCH_IFACE=bborch0
# The gateway (data-plane) VM's own TAP — the second infra link, split
# from the orchestrator now that #469 got the DB off the data plane (PRD
# 0070 "Separating the planes"). It mirrors the orchestrator link: NAT'd
# internet egress (the gateway forwards agent egress upstream) plus a
# forward path to reach the orchestrator's control plane. Its /31 is the
# next one above the orchestrator link (host x.y.255.2, guest x.y.255.3).
# Agents DNAT their gateway-port traffic here (not to the orchestrator),
# so a breached agent has no L3 route to the control plane.
BOT_BOTTLE_FC_GW_IFACE=bbgw0
+9 -35
View File
@@ -79,19 +79,12 @@ def _cfg(key: str) -> str:
IFACE_PREFIX = _cfg("BOT_BOTTLE_FC_IFACE_PREFIX")
NFT_TABLE = _cfg("BOT_BOTTLE_FC_NFT_TABLE")
# The orchestrator VM's dedicated TAP — outside the bbfc* agent pool
# and, unlike it, NAT'd to the internet (see `orch_slot`). The
# The orchestrator/gateway VM's dedicated TAP — outside the bbfc* agent
# pool and, unlike it, NAT'd to the internet (see `orch_slot`). The
# orchestrator is trusted infra: it builds agent images in-VM (buildah
# needs to FROM-pull + apt/npm).
# needs to FROM-pull + apt/npm) and forwards agent egress upstream.
ORCH_IFACE = _cfg("BOT_BOTTLE_FC_ORCH_IFACE")
# The gateway (data-plane) VM's dedicated TAP — the second infra link
# (see `gw_slot`), split from the orchestrator per PRD 0070. Like the
# orchestrator link it is NAT'd to the internet (the gateway forwards
# agent egress upstream); unlike it, agents DNAT here, never to the
# orchestrator, so a breached agent has no L3 route to the control plane.
GW_IFACE = _cfg("BOT_BOTTLE_FC_GW_IFACE")
def pool_size() -> int:
return int(_cfg("BOT_BOTTLE_FC_POOL_SIZE"))
@@ -137,12 +130,12 @@ def all_slots() -> list[Slot]:
def orch_slot() -> Slot:
"""The orchestrator VM's dedicated link — its own TAP (`ORCH_IFACE`)
on a /31 at the TOP of the IP_BASE /16 (host x.y.255.0, guest
x.y.255.1), well clear of the agent pool near the bottom of the
block. Unlike a pool `Slot`, this link is NAT'd out to the internet
by the setup (the orchestrator is trusted infra), so it is
deliberately *not* one of the isolated `bbfc*` slots.
"""The orchestrator/gateway VM's dedicated link — its own TAP
(`ORCH_IFACE`) on a /31 at the TOP of the IP_BASE /16 (host
x.y.255.0, guest x.y.255.1), well clear of the agent pool near the
bottom of the block. Unlike a pool `Slot`, this link is NAT'd out to
the internet by the setup (the orchestrator is trusted infra), so it
is deliberately *not* one of the isolated `bbfc*` slots.
`index` is -1 (sentinel: not a pool index)."""
base16 = int(ipaddress.IPv4Address(ip_base())) & 0xFFFF0000
@@ -155,25 +148,6 @@ def orch_slot() -> Slot:
)
def gw_slot() -> Slot:
"""The gateway (data-plane) VM's dedicated link — its own TAP
(`GW_IFACE`) on the /31 immediately above the orchestrator link (host
x.y.255.2, guest x.y.255.3), still clear of the agent pool at the
bottom of the block. Mirrors `orch_slot`: NAT'd to the internet (the
gateway forwards agent egress upstream), not one of the isolated
`bbfc*` slots. Agents DNAT their gateway-port traffic to this guest.
`index` is -2 (sentinel: not a pool index)."""
base16 = int(ipaddress.IPv4Address(ip_base())) & 0xFFFF0000
host = base16 + 0xFF02
return Slot(
index=-2,
iface=GW_IFACE,
host_ip=str(ipaddress.IPv4Address(host)),
guest_ip=str(ipaddress.IPv4Address(host + 1)),
)
# --- fail-closed verification ---------------------------------------
def _run_ok(argv: list[str]) -> bool:
@@ -1,159 +0,0 @@
"""The Firecracker orchestrator (control plane) as a microVM (PRD 0070).
`FirecrackerOrchestrator` is the Firecracker implementation of the backend-neutral
`Orchestrator` service, and owns the control plane's host-side logic directly:
booting the orchestrator microVM on its NAT'd link with the persistent registry
volume (/dev/vdb the sole opener of `bot-bottle.db`), seeding the host-canonical
signing key over SSH, and waiting for `/health`. The host CLI reaches it at its
guest IP, and so does the gateway (`bb_orch` cmdline), so `url()` == `gateway_url()`.
The plane-agnostic VM substrate the gateway VM also uses stays in `infra_vm`
booting a VM from a per-plane rootfs (`boot_vm`), the stable SSH keypair, the
secret-push retry, the PID lifecycle, the adoption/version helpers, and the
per-plane guest inits (`role_init`). The pair coordinator (adopt-or-boot-both
under a singleton flock) is `FirecrackerInfraService` (`infra.py`).
"""
from __future__ import annotations
import subprocess
import time
from pathlib import Path
from ...log import die, info
from ...orchestrator.lifecycle import (
DEFAULT_STARTUP_TIMEOUT_SECONDS,
Orchestrator,
)
from . import firecracker_vm, infra_vm, netpool
from .infra_vm import ORCHESTRATOR_PORT
# The orchestrator microVM's name (its run dir). Fixed per host — one
# control-plane VM.
ORCHESTRATOR_NAME = "bot-bottle-orchestrator"
# Memory ceiling (fixed at boot, demand-paged). The orchestrator keeps the build
# headroom (buildah's 2-4 GB working set during in-VM agent builds) — PRD 0070
# "Memory: fixed ceilings".
_ORCH_MEM_MIB = 4096
_HEALTH_POLL_SECONDS = 0.5
# The registry "volume": a host-side ext4 file attached to the orchestrator VM as
# a second virtio-block device (guest /dev/vdb), mounted at the control plane's DB
# dir. It outlives the ephemeral rootfs, so the bottle registry survives an
# orchestrator-VM restart — the firecracker analogue of a docker volume. A plain
# ext4 file: `sudo mount -o loop <path>` on the host (VM stopped) to inspect
# bot-bottle.db. The gateway VM has no such volume — the data plane never opens
# the DB (#469).
_REGISTRY_SIZE = "512M"
def registry_volume_path() -> Path:
return infra_vm._infra_dir() / "registry.ext4"
class FirecrackerOrchestrator(Orchestrator):
"""The control plane as a Firecracker microVM on the orchestrator link.
The orchestrator rootfs is built/downloaded by `infra_vm.ensure_built` (the
ABC's `ensure_built` no-op here); `ensure_running` boots the VM, seeds the
signing key, and blocks until `/health` answers."""
name = ORCHESTRATOR_NAME
def __init__(self, vm: infra_vm.InfraVm | None = None) -> None:
# The live VM handle when this process booted it; None when adapting an
# already-running orchestrator (health/URL go through the fixed link).
self._vm = vm
def url(self) -> str:
"""The orchestrator VM's guest IP URL — where the host CLI reaches the
control plane."""
return f"http://{netpool.orch_slot().guest_ip}:{ORCHESTRATOR_PORT}"
def gateway_url(self) -> str:
"""Same guest IP the host uses — the gateway resolves the orchestrator at
`bb_orch=<guest_ip>` off its cmdline."""
return self.url()
def ssh_target(self) -> tuple[Path, str]:
"""(private key, guest IP) for SSHing into the orchestrator VM. In-VM
agent-image builds (buildah) run here the build host drives them over
SSH with the stable infra key."""
return infra_vm._infra_dir() / "id_ed25519", netpool.orch_slot().guest_ip
def is_running(self) -> bool:
return infra_vm._pidfile_alive(infra_vm._orch_dir())
def stop(self) -> None:
"""Stop only the orchestrator VM (idempotent). A dead orchestrator fails
`infra_vm.adoptable`'s health check, so no version marker to clear."""
infra_vm._kill_pidfile(infra_vm._orch_dir())
infra_vm._pid_file(infra_vm._orch_dir()).unlink(missing_ok=True)
def ensure_running(
self, *, startup_timeout: float = DEFAULT_STARTUP_TIMEOUT_SECONDS,
) -> None:
"""Boot the orchestrator VM on its link with the persistent registry
volume, seed the signing key over SSH, and block until `/health` answers.
Idempotent a live, healthy control plane is left alone (the pair
coordinator otherwise `stop()`s it before calling, so this boots fresh)."""
if self.is_running() and self.is_healthy():
return
vm = infra_vm.boot_vm(
name=ORCHESTRATOR_NAME, slot=netpool.orch_slot(),
run_dir=infra_vm._orch_dir(), role="orchestrator", mem_mib=_ORCH_MEM_MIB,
data_drive=self._ensure_registry_volume(),
)
# Push the host-canonical signing key (the init waits for it before
# starting the control plane). It comes through the shared provisioning
# contract (#476) — the same host token file every backend uses, so a
# co-running docker/macOS control plane keeps working and the guest
# verifies tokens with the same key the CLI signs from; fail-closed, so
# the guest is never handed an empty key that would run it OPEN.
infra_vm.push_secret(
vm, self.control_plane_key(), infra_vm._GUEST_SIGNING_KEY_PATH,
"the control-plane signing key to the orchestrator VM "
"(its control plane will not start)",
)
self._vm = vm
self._wait_for_health(vm, timeout=startup_timeout)
def _wait_for_health(
self, vm: infra_vm.InfraVm, *, timeout: float,
) -> None:
"""Poll `/health` until it answers 200 or the deadline passes. Dies (with
the console tail) if the VMM exits early."""
url = f"{self.url()}/health"
deadline = time.monotonic() + timeout
while time.monotonic() < deadline:
if vm.vm is not None and not vm.vm.is_alive():
die(f"orchestrator VM exited during boot (rc={vm.vm.process.returncode}).\n"
f"{firecracker_vm._console_tail(vm.vm.console_log)}")
if self.is_healthy():
info(f"orchestrator control plane healthy at {self.url()}")
return
time.sleep(_HEALTH_POLL_SECONDS)
tail = (firecracker_vm._console_tail(vm.vm.console_log)
if vm.vm is not None else "")
die(f"orchestrator control plane at {url} did not become healthy within "
f"{timeout:.0f}s.\n{tail}")
def _ensure_registry_volume(self) -> Path:
"""Create the empty ext4 registry volume on first use; reuse it after."""
vol = registry_volume_path()
if vol.exists():
return vol
info(f"creating infra registry volume {vol} ({_REGISTRY_SIZE})")
proc = subprocess.run(
["mke2fs", "-q", "-t", "ext4", "-F", str(vol), _REGISTRY_SIZE],
capture_output=True, text=True, check=False,
)
if proc.returncode != 0:
vol.unlink(missing_ok=True)
die(f"creating registry volume failed: {proc.stderr.strip()}")
return vol
__all__ = ["FirecrackerOrchestrator", "ORCHESTRATOR_NAME", "registry_volume_path"]
+56 -150
View File
@@ -1,22 +1,19 @@
"""Build the infra rootfs artifacts and publish them as Gitea generic packages.
"""Build the infra rootfs and publish it as a Gitea generic package.
The off-host (build / CI) half of PRD 0069 Stage 2: this DOES use Docker, but
never on the launch host. It runs the same pipeline the launch host used to run
locally `docker build` the fixed images, export each per-plane rootfs, inject
the guest boot, `mke2fs` to an ext4 then gzips each and PUTs it (plus a
`.sha256`) to `/api/packages/<owner>/generic/bot-bottle-firecracker-<role>/<version>/`.
locally `docker build` the three fixed images, export to a rootfs dir, inject
the guest boot, `mke2fs` to an ext4 with the buildah build slack then gzips
the ext4 and PUTs it (plus a `.sha256`) to
`/api/packages/<owner>/generic/bot-bottle-firecracker-infra/<version>/`.
There are two artifacts, one per plane (`orchestrator`, `gateway`); the
orchestrator rootfs carries buildah, the gateway rootfs is slim. Each
`<version>` is `infra_artifact.infra_artifact_version(...)`, the content hash of
that rootfs's inputs, so a launch host at the same code checkout resolves the
exact artifacts this produced.
The `<version>` is `infra_artifact.infra_artifact_version(...)`, the content
hash of the rootfs inputs, so a launch host at the same code checkout resolves
the exact artifact this produced.
python3 -m bot_bottle.backend.firecracker.publish_infra --output DIR
python3 -m bot_bottle.backend.firecracker.publish_infra --publish-dir DIR
python3 -m bot_bottle.backend.firecracker.publish_infra [--dry-run] [--force]
A candidate bundle holds each role under its own `DIR/<role>/` subdir. Auth: a
token with `write:package` on the target owner, from
Auth: a token with `write:package` on the target owner, from
`BOT_BOTTLE_INFRA_ARTIFACT_TOKEN`.
"""
@@ -27,6 +24,7 @@ import gzip
import hashlib
import shutil
import sys
import tempfile
import urllib.error
import urllib.request
from pathlib import Path
@@ -35,27 +33,17 @@ from . import infra_artifact, infra_vm, util
_CHUNK = 1 << 20
_GZ_NAME = "rootfs.ext4.gz"
_SHA_NAME = "rootfs.ext4.gz.sha256"
# A human-readable description shipped alongside each artifact — generic packages
# A human-readable description shipped alongside the artifact — generic packages
# have no description field, so this file *is* the description on the package
# page. Uploaded on every publish so it never goes stale.
_ABOUT_NAME = "about.txt"
def _about_text(role: str) -> str:
return (
f"bot-bottle firecracker {role} rootfs (PRD 0069 Stage 2 / PRD 0070): "
f"the per-host {role} infra VM. Prebuilt off-host, gzip ext4; the launch "
f"host downloads + sha256-verifies + boots it, no host Docker. The "
f"version tag is a content hash of the rootfs inputs. Files: "
f"{_GZ_NAME} + {_SHA_NAME}.\n"
)
def _role_version(role: str) -> str:
return infra_artifact.infra_artifact_version(infra_vm.role_init(role), role)
_ABOUT_TEXT = (
"bot-bottle infra rootfs for the Firecracker backend (PRD 0069 Stage 2, "
"#348): the per-host infra VM (orchestrator control plane + gateway + "
"buildah). Prebuilt off-host, gzip ext4; the launch host downloads + "
"sha256-verifies + boots it, no host Docker. The version tag is a content "
"hash of the rootfs inputs. Files: rootfs.ext4.gz + rootfs.ext4.gz.sha256.\n"
)
def _gzip(src: Path, dest: Path) -> None:
@@ -121,141 +109,59 @@ def _delete(url: str, token: str) -> None:
raise SystemExit(f"registry unreachable: {url} ({e.reason})")
def build_role_artifact(role: str, role_dir: Path) -> str:
"""Build `role`'s rootfs ext4, gzip it, and write the checksum + version into
`role_dir`. Returns the version. Assumes the docker images are already
built (`infra_vm.build_infra_images_with_docker`). Uses host Docker."""
version = _role_version(role)
print(f"building {role} rootfs artifact {version} (docker)")
base = infra_vm.build_rootfs_dir(role)
def build_artifact(out_dir: Path) -> tuple[str, Path, Path]:
"""Build the infra rootfs ext4, gzip it, and write the checksum. Returns
`(version, gz_path, sha_path)`. Uses host Docker (off-host / CI)."""
version = infra_artifact.infra_artifact_version(infra_vm._infra_init())
print(f"building infra rootfs artifact {version} (docker)")
infra_vm.build_infra_images_with_docker()
base = infra_vm.build_infra_rootfs_dir()
ext4 = role_dir / "rootfs.ext4"
util.build_rootfs_ext4(base, ext4, slack_mib=infra_vm._ROOTFS_SLACK_MIB[role])
gz = role_dir / _GZ_NAME
print(f"compressing {role} rootfs")
ext4 = out_dir / "rootfs.ext4"
util.build_rootfs_ext4(base, ext4, slack_mib=8192)
gz = out_dir / "rootfs.ext4.gz"
print("compressing rootfs")
_gzip(ext4, gz)
ext4.unlink(missing_ok=True)
sha = role_dir / _SHA_NAME
sha = out_dir / "rootfs.ext4.gz.sha256"
digest = _sha256(gz)
sha.write_text(f"{digest} {_GZ_NAME}\n")
(role_dir / "version.txt").write_text(version + "\n", encoding="utf-8")
print(f" {role}/{gz.name}: {gz.stat().st_size / 1e6:.0f} MB sha256={digest}")
return version
def _try_download_published(role: str, role_dir: Path) -> str | None:
"""If `role`'s artifact for this version is already in the registry, download
the gz + sha into `role_dir` and return the version. None when not yet
published."""
version = _role_version(role)
sha_url = infra_artifact.artifact_url(version, _SHA_NAME, role=role)
try:
with urllib.request.urlopen(infra_artifact._open(sha_url)):
pass
except urllib.error.HTTPError as e:
if e.code == 404:
return None
raise SystemExit(f"registry check failed (HTTP {e.code}): {sha_url}")
except urllib.error.URLError as e:
raise SystemExit(f"registry unreachable: {sha_url} ({e.reason})")
print(f"{role} rootfs {version} already published — downloading instead of building")
infra_artifact._download(
infra_artifact.artifact_url(version, _GZ_NAME, role=role), role_dir / _GZ_NAME)
infra_artifact._download(sha_url, role_dir / _SHA_NAME)
(role_dir / "version.txt").write_text(version + "\n", encoding="utf-8")
return version
def _publish_bundle(role: str, role_dir: Path, token: str) -> str:
version_file = role_dir / "version.txt"
# Guard the read so a missing version.txt is a clean error, not a raw
# FileNotFoundError.
if not version_file.is_file():
raise SystemExit(f"incomplete {role} artifact bundle: {role_dir}")
version = version_file.read_text(encoding="utf-8").strip()
expected = _role_version(role)
if version != expected:
raise SystemExit(
f"{role} artifact bundle version {version!r} does not match checkout {expected!r}"
)
gz = role_dir / _GZ_NAME
sha = role_dir / _SHA_NAME
if not gz.is_file() or not sha.is_file():
raise SystemExit(f"incomplete {role} artifact bundle: {role_dir}")
expected_sha = sha.read_text().split()[0].strip().lower()
if _sha256(gz) != expected_sha:
raise SystemExit(f"{role} artifact bundle checksum mismatch")
gz_url = infra_artifact.artifact_url(version, _GZ_NAME, role=role)
sha_url = infra_artifact.artifact_url(version, _SHA_NAME, role=role)
about_url = infra_artifact.artifact_url(version, _ABOUT_NAME, role=role)
# Publishing is idempotent. If this exact complete artifact is already
# present, a re-publish is a no-op. Otherwise clear any partial upload left
# by an interrupted prior attempt and upload the complete set.
try:
with urllib.request.urlopen(infra_artifact._open(sha_url)) as resp:
remote_sha = resp.read().decode("utf-8").split()[0].strip().lower()
except urllib.error.HTTPError as e:
if e.code != 404:
raise SystemExit(f"checking existing {role} artifact failed (HTTP {e.code})")
remote_sha = ""
except urllib.error.URLError as e:
raise SystemExit(f"registry unreachable: {sha_url} ({e.reason})")
if remote_sha == expected_sha:
print(f"{role} rootfs {version} already published")
return version
for url in (gz_url, sha_url, about_url):
_delete(url, token)
_put(gz_url, gz, token)
_put(sha_url, sha.read_bytes(), token)
_put(about_url, _about_text(role).encode(), token)
return version
sha.write_text(f"{digest} rootfs.ext4.gz\n")
print(f" {gz.name}: {gz.stat().st_size / 1e6:.0f} MB sha256={digest}")
return version, gz, sha
def main(argv: list[str] | None = None) -> int:
parser = argparse.ArgumentParser(
prog="publish_infra", description="Build + publish the infra rootfs artifacts.")
mode = parser.add_mutually_exclusive_group(required=True)
mode.add_argument("--output", type=Path,
help="build candidate bundles in DIR/<role>/ without publishing")
mode.add_argument("--publish-dir", type=Path,
help="publish already-built + tested candidate bundles under DIR")
parser.add_argument("--reuse-published", action="store_true",
help="with --output: download from registry if already published instead of building")
prog="publish_infra", description="Build + publish the infra rootfs artifact.")
parser.add_argument("--dry-run", action="store_true",
help="build the artifact but do not upload")
parser.add_argument("--force", action="store_true",
help="overwrite an already-published artifact of this version")
args = parser.parse_args(argv)
_, _, token = infra_artifact._config()
if args.publish_dir is not None and not token:
if not args.dry_run and not token:
raise SystemExit(
"no publish token: set BOT_BOTTLE_INFRA_ARTIFACT_TOKEN to a token "
"with write:package")
if args.output is not None:
# Build (or reuse) all roles. Images are built once, up front, only when
# something actually needs building.
pending = []
for role in infra_artifact.ROLES:
role_dir = args.output / role
role_dir.mkdir(parents=True, exist_ok=True)
if args.reuse_published and _try_download_published(role, role_dir):
print(f"reused published {role} rootfs candidate")
continue
pending.append(role)
if pending:
print("building infra images (docker)")
infra_vm.build_infra_images_with_docker()
for role in pending:
build_role_artifact(role, args.output / role)
print(f"built {role} rootfs candidate")
return 0
assert args.publish_dir is not None
for role in infra_artifact.ROLES:
version = _publish_bundle(role, args.publish_dir / role, token)
print(f"published {role} rootfs {version}")
with tempfile.TemporaryDirectory(prefix="bb-publish-infra.") as tmp:
version, gz, sha = build_artifact(Path(tmp))
gz_url = infra_artifact.artifact_url(version, gz.name)
sha_url = infra_artifact.artifact_url(version, sha.name)
about_url = infra_artifact.artifact_url(version, _ABOUT_NAME)
if args.dry_run:
print(f"dry-run: would upload -> {gz_url}")
return 0
if args.force:
_delete(gz_url, token)
_delete(sha_url, token)
_delete(about_url, token)
_put(gz_url, gz, token) # streamed from disk (hundreds of MB)
_put(sha_url, sha.read_bytes(), token) # tiny, in-memory is fine
_put(about_url, _ABOUT_TEXT.encode(), token) # package description
print(f"published infra rootfs {version}")
return 0
@@ -9,7 +9,7 @@ from ...egress import EgressPlan
from ...env import ResolvedEnv
from ...git_gate import GitGatePlan
from ...manifest import Manifest
from ...supervisor.plan import SupervisePlan
from ...supervise import SupervisePlan
from .. import BottleSpec
from . import util
from .bottle_plan import FirecrackerBottlePlan
+6 -86
View File
@@ -13,7 +13,6 @@ generic `./cli.py backend {setup,status}` command dispatches to.
from __future__ import annotations
import fcntl
import os
import shutil
import subprocess
@@ -23,9 +22,6 @@ from pathlib import Path
from . import netpool
from . import util
# KVM_GET_API_VERSION = _IO(KVMIO=0xAE, 0x00): cheapest proof of KVM access.
_KVM_GET_API_VERSION = 0xAE00
_FC_RELEASES = "https://github.com/firecracker-microvm/firecracker/releases"
_UNIT_PATH = Path("/etc/systemd/system") / netpool.SYSTEMD_UNIT
@@ -223,90 +219,14 @@ def teardown() -> int:
return 0
def _firecracker_binary_ok() -> bool:
"""True iff the firecracker binary is on PATH and `--version` exits 0."""
if shutil.which("firecracker") is None:
return False
try:
return subprocess.run(
["firecracker", "--version"],
capture_output=True, check=False, timeout=5,
).returncode == 0
except (OSError, subprocess.TimeoutExpired):
return False
def _kvm_accessible() -> bool:
"""True iff /dev/kvm can be opened read-write and responds to KVM_GET_API_VERSION.
VM creation requires write access; opening read-only may satisfy the
ioctl but fails at boot time, so O_RDWR is the permission check."""
if not os.path.exists(util._KVM_DEVICE):
return False
try:
fd = os.open(util._KVM_DEVICE, os.O_RDWR | os.O_CLOEXEC)
try:
fcntl.ioctl(fd, _KVM_GET_API_VERSION)
finally:
os.close(fd)
return True
except OSError:
return False
def status() -> int:
# Readiness == what the launch preflight hard-requires: the binary
# executable, /dev/kvm accessible, the TAP pool present, and no range
# overlap. Listing the nft table usually needs root, so — like the
# preflight — an unconfirmable table is reported but NOT treated as
# not-ready; the post-boot isolation probe is the authoritative check.
# This keeps an unprivileged `backend status` usable as a launch gate.
# Readiness == what the launch preflight hard-requires: the TAP pool
# present (unprivileged, authoritative) and no range overlap. Listing
# the nft table usually needs root, so — like the preflight — an
# unconfirmable table is reported but NOT treated as not-ready; the
# post-boot isolation probe is the authoritative check. This keeps an
# unprivileged `backend status` usable as a launch gate.
ok = True
if _firecracker_binary_ok():
sys.stderr.write(f"firecracker binary: ok ({shutil.which('firecracker')})\n")
else:
fc_path = shutil.which("firecracker")
if fc_path is None:
sys.stderr.write("firecracker binary: NOT found on PATH\n")
else:
sys.stderr.write(
f"firecracker binary: found ({fc_path}) but `--version` failed\n"
)
ok = False
if _kvm_accessible():
sys.stderr.write(f"KVM: {util._KVM_DEVICE} accessible\n")
else:
if not os.path.exists(util._KVM_DEVICE):
sys.stderr.write(f"KVM: {util._KVM_DEVICE} not present\n")
else:
sys.stderr.write(
f"KVM: {util._KVM_DEVICE} not accessible (open/ioctl failed)\n"
)
ok = False
kernel = util.kernel_path()
if kernel.is_file():
sys.stderr.write(f"guest kernel: {kernel}\n")
else:
sys.stderr.write(
f"guest kernel: NOT found at {kernel} "
f"(set BOT_BOTTLE_FC_KERNEL or cache a vmlinux there)\n"
)
ok = False
dropbear = util.dropbear_path()
if dropbear.is_file():
sys.stderr.write(f"dropbear: {dropbear}\n")
else:
sys.stderr.write(
f"dropbear: NOT found at {dropbear} "
f"(set BOT_BOTTLE_FC_DROPBEAR or cache a static binary)\n"
)
ok = False
mke2fs = shutil.which("mke2fs")
if mke2fs is not None:
sys.stderr.write(f"mke2fs: {mke2fs}\n")
else:
sys.stderr.write("mke2fs: NOT found on PATH (install e2fsprogs)\n")
ok = False
missing = netpool.missing_taps()
total = netpool.pool_size()
if missing:
+3 -17
View File
@@ -88,7 +88,7 @@ def require_firecracker() -> None:
booting a VM without it."""
if not is_linux():
die("firecracker backend is only supported on Linux (KVM). "
"On macOS use the macos-container backend.")
"On macOS use --backend=macos-container.")
if shutil.which("firecracker") is None:
info("Firecracker is required but was not found on PATH.")
info("Install: https://github.com/firecracker-microvm/firecracker/releases")
@@ -258,8 +258,8 @@ def build_committed_rootfs_dir(tar_path: Path) -> Path:
def inject_guest_boot(rootfs: Path, init_script: str | None = None) -> None:
"""Drop the static dropbear and the PID-1 init into the rootfs.
`init_script` defaults to the SSH-only agent init; each infra VM
passes its own per-plane init (orchestrator or gateway).
`init_script` defaults to the SSH-only agent init; the infra VM
passes its own (control plane + gateway) init.
A committed snapshot is guest-controlled, so `bb-dropbear`/`bb-init`
may already exist as symlinks aimed at a host file (e.g. bb-init ->
@@ -368,17 +368,6 @@ mount -t devtmpfs dev /dev 2>/dev/null
mkdir -p /dev/pts && mount -t devpts devpts /dev/pts 2>/dev/null
mount -o remount,rw / 2>/dev/null
# /tmp must be world-writable + sticky. The rootless rootfs build can land
# it 0755/root-owned, leaving the agent (uid 1000 node) unable to create
# scratch dirs there — git worktrees, build temp, `git init /tmp/...`, etc.
mkdir -p /tmp && chmod 1777 /tmp
# Rootfs export also maps the image's original owners to the unprivileged
# host build uid. That uid is not guaranteed to be node's uid in the guest;
# restore the home-directory boundary before any SSH provisioning runs.
chown node:node /home/node 2>/dev/null || true
chmod 755 /home/node 2>/dev/null || true
# Install the per-bottle SSH pubkey from the kernel cmdline.
KEY=$(sed -n 's/.*bb_pubkey=\([^ ]*\).*/\1/p' /proc/cmdline | base64 -d 2>/dev/null)
if [ -n "$KEY" ]; then
@@ -399,9 +388,6 @@ fi
chown -R 0:0 /root 2>/dev/null || true
mkdir -p /etc/dropbear /run
# Keep restart-recovery key material memory-backed, separate from both the
# agent rootfs and the infra VM's persistent registry volume.
mount -t tmpfs -o mode=0755 tmpfs /run 2>/dev/null || true
# -R: generate host keys on demand. -E: log auth failures to stderr,
# captured in the host-side console.log for debugging.
/bb-dropbear -R -E -p 22 &
-58
View File
@@ -1,58 +0,0 @@
"""The per-host infra service contract (PRD 0070).
`InfraService` is the backend-neutral composer of the per-host **pair**: the
`Orchestrator` (control plane) + `Gateway` (data plane) services, brought up as
an idempotent per-host singleton. One concrete impl per backend
(`backend/*/infra.py`), mirroring how `Orchestrator` and `Gateway` each have a
per-backend impl.
The two backend-specific accessors (`orchestrator()` / `gateway()`) are the
source of truth for how to reach each plane; the convenience `url()` /
`is_healthy()` just delegate to the orchestrator.
"""
from __future__ import annotations
import abc
from ..gateway import Gateway
from ..orchestrator.lifecycle import DEFAULT_STARTUP_TIMEOUT_SECONDS, Orchestrator
class InfraService(abc.ABC):
"""Compose + bring up the per-host orchestrator + gateway pair.
Backend-neutral: docker/macOS run the pair as two containers, firecracker as
two microVMs. Callers read reach-info off `orchestrator()` / `gateway()`."""
@abc.abstractmethod
def orchestrator(self) -> Orchestrator:
"""The control-plane service. Cheap to reconstruct."""
@abc.abstractmethod
def gateway(self) -> Gateway:
"""The data-plane service. Cheap to reconstruct."""
@abc.abstractmethod
def ensure_running(
self, *, startup_timeout: float = DEFAULT_STARTUP_TIMEOUT_SECONDS,
) -> str:
"""Bring the orchestrator + gateway pair up (idempotent per-host
singleton a healthy, current pair is left running). Returns the host
control-plane URL. Raises `OrchestratorStartError` on control-plane
startup timeout."""
@abc.abstractmethod
def stop(self) -> None:
"""Remove both the orchestrator and the gateway. Idempotent."""
def url(self) -> str:
"""The host-facing control-plane URL the CLI reaches (the orchestrator's)."""
return self.orchestrator().url()
def is_healthy(self) -> bool:
"""True iff the control plane answers `/health`."""
return self.orchestrator().is_healthy()
__all__ = ["InfraService"]
+4 -23
View File
@@ -1,29 +1,10 @@
"""macOS Apple Container backend.
Selectable via `BOT_BOTTLE_BACKEND=macos-container`. This package owns the Apple
`container` CLI integration; launch remains gated until the gateway network
enforcement shape is implemented.
Thin by design: `MacosContainerBottleBackend` is re-exported lazily via
`__getattr__`, so importing a leaf module under this package doesn't drag the
backend (and the framework it pulls) into memory.
Selectable via `BOT_BOTTLE_BACKEND=macos-container`. This package owns
the Apple `container` CLI integration; launch remains gated until the
gateway network enforcement shape is implemented.
"""
from __future__ import annotations
from typing import TYPE_CHECKING, Any
if TYPE_CHECKING:
from .backend import MacosContainerBottleBackend
def __getattr__(name: str) -> Any:
if name == "MacosContainerBottleBackend":
from .backend import MacosContainerBottleBackend
globals()[name] = MacosContainerBottleBackend
return MacosContainerBottleBackend
raise AttributeError(f"module {__name__!r} has no attribute {name!r}")
from .backend import MacosContainerBottleBackend
__all__ = ["MacosContainerBottleBackend"]
+10 -21
View File
@@ -2,8 +2,7 @@
from __future__ import annotations
import io
from contextlib import contextmanager, redirect_stderr
from contextlib import contextmanager
from pathlib import Path
from typing import Generator, Sequence
@@ -11,9 +10,9 @@ from ...agent_provider import AgentProvisionPlan
from ...egress import EgressPlan
from ...env import ResolvedEnv
from ...git_gate import GitGatePlan
from ...supervisor.plan import SupervisePlan
from ...supervise import SupervisePlan
from ...manifest import Manifest
from .. import ActiveAgent, BottleBackend, BottleImages, BottleSpec
from .. import ActiveAgent, BottleBackend, BottleSpec
from . import cleanup as _cleanup
from . import enumerate as _enumerate
from . import launch as _launch
@@ -32,7 +31,6 @@ class MacosContainerBottleBackend(
`--backend=macos-container`."""
name = "macos-container"
supports_nested_containers = True
@classmethod
def is_available(cls) -> bool:
@@ -44,11 +42,8 @@ class MacosContainerBottleBackend(
return _setup.setup()
@classmethod
def status(cls, *, quiet: bool = False) -> int:
def status(cls) -> int:
from . import setup as _setup
if quiet:
with redirect_stderr(io.StringIO()):
return _setup.status()
return _setup.status()
@classmethod
@@ -87,26 +82,20 @@ class MacosContainerBottleBackend(
stage_dir=stage_dir,
)
def prelaunch_checks(self, plan: MacosContainerBottlePlan) -> None:
_launch.stale_checks(plan)
def _build_or_load_images(self, plan: MacosContainerBottlePlan) -> BottleImages:
return _launch.build_or_load_images(plan)
@contextmanager
def _launch_impl(
self, plan: MacosContainerBottlePlan, images: BottleImages
def launch(
self, plan: MacosContainerBottlePlan
) -> Generator[MacosContainerBottle, None, None]:
with _launch.launch(plan, images, provision=self.provision) as bottle:
with _launch.launch(plan, provision=self.provision) as bottle:
yield bottle
def ensure_orchestrator(self) -> str:
"""Bring up the per-host pair (orchestrator + gateway containers) and
"""Bring up the per-host infra container (control plane + gateway) and
return its control-plane URL the on-demand entry point operator tools
(`supervise`) call when no control plane is running yet. Mirrors
firecracker's infra bring-up."""
firecracker's infra-VM bring-up."""
from .infra import MacosInfraService
return MacosInfraService().ensure_running()
return MacosInfraService().ensure_running().control_plane_url
def prepare_cleanup(self) -> MacosContainerBottleCleanupPlan:
return _cleanup.prepare_cleanup()
+3 -8
View File
@@ -68,14 +68,9 @@ class MacosContainerBottle(Bottle):
# reaches the agent (PRD 0070): registration mints it *after* the
# container exists — its source IP is the registration key and Apple
# Container assigns that by DHCP — so it cannot be in the run-time env
# the way docker's compose spec does it.
#
# `container exec --env` does NOT override a run-time value — it
# appends, leaving duplicate entries in the agent's `environ` whose
# resolution is runtime-specific (Node last-wins, Rust first-wins). So
# nothing here may rely on superseding: the proxy vars are supplied
# *only* at exec time and are deliberately absent from the run-time
# env. See `launch._agent_env_entries`.
# the way docker's compose spec does it. `container exec --env` wins
# over the run-time value, so the token-bearing proxy URL set here
# supersedes the token-less one baked in at launch.
self._exec_env = dict(exec_env or {})
self._closed = False
@@ -20,12 +20,6 @@ class MacosContainerBottlePlan(BottlePlan):
# bottle is registered. See launch.py's stamp for why it lives here and not
# only in the exec-time proxy env.
identity_token: str = ""
# Guest-local container engine (issue #392). Gates the derived image, the
# device-mode relaxation, and the resident podman service.
nested_containers: bool = False
# Generated before `container run` so it becomes part of the container's
# configured environment and can be read back after an infra restart.
env_var_secret: str = ""
@property
def container_name(self) -> str:
@@ -15,10 +15,8 @@ caller has to start the agent in between. `ensure_gateway` runs first because
the agent's proxy env needs the gateway's address at `container run` time; the
agent's *own* address (the attribution key) only exists afterwards.
The control plane and the gateway are **separate containers** here (see
`infra`): the orchestrator on the host-only control network, the gateway on the
agent network `gateway_ip` is the gateway container's agent-network address,
distinct from the orchestrator's control-network host.
The control plane and the gateway are one **infra container** here (see
`infra`), so `gateway_ip` and the control-plane host are the same address.
The consequence for the identity token: it is minted by registration, i.e.
*after* the agent container exists, so it cannot be baked into the run-time
@@ -38,15 +36,11 @@ from dataclasses import dataclass
from ...egress import EgressPlan
from ...git_gate import GitGatePlan
from ...log import info
from ...orchestrator.client import OrchestratorClient, OrchestratorClientError
from ...orchestrator.reprovision import reprovision_bottles
from ...orchestrator.store.secret_store import ENV_VAR_SECRET_NAME
from ..provision_bottle import deprovision_bottle, provision_bottle
from . import util as container_mod
from .enumerate import CONTAINER_NAME_PREFIX, EnumerationError, enumerate_active
from ...orchestrator.client import OrchestratorClient
from ...orchestrator.registration import registration_inputs
from ..docker.gateway_provision import deprovision_git_gate, provision_git_gate
from .gateway import GATEWAY_NETWORK
from .gateway_transport import MacosGatewayTransport
from .gateway_provision import AppleGatewayTransport
from .infra import MacosInfraService, OrchestratorStartError
@@ -56,9 +50,9 @@ class ConsolidatedLaunchError(RuntimeError):
@dataclass(frozen=True)
class GatewayEndpoint:
"""What the agent `container run` needs to reach the shared gateway.
`gateway_ip` is the gateway container's agent-network address (the agent's
proxy target); `orchestrator_url` points at the separate control plane."""
"""What the agent `container run` needs to reach the shared gateway (the
infra container). `gateway_ip` is that container's host-only address, the
same host the control-plane URL points at."""
orchestrator_url: str
gateway_ip: str # the gateway's address — the agent's proxy target
@@ -76,73 +70,23 @@ class LaunchContext:
gateway_ip: str
network: str
orchestrator_url: str
env_var_secret: str = "" # encryption key injected into the agent's env
def ensure_gateway(
*, service: MacosInfraService | None = None,
) -> GatewayEndpoint:
"""Ensure the per-host pair (orchestrator + gateway containers) is up and
report how to reach the gateway. Idempotent one singleton pair, so N bottle
launches share it. Call before starting the agent container: the agent's
proxy env needs `gateway_ip` at run time."""
"""Ensure the per-host infra container (control plane + gateway) is up and
report how to reach it. Idempotent one singleton, so N bottle launches
share it. Call before starting the agent container: the agent's proxy env
needs `gateway_ip` at run time."""
service = service or MacosInfraService()
orchestrator_url = service.ensure_running()
endpoint = GatewayEndpoint(
orchestrator_url=orchestrator_url,
gateway_ip=service.gateway().address(),
infra = service.ensure_running()
return GatewayEndpoint(
orchestrator_url=infra.control_plane_url,
gateway_ip=infra.gateway_ip,
gateway_ca_pem=service.ca_cert_pem(),
network=service.network,
)
_reprovision_running_bottles(endpoint)
return endpoint
def _reprovision_running_bottles(endpoint: GatewayEndpoint) -> None:
"""Recover keys from live Apple containers and restore gateway tokens."""
try:
secrets_by_ip: dict[str, str] = {}
for agent in enumerate_active():
name = f"{CONTAINER_NAME_PREFIX}{agent.slug}"
source_ip = container_mod.inspect_container_network_ip(name, endpoint.network)
if not source_ip:
continue
secret = container_mod.read_container_env(name, ENV_VAR_SECRET_NAME)
if secret:
secrets_by_ip[source_ip] = secret
count = reprovision_bottles(
OrchestratorClient(endpoint.orchestrator_url), secrets_by_ip,
)
if count:
info(f"reprovisioned egress tokens for {count} macOS bottle(s)")
except (OrchestratorClientError, EnumerationError, OSError) as exc:
info(f"egress token reprovision skipped: {exc}")
def live_source_ips(network: str) -> list[str]:
"""Every running agent container's address on `network`.
The reconciliation input: the orchestrator lives inside the infra
container and cannot enumerate the host's containers, so the host has to
tell it which bottles are actually up. Containers that have not been
assigned an address yet contribute nothing the reap's grace window, not
this list, is what protects an in-flight launch.
Raises `EnumerationError` when the live set cannot be determined
authoritatively: either the container listing fails or any individual
inspect fails. Callers must skip reconciliation in that case to avoid
unregistering healthy bottles."""
ips: list[str] = []
for agent in enumerate_active():
name = f"{CONTAINER_NAME_PREFIX}{agent.slug}"
ip = container_mod.inspect_container_network_ip(name, network)
if ip is None:
raise EnumerationError(
f"container inspect {name!r} failed; live set is not authoritative"
)
if ip:
ips.append(ip)
return ips
def register_agent(
@@ -153,28 +97,23 @@ def register_agent(
endpoint: GatewayEndpoint,
image_ref: str = "",
tokens: dict[str, str] | None = None,
env_var_secret: str | None = None,
) -> LaunchContext:
"""Register the (already running) agent by its address and provision its
git-gate state into the gateway. `source_ip` must be read from the live
container it is the attribution key the gateway resolves policy by.
Raises on failure; the caller tears down."""
client = OrchestratorClient(endpoint.orchestrator_url)
# Self-heal before registering: a launcher that died hard (SIGKILL, closed
# terminal, host sleep) never ran its teardown callback, leaving an active
# row with no container. vmnet recycles addresses, so such a row can
# collide with this bottle's — and `by_source_ip` fail-closes on ambiguity,
# which would resolve no policy at all and deny every host. Best-effort: a
# reconciliation failure must not block an otherwise-fine launch.
try:
client.reconcile(live_source_ips(endpoint.network))
except (OrchestratorClientError, EnumerationError) as e:
info(f"registry reconciliation skipped: {e}")
reg = provision_bottle(
client, source_ip, egress_plan, git_gate_plan, MacosGatewayTransport(),
image_ref=image_ref, tokens=tokens,
env_var_secret=env_var_secret,
inputs = registration_inputs(egress_plan)
reg = client.register_bottle(
source_ip, image_ref=image_ref, policy=inputs.policy,
metadata=inputs.metadata, tokens=tokens,
)
try:
provision_git_gate(AppleGatewayTransport(), reg.bottle_id, git_gate_plan)
except Exception:
# Roll the registration back so a provisioning failure leaves no orphan.
client.teardown_bottle(reg.bottle_id)
raise
return LaunchContext(
bottle_id=reg.bottle_id,
identity_token=reg.identity_token,
@@ -182,27 +121,23 @@ def register_agent(
gateway_ip=endpoint.gateway_ip,
network=endpoint.network,
orchestrator_url=endpoint.orchestrator_url,
env_var_secret=reg.env_var_secret,
)
def deprovision_consolidated(
bottle_id: str, *, orchestrator_url: str, timeout: float | None = None,
) -> None:
def teardown_consolidated(bottle_id: str, *, orchestrator_url: str) -> None:
"""Deregister the bottle and remove its git-gate state from the gateway.
Both steps are idempotent so this is safe from a cleanup trap. Does NOT
stop the gateway it's a persistent per-host singleton."""
deprovision_bottle(bottle_id, MacosGatewayTransport(),
orchestrator_url=orchestrator_url, timeout=timeout)
OrchestratorClient(orchestrator_url).teardown_bottle(bottle_id)
deprovision_git_gate(AppleGatewayTransport(), bottle_id)
__all__ = [
"GatewayEndpoint",
"LaunchContext",
"ensure_gateway",
"live_source_ips",
"register_agent",
"deprovision_consolidated",
"teardown_consolidated",
"ConsolidatedLaunchError",
"OrchestratorStartError",
"GATEWAY_NETWORK",
@@ -6,22 +6,13 @@ import subprocess
from ...bottle_state import read_metadata
from .. import ActiveAgent
from .infra import INFRA_NAME, ORCHESTRATOR_NAME
from .infra import INFRA_NAME
# The name every agent container carries: `bot-bottle-<slug>`. Exported
# because callers that act on a running bottle (gateway-host rewrites,
# registry reconciliation) have to map an enumerated slug back to a
# container name.
CONTAINER_NAME_PREFIX = "bot-bottle-"
# The two shared per-host infra containers (orchestrator + gateway) carry the
# same `bot-bottle-` prefix as agent containers but are infrastructure, not
# bottles — one pair serves every agent, so enumerating either as an agent would
# invent a phantom bottle per host.
_INFRA_NAMES = frozenset({INFRA_NAME, ORCHESTRATOR_NAME})
class EnumerationError(RuntimeError):
"""container list failed; the resulting live set is not authoritative."""
_PREFIX = "bot-bottle-"
# The shared per-host infra container carries the same prefix as agent
# containers but is infrastructure, not a bottle — one control plane + gateway
# serves every agent, so listing it as an agent would invent one per host.
_INFRA_NAMES = frozenset({INFRA_NAME})
def enumerate_active() -> list[ActiveAgent]:
@@ -32,15 +23,12 @@ def enumerate_active() -> list[ActiveAgent]:
check=False,
)
if result.returncode != 0:
raise EnumerationError(
f"container list failed: "
f"{(result.stderr or '').strip() or '<no stderr>'}"
)
return []
out: list[ActiveAgent] = []
for name in sorted(line.strip() for line in result.stdout.splitlines()):
if not name.startswith(CONTAINER_NAME_PREFIX) or name in _INFRA_NAMES:
if not name.startswith(_PREFIX) or name in _INFRA_NAMES:
continue
slug = name[len(CONTAINER_NAME_PREFIX):]
slug = name[len(_PREFIX):]
metadata = read_metadata(slug)
out.append(ActiveAgent(
backend_name="macos-container",
+14 -180
View File
@@ -1,212 +1,46 @@
"""The macOS gateway data plane as an Apple container (PRD 0070).
"""Shared network/image constants for the macOS consolidated infra container.
`MacosGateway` is the Apple-Container implementation of the backend-neutral
`Gateway` service. It runs the gateway daemons (egress / git-http / supervise)
in a single triple-homed container and never opens `bot-bottle.db` it reaches
the supervise queue over the control-plane RPC (#469), so it holds no signing
key, only the pre-minted `gateway` token the orchestrator hands it via
`connect_to_orchestrator`.
This module also holds the pieces the infra service + launch/provision glue
share: the network names, the gateway image, and the network-creation helper.
The gateway data plane no longer runs as its own Apple container it shares a
single per-host **infra container** with the control plane (see `infra`),
because two Apple-Container guests writing one `bot-bottle.db` over virtiofs
would race incoherent `fcntl` locks. This module holds the pieces both the
infra service and the launch/provision glue need: the network names, the
gateway image, and the network-creation helper.
"""
from __future__ import annotations
import os
from pathlib import Path
from ...gateway import (
DEFAULT_CA_TIMEOUT_SECONDS,
GATEWAY_CA_CERT,
MITMPROXY_HOME,
Gateway,
GatewayError,
GatewayTransport,
)
from ...paths import (
ORCHESTRATOR_AUTH_JWT_ENV,
host_gateway_ca_dir,
)
from .. import util as backend_util
from ...orchestrator.gateway import GatewayError
from . import util as container_mod
# The shared host-only network the gateway container and every agent bottle sit
# The shared host-only network the infra container and every agent bottle sit
# on. The agent's address here is the attribution key. Distinct from the docker
# names so both backends can coexist on one host.
GATEWAY_NETWORK = "bot-bottle-mac-gateway"
# The NAT network that gives the gateway (and only it) a route out.
# The NAT network that gives the infra container (and only it) a route out.
GATEWAY_EGRESS_NETWORK = "bot-bottle-mac-egress"
# The control network the gateway reaches the orchestrator over (host-only).
# Only the orchestrator + gateway join it; agents never do, so agents have no
# route to the control plane (PRD 0070 "Separating the planes").
CONTROL_NETWORK = "bot-bottle-mac-control"
# The gateway (data plane) container. The name predates the split — it was the
# combined "infra" container — and is kept so callers importing it still resolve
# the gateway (probe / reprovision attribute against it).
GATEWAY_NAME = "bot-bottle-mac-infra"
GATEWAY_LABEL = "bot-bottle-mac-infra=1"
# The gateway subset the consolidated model runs (no per-bottle git:// daemon).
GATEWAY_DAEMONS = "egress,git-http,supervise"
GATEWAY_IMAGE = os.environ.get("BOT_BOTTLE_GATEWAY_IMAGE", "bot-bottle-gateway:latest")
_REPO_ROOT = Path(__file__).resolve().parents[3]
DEFAULT_CA_TIMEOUT_SECONDS = 30.0
def ensure_networks(
network: str = GATEWAY_NETWORK,
egress_network: str = GATEWAY_EGRESS_NETWORK,
control_network: str = CONTROL_NETWORK,
network: str = GATEWAY_NETWORK, egress_network: str = GATEWAY_EGRESS_NETWORK,
) -> None:
"""Create the shared host-only agent network, the NAT egress network, and
the host-only control network. Idempotent `create_network` tolerates
'already exists'."""
"""Create the shared host-only network + the NAT egress network. Idempotent
`create_network` tolerates 'already exists'."""
container_mod.create_network(egress_network)
container_mod.create_network(network, internal=True)
container_mod.create_network(control_network, internal=True)
class MacosGateway(Gateway):
"""The consolidated gateway as a single Apple container, triple-homed on the
NAT egress, host-only agent, and control networks.
`ensure_built` builds `Dockerfile.gateway`; the networks are ensured by the
composer. `connect_to_orchestrator` runs the container carrying the mitmproxy
CA + the pre-minted `gateway` token."""
def __init__(
self,
image_ref: str = GATEWAY_IMAGE,
*,
name: str = GATEWAY_NAME,
network: str = GATEWAY_NETWORK,
egress_network: str = GATEWAY_EGRESS_NETWORK,
control_network: str = CONTROL_NETWORK,
repo_root: Path = _REPO_ROOT,
) -> None:
self.image_ref = image_ref
self.name = name
self.network = network
self.egress_network = egress_network
self.control_network = control_network
self._repo_root = repo_root
# Set by `connect_to_orchestrator`: the URL the daemons resolve policy
# against + the pre-minted `gateway` token they present. The gateway
# never mints, so it never holds the signing key (#469).
self._orchestrator_url = ""
self._gateway_token = ""
def ensure_built(self) -> None:
"""Build the data-plane image from `Dockerfile.gateway`."""
container_mod.build_image(
self.image_ref, str(self._repo_root), dockerfile="Dockerfile.gateway")
def connect_to_orchestrator(self, orchestrator_url: str, gateway_token: str) -> None:
"""Bind the gateway to this orchestrator and (re)start it, dual-homed on
the agent + control networks, resolving policy against `orchestrator_url`
(the orchestrator's control-network address — Apple has no container
DNS) and presenting `gateway_token`."""
self._orchestrator_url = orchestrator_url
self._gateway_token = gateway_token
# Fail closed on a missing policy source or token: the resolver-only data
# plane (PRD 0070) would only crash-loop its daemons without an
# orchestrator URL, and it cannot mint the token it presents (#469).
if not self._orchestrator_url:
raise GatewayError(
"gateway requires an orchestrator URL to run "
"(resolver-only data plane; no single-tenant fallback)"
)
if not self._gateway_token:
raise GatewayError(
"gateway requires a pre-minted `gateway` token to run "
"(the orchestrator mints it; the gateway never holds the key)"
)
container_mod.force_remove_container(self.name)
argv = [
"container", "run", "--detach",
"--name", self.name,
"--label", "bot-bottle.backend=macos-container",
"--label", GATEWAY_LABEL,
# NAT egress FIRST (default route out); the host-only agent network
# is where agents reach the gateway; the control network reaches the
# orchestrator.
"--network", self.egress_network,
"--network", self.network,
"--network", self.control_network,
"--dns", container_mod.dns_server(),
# The mitmproxy CA on a host bind-mount (survives recreation +
# volume pruning — issue #450). No DB mount: the data plane never
# opens bot-bottle.db (#469).
"--mount",
container_mod.bind_mount_spec(str(host_gateway_ca_dir()), MITMPROXY_HOME),
"--env", f"BOT_BOTTLE_GATEWAY_DAEMONS={GATEWAY_DAEMONS}",
"--env", f"BOT_BOTTLE_ORCHESTRATOR_URL={self._orchestrator_url}",
# The pre-minted `gateway` JWT (never the signing key). Bare
# `--env NAME` inherits the value from run_env below.
"--env", ORCHESTRATOR_AUTH_JWT_ENV,
self.image_ref,
]
run_env = {**os.environ, ORCHESTRATOR_AUTH_JWT_ENV: self._gateway_token}
result = container_mod.run_container_argv(argv, env=run_env)
if result.returncode != 0:
raise GatewayError(
f"gateway container failed to start: "
f"{(result.stderr or '').strip() or '<no stderr>'}"
)
def is_running(self) -> bool:
return container_mod.container_is_running(self.name)
def stop(self) -> None:
"""Remove the gateway container (idempotent)."""
container_mod.force_remove_container(self.name)
def address(self) -> str:
"""The gateway's agent-network address — the proxy / git-http / supervise
target agents dial (also the source IP the gateway attributes by)."""
ip = container_mod.try_container_ipv4_on_network(self.name, self.network)
if not ip:
raise GatewayError(
f"gateway {self.name} has no address on {self.network}"
)
return ip
def ca_cert_pem(self, *, timeout: float = DEFAULT_CA_TIMEOUT_SECONDS) -> str:
"""The gateway's mitmproxy CA (PEM) agents install to trust its TLS
interception. Read from the gateway container; polls because mitmproxy
writes it a beat after start."""
def _fetch() -> str | None:
result = container_mod.run_container_argv(
["container", "exec", self.name, "cat", GATEWAY_CA_CERT])
return result.stdout if result.returncode == 0 and result.stdout.strip() else None
try:
return backend_util.poll_ca_cert(_fetch, timeout=timeout)
except TimeoutError as exc:
raise GatewayError(
f"gateway CA not available in {self.name} after {timeout:g}s"
) from exc
def provisioning_transport(self) -> GatewayTransport:
"""The exec/cp transport git-gate provisioning stages per-bottle repos +
deploy keys through (over the `container` CLI)."""
# Local import: gateway_transport imports GATEWAY_NAME from this module,
# so importing MacosGatewayTransport at module scope would cycle.
from .gateway_transport import MacosGatewayTransport
return MacosGatewayTransport(self.name)
__all__ = [
"GATEWAY_NETWORK",
"GATEWAY_EGRESS_NETWORK",
"CONTROL_NETWORK",
"GATEWAY_NAME",
"GATEWAY_LABEL",
"GATEWAY_DAEMONS",
"GATEWAY_IMAGE",
"GatewayError",
"DEFAULT_CA_TIMEOUT_SECONDS",
"ensure_networks",
"MacosGateway",
]
@@ -1,96 +0,0 @@
"""Stable gateway name for macOS agents, via each bottle's `/etc/hosts`.
The shared gateway's address is assigned by vmnet's DHCP and changes whenever
the gateway container is recreated a source-hash bump, an image upgrade, a
crash. Every agent-facing URL (egress proxy, git-http, supervise) embeds that
address, and the proxy URL reaches the agent as **process environment** at
`container exec` time. A running process's `environ` cannot be rewritten from
outside, so a moved gateway used to strand every running bottle permanently:
not degraded, unreachable, until the bottle was relaunched and its session
thrown away.
So the agent never learns the address. It is given a stable *name*
(`GATEWAY_HOSTNAME`) in every URL, resolved through its own `/etc/hosts`.
Unlike `environ`, that is a file it can be rewritten inside a container that
is already running, so a gateway that comes back at a new address is picked up
by live bottles instead of orphaning them.
Apple Container 1.0 offers no container-name DNS on a user network (the only
nameserver an agent sees is vmnet's, which does not know container names) and
`container run` has no `--add-host`, so the entry is written by exec after the
container starts.
Writing it needs root, and the agent runs as `node`: the agent therefore
cannot repoint its own gateway name, while the host (which drives `container
exec --user root`) can. That asymmetry is deliberate keep it.
"""
from __future__ import annotations
from ...log import warn
from . import util as container_mod
from .enumerate import CONTAINER_NAME_PREFIX, enumerate_active
# The name every agent-facing gateway URL uses. Must not collide with a real
# DNS name the agent might resolve; it is bottle-local by construction.
GATEWAY_HOSTNAME = "bot-bottle-gateway"
# Marker so the rewrite is idempotent and only ever touches our own line —
# the rest of /etc/hosts (localhost, the container's own name) is preserved.
_MARKER = "# bot-bottle gateway"
def _rewrite_script(gateway_ip: str) -> str:
"""A shell one-liner that replaces our managed line in `/etc/hosts`.
Rewrites in place via a temp file + `cat` rather than `mv`, so the file
keeps its original inode, ownership, and mode a bind-mounted or
pre-created `/etc/hosts` must not be replaced by a root-owned 0644 copy
that the runtime then refuses to update.
"""
return (
"set -e; "
f"grep -v '{_MARKER}' /etc/hosts > /tmp/.bb-hosts || true; "
f"printf '%s %s %s\\n' '{gateway_ip}' '{GATEWAY_HOSTNAME}' "
f"'{_MARKER}' >> /tmp/.bb-hosts; "
"cat /tmp/.bb-hosts > /etc/hosts; "
"rm -f /tmp/.bb-hosts"
)
def set_gateway_host(container_name: str, gateway_ip: str) -> None:
"""Point `GATEWAY_HOSTNAME` at `gateway_ip` inside one running container.
Must run before the agent is exec'd: the agent's proxy URL names the
gateway, so the entry has to exist for its first connection. Idempotent
re-running with the same address is a no-op in effect.
"""
container_mod.exec_container_as_root(
container_name, ["sh", "-c", _rewrite_script(gateway_ip)],
)
def refresh_gateway_host(gateway_ip: str) -> list[str]:
"""Re-point every running bottle at the current gateway address.
Called once the shared gateway is known to be up, so a bottle stranded by
an earlier gateway restart re-attaches instead of needing a relaunch.
Returns the containers updated.
Best-effort per bottle: one container that refuses the write (already
exiting, say) must not stop the others from being repaired, and must not
fail the launch that triggered the sweep.
"""
updated: list[str] = []
for agent in enumerate_active():
name = f"{CONTAINER_NAME_PREFIX}{agent.slug}"
try:
set_gateway_host(name, gateway_ip)
updated.append(name)
# One bad bottle must not stop the sweep, so this is deliberately broad.
except Exception as e: # noqa: BLE001 # pylint: disable=broad-exception-caught
warn(f"could not re-point {name} at the gateway: {e}")
return updated
__all__ = ["GATEWAY_HOSTNAME", "set_gateway_host", "refresh_gateway_host"]
@@ -1,22 +1,23 @@
"""The `GatewayTransport` for the Apple gateway container (PRD 0070).
"""`GatewayTransport` for the Apple infra container (PRD 0070).
How the launcher stages files + runs commands in the running gateway container:
the `container` CLI's exec/cp equivalents. The backend-neutral provisioning
logic that drives it lives in `backend.provision_gateway`; Docker uses
`docker exec`/`docker cp` and Firecracker uses SSH.
The provisioning *logic* (per-bottle creds dirs, namespaced repo init) is
backend-neutral and lives in `backend.docker.gateway_provision`; this is only
the transport how files and commands reach the running gateway. Docker uses
`docker exec`/`docker cp` and Firecracker uses SSH; Apple uses the `container`
CLI's equivalents against the infra container that hosts the gateway daemons.
"""
from __future__ import annotations
from ...gateway import GatewayProvisionError
from ..docker.gateway_provision import GatewayProvisionError
from . import util as container_mod
from .gateway import GATEWAY_NAME
from .infra import INFRA_NAME
class MacosGatewayTransport:
"""`GatewayTransport` for the gateway daemons in the Apple gateway container."""
class AppleGatewayTransport:
"""`GatewayTransport` for the gateway daemons in the Apple infra container."""
def __init__(self, gateway: str = GATEWAY_NAME) -> None:
def __init__(self, gateway: str = INFRA_NAME) -> None:
self.gateway = gateway
def exec(self, argv: list[str]) -> None:
@@ -40,4 +41,4 @@ class MacosGatewayTransport:
)
__all__ = ["MacosGatewayTransport"]
__all__ = ["AppleGatewayTransport", "GatewayProvisionError"]
+244 -100
View File
@@ -1,65 +1,123 @@
"""The per-host control plane + gateway for the macOS backend (PRD 0070).
"""The per-host infra container for the macOS backend (PRD 0070).
Two Apple containers the orchestrator (control plane) and the gateway (data
plane) split now that #469 got the DB off the data plane. The single-container
model existed only because two Apple-Container guests writing one `bot-bottle.db`
over virtiofs would race incoherent `fcntl` locks; with the data plane no longer
opening the DB at all, only the orchestrator does, so the split is safe.
A single persistent Apple container that runs BOTH the orchestrator control
plane and the gateway data plane the macOS analogue of the Firecracker infra
VM (`backend/firecracker/infra_vm.py`), not the docker backend's two separate
containers.
* `bot-bottle-mac-orchestrator` the lean control plane (`MacosOrchestrator`).
Joins the host-only **control network** (`bot-bottle-mac-control`) only. Sole
opener of the container-only DB volume; holds the signing key. The host CLI
reaches it at its control-network address; the gateway reaches it there too.
* `bot-bottle-mac-infra` the gateway data plane (`MacosGateway`). Triple-homed:
the NAT egress network (route out), the host-only agent network (agents + CLI
reach the gateway), and the control network (reach the orchestrator by IP
Apple has no container DNS). Holds the mitmproxy CA + the `gateway` JWT.
Why one container, not two: Apple Containers are lightweight VMs, each with its
own kernel. The docker backend runs the orchestrator and gateway as two
containers safely because they share the host kernel, so their concurrent
writes to the one `bot-bottle.db` (the orchestrator's registry + the gateway
supervise daemon's queue) are serialized by coherent `fcntl` locks. Across two
*guest* kernels sharing a virtiofs-mounted DB those locks are not coherent, and
concurrent writers can corrupt the file. Firecracker solved this by putting
both services in one guest with the DB on a device only that guest mounts; this
does the same with Apple primitives.
`MacosInfraService` composes the two services and brings them up as an idempotent
per-host pair. Agents sit on the agent network only, never the control network,
so they have no route to the control plane (the L3 block, not just the JWT).
Two consequences fall out of the single container, both simplifications:
- **No DNS dance.** The control plane and the gateway daemons reach each other
over `127.0.0.1`, so nothing depends on Apple's (absent) container DNS and
there is no orchestrator-before-gateway ordering to get right.
- **The DB is never host-shared.** It lives on a container-only volume, so no
host process opens the live file. The host CLI reaches registry + supervise
state through the control-plane HTTP surface (`cli/supervise.py` already uses
`OrchestratorClient`), exactly as it does for firecracker.
The control-plane source is bind-mounted (like the docker orchestrator), so a
code change takes effect on the next launch without an image rebuild; the
gateway daemons are baked in the gateway image and rebuild through its own
digest check.
"""
from __future__ import annotations
import os
import time
import urllib.error
import urllib.request
from dataclasses import dataclass
from pathlib import Path
from ... import log
from ...orchestrator.gateway import GATEWAY_CA_CERT
from ...orchestrator.lifecycle import (
DEFAULT_PORT,
DEFAULT_STARTUP_TIMEOUT_SECONDS,
OrchestratorStartError,
source_hash,
)
from ...paths import (
CONTROL_PLANE_TOKEN_ENV,
HOST_DB_FILENAME,
host_control_plane_token,
)
from ..infra_service import InfraService
from . import util as container_mod
from .gateway import (
CONTROL_NETWORK,
DEFAULT_CA_TIMEOUT_SECONDS,
GATEWAY_EGRESS_NETWORK,
GATEWAY_IMAGE,
GATEWAY_NAME,
GATEWAY_NETWORK,
MacosGateway,
GatewayError,
ensure_networks,
)
from .orchestrator import (
ORCHESTRATOR_DB_VOLUME,
ORCHESTRATOR_IMAGE,
ORCHESTRATOR_NAME,
MacosOrchestrator,
probe_orchestrator_url,
)
# `INFRA_NAME` is kept — now aliasing the gateway container — for callers that
# still import it (probe / reprovision attribute against the gateway).
INFRA_NAME = GATEWAY_NAME
# The one per-host infra container: control plane + gateway data plane.
INFRA_NAME = "bot-bottle-mac-infra"
INFRA_LABEL = "bot-bottle-mac-infra=1"
# Container-only volume holding bot-bottle.db. No host bind-mount, so the DB is
# written by exactly one kernel (this container's). Survives recreation.
INFRA_DB_VOLUME = "bot-bottle-mac-db"
# BOT_BOTTLE_ROOT inside the container; host_db_path() resolves the DB to
# <root>/db/<filename> and the supervise daemon writes the same file.
_DB_ROOT_IN_CONTAINER = "/var/lib/bot-bottle"
_DB_PATH_IN_CONTAINER = f"{_DB_ROOT_IN_CONTAINER}/db/{HOST_DB_FILENAME}"
_SRC_IN_CONTAINER = "/bot-bottle-src"
_REPO_ROOT = Path(__file__).resolve().parents[3]
_HEALTH_POLL_SECONDS = 0.25
_HEALTH_REQUEST_TIMEOUT_SECONDS = 1.0
_CA_POLL_SECONDS = 0.5
class MacosInfraService(InfraService):
"""Composes the per-host orchestrator + gateway containers. Callers use
`ensure_running()` (returns the control-plane URL), the `orchestrator()` /
`gateway()` accessors, and `ca_cert_pem()`."""
# The gateway subset the consolidated model runs (no per-bottle git:// daemon).
_GATEWAY_DAEMONS = "egress,git-http,supervise"
def _init_script(port: int) -> str:
"""PID-1 init: start the control plane and the gateway daemons, both in
this container, reaching each other over loopback. Backgrounded so `wait`
reaps as PID 1. No `set -e` a transient daemon failure must not kill the
whole container (gateway_init applies the same 'stay up' policy)."""
return (
"export PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin\n"
f"mkdir -p $(dirname {_DB_PATH_IN_CONTAINER})\n"
# Control plane, from the bind-mounted source (stdlib-only package).
f"( cd {_SRC_IN_CONTAINER} && BOT_BOTTLE_ROOT={_DB_ROOT_IN_CONTAINER} "
f"python3 -m bot_bottle.orchestrator --host 0.0.0.0 --port {port} "
"--broker stub ) &\n"
# Gateway data plane, multi-tenant against the local control plane.
f"( cd /app && BOT_BOTTLE_GATEWAY_DAEMONS={_GATEWAY_DAEMONS} "
f"BOT_BOTTLE_ORCHESTRATOR_URL=http://127.0.0.1:{port} "
f"SUPERVISE_DB_PATH={_DB_PATH_IN_CONTAINER} python3 /app/gateway_init.py ) &\n"
"while : ; do wait ; done\n"
)
@dataclass(frozen=True)
class InfraEndpoint:
"""How to reach the running infra container. The control plane and the
gateway are the same container, so one address serves both."""
control_plane_url: str # http://<infra ip>:8099 — host CLI + registration
gateway_ip: str # same container; agents' proxy / git-http / MCP target
class MacosInfraService:
"""Manages the single per-host infra container. Callers use
`ensure_running()` (returns the endpoint) and `ca_cert_pem()`."""
def __init__(
self,
@@ -67,95 +125,181 @@ class MacosInfraService(InfraService):
port: int = DEFAULT_PORT,
network: str = GATEWAY_NETWORK,
egress_network: str = GATEWAY_EGRESS_NETWORK,
control_network: str = CONTROL_NETWORK,
gateway_image: str = GATEWAY_IMAGE,
orchestrator_image: str = ORCHESTRATOR_IMAGE,
image: str = GATEWAY_IMAGE,
repo_root: Path = _REPO_ROOT,
orchestrator_name: str = ORCHESTRATOR_NAME,
gateway_name: str = INFRA_NAME,
db_volume: str = ORCHESTRATOR_DB_VOLUME,
name: str = INFRA_NAME,
db_volume: str = INFRA_DB_VOLUME,
) -> None:
self.port = port
self.network = network
self.egress_network = egress_network
self.control_network = control_network
self.gateway_image = gateway_image
self.orchestrator_image = orchestrator_image
self.image = image
self._repo_root = repo_root
self._orchestrator_name = orchestrator_name
self._gateway_name = gateway_name
self._name = name
self._db_volume = db_volume
def orchestrator(self) -> MacosOrchestrator:
"""The control-plane service on the host-only control network. Cheap to
reconstruct the launch flow reads its `url()` / `gateway_url()` /
`mint_gateway_token()` off it."""
return MacosOrchestrator(
self.orchestrator_image,
name=self._orchestrator_name,
port=self.port,
control_network=self.control_network,
repo_root=self._repo_root,
db_volume=self._db_volume,
)
def _resolve_url(self) -> str:
"""The control-plane URL, or "" while the container has no address."""
ip = container_mod.try_container_ipv4_on_network(self._name, self.network)
return f"http://{ip}:{self.port}" if ip else ""
def gateway(self) -> MacosGateway:
"""The data-plane gateway service, triple-homed on the egress + agent +
control networks. Cheap to reconstruct the launch flow reads its
`address()` / CA / provisioning transport off it, and `ensure_running`
connects it to the control plane."""
return MacosGateway(
self.gateway_image,
name=self._gateway_name,
network=self.network,
egress_network=self.egress_network,
control_network=self.control_network,
repo_root=self._repo_root,
def is_healthy(
self, url: str, *, timeout: float = _HEALTH_REQUEST_TIMEOUT_SECONDS,
) -> bool:
if not url:
return False
try:
with urllib.request.urlopen(f"{url}/health", timeout=timeout) as resp:
return resp.status == 200
except (urllib.error.URLError, TimeoutError, OSError):
return False
def _source_current(self, current_hash: str) -> bool:
"""True iff the running infra container was created from the current
bind-mounted control-plane source. The control-plane process loads that
code at startup and won't reload it, so a stale container keeps serving
OLD code."""
if not container_mod.container_is_running(self._name):
return False
env = container_mod.container_env(self._name)
if not env:
return True # can't compare → don't churn a working container
return env.get("BOT_BOTTLE_SOURCE_HASH") == current_hash
def _running_healthy_endpoint(self, current_hash: str) -> InfraEndpoint | None:
"""The endpoint if the running container is BOTH source-current and
answering /health, else None ( recreate). Health, not just the source
label, is what lets a wedged-but-current container self-heal instead of
being polled to death forever."""
if not self._source_current(current_hash):
return None
url = self._resolve_url()
if url and self.is_healthy(url):
return InfraEndpoint(control_plane_url=url, gateway_ip=_ip_of(url))
return None
def ensure_built(self) -> None:
"""Ensure the gateway data-plane image exists. The control-plane source
is bind-mounted, not baked, so only the gateway image needs building."""
container_mod.build_image(
self.image, str(self._repo_root), dockerfile="Dockerfile.gateway",
)
def ensure_running(
self, *, startup_timeout: float = DEFAULT_STARTUP_TIMEOUT_SECONDS,
) -> str:
"""Ensure the orchestrator + gateway containers are up; return the host
control-plane URL. Idempotent per-host singleton a healthy orchestrator
on current source is left untouched. Raises `OrchestratorStartError` on
control-plane startup timeout."""
# The networks (host-only agent + NAT egress + host-only control) are a
# shared concern — create them before either plane comes up.
ensure_networks(self.network, self.egress_network, self.control_network)
) -> InfraEndpoint:
"""Ensure the single infra container is up; return how to reach it.
Idempotent per-host singleton a healthy container on current source
is left untouched, so N launches share the one control plane + gateway.
Raises `OrchestratorStartError` on startup timeout."""
current_hash = source_hash(self._repo_root)
endpoint = self._running_healthy_endpoint(current_hash)
if endpoint is not None:
return endpoint
self.ensure_built()
log.info("starting infra container", context={"name": self._name})
self._run_container(current_hash)
return self._wait_healthy(startup_timeout)
orchestrator = self.orchestrator()
gateway = self.gateway()
orchestrator.ensure_built()
gateway.ensure_built()
def _run_container(self, current_hash: str) -> None:
ensure_networks(self.network, self.egress_network)
container_mod.force_remove_container(self._name)
argv = [
"container", "run", "--detach",
"--name", self._name,
"--label", "bot-bottle.backend=macos-container",
"--label", INFRA_LABEL,
# NAT network FIRST so the gateway's egress has a default route;
# the host-only network is where agents (and the host CLI) reach it.
"--network", self.egress_network,
"--network", self.network,
"--dns", container_mod.dns_server(),
# Container-only DB volume: one kernel writes bot-bottle.db, never
# shared with the host or another guest.
"--volume", f"{self._db_volume}:{_DB_ROOT_IN_CONTAINER}",
# Bind-mount the control-plane source (read-only); a code change
# takes effect on relaunch with no image rebuild.
"--mount",
container_mod.bind_mount_spec(
str(self._repo_root), _SRC_IN_CONTAINER, readonly=True),
# Baked onto the container so `_source_current` can detect a real
# control-plane code change and recreate.
"--env", f"BOT_BOTTLE_SOURCE_HASH={current_hash}",
# The control-plane secret, for BOTH the control plane (to require
# it) and the gateway's PolicyResolver (to present it) — they share
# this one container. Bare `--env NAME` inherits the value from the
# run process below, so the secret never lands on argv or in
# `container inspect`'s command line. The agent runs in a SEPARATE
# container that is never given this var, which is the whole point.
"--env", CONTROL_PLANE_TOKEN_ENV,
"--entrypoint", "sh",
self.image,
"-c", _init_script(self.port),
]
run_env = {**os.environ, CONTROL_PLANE_TOKEN_ENV: host_control_plane_token()}
result = container_mod.run_container_argv(argv, env=run_env)
if result.returncode != 0:
raise OrchestratorStartError(
f"infra container failed to start: "
f"{(result.stderr or '').strip() or '<no stderr>'}"
)
orchestrator.ensure_running(startup_timeout=startup_timeout)
# (Re)ensure the gateway once the control plane it resolves against is
# healthy — it needs the orchestrator's control-network address. The
# orchestrator (which holds the signing key) mints the role-scoped
# `gateway` JWT and hands it to the gateway, which never sees the key
# (#469).
url = orchestrator.url()
gateway.connect_to_orchestrator(url, orchestrator.mint_gateway_token())
return url
def _wait_healthy(self, startup_timeout: float) -> InfraEndpoint:
deadline = time.monotonic() + startup_timeout
while True:
url = self._resolve_url()
if url and self.is_healthy(url):
log.info("infra container healthy", context={"url": url})
return InfraEndpoint(control_plane_url=url, gateway_ip=_ip_of(url))
if time.monotonic() >= deadline:
raise OrchestratorStartError(
f"infra container did not become healthy within "
f"{startup_timeout:g}s"
)
time.sleep(_HEALTH_POLL_SECONDS)
def ca_cert_pem(self, *, timeout: float = DEFAULT_CA_TIMEOUT_SECONDS) -> str:
"""The gateway's mitmproxy CA (PEM) agents install to trust its TLS
interception delegated to the gateway service (reads it out of the
gateway container, polling until mitmproxy writes it)."""
return self.gateway().ca_cert_pem(timeout=timeout)
interception. Read out of the container (the CA lives on a
container-internal path, not a host mount); polls because mitmproxy
writes it a beat after start."""
deadline = time.monotonic() + timeout
while True:
result = container_mod.run_container_argv(
["container", "exec", self._name, "cat", GATEWAY_CA_CERT])
if result.returncode == 0 and result.stdout.strip():
return result.stdout
if time.monotonic() >= deadline:
raise GatewayError(
f"gateway CA not available in {self._name} after {timeout:g}s: "
f"{(result.stderr or '').strip() or 'empty'}"
)
time.sleep(_CA_POLL_SECONDS)
def stop(self) -> None:
"""Remove both containers (idempotent). The DB volume persists."""
container_mod.force_remove_container(self._gateway_name)
container_mod.force_remove_container(self._orchestrator_name)
"""Remove the infra container (idempotent). The DB volume persists."""
container_mod.force_remove_container(self._name)
def _ip_of(url: str) -> str:
"""The host from an http://host:port URL."""
return url.split("://", 1)[-1].rsplit(":", 1)[0]
def probe_control_plane_url(port: int = DEFAULT_PORT) -> str:
"""The running infra container's control-plane URL, or "" if it isn't up.
Used by host-side control-plane discovery (`discover_orchestrator_url`);
safe to call on any host returns "" when the container or the `container`
CLI isn't present."""
ip = container_mod.try_container_ipv4_on_network(INFRA_NAME, GATEWAY_NETWORK)
return f"http://{ip}:{port}" if ip else ""
__all__ = [
"MacosInfraService",
"InfraEndpoint",
"OrchestratorStartError",
"ORCHESTRATOR_NAME",
"GatewayError",
"INFRA_NAME",
"probe_orchestrator_url",
"INFRA_DB_VOLUME",
]
+57 -162
View File
@@ -44,100 +44,45 @@ from ...bottle_state import (
git_gate_state_dir,
read_committed_image,
)
from ...egress import Egress
from ...git_gate import GitGate
from ...gateway.git_gate.http_backend import DEFAULT_PORT as _GIT_HTTP_PORT
from ...image_cache import check_stale
from ...egress import (
egress_agent_env_entries,
egress_resolve_token_values,
)
from ...git_gate import (
provision_git_gate_dynamic_keys,
revoke_git_gate_provisioned_keys,
)
from ...git_http_backend import DEFAULT_PORT as _GIT_HTTP_PORT
from ...log import die, info, warn
from .. import BottleImages
from ...supervisor.types import SUPERVISE_PORT
from ...supervise import SUPERVISE_PORT
from ..docker.egress import EGRESS_PORT
from ..util import AGENT_CA_BUNDLE, AGENT_CA_PATH
from . import util as container_mod
from .bottle import MacosContainerBottle
from .gateway_hosts import (
GATEWAY_HOSTNAME,
refresh_gateway_host,
set_gateway_host,
)
from . import nested_containers as nested_containers_mod
from .bottle_plan import MacosContainerBottlePlan
from ...orchestrator.store.config_store import resolve_teardown_timeout
from ...orchestrator.store.secret_store import ENV_VAR_SECRET_NAME, new_env_var_secret
from .consolidated_launch import (
GatewayEndpoint,
ensure_gateway,
register_agent,
deprovision_consolidated,
teardown_consolidated,
)
_REPO_DIR = str(Path(__file__).resolve().parent.parent.parent.parent)
_AGENT_SLEEP_SECONDS = "2147483647"
def build_or_load_images(plan: MacosContainerBottlePlan) -> BottleImages:
"""Resolve the agent image ref for this plan. The gateway's own image is
built by `ensure_gateway` it belongs to the shared singleton."""
return BottleImages(agent=_layer_nested_containers(plan, _agent_image(plan)))
def _agent_image(plan: MacosContainerBottlePlan) -> str:
committed = read_committed_image(plan.slug)
if committed and container_mod.image_exists(committed):
info(f"using committed image {committed!r}")
return committed
if plan.spec.image_policy == "cached":
if not container_mod.image_exists(plan.image):
die(
f"cached agent image {plan.image!r} not found; "
"run without --cached-images to build it"
)
info(f"using cached agent image {plan.image!r}")
return plan.image
container_mod.build_image(plan.image, _REPO_DIR, dockerfile=plan.dockerfile_path)
return plan.image
def _layer_nested_containers(
plan: MacosContainerBottlePlan, agent_image: str,
) -> str:
"""Add the guest-local container tooling on top of the agent image.
A separate derived tag, not the provider Dockerfile, so bottles that never
ask for nested containers carry none of its weight.
"""
if not plan.nested_containers:
return agent_image
derived = f"{agent_image}{nested_containers_mod.IMAGE_SUFFIX}"
if plan.spec.image_policy == "cached":
if not container_mod.image_exists(derived):
die(
f"cached nested-container image {derived!r} not found; "
"run without --cached-images to build it"
)
info(f"using cached nested-container image {derived!r}")
return derived
return nested_containers_mod.build_image(agent_image, container_mod.build_image)
@contextmanager
def launch(
plan: MacosContainerBottlePlan,
images: BottleImages,
*,
provision: Callable[[MacosContainerBottlePlan, "MacosContainerBottle"], str | None],
) -> Generator[MacosContainerBottle, None, None]:
"""Run, register, provision, and yield an Apple Container bottle on the
shared per-host gateway."""
"""Build, run, register, provision, and yield an Apple Container bottle on
the shared per-host gateway."""
stack = ExitStack()
bottle_for_revoke = plan.manifest.bottle
git_gate_dir_for_revoke = git_gate_state_dir(plan.slug)
plan = dataclasses.replace(
plan,
agent_provision=dataclasses.replace(plan.agent_provision, image=str(images.agent)),
)
def teardown() -> None:
teardown_exc: BaseException | None = None
try:
@@ -145,26 +90,22 @@ def launch(
except BaseException as exc: # noqa: W0718 - teardown must continue
teardown_exc = exc
warn(f"macos-container teardown failed: {exc!r}")
GitGate().revoke_provisioned_keys(bottle_for_revoke, git_gate_dir_for_revoke)
revoke_git_gate_provisioned_keys(bottle_for_revoke, git_gate_dir_for_revoke)
if teardown_exc is not None:
raise teardown_exc
try:
plan = _build_images(plan)
# Step 1: the per-host singletons. Must precede the agent run — its
# proxy env needs the gateway's address at `container run` time.
endpoint = ensure_gateway()
# The gateway's address may have changed since these bottles launched
# (any infra recreate re-runs DHCP). They name the gateway rather than
# address it, so re-pointing /etc/hosts re-attaches them in place
# instead of leaving them stranded until relaunch.
refresh_gateway_host(endpoint.gateway_ip)
# Step 2: mint this bottle's deploy keys, then point it at the SHARED
# gateway's CA + git-http/supervise ports.
plan = _provision_git_gate_keys(plan)
plan = _install_gateway_ca(plan, endpoint)
plan = _stamp_agent_urls(plan, endpoint)
plan = dataclasses.replace(plan, env_var_secret=new_env_var_secret())
# Step 3: run the agent. It has no identity token yet — registration
# needs the address this run assigns.
@@ -176,9 +117,6 @@ def launch(
# attribution key; `--cap-drop CAP_NET_RAW` at run is what makes it
# unforgeable. Poll: `container run --detach` can return before vmnet's
# DHCP has assigned the address.
# Resolve the gateway name before anything execs: every agent-facing
# URL uses it, so the entry must exist for the first connection.
set_gateway_host(plan.container_name, endpoint.gateway_ip)
source_ip = container_mod.wait_container_ipv4_on_network(
plan.container_name, endpoint.network,
)
@@ -188,10 +126,9 @@ def launch(
f"{endpoint.network}"
)
effective_env = {**os.environ, **plan.agent_provision.provisioned_env}
token_values = Egress().resolve_token_values(
token_values = egress_resolve_token_values(
plan.egress_plan.token_env_map, effective_env,
)
teardown_timeout = resolve_teardown_timeout()
ctx = register_agent(
plan.egress_plan,
plan.git_gate_plan,
@@ -199,12 +136,10 @@ def launch(
endpoint=endpoint,
image_ref=plan.image,
tokens=token_values,
env_var_secret=plan.env_var_secret,
)
stack.callback(
deprovision_consolidated, ctx.bottle_id,
teardown_consolidated, ctx.bottle_id,
orchestrator_url=ctx.orchestrator_url,
timeout=teardown_timeout,
)
info(
f"agent {plan.container_name} registered "
@@ -220,10 +155,6 @@ def launch(
# token above, so — unlike the run-time env — the plan CAN carry it.
plan = dataclasses.replace(plan, identity_token=ctx.identity_token)
exec_env = {
**_identity_proxy_env(endpoint, ctx.identity_token),
**nested_containers_mod.guest_env(plan.nested_containers),
}
bottle = MacosContainerBottle(
plan.container_name,
teardown,
@@ -237,38 +168,31 @@ def launch(
),
terminal_color=plan.spec.color,
agent_workdir=plan.workspace_plan.workdir,
exec_env=exec_env,
exec_env=_identity_proxy_env(endpoint, ctx.identity_token),
)
bottle.prompt_path = provision(plan, bottle)
if plan.nested_containers:
nested_containers_mod.prepare_guest_devices(
plan.container_name, container_mod.exec_container_as_root,
)
nested_containers_mod.start(bottle)
yield bottle
finally:
teardown()
def stale_checks(plan: MacosContainerBottlePlan) -> None:
"""Raise StaleImageError if a cached image is older than the configured
threshold. Only runs when image_policy is 'cached'. Called by the backend
class's _image_stale_checks before _launch_impl starts any resources."""
if plan.spec.image_policy != "cached":
return
def _build_images(plan: MacosContainerBottlePlan) -> MacosContainerBottlePlan:
"""Build the agent image. The gateway's own image is built by
`ensure_gateway` it belongs to the shared singleton, not to a bottle."""
committed = read_committed_image(plan.slug)
if committed and container_mod.image_exists(committed):
ts = container_mod.image_created_at(committed)
if ts is not None:
check_stale(f"agent image {committed!r}", ts)
return
if container_mod.image_exists(plan.image):
ts = container_mod.image_created_at(plan.image)
if ts is not None:
check_stale(f"agent image {plan.image!r}", ts)
info(f"using committed image {committed!r}")
return dataclasses.replace(
plan,
agent_provision=dataclasses.replace(
plan.agent_provision, image=committed,
),
)
container_mod.build_image(
plan.image, _REPO_DIR, dockerfile=plan.dockerfile_path,
)
return plan
def _provision_git_gate_keys(
@@ -276,7 +200,7 @@ def _provision_git_gate_keys(
) -> MacosContainerBottlePlan:
if not plan.git_gate_plan.upstreams:
return plan
git_gate_plan = GitGate().provision_dynamic_keys(
git_gate_plan = provision_git_gate_dynamic_keys(
plan.manifest.bottle,
plan.git_gate_plan,
git_gate_state_dir(plan.slug),
@@ -307,20 +231,13 @@ def _stamp_agent_urls(
) -> MacosContainerBottlePlan:
"""Point the agent's git-gate insteadOf rewrites + supervise MCP at the
shared gateway's ports. Both bypass the egress proxy (NO_PROXY covers the
gateway name).
Addressed by `GATEWAY_HOSTNAME`, never by IP: these URLs are baked into
the agent's gitconfig and MCP config at provision time, so an address here
would strand the bottle the moment the gateway moved. The name is resolved
per connection through `/etc/hosts`, which stays rewritable while the
bottle runs."""
del endpoint # addressed by name; the address reaches the bottle via /etc/hosts
gateway address)."""
git_gate_url = (
f"http://{GATEWAY_HOSTNAME}:{_GIT_HTTP_PORT}"
f"http://{endpoint.gateway_ip}:{_GIT_HTTP_PORT}"
if plan.git_gate_plan.upstreams else ""
)
supervise_url = (
f"http://{GATEWAY_HOSTNAME}:{SUPERVISE_PORT}/"
f"http://{endpoint.gateway_ip}:{SUPERVISE_PORT}/"
if plan.supervise_plan is not None else ""
)
return dataclasses.replace(
@@ -330,43 +247,30 @@ def _stamp_agent_urls(
)
def _proxy_url(identity_token: str = "") -> str:
def _proxy_url(gateway_ip: str, identity_token: str = "") -> str:
"""The agent's egress proxy URL. The identity token rides as proxy
credentials the gateway reads Proxy-Authorization, resolves the
(source_ip, token) pair against the control plane, and strips it before
upstream. Without a valid pair `/resolve` denies the request (#366).
Names the gateway rather than addressing it: this URL reaches the agent as
process environment, which cannot be rewritten once the agent is running,
so an address baked here is unfixable if the gateway moves."""
upstream. Without a valid pair `/resolve` denies the request (#366)."""
cred = f"bottle:{identity_token}@" if identity_token else ""
return f"http://{cred}{GATEWAY_HOSTNAME}:{EGRESS_PORT}"
return f"http://{cred}{gateway_ip}:{EGRESS_PORT}"
def _no_proxy() -> str:
def _no_proxy(gateway_ip: str) -> str:
# git-http + supervise live on the gateway and must NOT go through the
# egress proxy — the agent reaches them directly by name. Deliberately
# address-free: NO_PROXY is baked into the run-time env and is therefore
# just as unfixable as the proxy URL if the gateway moves.
return f"localhost,127.0.0.1,{GATEWAY_HOSTNAME}"
# egress proxy — the agent reaches them directly by its address.
return f"localhost,127.0.0.1,{gateway_ip}"
def _identity_proxy_env(
endpoint: GatewayEndpoint, identity_token: str,
) -> dict[str, str]:
"""The token-bearing proxy env applied at `container exec` — the only way
to get the token in, since it does not exist until after the container
runs (registration keys on the DHCP-assigned address).
This is the *sole* source of `*_PROXY` for the agent. It deliberately does
not rely on overriding a run-time value: `container exec --env` appends
rather than replaces, so a run-time `HTTPS_PROXY` would survive alongside
this one and first-wins runtimes would read the wrong entry. See
`_agent_env_entries`."""
"""The token-bearing proxy env applied at `container exec`. It supersedes
the token-less run-time value (exec `--env` wins), which is the only way to
get the token in: it does not exist until after the container runs."""
if not identity_token:
return {}
del endpoint # the gateway is named, not addressed
url = _proxy_url(identity_token)
url = _proxy_url(endpoint.gateway_ip, identity_token)
return {
"HTTPS_PROXY": url, "HTTP_PROXY": url,
"https_proxy": url, "http_proxy": url,
@@ -413,23 +317,16 @@ def _agent_run_argv(
def _agent_env_entries(
plan: MacosContainerBottlePlan, endpoint: GatewayEndpoint,
) -> tuple[str, ...]:
# No `*_PROXY` here on purpose. The token-bearing URL is applied at
# `container exec` (`_identity_proxy_env`), and Apple's `container exec
# --env` **appends** to the run-time environment rather than replacing it:
# setting a token-less value here leaves two `HTTPS_PROXY` entries in the
# agent's `environ`, token-less first. Which one a runtime reads is then
# pure luck — Node takes the last (and worked), Rust's `std::env::var`
# takes the first, so Codex proxied without its identity token and
# `/resolve` fail-closed on every request.
#
# A token-less proxy URL has no legitimate consumer anyway: the init
# process is `sleep` and everything that egresses arrives via exec. Its
# only value was a tidy 403 for unattributed callers, which is not worth
# silently dropping attribution for. Without it a process that egresses
# before the exec-time env still fails closed — the agent network is
# host-only, so there is no route off it except the gateway.
no_proxy = _no_proxy()
# Token-less at run time — the token does not exist yet (see
# `_identity_proxy_env`). Anything egressing before the exec-time override
# is denied by `/resolve`, which is the safe direction.
proxy_url = _proxy_url(endpoint.gateway_ip)
no_proxy = _no_proxy(endpoint.gateway_ip)
env = [
f"HTTPS_PROXY={proxy_url}",
f"HTTP_PROXY={proxy_url}",
f"https_proxy={proxy_url}",
f"http_proxy={proxy_url}",
f"NO_PROXY={no_proxy}",
f"no_proxy={no_proxy}",
f"NODE_EXTRA_CA_CERTS={AGENT_CA_PATH}",
@@ -440,15 +337,13 @@ def _agent_env_entries(
env.append(f"GIT_GATE_URL={plan.agent_git_gate_url}")
if plan.agent_supervise_url:
env.append(f"MCP_SUPERVISE_URL={plan.agent_supervise_url}")
if getattr(plan, "env_var_secret", ""):
env.append(f"{ENV_VAR_SECRET_NAME}={plan.env_var_secret}")
for name, value in sorted(plan.agent_provision.guest_env.items()):
env.append(f"{name}={value}")
# Forwarded vars: bare name → inherits from the `container run` process env
# so the secret value never lands on argv.
for name in sorted(plan.forwarded_env.keys()):
env.append(name)
env.extend(Egress().agent_env_entries(plan.egress_plan))
env.extend(egress_agent_env_entries(plan.egress_plan))
return tuple(env)
@@ -1,210 +0,0 @@
#!/bin/sh
set -eu
uid="$(id -u)"
if [ "$uid" -eq 0 ]; then
echo "refusing to run the guest container engine as root" >&2
exit 1
fi
# Every piece of podman 5's networking stack is checked here, because each
# one fails at a different and misleading layer if it is absent: no pasta and
# nothing starts at all; no nft and netavark cannot build the bridge every
# compose file expects; no aardvark-dns and DNS inside nested containers fails
# while everything else looks healthy.
for command in podman docker fuse-overlayfs pasta nft slirp4netns; do
command -v "$command" >/dev/null 2>&1 || {
echo "missing nested-container prerequisite: $command" >&2
exit 1
}
done
# The inverse of the rootless-Docker check, and the whole point of the podman
# variant: a subordinate range would push podman onto newuidmap, which cannot
# write a multi-range uid_map without CAP_SYS_ADMIN in this guest. An empty
# range keeps it on the single-UID self-mapping an unprivileged process may
# write itself.
if grep -q "^$(id -un):" /etc/subuid 2>/dev/null; then
echo "unexpected subordinate UID range for $(id -un): podman would" >&2
echo "require CAP_SYS_ADMIN via newuidmap in this guest" >&2
exit 1
fi
for device in /dev/fuse /dev/net/tun; do
[ -r "$device" ] && [ -w "$device" ] || {
echo "device $device is not readable/writable by $(id -un)" >&2
exit 1
}
done
# Short by necessity, not by accident: conmon's attach socket lives under
# this directory and must fit in a 108-byte sun_path. See nested_containers.py.
# Must stay in step with AGENT_CA_BUNDLE in bot_bottle/backend/util.py; a unit
# test pins the two together.
CA_BUNDLE="/etc/ssl/certs/ca-certificates.crt"
[ -r "$CA_BUNDLE" ] || {
echo "gateway CA bundle $CA_BUNDLE is missing or unreadable" >&2
exit 1
}
# The proxy URL the agent inherits names `bot-bottle-gateway`, which resolves
# only through this bottle's /etc/hosts. A nested container gets its own hosts
# file, so it cannot resolve the name and dies at "Could not resolve proxy".
#
# podman's containers.conf `hosts_file` would fix that, except the
# Docker-compatible API ignores it — it only takes effect for native
# `podman run`, and the agent types `docker`. So the name is resolved *here*
# and the address, not the name, goes into the proxy URL the nested container
# receives. Verified on macOS 26 / podman 5.4.2: with the address in place,
# https://quay.io returns 200 and a non-allowlisted host still gets 403, so
# the egress boundary applies inside nested containers too.
GATEWAY_NAME="bot-bottle-gateway"
gateway_ip="$(
awk -v name="$GATEWAY_NAME" '$2 == name { print $1; exit }' /etc/hosts
)"
[ -n "$gateway_ip" ] || {
echo "no /etc/hosts entry for $GATEWAY_NAME; the gateway address is" >&2
echo "needed so nested containers can reach the egress proxy" >&2
exit 1
}
export XDG_RUNTIME_DIR="${XDG_RUNTIME_DIR:-/tmp/bbp}"
config="$HOME/.config/containers"
mkdir -p "$XDG_RUNTIME_DIR" "$config"
chmod 700 "$XDG_RUNTIME_DIR"
# ignore_chown_errors is required, not incidental: with a single-UID mapping
# there is no second UID for image layers to be chowned to, so layers that
# record other owners would otherwise fail to extract.
cat > "$config/storage.conf" <<'CONF'
[storage]
driver="overlay"
[storage.options.overlay]
mount_program="/usr/bin/fuse-overlayfs"
ignore_chown_errors="true"
CONF
# No cgroup delegation reaches this guest, so asking podman to manage cgroups
# fails; events_logger=file avoids the journald socket that is equally absent.
#
# The rest of this config is what lets a nested container reach the network:
#
# hosts_file only takes effect for native `podman run` — the
# Docker-compatible API ignores it, and the agent types
# `docker`. Kept anyway because it costs nothing and makes
# podman-native use behave; the compat path is covered by the
# address-bearing proxy URL below.
# volumes/env the gateway TLS-intercepts, so a container that does not
# trust the bottle's CA bundle gets "unable to get local issuer
# certificate". Mounting the bundle read-only and pointing the
# usual env vars at it covers curl, wget, python, and node
# without distro-specific trust commands.
#
# The proxy URL carries the bottle's identity token. podman already forwards
# that same URL into every nested container from the agent's own environment,
# so writing it to a 0600 file inside this disposable VM hands it to nobody
# new. It is never echoed.
CA_BUNDLE="$CA_BUNDLE" GATEWAY_NAME="$GATEWAY_NAME" GATEWAY_IP="$gateway_ip" \
CONTAINERS_CONF="$config/containers.conf" python3 - <<'PY'
import os
from pathlib import Path
ca = os.environ["CA_BUNDLE"]
name = os.environ["GATEWAY_NAME"]
ip = os.environ["GATEWAY_IP"]
entries = [
f"SSL_CERT_FILE={ca}",
f"CURL_CA_BUNDLE={ca}",
f"REQUESTS_CA_BUNDLE={ca}",
f"NODE_EXTRA_CA_CERTS={ca}",
]
# The gateway name resolves only through the bottle's /etc/hosts, which a
# nested container does not inherit, so hand it the address instead.
for var in ("HTTP_PROXY", "HTTPS_PROXY", "http_proxy", "https_proxy"):
value = os.environ.get(var)
if value:
entries.append(f"{var}={value.replace(name, ip)}")
# NO_PROXY keeps the name: it is matched against what a client asks for, and
# code inside a nested container still says "bot-bottle-gateway".
for var in ("NO_PROXY", "no_proxy"):
value = os.environ.get(var)
if value:
entries.append(f"{var}={value}")
path = Path(os.environ["CONTAINERS_CONF"])
path.write_text("\n".join([
"[containers]",
'cgroups="disabled"',
# podman copies the host's proxy vars into every container by default,
# and that copy *wins* over the env below — putting the unresolvable
# gateway name back. Turn it off so the address-bearing URLs stand.
"http_proxy=false",
'hosts_file="/etc/hosts"',
f'volumes=["{ca}:{ca}:ro"]',
"env=[",
*[f' "{entry}",' for entry in entries],
"]",
"[engine]",
'cgroup_manager="cgroupfs"',
'events_logger="file"',
"",
]), encoding="utf-8")
path.chmod(0o600)
PY
# Registry pulls egress through the bottle's proxy like everything else. The
# token-bearing proxy URL is already in the agent's environment; persisting it
# inside this disposable VM does not broaden its authority.
#
# This file is also what the Docker CLI copies into every container it starts,
# and being client-side it beats anything the podman service does — it is why
# containers.conf `env`, `http_proxy=false`, and the service's own environment
# all failed to change what a nested container saw. The address goes in here
# for the same reason it goes everywhere else: `bot-bottle-gateway` resolves
# in the bottle, never inside a nested container.
GATEWAY_NAME="$GATEWAY_NAME" GATEWAY_IP="$gateway_ip" python3 - <<'PY'
import json
import os
from pathlib import Path
name = os.environ["GATEWAY_NAME"]
ip = os.environ["GATEWAY_IP"]
proxy = os.environ.get("HTTPS_PROXY") or os.environ.get("https_proxy", "")
# NO_PROXY keeps the name: it is matched against what a client asks for, and
# code inside a nested container still says "bot-bottle-gateway".
no_proxy = os.environ.get("NO_PROXY") or os.environ.get("no_proxy", "")
config = {"proxies": {"default": {
"httpProxy": proxy.replace(name, ip),
"httpsProxy": proxy.replace(name, ip),
"noProxy": no_proxy,
}}}
path = Path.home() / ".docker" / "config.json"
path.parent.mkdir(parents=True, exist_ok=True)
path.write_text(json.dumps(config), encoding="utf-8")
path.chmod(0o600)
PY
if docker info >/dev/null 2>&1; then
exit 0
fi
# Belt to the ~/.docker/config.json braces above, which is what actually
# decides this for `docker run`. The service environment is what podman falls
# back to for anything the CLI does not stamp — its own registry pulls, and
# containers created through the API by something other than the Docker CLI.
# Cheap, and it keeps the address consistent across both paths.
#
# Assigned via parameter expansion, never echoed: these carry the bottle's
# identity token.
for var in HTTP_PROXY HTTPS_PROXY http_proxy https_proxy; do
eval "value=\${$var:-}"
[ -n "$value" ] || continue
eval "export $var=\"\${value%%$GATEWAY_NAME*}$gateway_ip\${value#*$GATEWAY_NAME}\""
done
log=/tmp/bot-bottle-nested-containers.log
nohup podman system service --time=0 \
"unix://$XDG_RUNTIME_DIR/podman.sock" \
>"$log" 2>&1 </dev/null &
@@ -1,161 +0,0 @@
"""Guest-local container engine for Apple-container bottles (issue #392).
The service and every nested container remain inside the existing per-bottle
VM. This module refuses to compensate for missing prerequisites with outer
capabilities, a privileged container, or a host Docker socket.
Podman is used rather than rootless Docker for one specific reason: Apple
Container's capability bounding set omits `CAP_SYS_ADMIN`, which the kernel
requires to write a multi-range `uid_map` via `newuidmap`. Rootless Docker
has no path that avoids that write. Podman does with no subordinate UID
range configured it falls back to a single-UID self-mapping, which an
unprivileged process may write itself. See
`docs/research/rootless-docker-in-apple-container-spike.md`.
That fallback is why `build_image` *removes* the agent user's `/etc/subuid`
and `/etc/subgid` entries instead of adding them: their presence is precisely
what would send podman down the `newuidmap` path that cannot work here.
The agent still talks to `docker` and `docker compose`; those speak to
podman's Docker-compatible API socket, so nothing in the agent's habits
changes.
"""
from __future__ import annotations
import shlex
import shutil
import tempfile
import time
from pathlib import Path
from typing import Callable
from ...log import die, info
_INIT = "/usr/local/libexec/bot-bottle/nested-containers-init"
# Deliberately cryptic and short. podman derives conmon's attach socket as
# `$XDG_RUNTIME_DIR/libpod/tmp/socket/<64-hex-id>/attach`, and a Unix socket
# path may not exceed 108 bytes (`sun_path`). The descriptive
# `/tmp/bot-bottle-podman-run` produced a 116-byte path — over the limit, so
# attach would have broken as soon as anything got far enough to attach. Do
# not lengthen this for readability; it buys 8 bytes of headroom.
_RUNTIME_DIR = "/tmp/bbp"
_SOCKET = f"{_RUNTIME_DIR}/podman.sock"
_LOG = "/tmp/bot-bottle-nested-containers.log"
IMAGE_SUFFIX = "-nested-containers"
READY_RETRIES = 30
# Apple Container creates both device nodes 0600 root:root, so the agent user
# cannot open them: /dev/fuse blocks the fuse-overlayfs storage driver and
# /dev/net/tun blocks slirp4netns, which rootless podman uses for the default
# bridge network that stock compose files expect. Relaxing the modes needs no
# capability the bottle does not already hold — unlike CAP_SYS_ADMIN, which is
# what killed the rootless-Docker approach.
_GUEST_DEVICES = ("/dev/fuse", "/dev/net/tun")
def build_image(
base_image: str,
build: Callable[..., None],
) -> str:
"""Layer the nested-container tooling onto an already-built agent image.
Podman and its networking stack live here rather than in the base agent
images so that bottles without the flag pay no image-size cost.
# TODO(#394): replace this hand-rolled Dockerfile with a docker-layer
# abstraction once that infrastructure exists.
"""
image = f"{base_image}{IMAGE_SUFFIX}"
init_script = Path(__file__).with_name("nested-containers-init.sh")
with tempfile.TemporaryDirectory(prefix="bot-bottle-nested-containers.") as tmp:
context = Path(tmp)
shutil.copy2(init_script, context / "nested-containers-init.sh")
(context / "Dockerfile").write_text(
"FROM docker:28-cli AS docker_cli\n"
f"FROM {base_image}\n"
"USER root\n"
"COPY --from=docker_cli /usr/local/bin/docker /usr/local/bin/docker\n"
"COPY --from=docker_cli /usr/local/libexec/docker/cli-plugins/"
"docker-compose /usr/local/libexec/docker/cli-plugins/docker-compose\n"
"RUN apt-get update \\\n"
# podman 5's networking stack, installed explicitly because
# --no-install-recommends omits it and each missing piece fails
# at a different, misleading layer:
# podman -> moved here from the base agent images so that
# bottles without nested_containers pay no cost
# passt -> `pasta`, the default rootless netns helper
# (podman 4 used slirp4netns); without it
# nothing starts: "could not find pasta"
# nftables -> `nft`, which netavark shells out to for the
# bridge network every compose file expects
# aardvark-dns -> name resolution *inside* nested containers;
# without it DNS fails while everything else
# looks healthy
# slirp4netns stays as the documented fallback for pasta.
" && apt-get install -y --no-install-recommends "
"aardvark-dns fuse-overlayfs netavark nftables passt podman "
"slirp4netns uidmap \\\n"
" && rm -rf /var/lib/apt/lists/* \\\n"
# Deliberate: an empty subordinate range keeps podman on the
# single-UID mapping that needs no CAP_SYS_ADMIN. Adding ranges
# here would reintroduce the newuidmap failure this design exists
# to route around.
" && sed -i '/^node:/d' /etc/subuid /etc/subgid\n"
"COPY nested-containers-init.sh "
"/usr/local/libexec/bot-bottle/nested-containers-init\n"
"RUN chmod 0755 /usr/local/libexec/bot-bottle/nested-containers-init\n"
"USER node\n",
encoding="utf-8",
)
build(image, str(context), dockerfile=str(context / "Dockerfile"))
return image
def guest_env(enabled: bool) -> dict[str, str]:
"""Environment consumed by the Docker CLI inside an enabled bottle."""
if not enabled:
return {}
return {
"DOCKER_HOST": f"unix://{_SOCKET}",
"XDG_RUNTIME_DIR": _RUNTIME_DIR,
}
def prepare_guest_devices(container_name: str, exec_as_root: Callable[..., None]) -> None:
"""Make /dev/fuse and /dev/net/tun openable by the agent user.
Runs as root inside the bottle because the agent must not be able to
re-mode device nodes itself. No outer capability is involved.
"""
exec_as_root(
container_name,
["sh", "-c", f"chmod 0666 {' '.join(_GUEST_DEVICES)}"],
)
def start(bottle: object) -> None:
"""Start and verify the unprivileged service through the bottle exec API."""
info("starting guest-local container engine")
result = bottle.exec(shlex.quote(_INIT)) # type: ignore[attr-defined]
if result.returncode != 0:
detail = (result.stderr or result.stdout or "").strip()
die(f"nested-container bootstrap failed: {detail or '<no output>'}")
for _ in range(READY_RETRIES):
result = bottle.exec("docker info >/dev/null 2>&1") # type: ignore[attr-defined]
if result.returncode == 0:
info("guest-local container engine is ready")
return
time.sleep(0.2)
logs = bottle.exec( # type: ignore[attr-defined]
f"tail -n 80 {_LOG} 2>/dev/null || true"
)
die(
"guest-local container engine did not become ready without additional "
f"outer privileges:\n{(logs.stdout or logs.stderr or '<no log>').strip()}"
)
__all__ = ["build_image", "guest_env", "prepare_guest_devices", "start"]
@@ -1,204 +0,0 @@
"""The macOS orchestrator (control plane) as an Apple container (PRD 0070).
`MacosOrchestrator` is the Apple-Container implementation of the backend-neutral
`Orchestrator` service. The lean control-plane container joins the host-only
control network only (agents are never on it), mounts a container-only DB volume
(exactly one kernel writes `bot-bottle.db`), and holds the signing key (#469).
The host CLI and the gateway both reach it at its control-network address
Apple has no container DNS, so there's a single resolved URL, not docker's
loopback-vs-name split.
"""
from __future__ import annotations
import os
import time
import urllib.error
import urllib.request
from pathlib import Path
from ... import log
from ...paths import ORCHESTRATOR_TOKEN_ENV
from ...orchestrator.lifecycle import (
DEFAULT_HEALTH_TIMEOUT_SECONDS,
DEFAULT_PORT,
DEFAULT_STARTUP_TIMEOUT_SECONDS,
Orchestrator,
OrchestratorStartError,
source_hash,
)
from . import util as container_mod
from .gateway import CONTROL_NETWORK
ORCHESTRATOR_IMAGE = os.environ.get(
"BOT_BOTTLE_ORCHESTRATOR_IMAGE", "bot-bottle-orchestrator:latest"
)
ORCHESTRATOR_NAME = "bot-bottle-mac-orchestrator"
ORCHESTRATOR_LABEL = "bot-bottle-mac-orchestrator=1"
# Container-only volume holding bot-bottle.db, mounted ONLY into the
# orchestrator. One kernel writes it (never host-shared or cross-guest).
ORCHESTRATOR_DB_VOLUME = "bot-bottle-mac-db"
# BOT_BOTTLE_ROOT inside the orchestrator; host_db_path() resolves the DB to
# <root>/db/<filename>.
_DB_ROOT_IN_CONTAINER = "/var/lib/bot-bottle"
_SRC_IN_CONTAINER = "/bot-bottle-src"
_HEALTH_POLL_SECONDS = 0.25
_REPO_ROOT = Path(__file__).resolve().parents[3]
class MacosOrchestrator(Orchestrator):
"""The control plane as an Apple container on the host-only control network.
`ensure_built` builds `Dockerfile.orchestrator`; `ensure_running` starts it
and blocks until `/health` answers at its control-network address."""
def __init__(
self,
image_ref: str = ORCHESTRATOR_IMAGE,
*,
name: str = ORCHESTRATOR_NAME,
label: str = ORCHESTRATOR_LABEL,
port: int = DEFAULT_PORT,
control_network: str = CONTROL_NETWORK,
repo_root: Path = _REPO_ROOT,
db_volume: str = ORCHESTRATOR_DB_VOLUME,
) -> None:
self.image_ref = image_ref
self.name = name
self.label = label
self.port = port
self.control_network = control_network
self._repo_root = repo_root
self._db_volume = db_volume
def url(self) -> str:
"""The orchestrator's control-network address (host CLI + registration),
or "" while it has no address yet. Apple has no container DNS, so this is
also what the gateway resolves against (`gateway_url`)."""
ip = container_mod.try_container_ipv4_on_network(self.name, self.control_network)
return f"http://{ip}:{self.port}" if ip else ""
def gateway_url(self) -> str:
"""Same address the host uses — the gateway reaches the orchestrator by
control-network IP (Apple has no container DNS)."""
return self.url()
def is_healthy(self, *, timeout: float = DEFAULT_HEALTH_TIMEOUT_SECONDS) -> bool:
url = self.url()
if not url:
return False
try:
with urllib.request.urlopen(f"{url}/health", timeout=timeout) as resp:
return resp.status == 200
except (urllib.error.URLError, TimeoutError, OSError):
return False
def is_running(self) -> bool:
return container_mod.container_is_running(self.name)
def _source_current(self, current_hash: str) -> bool:
"""True iff the running orchestrator was created from the current
bind-mounted control-plane source (it loads that code at startup and
won't reload it)."""
if not self.is_running():
return False
env = container_mod.container_env(self.name)
if not env:
return True # can't compare → don't churn a working container
return env.get("BOT_BOTTLE_SOURCE_HASH") == current_hash
def ensure_built(self) -> None:
"""Build the control-plane image. The source is bind-mounted so a code
change takes effect without a rebuild; the image still carries the
package for its entrypoint."""
container_mod.build_image(
self.image_ref, str(self._repo_root), dockerfile="Dockerfile.orchestrator")
def ensure_running(
self, *, startup_timeout: float = DEFAULT_STARTUP_TIMEOUT_SECONDS,
) -> None:
"""Ensure the control-plane container is up on current source; block
until healthy. Idempotent a healthy orchestrator on current source is
left untouched. Raises `OrchestratorStartError` on startup timeout.
The control network must already exist (the composer ensures it)."""
current_hash = source_hash(self._repo_root)
if self._source_current(current_hash) and self.is_healthy():
return
log.info("starting orchestrator container", context={"name": self.name})
self._run_container(current_hash)
self._wait_healthy(startup_timeout)
def _run_container(self, current_hash: str) -> None:
container_mod.force_remove_container(self.name)
# The signing key comes through the shared provisioning contract (#476),
# which fail-closes rather than yield an empty key that would run OPEN.
_signing_key = self.control_plane_key()
argv = [
"container", "run", "--detach",
"--name", self.name,
"--label", "bot-bottle.backend=macos-container",
"--label", self.label,
# Control network only — agents are never on it (L3-isolated).
"--network", self.control_network,
"--dns", container_mod.dns_server(),
# Container-only DB volume: exactly one kernel writes bot-bottle.db.
"--volume", f"{self._db_volume}:{_DB_ROOT_IN_CONTAINER}",
# Live control-plane source (a code change takes effect on relaunch).
"--mount",
container_mod.bind_mount_spec(
str(self._repo_root), _SRC_IN_CONTAINER, readonly=True),
"--env", f"PYTHONPATH={_SRC_IN_CONTAINER}",
"--env", f"BOT_BOTTLE_ROOT={_DB_ROOT_IN_CONTAINER}",
# Detect a real control-plane code change and recreate.
"--env", f"BOT_BOTTLE_SOURCE_HASH={current_hash}",
# The signing key — held ONLY by the orchestrator (issue #469). Bare
# `--env NAME` keeps the value off argv / `container inspect`.
"--env", ORCHESTRATOR_TOKEN_ENV,
self.image_ref,
# Dockerfile.orchestrator ENTRYPOINT is `-m bot_bottle.orchestrator`.
"--host", "0.0.0.0", "--port", str(self.port), "--broker", "stub",
]
result = container_mod.run_container_argv(
argv, env={**os.environ, ORCHESTRATOR_TOKEN_ENV: _signing_key})
if result.returncode != 0:
raise OrchestratorStartError(
f"orchestrator container failed to start: "
f"{(result.stderr or '').strip() or '<no stderr>'}"
)
def _wait_healthy(self, startup_timeout: float) -> None:
deadline = time.monotonic() + startup_timeout
while True:
if self.is_healthy():
log.info("orchestrator healthy", context={"url": self.url()})
return
if time.monotonic() >= deadline:
raise OrchestratorStartError(
f"orchestrator did not become healthy within "
f"{startup_timeout:g}s"
)
time.sleep(_HEALTH_POLL_SECONDS)
def stop(self) -> None:
"""Remove the control-plane container (idempotent). The DB volume
persists."""
container_mod.force_remove_container(self.name)
def probe_orchestrator_url(port: int = DEFAULT_PORT) -> str:
"""The running orchestrator's control-plane URL, or "" if it isn't up. Used
by host-side control-plane discovery; safe on any host (returns "" when the
container or the `container` CLI isn't present)."""
ip = container_mod.try_container_ipv4_on_network(ORCHESTRATOR_NAME, CONTROL_NETWORK)
return f"http://{ip}:{port}" if ip else ""
__all__ = [
"MacosOrchestrator",
"ORCHESTRATOR_NAME",
"ORCHESTRATOR_LABEL",
"ORCHESTRATOR_IMAGE",
"ORCHESTRATOR_DB_VOLUME",
"probe_orchestrator_url",
]
@@ -8,7 +8,7 @@ from ...agent_provider import AgentProvisionPlan
from ...egress import EgressPlan
from ...env import ResolvedEnv
from ...git_gate import GitGatePlan
from ...supervisor.plan import SupervisePlan
from ...supervise import SupervisePlan
from ...manifest import Manifest
from .. import BottleSpec
from . import util as container_mod
@@ -44,5 +44,4 @@ def resolve_plan(
egress_plan=egress_plan,
supervise_plan=supervise_plan,
agent_provision=agent_provision_plan,
nested_containers=manifest.bottle.nested_containers,
)
@@ -10,7 +10,6 @@ import shutil
import subprocess
import tempfile
import time
from datetime import datetime, timezone
from typing import Iterable
from ...log import die, info
@@ -361,27 +360,6 @@ def exec_container(name: str, argv: list[str]) -> None:
)
def read_container_env(name: str, env_name: str) -> str:
"""Read one configured env value from a running container, or ``""``."""
result = _run_container_op([_CONTAINER, "exec", name, "printenv", env_name])
return result.stdout.strip() if result.returncode == 0 else ""
def exec_container_as_root(name: str, argv: list[str]) -> None:
"""`exec_container`, but as uid 0 inside the container.
For host-driven maintenance the agent itself must not be able to perform
rewriting `/etc/hosts` to point the gateway name at an address. The agent
runs as `node`, so it cannot repoint its own gateway; the host can.
"""
result = _run_container_op([_CONTAINER, "exec", "--user", "root", name, *argv])
if result.returncode != 0:
die(
f"container exec (root) in {name} failed: "
f"{(result.stderr or '').strip() or '<no stderr>'}"
)
def _run_container_op(cmd: list[str]) -> subprocess.CompletedProcess[str]:
result = subprocess.run(
cmd,
@@ -579,41 +557,6 @@ def try_container_ipv4_on_network(name: str, network: str) -> str:
return ""
def inspect_container_network_ip(name: str, network: str) -> str | None:
"""IP of `name` on `network`, distinguishing inspect failure from "not yet".
Returns:
- the IP string when the container has one on `network`
- "" when inspect succeeds but no address is assigned yet (in-flight DHCP)
- None when the inspect command itself fails (authoritative list impossible)
"""
result = subprocess.run(
[_CONTAINER, "inspect", name],
capture_output=True, text=True, check=False,
)
if result.returncode != 0:
return None
try:
data = json.loads(result.stdout or "[]")
except json.JSONDecodeError:
return None
if isinstance(data, list):
data = data[0] if data else {}
if not isinstance(data, dict):
return None
status = data.get("status")
networks = status.get("networks") if isinstance(status, dict) else None
if not isinstance(networks, list):
return ""
for entry in networks:
if not isinstance(entry, dict) or entry.get("network") != network:
continue
raw = entry.get("ipv4Address")
if isinstance(raw, str) and raw:
return raw.split("/", 1)[0]
return ""
def wait_container_ipv4_on_network(
name: str, network: str, *, timeout: float = 15.0, poll: float = 0.25,
) -> str:
@@ -668,39 +611,6 @@ def image_id(ref: str) -> str:
raise AssertionError("unreachable")
def image_created_at(ref: str) -> datetime | None:
"""Return the image creation timestamp as an aware UTC datetime, or None
when the field is absent or unparseable (e.g. FROM-scratch images, images
pulled from registries that omit the field). Callers should skip the stale
check when None is returned rather than treating it as an error."""
result = subprocess.run(
[_CONTAINER, "image", "inspect", ref],
capture_output=True,
text=True,
check=False,
)
if result.returncode != 0:
die(
f"container image inspect for {ref!r} failed: "
f"{(result.stderr or '').strip() or '<no stderr>'}"
)
try:
data = json.loads(result.stdout or "{}")
except json.JSONDecodeError as exc:
die(f"container image inspect for {ref!r} returned malformed JSON: {exc}")
if isinstance(data, list) and data:
data = data[0]
if isinstance(data, dict):
value = data.get("created") or data.get("Created")
if isinstance(value, str) and value:
try:
ts = value.rstrip("Z")
return datetime.fromisoformat(ts).replace(tzinfo=timezone.utc)
except ValueError:
pass
return None
def save(ref: str, output: str) -> None:
subprocess.run([_CONTAINER, "image", "save", ref, "-o", output], check=True)
-71
View File
@@ -1,71 +0,0 @@
"""Bottle-level provisioning for the consolidated launch sequence (PRD 0070).
Register a bottle with the orchestrator and provision its git-gate state into
the shared gateway (`provision_bottle`), and the inverse teardown
(`deprovision_bottle`). Backend-neutral each backend's consolidated_launch
drives these through its own `GatewayTransport` rather than re-implementing
them. The git-gate half of the work lives in `provision_gateway`.
"""
from __future__ import annotations
import dataclasses
from ..egress import EgressPlan
from ..git_gate import GitGatePlan
from ..orchestrator.client import OrchestratorClient, RegisteredBottle
from ..orchestrator.registration import registration_inputs
from ..orchestrator.store.secret_store import new_env_var_secret
from .provision_gateway import GatewayTransport, deprovision_git_gate, provision_git_gate
def provision_bottle(
client: OrchestratorClient,
source_ip: str,
egress_plan: EgressPlan,
git_gate_plan: GitGatePlan,
transport: GatewayTransport,
*,
image_ref: str = "",
tokens: dict[str, str] | None = None,
env_var_secret: str | None = None,
) -> RegisteredBottle:
"""Register the bottle and provision its git-gate state. Rolls back the
registration if provisioning fails so no orphan is left.
Generates a fresh ENV_VAR_SECRET, passes it to the orchestrator so it can
encrypt the token values at rest, and stamps the secret onto the returned
``RegisteredBottle`` so callers can inject it into the agent container's
environment."""
inputs = registration_inputs(egress_plan)
env_var_secret = env_var_secret or new_env_var_secret()
reg = client.register_bottle(
source_ip, image_ref=image_ref, policy=inputs.policy,
metadata=inputs.metadata, tokens=tokens, env_var_secret=env_var_secret,
)
try:
provision_git_gate(transport, reg.bottle_id, git_gate_plan)
except Exception:
client.teardown_bottle(reg.bottle_id)
raise
return dataclasses.replace(reg, env_var_secret=env_var_secret)
def deprovision_bottle(
bottle_id: str,
transport: GatewayTransport,
*,
orchestrator_url: str,
timeout: float | None = None,
) -> None:
"""Deregister the bottle and remove its git-gate state. Both steps are
idempotent so this is safe from a cleanup trap."""
from ..orchestrator.store.config_store import DEFAULT_TEARDOWN_TIMEOUT_SECONDS
OrchestratorClient(
orchestrator_url,
timeout=timeout if timeout is not None else DEFAULT_TEARDOWN_TIMEOUT_SECONDS,
).teardown_bottle(bottle_id)
deprovision_git_gate(transport, bottle_id)
__all__ = ["provision_bottle", "deprovision_bottle"]
+2 -21
View File
@@ -26,10 +26,8 @@ from ..bottle_state import (
)
from ..egress import Egress, EgressPlan
from ..git_gate import GitGate, GitGatePlan
from ..log import die
from ..manifest import Manifest, ManifestBottle
from ..supervisor.plan import SupervisePlan
from ..orchestrator.supervisor import Supervisor
from ..supervise import Supervise, SupervisePlan
from . import BottleSpec
@@ -102,7 +100,7 @@ def prepare_supervise(bottle: ManifestBottle, slug: str) -> SupervisePlan | None
return None
supervise_dir = supervise_state_dir(slug)
supervise_dir.mkdir(parents=True, exist_ok=True)
return Supervisor().prepare(slug, supervise_dir)
return Supervise().prepare(slug, supervise_dir)
def merge_provision_env_vars(provision: AgentProvisionPlan) -> AgentProvisionPlan:
@@ -114,22 +112,6 @@ def merge_provision_env_vars(provision: AgentProvisionPlan) -> AgentProvisionPla
return replace(provision, guest_env=merged)
def reject_nested_containers(backend: str, manifest: Manifest) -> None:
"""Fail loudly when a backend cannot honor `nested_containers: true`.
Silently ignoring it would hand the agent a bottle where `docker` is not
there and the only sound alternatives on these backends (a host daemon
socket, a privileged container) are exactly what issue #392 rules out.
"""
if not manifest.bottle.nested_containers:
return
die(
f"nested_containers is not supported on the {backend} backend. "
"Only macos-container runs a guest-local container engine today; "
"mounting the host Docker socket is not an option bot-bottle offers."
)
def resolve_manifest_dockerfile(path_value: str, spec: BottleSpec) -> str:
"""Resolve a manifest-supplied dockerfile path relative to user_cwd."""
path = Path(os.path.expanduser(path_value))
@@ -140,7 +122,6 @@ def resolve_manifest_dockerfile(path_value: str, spec: BottleSpec) -> str:
__all__ = [
"merge_provision_env_vars",
"reject_nested_containers",
"mint_slug",
"prepare_agent_state_dir",
"prepare_egress",
-211
View File
@@ -1,211 +0,0 @@
"""Backend registry, selection, and active-agent enumeration.
Resolves which bottle backend to use (explicit name / `BOT_BOTTLE_BACKEND` /
auto-select), and enumerates running agents across every available backend. The
three concrete backends are imported lazily inside `_get_backends` so this
module and anything that only needs to *select* a backend stays cheap.
"""
from __future__ import annotations
import os
import sys
from typing import Any
from ..log import die, info, warn
from ..util import read_tty_line
from .base import ActiveAgent, BackendStatus, BottleBackend
# _backends is None until the first call to _get_backends(), at which
# point all three concrete backend classes are imported and instantiated.
# Keeping the imports out of module scope means that importing any
# backend sub-module (e.g. `backend.docker.util`) no longer drags the
# firecracker and macos-container implementations into memory.
#
# Tests may replace _backends with a {name: fake} dict via patch.object;
# _get_backends() returns the current module-level value as-is when it
# is not None, so test fakes take effect without triggering real imports.
_backends: dict[str, BottleBackend[Any, Any]] | None = None
def _get_backends() -> dict[str, BottleBackend[Any, Any]]:
"""Return the registry of all backend instances, loading lazily on first call."""
global _backends # pylint: disable=global-statement
if _backends is None:
from .docker import DockerBottleBackend
from .firecracker import FirecrackerBottleBackend
from .macos_container import MacosContainerBottleBackend
_backends = {
"docker": DockerBottleBackend(),
"firecracker": FirecrackerBottleBackend(),
"macos-container": MacosContainerBottleBackend(),
}
return _backends
def get_bottle_backend(
name: str | None = None,
*,
prompt: bool = True,
) -> BottleBackend[Any, Any]:
"""Resolve the bottle backend.
`name` precedence:
1. explicit arg (e.g. resume passes the recorded backend name)
2. BOT_BOTTLE_BACKEND env var
3. auto-selection: VM backend first, docker fallback with prompt
`prompt` controls whether auto-selection may block on an interactive
[i/d/q] prompt when falling back to docker. Pass `prompt=False` in
non-interactive contexts (headless launches, CI) so the call dies
with an actionable message instead of hanging.
Dies with a pointer at the known backends if the chosen name
isn't implemented."""
resolved = name or os.environ.get("BOT_BOTTLE_BACKEND")
if resolved is None:
resolved = _auto_select_backend(prompt=prompt)
backends = _get_backends()
if resolved not in backends:
known = ", ".join(sorted(backends))
die(f"unknown backend {resolved!r}; known backends: {known}")
return backends[resolved]
def _platform_vm_suggestion() -> str:
"""Platform-appropriate VM backend name for install suggestions."""
return "macos-container" if sys.platform == "darwin" else "firecracker"
def _print_vm_install_instructions() -> None:
"""Print platform-appropriate VM backend install instructions to stderr."""
vm = _platform_vm_suggestion()
if vm == "macos-container":
info("Install Apple Container: https://github.com/apple/container/releases")
info("Then start the service: container system start")
else:
info("Install Firecracker: https://github.com/firecracker-microvm/firecracker/releases")
info("Configure the host: ./cli.py backend setup")
def _auto_select_backend(prompt: bool = True) -> str:
"""Tier-1 / tier-2 backend auto-selection.
Tier 1: VM backend macos-container on macOS when Apple Container is
installed; firecracker on KVM-capable Linux even before the binary is
present (its preflight prints an install pointer).
Tier 2: docker, with a security warning and an interactive prompt.
When `prompt=False` (headless / CI), dies with an actionable message
instead of blocking on a TTY read. When docker is also absent, prints
VM install instructions and exits.
"""
# --- Tier 1: VM backend -----------------------------------------
if has_backend("macos-container"):
return "macos-container"
# A KVM-capable Linux host defaults to firecracker even when the
# `firecracker` binary isn't installed yet: selecting it here routes
# start through firecracker's preflight, which prints an install
# pointer, instead of silently falling back to docker.
from .firecracker import FirecrackerBottleBackend
if FirecrackerBottleBackend.is_host_capable():
return "firecracker"
# --- Tier 2: docker fallback ------------------------------------
if not has_backend("docker"):
info("No backend available on this host.")
_print_vm_install_instructions()
die("no backend available; install a VM backend and re-run")
vm = _platform_vm_suggestion()
warn(
"docker is less secure than VM backends — "
"containers share the host kernel."
)
if not prompt:
die(
f"no VM backend available; set BOT_BOTTLE_BACKEND=docker to proceed "
f"with docker, or install the {vm!r} backend."
)
sys.stderr.write(
f"bot-bottle: For better isolation, install the {vm!r} backend.\n"
f" [i] show {vm} install instructions and exit\n"
" [d] use docker anyway\n"
" [q] quit\n"
"bot-bottle: choice [i/d/q]: "
)
sys.stderr.flush()
reply = read_tty_line().strip().lower()
if reply == "d":
return "docker"
if reply == "i":
_print_vm_install_instructions()
die("not proceeding with docker; install a VM backend or set BOT_BOTTLE_BACKEND=docker")
def known_backend_names() -> tuple[str, ...]:
"""Sorted tuple of all backend keys in `_get_backends()`. Used by
argparse (`--backend` choices) and the dashboard's backend
picker."""
return tuple(sorted(_get_backends()))
def has_backend(name: str) -> bool:
"""Whether the named backend's runtime prerequisites are
available on the current host. Cross-backend callers (list,
cleanup) skip unavailable backends so a docker-only host
doesn't fail when the firecracker backend isn't usable,
and vice versa.
Returns False for unknown names so callers can pass
arbitrary input without separate validation."""
backends = _get_backends()
if name not in backends:
return False
return backends[name].is_available()
def is_backend_available(name: str) -> bool:
"""Cheap availability check: is the backend's binary on PATH?
Suitable for cleanup enumeration and auto-selection does NOT probe
the daemon or network pool. Use is_backend_ready() for a full
readiness check before launching tests."""
return has_backend(name)
def is_backend_ready(name: str, *, quiet: bool = False) -> bool:
"""Full readiness check: passes all of the backend's status() checks.
When quiet=False the backend prints diagnostic output explaining what
is missing intended for test-suite guards that run at discovery time
so the operator sees a concrete failure reason for each skip.
Returns False for unknown backend names."""
backends = _get_backends()
if name not in backends:
return False
return backends[name].status(quiet=quiet) == BackendStatus.READY
def enumerate_active_agents() -> list[ActiveAgent]:
"""All currently-running agents, across every available
backend. Used by CLI `active` and the dashboard's agents
pane so neither has to know which backends exist. Skips
backends whose `is_available()` reports False.
Sorted by `(started_at, slug)` so the list is stable across
dashboard refresh ticks agents don't shift position while
the operator navigates with arrow keys. ISO 8601 timestamps
sort lexicographically in chronological order; `slug` is the
deterministic tiebreaker. Agents with missing metadata
(`started_at == ""`) sort first."""
out: list[ActiveAgent] = []
backends = _get_backends()
for name in sorted(backends):
if not backends[name].is_available():
continue
out.extend(backends[name].enumerate_active())
out.sort(key=lambda a: (a.started_at, a.slug))
return out
-20
View File
@@ -7,8 +7,6 @@ from __future__ import annotations
import hashlib
import os
import ssl
import time
from collections.abc import Callable
from pathlib import Path
from typing import TYPE_CHECKING
@@ -17,24 +15,6 @@ from ..log import die, info
if TYPE_CHECKING:
from ..egress import EgressPlan
_CA_POLL_INTERVAL = 0.5
def poll_ca_cert(fetch: Callable[[], str | None], *, timeout: float) -> str:
"""Poll `fetch` until it returns a non-empty PEM string or `timeout` expires.
`fetch` should return the PEM on success and `None` (or empty string) when
the cert is not yet available. Raises `TimeoutError` if the cert never
appears within `timeout` seconds."""
deadline = time.monotonic() + timeout
while True:
result = fetch()
if result:
return result
if time.monotonic() >= deadline:
raise TimeoutError(f"CA cert not available after {timeout:g}s")
time.sleep(_CA_POLL_INTERVAL)
# Debian-family CA layout, shared by every backend (all guest images
# are Debian-family). AGENT_CA_PATH is the source path that
+104 -9
View File
@@ -1,15 +1,110 @@
"""bot-bottle CLI package.
"""Main CLI dispatcher.
The subcommand handlers live in `commands/` and are assembled into the
COMMANDS registry by `commands/__init__.py`; the dispatcher `main()` lives
in `__main__.py`. They are re-exported here so `bot_bottle.cli.main`,
`bot_bottle.cli.COMMANDS`, and `bot_bottle.cli.NO_MIGRATION_COMMANDS` stay
importable (the repo-root `cli.py` entry point and the tests use them).
Commands: backend, cleanup, commit, edit, info, init, list, resume, start, supervise
"""
from __future__ import annotations
from .__main__ import main
from .commands import COMMANDS, NO_MIGRATION_COMMANDS
import sys
__all__ = ["main", "COMMANDS", "NO_MIGRATION_COMMANDS"]
from ..errors import MissingEnvVarError
from ..log import Die, die, error
from ..manifest import ManifestError
from ..store_manager import StoreManager
from ._common import PROG
from . import list as _list_mod
from .backend import cmd_backend
from .cleanup import cmd_cleanup
from .commit import cmd_commit
from .edit import cmd_edit
from .info import cmd_info
from .init import cmd_init
from .resume import cmd_resume
from .start import cmd_start
from .supervise import cmd_supervise
cmd_list = _list_mod.cmd_list
COMMANDS = {
"backend": cmd_backend,
"cleanup": cmd_cleanup,
"commit": cmd_commit,
"edit": cmd_edit,
"info": cmd_info,
"init": cmd_init,
"list": cmd_list,
"resume": cmd_resume,
"start": cmd_start,
"supervise": cmd_supervise,
}
def usage() -> None:
sys.stderr.write(f"usage: {PROG} <command> [args...]\n\n")
sys.stderr.write("Commands:\n")
sys.stderr.write(" backend set up / check / undo a backend's host prerequisites (setup|status|teardown)\n")
sys.stderr.write(" cleanup stop and remove all active bot-bottle containers\n")
sys.stderr.write(" commit snapshot a running bottle's container state to a Docker image\n")
sys.stderr.write(" edit open an agent in vim for editing\n")
sys.stderr.write(" info print env, skills, and prompt details for a named agent\n")
sys.stderr.write(" init interactively create a new agent and add it to bot-bottle.json\n")
sys.stderr.write(" list list available agents or active containers\n")
sys.stderr.write(
" resume re-launch a bottle by its identity "
"(continues state from PRD 0016)\n"
)
sys.stderr.write(
" start boot a container for a named agent and "
"attach an interactive session\n"
)
sys.stderr.write(
" supervise view + approve/modify/reject pending supervise "
"proposals (PRD 0013)\n\n"
)
sys.stderr.write(f"Run '{PROG} <command> --help' for command-specific usage.\n")
def main(argv: list[str] | None = None) -> int:
if argv is None:
argv = sys.argv[1:]
if not argv:
usage()
return 2
command = argv[0]
rest = argv[1:]
if command in ("-h", "--help"):
usage()
return 0
handler = COMMANDS.get(command)
if handler is None:
usage()
die(f"unknown command: {command}")
mgr = StoreManager.instance()
if not mgr.is_migrated():
sys.stderr.write("bot-bottle: database schema is out of date\n")
sys.stderr.write("Migrate now? [y/N] ")
sys.stderr.flush()
try:
answer = sys.stdin.readline().strip().lower()
except EOFError:
answer = ""
if answer != "y":
error("migration required — re-run and confirm to migrate")
return 1
mgr.migrate()
try:
return handler(rest) or 0
except MissingEnvVarError as e:
error(str(e))
return 1
except ManifestError as e:
error(str(e))
return 1
except Die as e:
return e.code if isinstance(e.code, int) else 1
except KeyboardInterrupt:
return 130
if __name__ == "__main__":
sys.exit(main())
-65
View File
@@ -1,65 +0,0 @@
"""Entry point + dispatcher for `python -m bot_bottle.cli`.
Maps `bot-bottle <command>` to its handler in the COMMANDS registry
(`bot_bottle.cli.commands`), enforces the schema-migration gate, and
translates handler exceptions into process exit codes. The repo-root
`cli.py` is the usual way in; this makes the package runnable too, so the
CLI works from an installed copy where there is no `cli.py` on disk.
"""
from __future__ import annotations
import sys
from ..errors import MissingEnvVarError
from ..log import Die, die, error
from ..manifest import ManifestError
from ..orchestrator.store.store_manager import StoreManager
from .commands import COMMANDS, NO_MIGRATION_COMMANDS
from .commands.help import cmd_help
def main(argv: list[str] | None = None) -> int:
if argv is None:
argv = sys.argv[1:]
if not argv:
cmd_help()
return 2
command = argv[0]
rest = argv[1:]
if command in ("-h", "--help"):
cmd_help()
return 0
handler = COMMANDS.get(command)
if handler is None:
cmd_help()
die(f"unknown command: {command}")
mgr = StoreManager.instance()
if command not in NO_MIGRATION_COMMANDS and not mgr.is_migrated():
sys.stderr.write("bot-bottle: database schema is out of date\n")
sys.stderr.write("Migrate now? [y/N] ")
sys.stderr.flush()
try:
answer = sys.stdin.readline().strip().lower()
except EOFError:
answer = ""
if answer != "y":
error("migration required — re-run and confirm to migrate")
return 1
mgr.migrate()
try:
return handler(rest) or 0
except MissingEnvVarError as e:
error(str(e))
return 1
except ManifestError as e:
error(str(e))
return 1
except Die as e:
return e.code if isinstance(e.code, int) else 1
except KeyboardInterrupt:
return 130
if __name__ == "__main__":
sys.exit(main())
+20
View File
@@ -0,0 +1,20 @@
"""Shared constants and tty helper for cli subcommands."""
from __future__ import annotations
import os
import sys
from pathlib import Path
PROG = "cli.py"
USER_CWD = os.getcwd()
REPO_DIR = str(Path(__file__).resolve().parent.parent.parent)
def read_tty_line() -> str:
"""Mirror `IFS= read -r REPLY </dev/tty`. Falls back to stdin."""
try:
with open("/dev/tty", "r", encoding="utf-8") as tty:
return tty.readline().rstrip("\n")
except OSError:
return sys.stdin.readline().rstrip("\n")
@@ -15,8 +15,8 @@ from __future__ import annotations
import argparse
from ...backend import get_bottle_backend, known_backend_names
from ..constants import PROG
from ..backend import get_bottle_backend, known_backend_names
from ._common import PROG
def cmd_backend(args: list[str]) -> int:
@@ -21,20 +21,16 @@ from __future__ import annotations
import sys
from ...backend import get_bottle_backend, has_backend, known_backend_names
from ...log import info
from ...util import read_tty_line
from ..backend import get_bottle_backend, known_backend_names
from ..log import info
from ._common import read_tty_line
def cmd_cleanup(_argv: list[str]) -> int:
# Order: stable backend iteration so the y/N output is
# deterministic across runs. Skip backends whose runtime
# isn't available on this host so e.g. macos-container
# doesn't error on Linux.
# deterministic across runs.
plans = [
(name, get_bottle_backend(name))
for name in known_backend_names()
if has_backend(name)
(name, get_bottle_backend(name)) for name in known_backend_names()
]
prepared = [(name, b, b.prepare_cleanup()) for name, b in plans]
-58
View File
@@ -1,58 +0,0 @@
"""CLI subcommand registry.
One module per `bot-bottle <command>`, each exposing a `cmd_<name>(argv)`
handler. This package `__init__` maps command names to their handlers
**lazily**: a short-lived CLI run dispatches exactly one command, so
importing all twelve handlers (and their transitive deps backend,
manifest, orchestrator, ) up front is wasted work. Each COMMANDS value is
a thin wrapper that imports its handler's module on first call. Shared CLI
helpers (`constants`, `tui`) stay one level up in the `cli` package.
"""
from __future__ import annotations
from importlib import import_module
from typing import Callable
# command name -> "<submodule>:<handler attr>". Kept as strings so building
# the registry imports nothing; the module loads only when dispatched.
_HANDLERS: dict[str, str] = {
"active": "active:cmd_active",
"backend": "backend:cmd_backend",
"cleanup": "cleanup:cmd_cleanup",
"commit": "commit:cmd_commit",
"edit": "edit:cmd_edit",
"help": "help:cmd_help",
"init": "init:cmd_init",
"list": "list:cmd_list",
"login": "login:cmd_login",
"resume": "resume:cmd_resume",
"start": "start:cmd_start",
"supervise": "supervise:cmd_supervise",
}
def _lazy(spec: str) -> Callable[[list[str]], "int | None"]:
"""Wrap a `<module>:<attr>` handler so its module is imported only when
the command is actually dispatched, not when the registry is built."""
module, attr = spec.split(":")
def run(argv: list[str]) -> "int | None":
handler = getattr(import_module(f".{module}", __name__), attr)
return handler(argv)
run.__name__ = attr
return run
COMMANDS = {name: _lazy(spec) for name, spec in _HANDLERS.items()}
# Commands that manage host prerequisites (or are otherwise store-free) and
# must run before — or without — a migrated DB. `backend` provisions/probes
# the host (TAP pool, /dev/kvm, firecracker) and never opens the store, so
# gating it on the schema breaks preflight on a fresh CI runner where stdin
# isn't a TTY and the migration prompt can't be answered. `help` and `login`
# likewise never touch the store.
NO_MIGRATION_COMMANDS = frozenset({"backend", "help", "login"})
__all__ = ["COMMANDS", "NO_MIGRATION_COMMANDS"]
-37
View File
@@ -1,37 +0,0 @@
"""help: print the top-level command list and usage.
Rendered by the dispatcher for the `help` command and for its
`-h`/`--help`, no-args, and unknown-command fallbacks. The per-command
summaries live here; keep them in sync with the COMMANDS table in
`bot_bottle.cli`.
"""
from __future__ import annotations
import sys
from ..constants import PROG
def cmd_help(argv: list[str] | None = None) -> int:
"""Write the top-level usage + command list to stderr. Returns 0;
the dispatcher chooses the process exit code per entry path (0 for an
explicit `help`/`-h`, 2 for the bare no-args usage error)."""
del argv # help takes no arguments
w = sys.stderr.write
w(f"usage: {PROG} <command> [args...]\n\n")
w("Commands:\n")
w(" active list currently-running bot-bottle bottles\n")
w(" backend set up / check / undo a backend's host prerequisites (setup|status|teardown)\n")
w(" cleanup stop and remove all active bot-bottle containers\n")
w(" commit snapshot a running bottle's container state to a Docker image\n")
w(" edit open an agent in vim for editing\n")
w(" help show this command list\n")
w(" init interactively create a new agent and add it to bot-bottle.json\n")
w(" list list available agents from bot-bottle.json\n")
w(" login register this host with a bot-bottle console\n")
w(" resume re-launch a bottle by its identity (continues state from PRD 0016)\n")
w(" start boot a container for a named agent and attach an interactive session\n")
w(" supervise view + approve/modify/reject pending supervise proposals (PRD 0013)\n\n")
w(f"Run '{PROG} <command> --help' for command-specific usage.\n")
return 0
-21
View File
@@ -1,21 +0,0 @@
"""list: list available agents."""
from __future__ import annotations
import os
import sys
from ...manifest import ManifestIndex
from ..constants import PROG
def cmd_list(argv: list[str]) -> int:
if argv and argv[0] in ("-h", "--help"):
sys.stderr.write(f"usage: {PROG} list\n")
sys.stderr.write("\nList all available agents from bot-bottle.json.\n")
return 0
manifest = ManifestIndex.resolve(os.getcwd())
for name in manifest.all_agent_names:
print(name)
return 0
-168
View File
@@ -1,168 +0,0 @@
"""bb login — register this host with a bot-bottle console.
Opens a device-authorization flow against the target console, waits for the
operator to approve, then writes access and refresh tokens to
~/.bot-bottle/console.json (or $BOT_BOTTLE_ROOT/console.json).
Usage:
bb login [--console-url URL] [--label LABEL]
Flags:
--console-url URL Target console URL (overrides BB_CONSOLE_URL env var)
--label LABEL Host label shown in the console (default: hostname)
"""
from __future__ import annotations
import json
import os
import socket
import sys
import tempfile
import time
import urllib.error
import urllib.request
from pathlib import Path
from typing import Any
from ...paths import bot_bottle_root
_CONSOLE_URL_ENV = "BB_CONSOLE_URL"
_POLL_SLEEP = 2 # seconds between polls; matches console's poll_interval default
def _usage() -> None:
sys.stderr.write(
"usage: bb login [--console-url URL] [--label LABEL]\n"
"\n"
"Options:\n"
" --console-url URL Console base URL (or BB_CONSOLE_URL env var)\n"
" --label LABEL Host label shown in the console (default: hostname)\n"
)
def _flag(argv: list[str], name: str) -> str | None:
for i, arg in enumerate(argv):
if arg == name and i + 1 < len(argv):
return argv[i + 1]
if arg.startswith(f"{name}="):
return arg[len(name) + 1:]
return None
def _post(url: str, payload: dict[str, Any]) -> dict[str, Any]:
data = json.dumps(payload).encode()
req = urllib.request.Request(
url, data=data, headers={"Content-Type": "application/json"}
)
with urllib.request.urlopen(req, timeout=10) as resp:
return json.loads(resp.read())
def _get(url: str) -> tuple[int, dict[str, Any]]:
req = urllib.request.Request(url)
try:
with urllib.request.urlopen(req, timeout=10) as resp:
return resp.status, json.loads(resp.read())
except urllib.error.HTTPError as e:
return e.code, {}
def _save_credentials(
console_url: str, host_id: str, access_token: str, refresh_token: str
) -> Path:
path = bot_bottle_root() / "console.json"
path.parent.mkdir(parents=True, exist_ok=True)
content = (
json.dumps(
{
"url": console_url,
"host_id": host_id,
"access_token": access_token,
"refresh_token": refresh_token,
},
indent=2,
)
+ "\n"
)
fd, tmp_path_str = tempfile.mkstemp(dir=path.parent, prefix=".console-")
tmp = Path(tmp_path_str)
try:
tmp.chmod(0o600)
with os.fdopen(fd, "w") as f:
f.write(content)
os.replace(tmp, path)
except OSError:
try:
tmp.unlink()
except OSError:
pass
raise
return path
def cmd_login(argv: list[str]) -> int:
if "--help" in argv or "-h" in argv:
_usage()
return 0
console_url = _flag(argv, "--console-url") or os.environ.get(_CONSOLE_URL_ENV)
if not console_url:
sys.stderr.write(
"bb login: --console-url or BB_CONSOLE_URL is required\n"
)
return 1
console_url = console_url.rstrip("/")
label = _flag(argv, "--label") or socket.gethostname()
try:
resp = _post(f"{console_url}/api/v1/hosts/authorize", {"label": label})
except (OSError, ValueError) as exc:
sys.stderr.write(f"bb login: failed to start authorization: {exc}\n")
return 1
device_code = resp["device_code"]
user_code = resp["user_code"]
expires_in = resp.get("expires_in", 300)
poll_sleep = max(1, min(int(resp.get("poll_interval", _POLL_SLEEP)), 60))
sys.stderr.write(
f"\nOpen this URL in your browser to authorize this host:\n\n"
f" {console_url}/hosts/authorize?code={user_code}\n\n"
f"Waiting for approval"
)
deadline = time.monotonic() + expires_in
while time.monotonic() < deadline:
sys.stderr.write(".")
sys.stderr.flush()
time.sleep(poll_sleep)
try:
code, result = _get(
f"{console_url}/api/v1/hosts/authorize/{device_code}"
)
except (OSError, ValueError):
continue
if code == 410:
break
st = result.get("status")
if st == "approved":
sys.stderr.write("\n\nApproved.\n")
path = _save_credentials(
console_url,
result["host_id"],
result["access_token"],
result["refresh_token"],
)
sys.stderr.write(f"Credentials saved to {path}\n")
return 0
if st == "denied":
sys.stderr.write("\n\nDenied by operator.\n")
return 1
sys.stderr.write("\n\nAuthorization timed out.\n")
return 1
@@ -12,12 +12,12 @@ from __future__ import annotations
import argparse
from ...backend import enumerate_active_agents
from ...backend.freeze import CommitCancelled, get_freezer
from ...bottle_state import read_metadata
from ...log import die
from ..constants import PROG
from .. import tui
from ..backend import enumerate_active_agents
from ..backend.freeze import CommitCancelled, get_freezer
from ..bottle_state import read_metadata
from ..log import die
from ._common import PROG
from . import tui
def cmd_commit(argv: list[str]) -> int:
@@ -27,7 +27,7 @@ def cmd_commit(argv: list[str]) -> int:
nargs="?",
default=None,
help=(
"bottle slug from `cli.py active` "
"bottle slug from `cli.py list active` "
"(omit to pick interactively)"
),
)
-8
View File
@@ -1,8 +0,0 @@
"""Shared CLI constants.
Kept as a leaf module (imports nothing from the `cli` package) so both the
dispatcher (`cli/__init__.py`) and the command modules it imports can share
`PROG` without a circular import.
"""
PROG = "cli.py"
@@ -7,8 +7,8 @@ import json
import os
from pathlib import Path
from ...log import die
from ..constants import PROG
from ..log import die
from ._common import PROG, USER_CWD
def cmd_edit(argv: list[str]) -> int:
@@ -20,7 +20,7 @@ def cmd_edit(argv: list[str]) -> int:
if args.scope == "user":
target_file = Path(os.environ["HOME"]) / "bot-bottle.json"
else:
target_file = Path(os.getcwd()) / "bot-bottle.json"
target_file = Path(USER_CWD) / "bot-bottle.json"
if not target_file.is_file():
die(f"{target_file} does not exist")
+49
View File
@@ -0,0 +1,49 @@
"""info: print env, skills, and prompt details for a named agent."""
from __future__ import annotations
import argparse
from ..log import info
from ..manifest import ManifestIndex
from ._common import PROG, USER_CWD
def cmd_info(argv: list[str]) -> int:
parser = argparse.ArgumentParser(prog=f"{PROG} info", add_help=True)
parser.add_argument("name", help="agent name defined in bot-bottle.json")
args = parser.parse_args(argv)
names = ManifestIndex.resolve(USER_CWD)
names.require_agent(args.name)
manifest = names.load_for_agent(args.name)
agent = manifest.agent
bottle = manifest.bottle
env_names = list(bottle.env.keys())
prompt_first_line = agent.prompt.splitlines()[0] if agent.prompt else ""
print()
info(f"agent : {args.name}")
info(f"env (names only): {', '.join(env_names) if env_names else '(none)'}")
info(f"skills : {' '.join(agent.skills) if agent.skills else '(none)'}")
info(
f"prompt : {len(agent.prompt)} chars; "
f"first line: {prompt_first_line or '(empty)'}"
)
info(f"bottle : {agent.bottle}")
identity = manifest.git_identity_summary()
if identity:
info(f" git identity : {identity}")
if bottle.git:
for e in bottle.git:
info(
f" git remote : {e.Name} -> {e.Upstream} "
f"(IdentityFile={e.IdentityFile})"
)
if e.KnownHostKey:
info(f" KnownHostKey: {e.KnownHostKey}")
else:
info(" git remotes : (none)")
print()
return 0
@@ -10,9 +10,8 @@ import sys
from pathlib import Path
from typing import Any
from ...log import die, info, warn
from ..constants import PROG
from ...util import read_tty_line
from ..log import die, info, warn
from ._common import PROG, USER_CWD, read_tty_line
def cmd_init(argv: list[str]) -> int:
@@ -23,7 +22,7 @@ def cmd_init(argv: list[str]) -> int:
if args.scope == "user":
target_file = Path(os.environ["HOME"]) / "bot-bottle.json"
else:
target_file = Path(os.getcwd()) / "bot-bottle.json"
target_file = Path(USER_CWD) / "bot-bottle.json"
print(file=sys.stderr)
info(f"bot-bottle init — adding a new agent to {target_file}")
@@ -1,12 +1,14 @@
"""active: list currently-running bot-bottle bottles."""
"""list: list available agents or active bottles."""
from __future__ import annotations
import argparse
import os
import sys
from ...backend import enumerate_active_agents
from ..constants import PROG
from ..backend import enumerate_active_agents
from ..manifest import ManifestIndex
from ._common import PROG, USER_CWD
_ANSI_COLOR_CODES: dict[str, str] = {
"red": "\033[91m",
@@ -32,13 +34,20 @@ def _ansi_label(text: str, color: str) -> str:
return f"{code}{text}{_ANSI_RESET}"
def cmd_active(argv: list[str]) -> int:
if argv and argv[0] in ("-h", "--help"):
sys.stderr.write(f"usage: {PROG} active\n")
sys.stderr.write("\nList all currently-running bot-bottle bottles.\n")
sys.stderr.write("Output: <backend>\\t<slug>\\t<label>\\t<services>\n")
def cmd_list(argv: list[str]) -> int:
parser = argparse.ArgumentParser(prog=f"{PROG} list", add_help=True)
parser.add_argument("scope", choices=["available", "active"])
args = parser.parse_args(argv)
if args.scope == "available":
manifest = ManifestIndex.resolve(USER_CWD)
for name in manifest.all_agent_names:
print(name)
return 0
# `active` enumerates every backend (docker, firecracker,
# macos-container) so non-docker bottles aren't hidden behind
# the env var.
active = enumerate_active_agents()
if not active:
print("no active bot-bottle bottles", file=sys.stderr)
@@ -15,13 +15,12 @@ to bring up the replacement from the recorded state.
from __future__ import annotations
import argparse
import os
from ...backend import BottleSpec
from ...bottle_state import read_metadata
from ...log import die
from ...manifest import ManifestIndex
from ..constants import PROG
from ..backend import BottleSpec
from ..bottle_state import read_metadata
from ..log import die
from ..manifest import ManifestIndex
from ._common import PROG, USER_CWD
from .start import _launch_bottle
@@ -41,14 +40,14 @@ def cmd_resume(argv: list[str]) -> int:
f"check ~/.bot-bottle/state/ or run `cli.py start` to create a new bottle"
)
manifest = ManifestIndex.resolve(os.getcwd())
manifest = ManifestIndex.resolve(USER_CWD)
manifest.require_agent(metadata.agent_name)
spec = BottleSpec(
manifest=manifest,
agent_name=metadata.agent_name,
copy_cwd=metadata.copy_cwd,
user_cwd=metadata.cwd or os.getcwd(),
user_cwd=metadata.cwd or USER_CWD,
identity=metadata.identity,
bottle_names=tuple(metadata.bottle_names),
)
@@ -14,7 +14,6 @@ the private orchestrator `_launch_bottle`.
from __future__ import annotations
import argparse
import io
import os
import shutil
import sys
@@ -22,26 +21,25 @@ import tempfile
from pathlib import Path
from typing import Callable
from ...agent_provider import get_provider, runtime_for
from ...backend import (
from ..agent_provider import get_provider, runtime_for
from ..backend import (
Bottle,
BottleSpec,
enumerate_active_agents,
get_bottle_backend,
known_backend_names,
)
from ...backend.docker import util as docker_mod
from ...backend.docker.bottle_plan import DockerBottlePlan
from ...bottle_state import (
from ..backend.docker import util as docker_mod
from ..backend.docker.bottle_plan import DockerBottlePlan
from ..bottle_state import (
cleanup_state,
is_preserved,
mark_preserved,
)
from ...image_cache import StaleImageError
from ...log import info, die
from ...manifest import Manifest, ManifestIndex
from ..constants import PROG
from ...util import read_tty_line
from .. import tui
from ..log import info, die
from ..manifest import Manifest, ManifestIndex
from ._common import PROG, USER_CWD, read_tty_line
from . import tui
def cmd_start(argv: list[str]) -> int:
@@ -59,6 +57,15 @@ def cmd_start(argv: list[str]) -> int:
"into a cached layer."
),
)
parser.add_argument(
"--backend",
choices=known_backend_names(),
default=None,
help=(
"backend to launch the bottle on (default: $BOT_BOTTLE_BACKEND "
"or host auto-selection). Overrides the env var when set."
),
)
parser.add_argument(
"--headless",
action="store_true",
@@ -67,14 +74,6 @@ def cmd_start(argv: list[str]) -> int:
"skip all prompts. For orchestrators, CI, and webhooks."
),
)
parser.add_argument(
"--cached-images",
action="store_true",
help=(
"quickstart with existing local agent and sidecar images; "
"only valid with --headless"
),
)
parser.add_argument(
"--bottle",
action="append",
@@ -107,8 +106,6 @@ def cmd_start(argv: list[str]) -> int:
help="agent name defined in bot-bottle.json (omit to pick interactively)",
)
args = parser.parse_args(argv)
if args.cached_images and not args.headless:
die("--cached-images is only supported with --headless")
dry_run = args.dry_run or os.environ.get("BOT_BOTTLE_DRY_RUN") == "1"
if args.no_cache or os.environ.get("BOT_BOTTLE_NO_CACHE") == "1":
@@ -117,11 +114,12 @@ def cmd_start(argv: list[str]) -> int:
# threading a no_cache field through every backend's plan dataclass.
os.environ["BOT_BOTTLE_NO_CACHE"] = "1"
manifest = ManifestIndex.resolve(os.getcwd())
manifest = ManifestIndex.resolve(USER_CWD)
backend_name: str | None = args.backend
if args.headless:
return _start_headless(
manifest, args, dry_run=dry_run
manifest, args, dry_run=dry_run, backend_name=backend_name
)
agent_name: str | None = args.name
@@ -160,23 +158,19 @@ def cmd_start(argv: list[str]) -> int:
label, color = tui.name_color_modal(default_label=agent_name)
label, color = _resolve_unique_label(label, color)
image_policy = _select_image_policy()
if image_policy is None:
return 0
spec = BottleSpec(
manifest=manifest,
agent_name=agent_name,
copy_cwd=args.cwd,
user_cwd=os.getcwd(),
user_cwd=USER_CWD,
label=label,
color=color,
bottle_names=bottle_names,
image_policy=image_policy,
)
return _launch_bottle(
spec,
dry_run=dry_run,
backend_name=backend_name,
)
@@ -188,6 +182,7 @@ def _start_headless(
args: argparse.Namespace,
*,
dry_run: bool,
backend_name: str | None,
) -> int:
"""Non-interactive launch path for orchestrators / CI / webhooks.
@@ -197,16 +192,6 @@ def _start_headless(
path, so the agent still execs on the inherited stdio/PTY an
orchestrator allocates that PTY and relays it to its
desktop/mobile clients."""
try:
stdin_fd = sys.stdin.fileno()
except io.UnsupportedOperation:
stdin_fd = -1
if not os.isatty(stdin_fd):
die(
"--headless requires a PTY on stdin; run via:\n"
" script -q /dev/null ./cli.py start ..."
)
agent_name = args.name
if not agent_name:
die("--headless requires an agent name: ./cli.py start <agent> --headless")
@@ -236,16 +221,16 @@ def _start_headless(
manifest=manifest,
agent_name=agent_name,
copy_cwd=args.cwd,
user_cwd=os.getcwd(),
user_cwd=USER_CWD,
label=label,
color=args.color or "",
bottle_names=bottle_names,
headless=True,
image_policy="cached" if args.cached_images else "fresh",
)
return _launch_bottle(
spec,
dry_run=dry_run,
backend_name=backend_name,
assume_yes=True,
headless_prompt_text=prompt,
)
@@ -283,18 +268,15 @@ def prepare_with_preflight(
injected callable, prompt y/N via the injected callable.
`backend_name` selects which backend prepares the plan
(`None` `$BOT_BOTTLE_BACKEND` host auto-selection).
When `spec.headless` is True the docker-fallback prompt is suppressed:
auto-selection dies with an actionable message rather than blocking
on a TTY read (which would hang CI, webhook dispatch, and orchestrators).
(`None` `$BOT_BOTTLE_BACKEND` host auto-selection). The CLI
passes whatever `--backend` resolved to.
Returns `(plan, identity)`. `plan` is None on dry-run or
operator-N, but `identity` is set as soon as `backend.prepare`
returns so callers can reap the prepare-time state dir via
`settle_state(identity)` in their finally exactly the existing
semantics."""
backend = get_bottle_backend(backend_name, prompt=not spec.headless)
backend = get_bottle_backend(backend_name)
plan = backend.prepare(spec, stage_dir=stage_dir)
identity = _identity_from_plan(plan)
@@ -381,8 +363,8 @@ def _peek_agent_bottle(manifest: ManifestIndex, agent_name: str) -> str:
return manifest.agents[agent_name].bottle
return ""
from ...manifest.loader import scan_agent_names
from ...yaml_subset import YamlSubsetError, parse_frontmatter
from ..manifest_loader import scan_agent_names
from ..yaml_subset import YamlSubsetError, parse_frontmatter
home_agents = scan_agent_names(manifest.home_md / "agents")
cwd_agents: dict[str, Path] = {}
@@ -424,13 +406,6 @@ def _text_prompt_yes() -> bool:
return reply in ("y", "Y", "yes", "YES")
def _select_image_policy() -> str | None:
return tui.filter_select(
["fresh", "cached"],
title="Select image startup mode",
)
def _text_render_preflight():
def _render(plan: DockerBottlePlan, backend_name: str) -> None:
print(file=sys.stderr)
@@ -450,7 +425,7 @@ def _bottle_lineage(manifest: ManifestIndex) -> dict[str, str]:
if not bottles_dir.is_dir():
return {}
from ...yaml_subset import YamlSubsetError, parse_frontmatter
from ..yaml_subset import YamlSubsetError, parse_frontmatter
extends_of: dict[str, str] = {}
for path in bottles_dir.glob("*.md"):
@@ -536,8 +511,6 @@ def _manifest_to_yaml(manifest: Manifest) -> str:
lines.append(f" scheme: {r.AuthScheme}")
lines.append(f" supervise: {'true' if bottle.supervise else 'false'}")
if bottle.nested_containers:
lines.append(" nested_containers: true")
return "\n".join(lines)
@@ -575,15 +548,6 @@ def _launch_bottle(
return 0
backend = get_bottle_backend(backend_name)
try:
backend.prelaunch_checks(plan)
except StaleImageError as exc:
if assume_yes:
die(str(exc))
sys.stderr.write(f"bot-bottle: {exc}\nLaunch anyway? [y/N] ")
sys.stderr.flush()
if read_tty_line() not in ("y", "Y", "yes", "YES"):
return 0
with backend.launch(plan) as bottle:
agent_provider_template = getattr(plan, "agent_provider_template", "claude")
extra_args: tuple[str, ...] = ()
@@ -602,6 +566,10 @@ def _launch_bottle(
f"session ended (exit {exit_code}); "
f"container {bottle.name} will be removed"
)
# While the container is still alive: always snapshot the
# transcript and — if the agent exited non-zero — mark
# the state for preservation. This picks up crashes /
# Ctrl-Cs / OOM kills before cleanup removes the state dir.
if agent_provider_template == "claude":
capture_claude_session_state(identity, exit_code)
return 0
@@ -19,22 +19,22 @@ from dataclasses import dataclass
from datetime import datetime, timezone
from pathlib import Path
from ...paths import bot_bottle_root
from ...log import Die, error, info
from ...orchestrator.client import (
from ..paths import bot_bottle_root
from ..log import Die, error, info
from ..orchestrator.client import (
OrchestratorClient,
OrchestratorClientError,
discover_orchestrator_url,
)
from ...supervisor.types import (
from ..supervise import (
Proposal,
TOOL_EGRESS_ALLOW,
TOOL_EGRESS_BLOCK,
TOOL_GITLEAKS_ALLOW,
TOOL_EGRESS_TOKEN_ALLOW,
)
from ..constants import PROG
from ._common import PROG
_REFRESH_INTERVAL_MS = 1000
@@ -81,7 +81,7 @@ def _resolve_orchestrator_url() -> str:
try:
return discover_orchestrator_url()
except OrchestratorClientError:
from ...backend import get_bottle_backend
from ..backend import get_bottle_backend
backend = get_bottle_backend()
info(f"no orchestrator control plane running; starting one ({backend.name})…")
return backend.ensure_orchestrator()
-17
View File
@@ -1,17 +0,0 @@
"""Shared wire-protocol constants for gateway-bundled modules.
Single source of truth for values that appear across the egress addon,
git-http backend, supervise server, and git-gate renderer. Importing
from this module instead of duplicating the literals means a rename is
a one-line change and is caught by the type checker at the import site."""
# App-layer identity token header. Delivered as proxy credentials
# (HTTPS_PROXY=http://<bottle_id>:<token>@gw) by launch; the egress
# addon reads and strips it, the supervise server and git-http backend
# read it for attribution, and none of them forward it upstream.
IDENTITY_HEADER = "x-bot-bottle-identity"
# Shared timeout (seconds) for all git-gate subprocess and CGI calls:
# git daemon (--timeout/--init-timeout), the access-hook subprocess in
# git_http_backend, and the git http-backend CGI subprocess.
GIT_GATE_TIMEOUT_SECS = 15
+2 -14
View File
@@ -10,7 +10,7 @@
# Current Node LTS; slim variant keeps the image small while still
# providing apt-get for any future additions.
FROM node:22-trixie-slim
FROM node:22-slim
# Install runtime system deps. claude-code shells out to git for several
# features (status checks, commits, PR creation) — without git in the
@@ -21,14 +21,7 @@ FROM node:22-trixie-slim
# to it) works against egress's bumped TLS without the agent needing
# local DNS.
RUN apt-get update \
&& apt-get install -y --no-install-recommends \
git \
ca-certificates \
curl \
openssh-client \
ripgrep \
iproute2 \
dnsutils \
&& apt-get install -y --no-install-recommends git ca-certificates curl ripgrep iproute2 dnsutils \
&& rm -rf /var/lib/apt/lists/*
# App-specific deps. Python isn't required by claude-code itself
@@ -46,11 +39,6 @@ RUN apt-get update \
RUN npm install -g --no-fund --no-audit @anthropic-ai/claude-code@2.1.172 \
&& npm cache clean --force
# Git reads both ~/.gitconfig and ~/.config/git/config. Keep its XDG config
# path traversable by the non-root runtime user so permission errors do not
# suppress bot-bottle's git-gate insteadOf rules.
RUN install -d -o node -g node -m 755 /home/node/.config /home/node/.config/git
# Run as a non-root user. The node image already provides a `node` user
# (uid 1000) with a home directory, which is where claude-code will write
# its session state.
+7 -33
View File
@@ -23,9 +23,8 @@ from ...agent_provider import (
provider_startup_args,
)
from ...backend.docker import util as docker_mod
from ...egress import CLAUDE_HOST_CREDENTIAL_TOKEN_REF, EgressRoute
from ...egress import EgressRoute
from ...log import die, info, warn
from .claude_auth import claude_host_access_token
if TYPE_CHECKING:
@@ -35,11 +34,6 @@ if TYPE_CHECKING:
_SUPERVISE_MCP_NAME = "supervise"
# App-layer identity token header (mirrors egress_addon / git_http_backend).
_IDENTITY_HEADER = "x-bot-bottle-identity"
# Placeholder stood in for the real identity token in the manual-recovery hint.
# The token is a per-bottle credential, so it must never be rendered into the
# host-side launch log (#476 review); the operator substitutes the value from
# inside the bottle (it rides in the agent's HTTPS_PROXY credentials).
_IDENTITY_TOKEN_PLACEHOLDER = "<bottle-identity-token>"
def _skills_dir(guest_home: str) -> str:
@@ -124,6 +118,7 @@ class ClaudeAgentProvider(AgentProvider):
color: str = "",
provider_settings: dict[str, object] | None = None,
) -> AgentProvisionPlan:
del forward_host_credentials, host_env
resolved_guest_env = dict(guest_env or {})
startup_args = provider_startup_args(provider_settings)
guest_home = self.guest_home
@@ -185,24 +180,13 @@ class ClaudeAgentProvider(AgentProvider):
claude_settings,
f"{guest_home}/.claude/settings.json",
))
provisioned_env: dict[str, str] = {}
if forward_host_credentials:
_host_env = host_env or dict(os.environ)
provisioned_env[CLAUDE_HOST_CREDENTIAL_TOKEN_REF] = (
claude_host_access_token(_host_env)
)
cred_token_ref = (
CLAUDE_HOST_CREDENTIAL_TOKEN_REF if forward_host_credentials
else auth_token
)
egress_routes = (EgressRoute(
host="api.anthropic.com",
auth_scheme="Bearer" if (auth_token or forward_host_credentials) else "",
token_ref=cred_token_ref,
auth_scheme="Bearer" if auth_token else "",
token_ref=auth_token,
),)
hidden_env_names: frozenset[str] = frozenset()
if auth_token or forward_host_credentials:
if auth_token:
env_vars["CLAUDE_CODE_OAUTH_TOKEN"] = "egress-placeholder"
hidden_env_names = frozenset({"CLAUDE_CODE_OAUTH_TOKEN"})
@@ -224,7 +208,6 @@ class ClaudeAgentProvider(AgentProvider):
files=tuple(files),
egress_routes=egress_routes,
hidden_env_names=hidden_env_names,
provisioned_env=provisioned_env,
)
def provision_skills(self, plan: "BottlePlan", bottle: "Bottle") -> None:
@@ -332,20 +315,11 @@ class ClaudeAgentProvider(AgentProvider):
user="node",
)
if r.returncode != 0:
# A placeholder — never the real token — keeps this per-bottle
# credential out of the host launch log (#476 review). The operator
# substitutes it from inside the bottle (it rides in the agent's
# HTTPS_PROXY credentials).
manual_header = (
f" --header {shlex.quote(f'{_IDENTITY_HEADER}: {_IDENTITY_TOKEN_PLACEHOLDER}')}"
if token else ""
)
warn(
f"`claude mcp add supervise` failed (exit {r.returncode}): "
f"{(r.stderr or r.stdout or '').strip()}. Inside the bottle, "
f"register manually (substitute the bottle's identity token) with: "
f"claude mcp add --scope user --transport http "
f"supervise {supervise_url}{manual_header}"
f"register manually with: "
f"claude mcp add --scope user --transport http supervise {supervise_url}"
)
def headless_prompt(self, prompt: str) -> list[str]:
-114
View File
@@ -1,114 +0,0 @@
"""Host Claude auth helpers.
Reads the host's Claude Code credentials and returns only the access
token needed by egress. Does not expose refresh tokens or raw payloads.
Credential storage by platform:
Linux ~/.claude/.credentials.json
macOS macOS Keychain, service "Claude Code-credentials"
(file path is tried first; Keychain is the fallback)
"""
from __future__ import annotations
import json
import os
import subprocess
import sys
from datetime import datetime, timezone
from pathlib import Path
from ...log import die
_KEYCHAIN_SERVICE = "Claude Code-credentials"
def claude_auth_path(host_env: dict[str, str] | None = None) -> Path:
env = os.environ if host_env is None else host_env
home = env.get("HOME")
if home:
return Path(home) / ".claude" / ".credentials.json"
return Path.home() / ".claude" / ".credentials.json"
def _read_keychain() -> dict[str, object] | None:
"""Try the macOS Keychain. Returns parsed JSON dict or None."""
if sys.platform != "darwin":
return None
try:
result = subprocess.run(
["security", "find-generic-password", "-s", _KEYCHAIN_SERVICE, "-w"],
capture_output=True,
text=True,
timeout=10,
)
except (FileNotFoundError, subprocess.TimeoutExpired):
return None
if result.returncode != 0 or not result.stdout.strip():
return None
try:
raw = json.loads(result.stdout.strip())
except json.JSONDecodeError:
return None
return raw if isinstance(raw, dict) else None
def claude_host_access_token(
host_env: dict[str, str] | None = None,
*,
now: datetime | None = None,
) -> str:
path = claude_auth_path(host_env)
raw: dict[str, object] | None = None
if path.is_file():
try:
raw = json.loads(path.read_text())
except (OSError, json.JSONDecodeError) as e:
die(f"claude host credentials: could not read valid JSON at {path}: {e}")
if not isinstance(raw, dict):
die(f"claude host credentials: {path} must contain a JSON object")
else:
raw = _read_keychain()
if raw is None:
die(
f"claude host credentials: auth file missing at {path} and "
f"macOS Keychain lookup for '{_KEYCHAIN_SERVICE}' failed. "
"Run `claude login` on the host or disable "
"agent_provider.forward_host_credentials."
)
oauth = raw.get("claudeAiOauth")
if not isinstance(oauth, dict):
die(
"claude host credentials: claudeAiOauth is missing from credentials. "
"Run `claude login` on the host or disable "
"agent_provider.forward_host_credentials."
)
access_token = oauth.get("accessToken")
if not isinstance(access_token, str) or not access_token:
die(
"claude host credentials: claudeAiOauth.accessToken is missing or empty. "
"Run `claude login` on the host and restart the bottle."
)
# expiresAt is in milliseconds
expires_at = oauth.get("expiresAt")
if isinstance(expires_at, (int, float)):
check_now = now or datetime.now(timezone.utc)
exp_dt = datetime.fromtimestamp(float(expires_at) / 1000.0, timezone.utc)
if exp_dt <= check_now:
die(
"claude host credentials: host Claude access token is expired. "
"Run `claude login` on the host and restart the bottle."
)
return access_token
__all__ = [
"claude_auth_path",
"claude_host_access_token",
]
+2 -10
View File
@@ -3,16 +3,10 @@
# Mirrors the default Claude image shape: Node LTS, git/network tooling,
# non-root node user, and the provider CLI installed for that user.
FROM node:22-trixie-slim
FROM node:22-slim
RUN apt-get update \
&& apt-get install -y --no-install-recommends \
git \
ca-certificates \
curl \
openssh-client \
procps \
ripgrep \
&& apt-get install -y --no-install-recommends git ca-certificates curl procps ripgrep \
&& rm -rf /var/lib/apt/lists/*
# App-specific deps. Python isn't required by codex itself
@@ -23,8 +17,6 @@ RUN apt-get update \
&& apt-get install -y --no-install-recommends python3 python3-pip python3-venv \
&& rm -rf /var/lib/apt/lists/*
RUN install -d -o node -g node -m 755 /home/node/.config /home/node/.config/git
USER node
WORKDIR /home/node
+2 -4
View File
@@ -2,7 +2,7 @@
#
# Node LTS, git/network tooling, and the Pi coding-agent CLI installed globally.
FROM node:22-trixie-slim
FROM node:22-slim
RUN apt-get update \
&& apt-get install -y --no-install-recommends \
@@ -10,7 +10,6 @@ RUN apt-get update \
ca-certificates \
curl \
fd-find \
openssh-client \
ripgrep \
&& ln -s /usr/bin/fdfind /usr/local/bin/fd \
&& rm -rf /var/lib/apt/lists/*
@@ -22,8 +21,7 @@ RUN apt-get update \
RUN npm install -g --ignore-scripts --no-fund --no-audit @earendil-works/pi-coding-agent \
&& npm cache clean --force
RUN install -d -o node -g node -m 755 /home/node/.config /home/node/.config/git \
&& mkdir -p /home/node/.pi/agent \
RUN mkdir -p /home/node/.pi/agent \
/home/node/.pi/context-mode/sessions \
/tmp/pi-subagents-uid-1000 \
&& chown -R node:node /home/node/.pi /tmp \

Some files were not shown because too many files have changed in this diff Show More