Compare commits

..

3 Commits

Author SHA1 Message Date
didericis-claude ea1f022f1a refactor: drop redundant single-parent fast path in _resolve_one_bottle
lint / lint (push) Successful in 1m47s
test / unit (pull_request) Successful in 31s
test / integration (pull_request) Successful in 16s
_fold_parents with one name returns after the first resolve; the
single-element branch was a verbatim copy of the general path.
2026-06-25 07:27:31 +00:00
didericis-claude f8f1bd6f32 fix: resolve pyright reportUnnecessaryIsInstance in _resolve_one_bottle
lint / lint (push) Successful in 1m45s
test / unit (pull_request) Successful in 31s
test / integration (pull_request) Successful in 16s
Validate list entries against object-typed raw_list before narrowing to
list[str], so the isinstance(pname, str) check is not redundant.
2026-06-25 07:08:37 +00:00
didericis-claude b8e2ce0b4d feat: support multiple parents in bottle extends:
lint / lint (push) Failing after 1m48s
test / unit (pull_request) Successful in 33s
test / integration (pull_request) Successful in 18s
Allow extends: to accept a list of bottle names in addition to a plain
string. Parents are resolved independently and folded left-to-right
into a single combined parent before the child is merged on top, so
orthogonal concerns (base env, networking, agent provider) can live in
separate bottles without forcing a linear chain.

Merge rules for the parent fold: env dict-merge with later winning on
collision; git-gate.user per-field overlay; git-gate.repos union by
name with later winning per-field on same name; egress.routes
concatenated; all scalar fields (supervise, agent_provider, egress.log)
use last-wins. The existing child-wins-over-all-parents rule is
unchanged. Cycle detection, diamond deduplication, and missing/invalid
parent errors all work across multi-parent graphs.

Closes #268
2026-06-25 07:01:31 +00:00
184 changed files with 7997 additions and 12624 deletions
-18
View File
@@ -1,18 +0,0 @@
[run]
branch = True
source = .
[report]
# Coverage policy: see docs/decisions/0004-coverage-policy.md.
#
# `omit` is reserved for genuinely interactive entry-point shells whose
# bodies are `read_tty_line()` / curses prompt loops — there is no
# behaviour to assert that a test wouldn't have to fake wholesale, so a
# test here would inflate the number without buying confidence. This is
# NOT a place to hide subprocess/backend orchestration: that code is
# security-relevant and is measured via the integration suite instead
# (run scripts/coverage.sh for the combined unit+integration number).
omit =
bot_bottle/cli/tui.py
bot_bottle/cli/init.py
tests/*
+1 -36
View File
@@ -39,14 +39,8 @@ jobs:
with:
python-version: "3.12"
- name: Install dev requirements
run: python3 -m pip install -r requirements-dev.txt
- name: Run unit tests
run: python3 -m coverage run -m unittest discover -t . -s tests/unit -v
- name: Report unit coverage
run: python3 -m coverage report -m
run: python3 -m unittest discover -t . -s tests/unit -v
integration:
runs-on: ubuntu-latest
@@ -70,32 +64,3 @@ jobs:
- name: Run integration tests
run: python3 -m unittest discover -t . -s tests/integration -v
# Combined unit+integration coverage + the diff-coverage gate.
# See docs/decisions/0004-coverage-policy.md. The hard gate is diff
# coverage (new/changed lines >= 90%); the combined + critical reports
# are informational and degrade gracefully when the runner has no
# Docker (integration tests skip, those modules just read lower).
coverage:
runs-on: ubuntu-latest
steps:
- name: Checkout
uses: actions/checkout@v4
with:
fetch-depth: 0
- name: Set up Python
uses: actions/setup-python@v5
with:
python-version: "3.12"
- name: Install dev requirements
run: python3 -m pip install -r requirements-dev.txt
- name: Combined coverage (unit + integration)
run: PYTHON=python3 bash scripts/coverage.sh critical
- name: Diff-coverage gate (changed lines >= 90%)
run: |
git fetch --no-tags origin main:refs/remotes/origin/main
python3 scripts/diff_coverage.py --base origin/main --min 90
+24 -26
View File
@@ -6,9 +6,8 @@ on:
- main
paths:
- '**.py'
- '.coveragerc'
# The core-coverage badge reads this list; refresh when it changes.
- 'scripts/critical-modules.txt'
- '.pylintrc'
- 'pyrightconfig.json'
workflow_dispatch:
jobs:
@@ -30,39 +29,38 @@ jobs:
python -m pip install --upgrade pip
pip install -r requirements-dev.txt
- name: Run coverage and extract percentage
id: coverage
- name: Run pylint and extract score
id: pylint
run: |
python -m coverage run -m unittest discover -t . -s tests/unit > /dev/null 2>&1 || true
PERCENT=$(python -m coverage report 2>/dev/null | grep '^TOTAL' | grep -oP '\d+(?=%)' | tail -1)
echo "percent=$PERCENT" >> $GITHUB_OUTPUT
echo "Coverage: $PERCENT%"
PYLINT_OUTPUT=$(python -m pylint bot_bottle/ 2>&1) || true
SCORE=$(echo "$PYLINT_OUTPUT" | grep -oP '(?<=rated at )\d+\.\d+/10' | head -1)
echo "score=$SCORE" >> $GITHUB_OUTPUT
echo "Pylint score: $SCORE"
- name: Extract core (critical-module) coverage percentage
id: core_coverage
- name: Run pyright and check errors
id: pyright
run: |
# Reuses the .coverage data from the previous step. The core list is
# the single source of truth in scripts/critical-modules.txt; every
# core module is unit-tested, so the unit-only run is accurate for it.
INCLUDE=$(grep -vE '^[[:space:]]*(#|$)' scripts/critical-modules.txt | paste -sd, -)
PERCENT=$(python -m coverage report --include="$INCLUDE" 2>/dev/null | grep '^TOTAL' | grep -oP '\d+(?=%)' | tail -1)
echo "percent=$PERCENT" >> $GITHUB_OUTPUT
echo "Core coverage: $PERCENT%"
PYRIGHT_OUTPUT=$(python -m pyright 2>&1) || true
ERRORS=$(echo "$PYRIGHT_OUTPUT" | grep -oP '\d+(?= error)' | head -1)
echo "errors=$ERRORS" >> $GITHUB_OUTPUT
echo "Pyright errors: $ERRORS"
- name: Update badges in README
run: |
COVERAGE_PERCENT="${{ steps.coverage.outputs.percent }}"
CORE_COVERAGE_PERCENT="${{ steps.core_coverage.outputs.percent }}"
PYLINT_SCORE="${{ steps.pylint.outputs.score }}"
PYRIGHT_ERRORS="${{ steps.pyright.outputs.errors }}"
if [ -n "$COVERAGE_PERCENT" ]; then
sed -i "s|/badge/coverage-[^)]*|/badge/coverage-${COVERAGE_PERCENT}%25-brightgreen|" README.md
PYLINT_SCORE_ENCODED=$(echo "$PYLINT_SCORE" | sed 's|/|%2F|g')
if [ -n "$PYLINT_SCORE_ENCODED" ]; then
sed -i "s|/badge/pylint-[^)]*|/badge/pylint-${PYLINT_SCORE_ENCODED}-brightgreen|" README.md
fi
if [ -n "$CORE_COVERAGE_PERCENT" ]; then
sed -i "s|/badge/core%20coverage-[^)]*|/badge/core%20coverage-${CORE_COVERAGE_PERCENT}%25-brightgreen|" README.md
if [ -n "$PYRIGHT_ERRORS" ]; then
sed -i "s|/badge/pyright-[^)]*|/badge/pyright-${PYRIGHT_ERRORS}%20errors-brightgreen|" README.md
fi
echo "Updated badges:"
grep -E "coverage" README.md | head -2
grep -E "pylint|pyright" README.md | head -2
- name: Commit and push badge updates
run: |
@@ -75,7 +73,7 @@ jobs:
else
echo "Badge changes detected, committing..."
git add README.md
MSG="chore: update quality badges"$'\n\n'"- Coverage: ${{ steps.coverage.outputs.percent }}%"$'\n'"- Core coverage: ${{ steps.core_coverage.outputs.percent }}%"$'\n\n'"[skip ci]"
MSG="chore: update quality badges"$'\n\n'"- Pylint: ${{ steps.pylint.outputs.score }}"$'\n'"- Pyright: ${{ steps.pyright.outputs.errors }} errors"$'\n\n'"[skip ci]"
git commit -m "$MSG"
git push
fi
-1
View File
@@ -22,4 +22,3 @@ venv/
.pytest_cache/
.mypy_cache/
.ruff_cache/
.coverage
+5 -5
View File
@@ -9,11 +9,11 @@ host. A Python CLI (entry point `cli.py`, package `bot_bottle/`) orchestrates
the runtime lifecycle and the copying of skills and env vars into it.
The default backend on compatible macOS hosts is macos-container:
agents and sidecar bundles run through Apple's `container` CLI without
requiring Docker. On KVM-capable Linux hosts the default is firecracker:
agents run in a Firecracker microVM reached over SSH on a point-to-point
TAP, while the sidecar bundle still uses Docker. The legacy Docker
backend remains available with `BOT_BOTTLE_BACKEND=docker` or
`--backend=docker`.
requiring Docker. The smolmachines backend remains available with
`BOT_BOTTLE_BACKEND=smolmachines` or `--backend=smolmachines`; agents
run in a libkrun micro-VM, while the sidecar bundle still uses Docker.
The legacy Docker backend remains available with `BOT_BOTTLE_BACKEND=docker`
or `--backend=docker`.
## Goals
+3 -10
View File
@@ -18,13 +18,13 @@
# /git-gate-entrypoint.sh docker-cp'd at start time
# /git-gate/creds/* docker-cp'd at start time
# /git/* bare repos, populated at runtime
# /run/supervise/bot-bottle.db bind-mounted at run time
# /run/supervise/queue/ bind-mounted at run time
# /home/mitmproxy/.mitmproxy/ mitmproxy CA dir
#
# Exposed ports inside the container:
# 9099 egress (mitmproxy, agent-facing HTTPS proxy)
# 9418 git-gate (git-daemon)
# 9420 git-gate smart HTTP (VM-backend agent-facing transport)
# 9420 git-gate smart HTTP (smolmachines agent-facing transport)
# 9100 supervise (MCP HTTP)
# Stage 1: gitleaks binary. The upstream gitleaks image is alpine
@@ -62,16 +62,9 @@ COPY --from=gitleaks-src /usr/bin/gitleaks /usr/bin/gitleaks
# top-level siblings (absolute imports), matching the prior
# Dockerfile.egress / Dockerfile.supervise layout.
COPY bot_bottle/egress_addon_core.py /app/egress_addon_core.py
COPY bot_bottle/egress_dlp_config.py /app/egress_dlp_config.py
COPY bot_bottle/egress_addon.py /app/egress_addon.py
COPY bot_bottle/dlp_detectors.py /app/dlp_detectors.py
COPY bot_bottle/yaml_subset.py /app/yaml_subset.py
COPY bot_bottle/migrations.py /app/migrations.py
COPY bot_bottle/db_store.py /app/db_store.py
COPY bot_bottle/supervise_types.py /app/supervise_types.py
COPY bot_bottle/queue_store.py /app/queue_store.py
COPY bot_bottle/audit_store.py /app/audit_store.py
COPY bot_bottle/store_manager.py /app/store_manager.py
COPY bot_bottle/supervise.py /app/supervise.py
COPY bot_bottle/supervise_server.py /app/supervise_server.py
COPY bot_bottle/sidecar_init.py /app/sidecar_init.py
@@ -87,7 +80,7 @@ RUN mkdir -p \
/etc/git-gate \
/git-gate/creds \
/git \
/run/supervise \
/run/supervise/queue \
/home/mitmproxy/.mitmproxy
# Documentation only — the compose renderer publishes whichever
+7 -22
View File
@@ -5,8 +5,8 @@
# bot-bottle
[![test](https://gitea.dideric.is/didericis/bot-bottle/actions/workflows/test.yml/badge.svg?branch=main)](https://gitea.dideric.is/didericis/bot-bottle/actions?workflow=test.yml)
[![coverage](https://img.shields.io/badge/coverage-84%25-brightgreen)](https://coverage.readthedocs.io/)
[![core coverage](https://img.shields.io/badge/core%20coverage-95%25-brightgreen)](https://gitea.dideric.is/didericis/bot-bottle/src/branch/main/docs/decisions/0004-coverage-policy.md)
[![pylint](https://img.shields.io/badge/pylint-9.93%2F10-brightgreen)](https://github.com/PyCQA/pylint)
[![pyright](https://img.shields.io/badge/pyright-0%20errors-brightgreen)](https://github.com/microsoft/pyright)
**Problem:** Developer wants to run a coding agent without supervision, but they don't want a prompt injected or misbehaving agent wrecking their environment or exfiltrating sensitive data.
@@ -25,14 +25,14 @@
- **Provider templates (Claude, Codex)** — `Dockerfile.claude` / `Dockerfile.codex`, or a bottle-supplied Dockerfile. Claude auth via long-lived OAuth token; Codex via opt-in host device-auth forwarding.
- **gVisor auto-detect** — on Linux hosts where `runsc` is registered with Docker, every bottle launches under it for a userspace syscall barrier; no manifest config required.
- **Apple Container backend (macOS default when available)** — runs the agent and sidecar bundle with Apple's `container` CLI, using a host-only agent network plus a separate sidecar egress network.
- **Firecracker backend (Linux default when available)** — runs the agent in a KVM Firecracker microVM reached over SSH on a point-to-point TAP, with the sidecar bundle in Docker. A dedicated, fail-closed `nftables` table isolates the guest, closing the raw DNS/IP exfiltration gap that exists in the legacy Docker backend. Requires KVM (`/dev/kvm`) and a one-time privileged network-pool setup.
- **Legacy Docker backend** — still available for examples, CI, and hosts without Apple Container or KVM via `BOT_BOTTLE_BACKEND=docker` or `--backend=docker`.
- **Smolmachines backend** — runs the agent in a libkrun micro-VM while the sidecar bundle stays in Docker. TSI and smolmachines DNS filtering close the raw DNS exfiltration gap that exists in the legacy Docker backend.
- **Legacy Docker backend** — still available for examples, CI, and hosts without Apple Container via `BOT_BOTTLE_BACKEND=docker` or `--backend=docker`.
## Architecture
On the default macOS Apple Container backend, a bottle is an agent container on a host-only internal network plus a sidecar bundle attached to both that internal network and a NAT egress network. The agent gets HTTP(S)_PROXY and CA bundle env vars pointing at the sidecar's internal-network IP, so HTTP/HTTPS traffic flows through the sidecar instead of direct egress. `bottle.git` / git-gate is intentionally deferred on this backend until a safe Apple Container key-delivery path exists.
On the Firecracker backend, a bottle is an agent microVM plus a Docker sidecar bundle for egress, git-gate, and supervise. The VM reaches the sidecars over a per-bottle point-to-point TAP link; a dedicated fail-closed `nftables` table (`inet bot_bottle_fc`) confines the guest to that link, so nothing leaves the box except through the sidecars. The TAP pool and nft table are provisioned once (root); per-launch needs no privilege.
On the smolmachines backend, a bottle is an agent micro-VM plus a Docker sidecar bundle for egress, git-gate, and supervise. The VM reaches the sidecars through a per-bottle loopback alias allowed by TSI; smolmachines handles DNS filtering below the guest OS.
On the legacy Docker backend, the same logical bottle is two containers per agent: an `agent` container and a `sidecars` container. They share a per-agent Docker `--internal` network; the agent has no default route off-box.
@@ -71,24 +71,9 @@ When the agent exits, `cli.py` tears down every sidecar and both networks; nothi
## Quickstart
On compatible macOS hosts, the default backend requires Apple's `container` CLI and does not require Docker. The Firecracker backend (Linux) requires Docker on the host for the sidecar bundle plus the `firecracker` binary and KVM. The legacy Docker backend requires Docker. Claude bottles also need a long-lived Claude Code OAuth token (`claude setup-token`) exported as `BOT_BOTTLE_CLAUDE_OAUTH_TOKEN`.
On compatible macOS hosts, the default backend requires Apple's `container` CLI and does not require Docker. The smolmachines backend requires Docker on the host for the sidecar bundle plus smolvm. The legacy Docker backend requires Docker. Claude bottles also need a long-lived Claude Code OAuth token (`claude setup-token`) exported as `BOT_BOTTLE_CLAUDE_OAUTH_TOKEN`.
Use `BOT_BOTTLE_BACKEND=docker ./cli.py start <agent>` on hosts where neither Apple Container nor KVM is available and Docker is the desired backend.
### Firecracker on Linux
On Linux, a KVM-capable host defaults to the Firecracker backend. It needs:
- **`/dev/kvm`** present and accessible. Load `kvm-intel` or `kvm-amd` (and enable virtualization in BIOS/firmware). The invoking user must be in the `kvm` group: `sudo usermod -aG kvm "$USER"` then re-login. bot-bottle preflights this and reports exactly what's missing.
- **`firecracker`** on `PATH`: grab a release from <https://github.com/firecracker-microvm/firecracker/releases>. Start flows print this pointer when the binary is missing.
- **Docker** for the sidecar bundle and image build.
- **A one-time privileged network setup** — the per-bottle TAP pool plus the fail-closed `nftables` isolation table. Run `./cli.py backend setup --backend=firecracker` for the host-appropriate config (a NixOS module, a `sudo` script elsewhere); `./cli.py backend status --backend=firecracker` reports what's present, including whether the pool range collides with an existing route. The pool defaults to `10.243.0.0/16` (an obscure RFC-1918 block that dodges docker/libvirt/LAN and, deliberately, Tailscale's `100.64.0.0/10` CGNAT range); override with `BOT_BOTTLE_FC_IP_BASE` if it clashes on your host.
```sh
BOT_BOTTLE_BACKEND=firecracker ./cli.py start <agent>
```
> **NixOS:** enable `virtualisation.docker`, ensure the KVM module is loaded (`boot.kernelModules = [ "kvm-intel" ];` or `kvm-amd`), and add your user to the `kvm` and `docker` groups. For the network pool, consume the flake module — `imports = [ inputs.bot-bottle.nixosModules.firecracker-netpool ]; services.bot-bottle-firecracker = { enable = true; owner = "you"; };` — then `nixos-rebuild switch` (imperative nft/TAP rules don't survive a rebuild; channel users can `imports = [ <bot-bottle>/nix/firecracker-netpool.nix ]`). `firecracker` isn't in nixpkgs by default as a user binary — install the release binary (pin the version) and put it on `PATH`.
Use `BOT_BOTTLE_BACKEND=docker ./cli.py start <agent>` on hosts where Apple Container is not installed and Docker is the desired backend.
```sh
./cli.py start <agent> # builds the image on first run, drops you into claude
+2 -14
View File
@@ -61,6 +61,7 @@ class AgentProviderRuntime:
prompt_mode: PromptMode
bypass_args: tuple[str, ...]
resume_args: tuple[str, ...]
remote_control_args: tuple[str, ...]
@dataclass(frozen=True)
@@ -209,15 +210,6 @@ class AgentProvider(ABC):
the supervise sidecar is reachable. No-op when
`plan.supervise_plan is None`."""
@abstractmethod
def headless_prompt(self, prompt: str) -> list[str]:
"""Return the agent CLI args that deliver `prompt` as the
initial task in a non-interactive (headless) session.
Called only when ``--prompt`` is passed to
``./cli.py start --headless``; the returned args are appended
after the provider's ``bypass_args`` and ``startup_args``."""
def provision_ca(self, bottle: "Bottle", plan: "BottlePlan") -> None:
"""Install the egress MITM CA into the agent's trust store.
@@ -227,10 +219,6 @@ class AgentProvider(ABC):
from .backend.util import AGENT_CA_PATH, log_ca_fingerprint, select_ca_cert
from .log import die
cert_host_path, label = select_ca_cert(plan.egress_plan)
# Ensure the target directory exists. A backend's rootfs build
# may not preserve the empty /usr/local/share/ca-certificates/
# directory; mkdir -p is idempotent and safe for all backends.
bottle.exec("mkdir -p /usr/local/share/ca-certificates", user="root")
bottle.cp_in(str(cert_host_path), AGENT_CA_PATH)
r = bottle.exec(
f"chmod 644 {AGENT_CA_PATH} && update-ca-certificates",
@@ -403,7 +391,7 @@ def prompt_args(
if prompt_mode == "append_file":
return ["--append-system-prompt-file", prompt_path]
if prompt_mode == "read_prompt_file":
if argv and ("resume" in argv or "remote-control" in argv):
if argv and "resume" in argv:
return []
return [f"Read and follow the instructions in {prompt_path}."]
if prompt_mode == "print_read_prompt_file":
-91
View File
@@ -1,91 +0,0 @@
"""SQLite-backed audit store for supervise (PRD 0013)."""
from __future__ import annotations
import sqlite3
from pathlib import Path
try:
from .supervise_types import AuditEntry, host_db_path
from .db_store import DbStore
from .migrations import TableMigrations
except ImportError:
from supervise_types import AuditEntry, host_db_path # type: ignore[import-not-found] # pylint: disable=import-error,no-name-in-module
from db_store import DbStore # type: ignore[import-not-found] # pylint: disable=import-error,no-name-in-module
from migrations import TableMigrations # type: ignore[import-not-found] # pylint: disable=import-error,no-name-in-module
class AuditStore(DbStore):
"""SQLite-backed persistent store for supervise audit entries."""
def __init__(self, db_path: Path | None = None) -> None:
# One entry per schema version: migrations[0] brings a fresh DB to
# version 1, [1] to version 2, etc. Add new entries at the end; never
# edit existing ones.
migrations = TableMigrations("audit_store", [
# v1 — initial schema
"""
CREATE TABLE IF NOT EXISTS supervise_audit_entries (
id INTEGER PRIMARY KEY AUTOINCREMENT,
timestamp TEXT NOT NULL,
bottle_slug TEXT NOT NULL,
component TEXT NOT NULL,
operator_action TEXT NOT NULL,
operator_notes TEXT NOT NULL,
justification TEXT NOT NULL,
diff TEXT NOT NULL
)
""",
])
super().__init__(db_path or host_db_path(), migrations)
def write_audit_entry(self, entry: AuditEntry) -> Path:
with self._connect() as conn:
conn.execute(
"""
INSERT INTO supervise_audit_entries (
timestamp, bottle_slug, component, operator_action,
operator_notes, justification, diff
) VALUES (?, ?, ?, ?, ?, ?, ?)
""",
(
entry.timestamp,
entry.bottle_slug,
entry.component,
entry.operator_action,
entry.operator_notes,
entry.justification,
entry.diff,
),
)
self._chmod()
return self.db_path
def read_audit_entries(self, component: str, slug: str) -> list[AuditEntry]:
if not self.db_path.is_file():
return []
with self._connect() as conn:
rows = conn.execute(
"""
SELECT * FROM supervise_audit_entries
WHERE component = ? AND bottle_slug = ?
ORDER BY id
""",
(component, slug),
).fetchall()
return [self._row_to_entry(row) for row in rows]
@staticmethod
def _row_to_entry(row: sqlite3.Row) -> AuditEntry:
return AuditEntry(
timestamp=row["timestamp"],
bottle_slug=row["bottle_slug"],
component=row["component"],
operator_action=row["operator_action"],
operator_notes=row["operator_notes"],
justification=row["justification"],
diff=row["diff"],
)
__all__ = ["AuditStore"]
+24 -84
View File
@@ -26,9 +26,8 @@ backend exposes five methods:
Selection is driven by `--backend` on `start` or BOT_BOTTLE_BACKEND
(env var). When neither is set, compatible macOS hosts default to
`macos-container`; Linux hosts with KVM default to `firecracker`;
otherwise `docker`. Per PRD 0003 the manifest does not carry a
backend field; the host picks.
`macos-container`; other hosts default to `smolmachines`. Per PRD 0003
the manifest does not carry a backend field; the host picks.
"""
from __future__ import annotations
@@ -73,12 +72,6 @@ class BottleSpec:
identity: str = ""
label: str = ""
color: str = ""
# Ordered bottle names selected at launch (issue #269). When non-empty
# they are merged in order and replace the agent's `bottle:` field.
bottle_names: tuple[str, ...] = ()
# True when launched via --headless (no TTY, no interactive prompts).
# The git-gate host-key preflight uses this to error rather than prompt.
headless: bool = False
@dataclass(frozen=True)
@@ -98,14 +91,14 @@ class BottlePlan(ABC):
@property
def git_gate_insteadof_host(self) -> str:
"""Host (and optional port) used in git-gate insteadOf URLs.
Docker uses the compose-network DNS alias; VM backends may
override with an IP:port when the guest has no DNS."""
Docker uses the compose-network DNS alias; smolmachines
overrides with a loopback IP:port since TSI has no DNS."""
return "git-gate"
@property
def git_gate_insteadof_scheme(self) -> str:
"""URL scheme for git-gate insteadOf rewrites. 'git' for
Docker (git daemon); VM backends may override (e.g. 'http'
Docker (git daemon); 'http' for smolmachines (HTTP proxy
over a published host port)."""
return "git"
egress_plan: EgressPlan
@@ -116,8 +109,9 @@ class BottlePlan(ABC):
def workspace_plan(self) -> WorkspacePlan:
return workspace_plan(self.spec, guest_home=self.guest_home)
def print(self) -> None:
def print(self, *, remote_control: bool) -> None:
"""Render the y/N preflight summary to stderr."""
del remote_control
spec = self.spec
manifest = self.manifest
agent = manifest.agent
@@ -136,11 +130,7 @@ class BottlePlan(ABC):
info(f"provider : {self.agent_provision.template}")
print_multi("env ", env_names)
print_multi("skills ", list(agent.skills))
effective_bottles = (
list(spec.bottle_names) if spec.bottle_names
else ([agent.bottle] if agent.bottle else [])
)
print_multi("bottle ", effective_bottles)
info(f"bottle : {agent.bottle}")
identity = manifest.git_identity_summary()
if identity:
@@ -183,8 +173,8 @@ class BottleCleanupPlan(ABC):
class ExecResult:
"""Captured result of `Bottle.exec`. Backend-neutral: the Docker
impl populates it from a `subprocess.CompletedProcess`, but a
VM backend could populate it from any source that produces a
returncode + captured streams."""
future fly/smolmachines backend could populate it from any source
that produces a returncode + captured streams."""
returncode: int
stdout: str
@@ -202,7 +192,7 @@ class ActiveAgent:
of sidecar daemons currently up for this bottle (`egress`,
`git-gate`, `supervise`); the dashboard uses it to
gate edit verbs. `backend_name` is the matching key in
`_BACKENDS` (`docker` / `firecracker` / `macos-container`) — used by the active-
`_BACKENDS` (`docker` / `smolmachines` / `macos-container`) — used by the active-
list rendering to disambiguate and by the dashboard's
re-attach path."""
@@ -304,13 +294,6 @@ class BottleBackend(ABC, Generic[PlanT, CleanupT]):
self._preflight()
from ..git_gate_host_key import preflight_host_keys
manifest = preflight_host_keys(
manifest,
headless=spec.headless,
home_md=spec.manifest.home_md,
)
manifest_bottle = manifest.bottle
manifest_agent_provider = manifest_bottle.agent_provider
agent_provider = get_provider(manifest_agent_provider.template)
@@ -381,7 +364,7 @@ class BottleBackend(ABC, Generic[PlanT, CleanupT]):
Returns the loaded Manifest for the selected agent. Subclasses with
additional preconditions should override and call
`super()._validate(spec)` first."""
manifest = spec.manifest.load_for_agent(spec.agent_name, spec.bottle_names)
manifest = spec.manifest.load_for_agent(spec.agent_name)
self._validate_skills(manifest.agent.skills)
self._validate_agent_provider_dockerfile(spec, manifest)
return manifest
@@ -407,12 +390,9 @@ class BottleBackend(ABC, Generic[PlanT, CleanupT]):
if not path.is_absolute():
path = Path(spec.user_cwd) / path
if not path.is_file():
effective = (
", ".join(spec.bottle_names) if spec.bottle_names else manifest.agent.bottle
)
die(
f"agent_provider.dockerfile for bottle "
f"'{effective}' not found: {path}"
f"'{manifest.agent.bottle}' not found: {path}"
)
@abstractmethod
@@ -507,7 +487,7 @@ class BottleBackend(ABC, Generic[PlanT, CleanupT]):
MCP entry inside the guest.
Default returns "" so backends without supervise support
don't have to implement it. Docker and firecracker override."""
don't have to implement it. Docker and smolmachines override."""
del plan
return ""
@@ -524,60 +504,27 @@ class BottleBackend(ABC, Generic[PlanT, CleanupT]):
def enumerate_active(self) -> Sequence[ActiveAgent]:
"""Return every currently-running agent on this backend.
Empty when none. Backend-specific: docker queries `docker
compose ls`; firecracker cross-references its running sidecar
containers against per-bottle metadata."""
compose ls`; smolmachines queries `smolvm machine ls --json`
+ cross-references its bundle container."""
@classmethod
@abstractmethod
def is_available(cls) -> bool:
"""Whether this backend's runtime prerequisites are satisfied
on the current host. Docker → `docker` on PATH; firecracker →
Linux + KVM. Used by the cross-backend
on the current host. Docker → `docker` on PATH; smolmachines
→ `smolvm` on PATH. Used by the cross-backend
`enumerate_active_agents` / `cmd_cleanup` to skip backends
the operator hasn't installed, so a docker-only host
doesn't fail when `cli.py list active` walks past
firecracker."""
@classmethod
@abstractmethod
def setup(cls) -> int:
"""Emit this backend's one-time host setup — privileged network
pool, daemon bring-up, install pointers, etc. — as
host-appropriate config or commands. Prints to stdout/stderr and
returns a shell exit code (0 = nothing to report / success). A
backend that needs no host setup prints a short note and returns
0. Invoked generically by `./cli.py backend setup [--backend=…]`
so operators can provision any backend without a
backend-specific command. Classmethod (like `is_available`) —
it's a host query, not per-bottle state."""
@classmethod
@abstractmethod
def status(cls) -> int:
"""Report whether this backend's prerequisites are satisfied on
the host — binaries, daemon reachability, network pool, range
conflicts, etc. Prints a human-readable summary; returns 0 when
the backend is ready to launch and non-zero when something is
missing. Invoked by `./cli.py backend status [--backend=…]`."""
@classmethod
@abstractmethod
def teardown(cls) -> int:
"""Undo `setup()` — the inverse operation, surfaced as
`./cli.py backend teardown [--backend=…]` (uninstall). Symmetric
with setup: where setup is advisory (prints the privileged
commands / declarative config to apply), teardown prints the
commands / config change to remove the host prerequisites. A
backend with no host setup prints a short note and returns 0.
Not called by the launch path or the test suite."""
smolmachines."""
# Import concrete backend classes AFTER the base types are defined, so
# each backend module can pull BottleSpec / BottlePlan / BottleBackend
# via `from . import ...` without hitting a partially-initialized module.
from .docker import DockerBottleBackend # noqa: E402 # pylint: disable=wrong-import-position
from .firecracker import FirecrackerBottleBackend # noqa: E402 # pylint: disable=wrong-import-position
from .macos_container import MacosContainerBottleBackend # noqa: E402 # pylint: disable=wrong-import-position
from .smolmachines import SmolmachinesBottleBackend # noqa: E402 # pylint: disable=wrong-import-position
# Freezer is imported after the backend classes for the same reason:
# Freezer.commit_slug constructs ActiveAgent, which must be fully
@@ -591,8 +538,8 @@ from .freeze import CommitCancelled, Freezer, get_freezer # noqa: E402 # pylin
# unparameterized methods (prepare → plan → launch(plan), cleanup, etc.).
_BACKENDS: dict[str, BottleBackend[Any, Any]] = {
"docker": DockerBottleBackend(),
"firecracker": FirecrackerBottleBackend(),
"macos-container": MacosContainerBottleBackend(),
"smolmachines": SmolmachinesBottleBackend(),
}
@@ -605,8 +552,7 @@ def get_bottle_backend(
1. explicit arg (CLI `--backend=<name>` passes through here)
2. BOT_BOTTLE_BACKEND env var
3. `macos-container` on compatible macOS hosts
4. `firecracker` on KVM-capable Linux hosts
5. default `docker`
4. default `smolmachines`
Dies with a pointer at the known backends if the chosen name
isn't implemented."""
@@ -620,13 +566,7 @@ def get_bottle_backend(
def _default_backend_name() -> str:
if has_backend("macos-container"):
return "macos-container"
# A KVM-capable Linux host defaults to firecracker even when the
# `firecracker` binary isn't installed yet: selecting it here routes
# start through firecracker's preflight, which prints an install
# pointer, instead of silently falling back to docker.
if FirecrackerBottleBackend.is_host_capable():
return "firecracker"
return "docker"
return "smolmachines"
def known_backend_names() -> tuple[str, ...]:
@@ -640,7 +580,7 @@ def has_backend(name: str) -> bool:
"""Whether the named backend's runtime prerequisites are
available on the current host. Cross-backend callers (list,
cleanup) skip unavailable backends so a docker-only host
doesn't fail when the firecracker backend isn't usable,
doesn't fail when the smolmachines backend isn't installed,
and vice versa.
Returns False for unknown names so callers can pass
-15
View File
@@ -54,21 +54,6 @@ class DockerBottleBackend(BottleBackend["DockerBottlePlan", "DockerBottleCleanup
launch."""
return shutil.which("docker") is not None
@classmethod
def setup(cls) -> int:
from . import setup as _setup
return _setup.setup()
@classmethod
def status(cls) -> int:
from . import setup as _setup
return _setup.status()
@classmethod
def teardown(cls) -> int:
from . import setup as _setup
return _setup.teardown()
def _preflight(self) -> None:
_resolve_plan.preflight()
@@ -0,0 +1,211 @@
"""capability_apply — host-side orchestrator for capability-block
remediation (PRD 0016).
On approval of a capability-block proposal, the dashboard calls
apply_capability_change(slug, new_dockerfile) which:
1. Snapshots the agent's transcript dir to
~/.bot-bottle/state/<slug>/transcript/ (best-effort).
2. Pushes the agent's working tree via `git push` (best-effort —
no upstream / no commits / no git repo all skip with a log).
3. Writes the new Dockerfile to
~/.bot-bottle/state/<slug>/Dockerfile (PRD 0016 Phase 1
state). The next `cli.py start <agent>` picks it up.
4. Force-removes the agent container + all sidecars + the
per-bottle networks. Idempotent — missing resources are not
errors.
Returns (before, after) Dockerfile contents so the dashboard can
record / render the diff. (capability-block has no audit log per
PRD 0013 — the per-bottle Dockerfile state is its own record.)
This is "fire-and-forget" from the agent's perspective: by the time
the dashboard writes the response file the supervise sidecar is
gone, so the agent's tool call connection drops without ever
receiving the response. The replacement agent (next manual
`cli.py start`) sees the new Dockerfile and starts from there.
v1 does not auto-relaunch — see PRD 0016's capability-block return
semantics open question.
"""
from __future__ import annotations
import shutil
import subprocess
from ...agent_provider import get_provider
from ...log import info, warn
from ...bottle_state import (
mark_preserved,
per_bottle_dockerfile,
transcript_snapshot_dir,
write_per_bottle_dockerfile,
)
from .sidecar_bundle import sidecar_bundle_container_name
# Agent home inside the container (per the repo Dockerfile's
# `USER node` + `WORKDIR /home/node`). Used to locate the transcript
# dir + the workspace dir for git push.
_AGENT_HOME_IN_CONTAINER = "/home/node"
_AGENT_TRANSCRIPT_IN_CONTAINER = f"{_AGENT_HOME_IN_CONTAINER}/.claude"
_AGENT_WORKSPACE_IN_CONTAINER = f"{_AGENT_HOME_IN_CONTAINER}/workspace"
# Per-bottle resource name patterns (mirroring prepare.py).
def _agent_container_name(slug: str) -> str:
return f"bot-bottle-{slug}"
def _per_bottle_container_names(slug: str) -> list[str]:
"""All container names that belong to this bottle. Missing
containers are silently skipped by the teardown helper, so it's
fine to include names that don't exist for a given bottle."""
return [
_agent_container_name(slug),
sidecar_bundle_container_name(slug),
]
def _per_bottle_network_names(slug: str) -> list[str]:
return [
f"bot-bottle-net-{slug}",
f"bot-bottle-egress-{slug}",
]
class CapabilityApplyError(RuntimeError):
"""Raised when the apply fails in a way that should keep the
proposal pending (so the operator can retry). Best-effort
failures (transcript snapshot, git push) do not raise — they
just log and proceed."""
# --- Public helpers --------------------------------------------------------
def fetch_current_dockerfile(slug: str) -> str:
"""Return the Dockerfile content the next `cli.py start <agent>`
would use for this bottle. If a per-bottle override exists, that
one; otherwise the repo's Dockerfile.
Used by the operator-edit verb to show the current source of
truth, and by apply_capability_change for the before-diff."""
override = per_bottle_dockerfile(slug)
if override is not None:
return override
repo_dockerfile = get_provider("claude").dockerfile
if repo_dockerfile.is_file():
return repo_dockerfile.read_text()
raise CapabilityApplyError(
f"no per-bottle Dockerfile for {slug} and no provider Dockerfile at "
f"{repo_dockerfile}"
)
def apply_capability_change(slug: str, new_dockerfile: str) -> tuple[str, str]:
"""End-to-end capability-block remediation. See module docstring
for the sequence. Returns (before, after) Dockerfile content."""
if not new_dockerfile.strip():
raise CapabilityApplyError("proposed Dockerfile is empty")
before = fetch_current_dockerfile(slug)
snapshot_transcript(slug)
_push_working_tree(slug)
write_per_bottle_dockerfile(slug, new_dockerfile)
# Set the preserve marker BEFORE teardown so cli.py's session-end
# cleanup sees it and keeps the state dir intact for the
# operator's `cli.py resume <identity>`. Without the marker the
# state dir would be deleted as part of normal session end.
mark_preserved(slug)
_teardown_bottle(slug)
return before, new_dockerfile
# --- Internals -------------------------------------------------------------
def snapshot_transcript(slug: str) -> None:
"""`docker cp` /home/node/.claude out of the agent container into
~/.bot-bottle/state/<slug>/transcript/. Best-effort: missing
container, missing dir, or cp error all log a warning and return.
The transcript is what `claude --resume` reads to pick up where
the agent left off.
Called from two places:
- capability-apply, before tearing the bottle down.
- cli.py's session-end path, before the launch context closes,
so a crash or normal exit also leaves a transcript on disk
(deleted along with the state dir on clean exit, kept on
crash or capability-block per the preserve marker)."""
container = _agent_container_name(slug)
dest = transcript_snapshot_dir(slug)
if dest.exists():
# Remove any prior snapshot so the new one is a clean copy.
shutil.rmtree(dest, ignore_errors=True)
dest.parent.mkdir(parents=True, exist_ok=True)
r = subprocess.run(
["docker", "cp", f"{container}:{_AGENT_TRANSCRIPT_IN_CONTAINER}", str(dest)],
capture_output=True, text=True, check=False,
)
if r.returncode != 0:
warn(
f"transcript snapshot skipped "
f"({(r.stderr or '').strip() or 'no transcript dir in container?'})"
)
return
info(f"transcript snapshotted to {dest}")
def _push_working_tree(slug: str) -> None:
"""`docker exec <agent> git push` from /home/node/workspace.
Best-effort: not-a-git-repo, no upstream, nothing-to-push, no
network all log a warning and return. The replacement bottle
will pick up whatever's actually upstream."""
container = _agent_container_name(slug)
r = subprocess.run(
[
"docker", "exec", container, "sh", "-c",
f"cd {_AGENT_WORKSPACE_IN_CONTAINER} && "
f"git rev-parse --is-inside-work-tree >/dev/null 2>&1 && "
f"git push origin HEAD 2>&1 || true",
],
capture_output=True, text=True, check=False,
)
if r.returncode != 0:
warn(
f"capability-apply: git push skipped "
f"({(r.stderr or '').strip() or 'docker exec failed'})"
)
return
output = (r.stdout or "").strip()
if output:
info(f"capability-apply: git push: {output}")
else:
info("capability-apply: git push ran (no output — likely not a git workspace)")
def _teardown_bottle(slug: str) -> None:
"""Force-remove all per-bottle docker resources. Idempotent —
`docker rm -f` / `docker network rm` silently ignore missing
names, so this can be called even mid-rebuild."""
info(f"capability-apply: tearing down bottle {slug}")
for name in _per_bottle_container_names(slug):
subprocess.run(
["docker", "rm", "-f", name],
stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL, check=False,
)
for net in _per_bottle_network_names(slug):
subprocess.run(
["docker", "network", "rm", net],
stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL, check=False,
)
__all__ = [
"CapabilityApplyError",
"apply_capability_change",
"fetch_current_dockerfile",
"snapshot_transcript",
]
+5 -4
View File
@@ -18,7 +18,8 @@ scan, just as a fallback bucket alongside the project list.
`cleanup` removes everything in the plan.
Active-agent enumeration lives in `backend/docker/enumerate.py`.
Active-agent enumeration lives in `backend/docker/enumerate.py`
(mirror of `backend/smolmachines/enumerate.py`).
"""
from __future__ import annotations
@@ -92,8 +93,8 @@ def _list_orphan_state_dirs(
`protected_identities` is the set of slugs that are live in
ANY backend — used so this docker-side check doesn't reap a
running non-docker bottle's state dir (the layout is shared
across backends)."""
running smolmachines bottle's state dir (the layout is shared
across both backends)."""
state_root = _supervise.bot_bottle_root() / "state"
if not state_root.is_dir():
return []
@@ -118,7 +119,7 @@ def prepare_cleanup() -> DockerBottleCleanupPlan:
Pulls the union of live identities across backends via
`enumerate_active_agents()` so the orphan-state-dir bucket
doesn't include slugs whose non-docker bottle is still up."""
doesn't include slugs whose smolmachines VM is still up."""
docker_mod.require_docker()
projects = list_compose_projects()
project_set = set(projects)
+15 -4
View File
@@ -34,7 +34,8 @@ from ...egress import (
from ...git_gate import GIT_GATE_HOSTNAME
from ...log import die, warn
from ...supervise import (
DB_PATH_IN_CONTAINER,
CURRENT_CONFIG_DIR_IN_AGENT,
QUEUE_DIR_IN_CONTAINER,
SUPERVISE_HOSTNAME,
SUPERVISE_PORT,
)
@@ -163,15 +164,16 @@ def _sidecar_bundle_service(plan: DockerBottlePlan) -> dict[str, Any]:
if sp is not None:
env += [
f"SUPERVISE_BOTTLE_SLUG={plan.slug}",
f"SUPERVISE_DB_PATH={DB_PATH_IN_CONTAINER}",
f"SUPERVISE_QUEUE_DIR={QUEUE_DIR_IN_CONTAINER}",
f"SUPERVISE_PORT={SUPERVISE_PORT}",
]
volumes.append({
"type": "bind",
"source": str(sp.db_path),
"target": DB_PATH_IN_CONTAINER,
"source": str(sp.queue_dir),
"target": QUEUE_DIR_IN_CONTAINER,
"read_only": False,
})
internal_aliases = [EGRESS_HOSTNAME]
if gp.upstreams:
internal_aliases.append(GIT_GATE_HOSTNAME)
@@ -231,6 +233,15 @@ def _agent_service(plan: DockerBottlePlan) -> dict[str, Any]:
if plan.use_runsc:
service["runtime"] = "runsc"
volumes: list[dict[str, Any]] = []
if plan.supervise_plan is not None:
volumes.append(_bind(
plan.supervise_plan.current_config_dir,
CURRENT_CONFIG_DIR_IN_AGENT,
))
if volumes:
service["volumes"] = volumes
# The init supervisor inside the bundle owns intra-bundle
# daemon ordering, so the agent only waits for the bundle
# container itself.
+2 -1
View File
@@ -1,6 +1,7 @@
"""Active-agent enumeration for the docker backend.
Returns `ActiveAgent` records the CLI `list active` command and the
Mirrors `backend/smolmachines/enumerate.py`: returns
`ActiveAgent` records the CLI `list active` command and the
dashboard agents pane consume. Empty when docker isn't reachable
— gated by `has_backend('docker')` at the cross-backend caller
so this module trusts that docker is available when called.
+1 -9
View File
@@ -37,10 +37,7 @@ from pathlib import Path
from typing import Callable, Generator
from ...egress import egress_resolve_token_values
from ...git_gate import (
provision_git_gate_dynamic_keys,
revoke_git_gate_provisioned_keys,
)
from ...git_gate import revoke_git_gate_provisioned_keys
from ...log import info, warn
from . import network as network_mod
from . import util as docker_mod
@@ -121,11 +118,6 @@ def launch(
git_gate_plan = plan.git_gate_plan
if git_gate_plan.upstreams:
git_gate_plan = provision_git_gate_dynamic_keys(
plan.manifest.bottle,
git_gate_plan,
git_gate_state_dir(plan.slug),
)
git_gate_plan = dataclasses.replace(
git_gate_plan,
internal_network=internal_network,
-89
View File
@@ -1,89 +0,0 @@
"""Host setup + status for the Docker backend.
Unlike Firecracker, the Docker backend needs no privileged one-time
host provisioning (no TAP pool / nft table) — networks and the sidecar
bundle are created per-launch. So `setup()` is mostly an install/daemon
pointer, and `status()` reports whether docker is usable.
This is intentionally minimal; a richer version (daemon config checks,
gVisor/runsc install guidance, rootless-docker hints) is tracked
separately. Reached via `DockerBottleBackend.setup` / `.status`, which
the generic `./cli.py backend {setup,status}` dispatches to.
"""
from __future__ import annotations
import shutil
import subprocess
import sys
from . import util as _util
def _docker_on_path() -> bool:
return shutil.which("docker") is not None
def _daemon_reachable() -> bool:
if not _docker_on_path():
return False
return subprocess.run(
["docker", "info"],
stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL, check=False,
).returncode == 0
def _print_install_pointer() -> None:
sys.stderr.write("Docker is required but was not found on PATH.\n")
sys.stderr.write(" macOS: Docker Desktop https://docs.docker.com/desktop/install/mac-install/\n")
sys.stderr.write(" Linux: Docker Engine https://docs.docker.com/engine/install/\n")
def setup() -> int:
if not _docker_on_path():
_print_install_pointer()
return 1
sys.stderr.write(
"Docker backend: no privileged host setup required — networks and "
"the sidecar bundle are created per-launch.\n"
)
if not _daemon_reachable():
sys.stderr.write(
"The Docker daemon isn't reachable; start it (e.g. Docker "
"Desktop, or `systemctl start docker`).\n"
)
return 1
if not _util.runsc_available():
sys.stderr.write(
"Optional: register the gVisor (`runsc`) runtime with Docker for "
"a userspace syscall barrier; bottles auto-detect and use it.\n"
)
return 0
def teardown() -> int:
sys.stderr.write(
"Docker backend: nothing to undo — it provisions no privileged host "
"state (networks and the sidecar bundle are per-launch and are "
"removed by `./cli.py cleanup`). Docker itself is left installed.\n"
)
return 0
def status() -> int:
ok = True
if _docker_on_path():
sys.stderr.write("docker on PATH: yes\n")
else:
sys.stderr.write("docker on PATH: NO\n")
ok = False
if ok and _daemon_reachable():
sys.stderr.write("docker daemon: reachable\n")
elif ok:
sys.stderr.write("docker daemon: UNREACHABLE\n")
ok = False
runsc = _docker_on_path() and _util.runsc_available()
sys.stderr.write(f"gVisor runsc runtime: {'registered' if runsc else 'not registered (optional)'}\n")
if not ok:
sys.stderr.write("\nRun: ./cli.py backend setup --backend=docker\n")
return 0 if ok else 1
+30
View File
@@ -167,6 +167,36 @@ def commit_container(container_name: str, image_tag: str) -> None:
info(f"committed {container_name!r}{image_tag!r}")
def image_id(ref: str) -> str:
"""Return the content-addressed image ID (e.g.
`sha256:abcd...`) for `ref`. The smolmachines backend keys its
`.smolmachine` artifact cache on this, so a Dockerfile change
that produces a new image automatically invalidates the cache."""
r = subprocess.run(
["docker", "image", "inspect", "--format", "{{.Id}}", ref],
capture_output=True,
text=True,
check=False,
)
if r.returncode != 0:
die(
f"docker image inspect for {ref!r} failed: "
f"{(r.stderr or '').strip() or '<no stderr>'}"
)
return r.stdout.strip()
def save(ref: str, output: str) -> None:
"""`docker save REF -o OUTPUT`. Writes a tarball of the image
layers + manifest to the host path. Used by smolmachines
prepare to hand the agent image to a containerized crane that
pushes it to the ephemeral registry — bypassing the docker
daemon's `docker push` (which on Docker Desktop can't reach a
host-loopback registry and refuses plain-HTTP pushes to
non-loopback hosts)."""
subprocess.run(["docker", "save", ref, "-o", output], check=True)
def _silent_run(cmd: Iterable[str]) -> int:
return subprocess.run(
list(cmd),
+2 -6
View File
@@ -11,7 +11,7 @@ from pathlib import Path
from ..bottle_state import egress_state_dir
from ..egress import EGRESS_ROUTES_FILENAME
from ..egress_addon_core import LOG_OFF, load_config
from ..egress_addon_core import load_routes
class EgressApplyError(RuntimeError):
@@ -33,15 +33,11 @@ class EgressApplicator(ABC):
@staticmethod
def validate_routes_content(content: str) -> None:
try:
config = load_config(content)
load_routes(content)
except ValueError as e:
raise EgressApplyError(
f"proposed routes.yaml is not valid: {e}"
) from e
if config.log != LOG_OFF:
raise EgressApplyError(
"proposed routes.yaml must not change egress logging"
)
@staticmethod
def _routes_path(slug: str) -> Path:
@@ -1,7 +0,0 @@
"""Firecracker backend: Linux KVM microVM isolation (issue #342)."""
from __future__ import annotations
from .backend import FirecrackerBottleBackend
__all__ = ["FirecrackerBottleBackend"]
-169
View File
@@ -1,169 +0,0 @@
"""Bottle handle for a Firecracker microVM, over SSH.
Every operation routes through SSH to the guest (dropbear, listening on
the TAP link). `exec` pipes a script over stdin and captures output;
`cp_in` uses scp; `exec_agent` uses `ssh -t` for an interactive PTY
session. `ssh -t` forwards the host terminal's SIGWINCH to the remote
PTY natively, so no separate resize bridge is needed.
Commands run as the image's `node` user via `runuser`, with HOME/USER/
PATH and the bottle env (HTTPS_PROXY at the sidecar, CA paths, …) set
per-invocation through `env` (the VM itself just runs the init; it has
no baked-in process env like a `docker run` container would).
"""
from __future__ import annotations
import os
import shlex
import subprocess
import sys
from pathlib import Path
from typing import Mapping, cast
from ...agent_provider import PromptMode, prompt_args
from .. import Bottle, ExecResult
from ..terminal import exec_shell_script
from . import util
_HOME_FOR = {"node": "/home/node", "root": "/root"}
_DEFAULT_PATH_FOR = {
"node": (
"/home/node/.local/bin:"
"/home/node/.codex/packages/standalone/current/bin:"
"/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"
),
"root": "/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
}
def _env_assignments_for(user: str, env: Mapping[str, str]) -> list[str]:
home = _HOME_FOR.get(user, f"/home/{user}")
out = [f"HOME={home}", f"USER={user}"]
if "PATH" not in env:
out.append(f"PATH={_DEFAULT_PATH_FOR.get(user, _DEFAULT_PATH_FOR['root'])}")
for k, v in env.items():
out.append(f"{k}={v}")
return out
class FirecrackerBottle(Bottle):
def __init__(
self,
name: str,
*,
private_key: Path,
guest_ip: str,
guest_env: Mapping[str, str] | None = None,
prompt_path_in_guest: str | None = None,
agent_command: str = "claude",
agent_prompt_mode: PromptMode = "append_file",
agent_provider_template: str = "claude",
terminal_title: str = "",
terminal_color: str = "",
agent_workdir: str = "/home/node",
) -> None:
self.name = name
self._private_key = private_key
self._guest_ip = guest_ip
self._guest_env = dict(guest_env or {})
self.prompt_path = prompt_path_in_guest
self._agent_prompt_mode = agent_prompt_mode
self.agent_command = agent_command
self.terminal_title = terminal_title
self.terminal_color = terminal_color
self.agent_provider_template = agent_provider_template
self.agent_workdir = agent_workdir
def _ssh(self, *, tty: bool) -> list[str]:
argv = util.ssh_base_argv(self._private_key, self._guest_ip)
if tty:
# -t: allocate a remote PTY and forward window-size changes.
argv.insert(1, "-t")
return argv
def _agent_remote_argv(self, argv: list[str]) -> list[str]:
"""The `runuser … <agent> …` command run inside the guest."""
full_argv = list(argv)
full_argv.extend(
prompt_args(
cast(PromptMode, self._agent_prompt_mode),
self.prompt_path,
argv=full_argv,
)
)
remote = ["runuser", "-u", "node", "--",
"env", *_env_assignments_for("node", self._guest_env)]
if self.agent_workdir and self.agent_workdir != _HOME_FOR["node"]:
remote += ["sh", "-lc", f"cd {self.agent_workdir} && exec \"$@\"",
"bot-bottle-agent"]
remote += [self.agent_command, *full_argv]
return remote
def agent_argv(self, argv: list[str], *, tty: bool = True) -> list[str]:
return [*self._ssh(tty=tty), "--", *self._agent_remote_argv(argv)]
def exec_agent(self, argv: list[str], *, tty: bool = True) -> int:
agent_argv = self.agent_argv(argv, tty=tty)
script = (
exec_shell_script(agent_argv, self.terminal_title, self.terminal_color)
if tty else None
)
if script is None:
return subprocess.run(agent_argv, check=False).returncode
return subprocess.run(["sh", "-lc", script], check=False).returncode
def exec(self, script: str, *, user: str = "node") -> ExecResult:
# Pipe the script over stdin (`sh -s`) so nothing needs shell
# quoting through the SSH command line.
remote = ["runuser", "-u", user, "--",
"env", *_env_assignments_for(user, self._guest_env),
"/bin/sh", "-s"]
result = subprocess.run(
[*self._ssh(tty=False), "--", *remote],
input=script, capture_output=True, text=True, check=False,
)
return ExecResult(
returncode=result.returncode,
stdout=result.stdout,
stderr=result.stderr,
)
def cp_in(self, host_path: str, container_path: str) -> None:
# Stream a tar over the SSH exec channel rather than scp: the
# guest runs dropbear, which ships no sftp-server/scp, but every
# agent image has `tar`. Copy so `container_path` becomes a copy
# of `host_path` (docker cp semantics — callers rm -rf the
# destination first).
host_parent = os.path.dirname(host_path.rstrip("/")) or "/"
host_base = os.path.basename(host_path.rstrip("/"))
guest_parent = os.path.dirname(container_path.rstrip("/")) or "/"
guest_base = os.path.basename(container_path.rstrip("/"))
remote = (
f"mkdir -p {shlex.quote(guest_parent)} && "
f"cd {shlex.quote(guest_parent)} && "
f"rm -rf {shlex.quote(guest_base)} && tar -xf - && "
f"{{ [ {shlex.quote(host_base)} = {shlex.quote(guest_base)} ] || "
f"mv {shlex.quote(host_base)} {shlex.quote(guest_base)}; }}"
)
tar = subprocess.Popen(
["tar", "-C", host_parent, "-cf", "-", host_base],
stdout=subprocess.PIPE,
)
ssh = subprocess.run(
[*self._ssh(tty=False), "--", "sh", "-c", remote],
stdin=tar.stdout, capture_output=True, text=True, check=False,
)
tar.wait()
if tar.returncode != 0 or ssh.returncode != 0:
sys.stderr.write(ssh.stderr)
raise subprocess.CalledProcessError(
ssh.returncode or tar.returncode,
["cp_in", host_path, container_path],
)
def close(self) -> None:
# Real teardown (VM terminate, TAP release) lives on the launch
# ExitStack; this is the idempotent alias the ABC expects.
pass
@@ -1,32 +0,0 @@
"""Cleanup plan for the Firecracker backend."""
from __future__ import annotations
from dataclasses import dataclass
from ...log import info
from .. import BottleCleanupPlan
@dataclass(frozen=True)
class FirecrackerBottleCleanupPlan(BottleCleanupPlan):
# PIDs of orphaned firecracker VMM processes and the sidecar
# containers left behind by previous bottles.
vm_pids: tuple[int, ...] = ()
containers: tuple[str, ...] = ()
run_dirs: tuple[str, ...] = ()
def print(self) -> None:
if self.empty:
info("firecracker cleanup: nothing to remove")
return
for pid in self.vm_pids:
info(f"firecracker VM process: pid {pid}")
for name in self.containers:
info(f"firecracker sidecar container: {name}")
for path in self.run_dirs:
info(f"firecracker run dir: {path}")
@property
def empty(self) -> bool:
return not (self.vm_pids or self.containers or self.run_dirs)
@@ -1,63 +0,0 @@
"""Plan type for the Firecracker backend."""
from __future__ import annotations
from dataclasses import dataclass, field
from pathlib import Path
from ...agent_provider import PromptMode
from .. import BottlePlan
@dataclass(frozen=True)
class FirecrackerBottlePlan(BottlePlan):
slug: str
forwarded_env: dict[str, str] = field(repr=False)
# Stamped by launch once the sidecar is up and its ports are
# published on the host-side TAP IP (empty at prepare time).
agent_proxy_url: str = ""
agent_git_gate_url: str = ""
agent_supervise_url: str = ""
@property
def container_name(self) -> str:
"""Instance name, reused for the sidecar container + VM run dir.
Matches the `bot-bottle-<slug>` convention the other backends
use so cleanup/enumerate discovery-by-prefix keeps working."""
return self.agent_provision.instance_name
@property
def image(self) -> str:
return self.agent_provision.image
@property
def dockerfile_path(self) -> str:
return self.agent_provision.dockerfile
@property
def prompt_file(self) -> Path:
return self.agent_provision.prompt_file
@property
def agent_command(self) -> str:
return self.agent_provision.command
@property
def agent_prompt_mode(self) -> PromptMode:
return self.agent_provision.prompt_mode
@property
def agent_provider_template(self) -> str:
return self.agent_provision.template
@property
def git_gate_insteadof_host(self) -> str:
if self.agent_git_gate_url.startswith("http://"):
return self.agent_git_gate_url.removeprefix("http://").rstrip("/")
return super().git_gate_insteadof_host
@property
def git_gate_insteadof_scheme(self) -> str:
if self.agent_git_gate_url.startswith("http://"):
return "http"
return super().git_gate_insteadof_scheme
-90
View File
@@ -1,90 +0,0 @@
"""Cleanup for the Firecracker backend.
Orphans are: firecracker VMM processes whose config lives under our run
dir, the `bot-bottle-sidecars-*` containers, and the per-bottle run
dirs. TAP slots free themselves (the flock drops when the launcher
exits), so there is nothing to reclaim there.
"""
from __future__ import annotations
import os
import shutil
import signal
import subprocess
from pathlib import Path
from ...log import info
from . import util
from .bottle_cleanup_plan import FirecrackerBottleCleanupPlan
_SIDECAR_PREFIX = "bot-bottle-sidecars-"
def _run_root() -> Path:
return util.cache_dir() / "run"
def _orphan_vm_pids() -> list[int]:
"""firecracker processes whose --config-file is under our run dir."""
run_root = str(_run_root())
result = subprocess.run(
["pgrep", "-a", "firecracker"],
capture_output=True, text=True, check=False,
)
if result.returncode != 0:
return []
pids: list[int] = []
for line in result.stdout.splitlines():
parts = line.split(None, 1)
if len(parts) != 2 or run_root not in parts[1]:
continue
try:
pids.append(int(parts[0]))
except ValueError:
continue
return pids
def _sidecar_containers() -> list[str]:
result = subprocess.run(
["docker", "ps", "-a", "--format", "{{.Names}}",
"--filter", f"name={_SIDECAR_PREFIX}"],
capture_output=True, text=True, check=False,
)
if result.returncode != 0:
return []
return sorted(n.strip() for n in result.stdout.splitlines() if n.strip())
def _run_dirs() -> list[str]:
run_root = _run_root()
if not run_root.is_dir():
return []
return sorted(str(p) for p in run_root.iterdir() if p.is_dir())
def prepare_cleanup() -> FirecrackerBottleCleanupPlan:
return FirecrackerBottleCleanupPlan(
vm_pids=tuple(_orphan_vm_pids()),
containers=tuple(_sidecar_containers()),
run_dirs=tuple(_run_dirs()),
)
def cleanup(plan: FirecrackerBottleCleanupPlan) -> None:
for pid in plan.vm_pids:
info(f"kill firecracker VM pid {pid}")
try:
os.kill(pid, signal.SIGTERM)
except ProcessLookupError:
pass
for name in plan.containers:
info(f"docker rm -f {name}")
subprocess.run(
["docker", "rm", "-f", name],
stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL, check=False,
)
for path in plan.run_dirs:
info(f"rm -rf {path}")
shutil.rmtree(path, ignore_errors=True)
@@ -1,43 +0,0 @@
"""Active-agent enumeration for the Firecracker backend.
The agent runs in a VM (no container to list), so a live bottle is
identified by its running sidecar container `bot-bottle-sidecars-<slug>`
— the same discovery-by-prefix the other backends use.
"""
from __future__ import annotations
import subprocess
from ...bottle_state import read_metadata
from .. import ActiveAgent
_SIDECAR_PREFIX = "bot-bottle-sidecars-"
def enumerate_active() -> list[ActiveAgent]:
result = subprocess.run(
["docker", "ps", "--format", "{{.Names}}",
"--filter", f"name={_SIDECAR_PREFIX}"],
capture_output=True, text=True, check=False,
)
if result.returncode != 0:
return []
out: list[ActiveAgent] = []
for name in sorted(n.strip() for n in result.stdout.splitlines() if n.strip()):
slug = name[len(_SIDECAR_PREFIX):]
metadata = read_metadata(slug)
if metadata is None or metadata.backend != "firecracker":
# Skip sidecars owned by another backend (docker shares the
# container-name prefix).
continue
out.append(ActiveAgent(
backend_name="firecracker",
slug=slug,
agent_name=metadata.agent_name,
started_at=metadata.started_at,
services=(),
label=metadata.label,
color=metadata.color,
))
return out
@@ -1,173 +0,0 @@
"""Firecracker microVM process lifecycle.
A bottle VM is one `firecracker` process booted from a JSON config
(kernel + writable rootfs ext4 + one TAP network interface). We run it
`--no-api`: all configuration is in the config file, and teardown is a
process signal — the microVM dies with its VMM, which is exactly the
ephemeral-bottle semantics we want. No API socket, no runtime
reconfiguration.
The guest gets its IP from the kernel `ip=` cmdline (kernel-level
autoconfig, no in-guest iproute2) and its SSH pubkey from a `bb_pubkey=`
cmdline arg the init decodes.
"""
from __future__ import annotations
import base64
import json
import signal
import subprocess
import time
from dataclasses import dataclass
from pathlib import Path
from ...log import die, info
from . import util
# Serial console off the critical path but captured to a log for
# debugging boot failures. `reboot=k panic=1 pci=off` are the standard
# Firecracker guest args; `i8042.*` skip the (absent) PS/2 probe to
# shave boot time.
_BASE_BOOT_ARGS = (
"console=ttyS0 reboot=k panic=1 pci=off "
"i8042.noaux i8042.nomux i8042.nopnp i8042.dumbkbd "
"root=/dev/vda rw init=/bb-init"
)
@dataclass
class VmHandle:
"""A running microVM: its VMM process, guest IP, and console log."""
process: subprocess.Popen[bytes]
guest_ip: str
console_log: Path
def is_alive(self) -> bool:
return self.process.poll() is None
def terminate(self) -> None:
"""Stop the VMM (and thus the guest). SIGTERM, then SIGKILL."""
if self.process.poll() is not None:
return
self.process.send_signal(signal.SIGTERM)
try:
self.process.wait(timeout=5)
except subprocess.TimeoutExpired:
self.process.kill()
self.process.wait(timeout=5)
def _boot_args(guest_ip: str, host_ip: str, pubkey: str) -> str:
# ip=<client>::<gw>:<netmask>::<dev>:off — /31 point-to-point link,
# so netmask is 255.255.255.254 and the gateway is the host TAP IP.
ip_arg = f"ip={guest_ip}::{host_ip}:255.255.255.254::eth0:off"
pub_b64 = base64.b64encode(pubkey.encode()).decode()
return f"{_BASE_BOOT_ARGS} {ip_arg} bb_pubkey={pub_b64}"
def _config(
*,
rootfs: Path,
tap: str,
guest_ip: str,
host_ip: str,
pubkey: str,
vcpus: int,
mem_mib: int,
guest_mac: str,
) -> dict[str, object]:
return {
"boot-source": {
"kernel_image_path": str(util.kernel_path()),
"boot_args": _boot_args(guest_ip, host_ip, pubkey),
},
"drives": [
{
"drive_id": "rootfs",
"path_on_host": str(rootfs),
"is_root_device": True,
"is_read_only": False,
}
],
"network-interfaces": [
{
"iface_id": "eth0",
"host_dev_name": tap,
"guest_mac": guest_mac,
}
],
"machine-config": {
"vcpu_count": vcpus,
"mem_size_mib": mem_mib,
},
}
def boot(
*,
name: str,
rootfs: Path,
tap: str,
guest_ip: str,
host_ip: str,
pubkey: str,
run_dir: Path,
vcpus: int = 2,
mem_mib: int = 2048,
guest_mac: str = "06:00:AC:10:00:02",
) -> VmHandle:
"""Write the config and launch the VMM. Returns once the process is
spawned; callers wait for SSH readiness separately."""
run_dir.mkdir(parents=True, exist_ok=True)
config_path = run_dir / "config.json"
console_log = run_dir / "console.log"
config_path.write_text(json.dumps(
_config(
rootfs=rootfs, tap=tap, guest_ip=guest_ip, host_ip=host_ip,
pubkey=pubkey, vcpus=vcpus, mem_mib=mem_mib, guest_mac=guest_mac,
),
indent=2,
))
info(f"booting microVM {name} on {tap} (guest {guest_ip})")
log_fh = console_log.open("wb")
process = subprocess.Popen(
["firecracker", "--no-api", "--config-file", str(config_path)],
stdout=log_fh, stderr=subprocess.STDOUT, stdin=subprocess.DEVNULL,
)
return VmHandle(process=process, guest_ip=guest_ip, console_log=console_log)
def wait_for_ssh(
vm: VmHandle, private_key: Path, *, timeout: float = 30.0,
) -> None:
"""Poll SSH until the guest accepts a command or the deadline
passes. Dies (with the console tail) if the VMM exits early or the
guest never comes up — a boot failure must be loud, not a hang."""
deadline = time.monotonic() + timeout
probe = util.ssh_base_argv(private_key, vm.guest_ip) + ["true"]
while time.monotonic() < deadline:
if not vm.is_alive():
die(f"microVM exited during boot (rc={vm.process.returncode}).\n"
f"{_console_tail(vm.console_log)}")
result = subprocess.run(
probe, stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL,
check=False,
)
if result.returncode == 0:
return
time.sleep(0.5)
die(f"microVM {vm.guest_ip} did not accept SSH within {timeout:.0f}s.\n"
f"{_console_tail(vm.console_log)}")
def _console_tail(console_log: Path, lines: int = 25) -> str:
try:
text = console_log.read_text(errors="replace").splitlines()
except OSError:
return "(no console log)"
tail = "\n".join(text[-lines:])
return f"--- guest console (last {lines} lines) ---\n{tail}"
-76
View File
@@ -1,76 +0,0 @@
"""FirecrackerFreezer — snapshot a running microVM to a Docker image.
The VM is live and can't be block-copied safely, so — like the macOS
backend — we stream the guest root filesystem out over the control
channel (SSH here) and rebuild an image from it. The bottle keeps
running after the snapshot.
"""
from __future__ import annotations
import json
import os
import subprocess
import tempfile
from pathlib import Path
from ...log import die, info
from .. import ActiveAgent
from ..freeze import Freezer
from . import util
class FirecrackerFreezer(Freezer):
backend_name = "firecracker"
def _freeze(self, agent: ActiveAgent) -> str:
run_dir = util.cache_dir() / "run" / agent.slug
private_key = run_dir / "bottle_id_ed25519"
guest_ip = _guest_ip_from_config(run_dir / "config.json")
if not private_key.is_file() or not guest_ip:
die(f"cannot freeze {agent.slug}: run dir {run_dir} is missing the "
f"SSH key or VM config (is the bottle still running?)")
image_tag = f"bot-bottle-committed-{agent.slug}:latest"
_commit_via_ssh(private_key, guest_ip, image_tag)
info(f"committed {agent.slug} -> {image_tag!r}")
return image_tag
def _export_hint(self, slug: str, image_ref: str) -> None:
info(f"to export for migration: docker image save {image_ref} "
f"-o {slug}.tar")
def _guest_ip_from_config(config_path: Path) -> str:
try:
cfg = json.loads(config_path.read_text())
except (OSError, json.JSONDecodeError):
return ""
boot_args = cfg.get("boot-source", {}).get("boot_args", "")
for token in boot_args.split():
if token.startswith("ip="):
# ip=<client>::<gw>:<mask>::<dev>:off
return token[len("ip="):].split(":", 1)[0]
return ""
def _commit_via_ssh(private_key: Path, guest_ip: str, image_tag: str) -> None:
with tempfile.TemporaryDirectory(prefix="bot-bottle-fc-commit.") as tmp:
rootfs_tar = os.path.join(tmp, "rootfs.tar")
ssh = util.ssh_base_argv(private_key, guest_ip)
with open(rootfs_tar, "wb") as tar_out:
result = subprocess.run(
[*ssh, "--", "tar", "--create", "--one-file-system",
"--exclude=./proc", "--exclude=./sys", "--exclude=./dev",
"--exclude=./run", "--file=-", "--directory=/", "."],
stdout=tar_out, stderr=subprocess.PIPE, check=False,
)
if result.returncode != 0:
die(f"ssh tar for {guest_ip} failed: "
f"{(result.stderr or b'').decode().strip() or '<no stderr>'}")
with open(os.path.join(tmp, "Dockerfile"), "w", encoding="utf-8") as f:
f.write("FROM scratch\nADD rootfs.tar /\nUSER node\nWORKDIR /home/node\n")
build = subprocess.run(
["docker", "build", "-t", image_tag, tmp], check=False,
)
if build.returncode != 0:
die(f"docker build for {image_tag!r} failed")
@@ -1,116 +0,0 @@
"""Empirical, post-boot isolation probe — the authoritative fail-closed
egress-boundary check.
The pre-boot nft check can't be trusted on every host (listing an
nftables table typically needs root; the launcher runs unprivileged).
So before the agent ever runs, we prove the boundary directly: open a
canary TCP listener on a host IP the VM must NOT be able to reach (the
host's primary non-TAP address), then have the guest try to connect. If
the isolation rules are in force the connection is dropped and times
out; if it succeeds, the VM can reach host services it shouldn't — a
sandbox escape — and we refuse to continue.
The listener is load-bearing: without it a "blocked" result could just
be connection-refused. With it, a reachable canary means a real leak
and a timeout means a real drop.
"""
from __future__ import annotations
import socket
import subprocess
import threading
from pathlib import Path
from ...log import die, info, warn
from . import util
def _host_primary_ip() -> str:
"""The source IP the host uses for off-box traffic (its LAN
address) — a canary the VM must not reach. Empty if the host has no
such route (isolated box)."""
result = subprocess.run(
["ip", "-o", "route", "get", "1.1.1.1"],
capture_output=True, text=True, check=False,
)
if result.returncode != 0:
return ""
tokens = result.stdout.split()
# `... src <addr> ...` — the address the host would use as source.
if "src" in tokens:
idx = tokens.index("src")
if idx + 1 < len(tokens):
return tokens[idx + 1]
return ""
# Guest-side connect test: exit 0 = reached the canary (LEAK), 1 =
# blocked (good), 2 = no usable tool (inconclusive -> fail-closed).
_GUEST_PROBE = r"""
h="$1"; p="$2"
if command -v python3 >/dev/null 2>&1; then
python3 - "$h" "$p" <<'PY'
import socket, sys
s = socket.socket(); s.settimeout(3)
sys.exit(0 if s.connect_ex((sys.argv[1], int(sys.argv[2]))) == 0 else 1)
PY
elif command -v bash >/dev/null 2>&1; then
timeout 3 bash -c "exec 3<>/dev/tcp/$h/$p" >/dev/null 2>&1
elif command -v nc >/dev/null 2>&1; then
nc -w3 -z "$h" "$p" >/dev/null 2>&1
else
exit 2
fi
"""
def verify_isolation(private_key: Path, guest_ip: str) -> None:
"""Prove the VM cannot reach the host's primary IP. Dies
(fail-closed) on a confirmed leak or if the guest has no tool to
run the test. Warns (does not fail) when the host has no canary
address to test against."""
canary_ip = _host_primary_ip()
if not canary_ip:
warn("isolation probe skipped: host has no non-TAP route to test "
"against (isolated box).")
return
listener = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
listener.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
listener.bind((canary_ip, 0))
listener.listen(1)
listener.settimeout(6)
canary_port = listener.getsockname()[1]
accepted: list[bool] = []
def _accept() -> None:
try:
conn, _ = listener.accept()
accepted.append(True)
conn.close()
except OSError:
pass
thread = threading.Thread(target=_accept, daemon=True)
thread.start()
ssh = util.ssh_base_argv(private_key, guest_ip)
result = subprocess.run(
[*ssh, "--", "sh", "-s", "--", canary_ip, str(canary_port)],
input=_GUEST_PROBE, capture_output=True, text=True, check=False,
)
thread.join(timeout=7)
listener.close()
if result.returncode == 0 or accepted:
die(f"ISOLATION FAILURE: the VM reached the host canary "
f"{canary_ip}:{canary_port}. The egress boundary is not in "
f"force — refusing to run the agent (fail-closed). Verify the "
f"nft table with: ./cli.py backend setup --backend=firecracker")
if result.returncode == 2:
die("isolation probe inconclusive: the guest has no python3/bash/nc "
"to run the connectivity test. Refusing to continue "
"(fail-closed).")
info(f"isolation verified: VM cannot reach host {canary_ip}")
-404
View File
@@ -1,404 +0,0 @@
"""Launch flow for the Firecracker backend.
Per bottle:
1. mint the egress CA, build the agent image (docker), export it to a
cached ext4 rootfs;
2. claim a free TAP pool slot (rootless flock);
3. bring up the Docker sidecar bundle, publishing egress / git-gate /
supervise on the slot's host-side TAP IP at fixed ports;
4. boot the microVM on that TAP; wait for SSH;
5. provision (CA, prompt, skills, workspace, git, supervise) over SSH.
Isolation is enforced by the operator-provisioned nft table (checked
fail-closed in preflight): a VM reaches only its sidecar (DNAT'd from
the host TAP IP) and nothing else. The agent's HTTPS_PROXY therefore
points at `http://<host_tap_ip>:9099`, its only route to the world.
"""
from __future__ import annotations
import dataclasses
import os
import subprocess
from contextlib import ExitStack, contextmanager
from pathlib import Path
from typing import Callable, Generator
from ...bottle_state import (
egress_state_dir,
git_gate_state_dir,
read_committed_image,
)
from ...egress import (
EGRESS_ROUTES_IN_CONTAINER,
egress_agent_env_entries,
egress_resolve_token_values,
egress_sidecar_env_entries,
)
from ...git_gate import (
provision_git_gate_dynamic_keys,
revoke_git_gate_provisioned_keys,
)
from ...log import die, info, warn
from ...supervise import DB_PATH_IN_CONTAINER, SUPERVISE_PORT
from ...util import expand_tilde
from ..docker.egress import (
EGRESS_CA_IN_CONTAINER,
EGRESS_PORT,
egress_tls_init,
)
from ..docker.git_gate import (
GIT_GATE_ACCESS_HOOK_IN_CONTAINER,
GIT_GATE_CREDS_DIR_IN_CONTAINER,
GIT_GATE_ENTRYPOINT_IN_CONTAINER,
GIT_GATE_HOOK_IN_CONTAINER,
)
from ..docker.sidecar_bundle import (
SIDECAR_BUNDLE_DOCKERFILE,
SIDECAR_BUNDLE_IMAGE,
)
from ..util import AGENT_CA_BUNDLE, AGENT_CA_PATH
from . import firecracker_vm, isolation_probe, netpool, util
from .bottle import FirecrackerBottle
from .bottle_plan import FirecrackerBottlePlan
_REPO_DIR = str(Path(__file__).resolve().parent.parent.parent.parent)
_GIT_HTTP_PORT = 9420
_GIT_GATE_READY_FILE = "/run/git-gate/ready"
def sidecar_container_name(slug: str) -> str:
return f"bot-bottle-sidecars-{slug}"
@contextmanager
def launch(
plan: FirecrackerBottlePlan,
*,
provision: Callable[[FirecrackerBottlePlan, "FirecrackerBottle"], str | None],
) -> Generator[FirecrackerBottle, None, None]:
stack = ExitStack()
bottle_for_revoke = plan.manifest.bottle
git_gate_dir_for_revoke = git_gate_state_dir(plan.slug)
def teardown() -> None:
teardown_exc: BaseException | None = None
try:
stack.close()
except BaseException as exc: # noqa: W0718 - teardown must continue
teardown_exc = exc
warn(f"firecracker teardown failed: {exc!r}")
revoke_git_gate_provisioned_keys(bottle_for_revoke, git_gate_dir_for_revoke)
if teardown_exc is not None:
raise teardown_exc
try:
plan = _mint_certs(plan)
plan = _build_agent_image(plan)
# Claim a TAP slot; the flock is held until teardown closes it.
slot, lock = netpool.allocate(plan.slug)
stack.callback(lock.close)
info(f"firecracker slot {slot.iface}: host={slot.host_ip} "
f"guest={slot.guest_ip}")
plan = _provision_git_gate_keys(plan)
sidecar_name = sidecar_container_name(plan.slug)
_force_remove_container(sidecar_name)
_start_sidecar_bundle(plan, sidecar_name, slot.host_ip)
stack.callback(_force_remove_container, sidecar_name)
_stage_git_gate(plan, sidecar_name)
plan = _stamp_agent_urls(plan, slot.host_ip)
# Build the per-bottle rootfs + SSH key, then boot.
base_dir = util.build_base_rootfs_dir(plan.image)
run_dir = util.cache_dir() / "run" / plan.slug
run_dir.mkdir(parents=True, exist_ok=True)
rootfs = run_dir / "rootfs.ext4"
util.build_rootfs_ext4(base_dir, rootfs)
private_key, pubkey = util.generate_keypair(run_dir)
vm = firecracker_vm.boot(
name=plan.container_name,
rootfs=rootfs,
tap=slot.iface,
guest_ip=slot.guest_ip,
host_ip=slot.host_ip,
pubkey=pubkey,
run_dir=run_dir,
)
stack.callback(vm.terminate)
firecracker_vm.wait_for_ssh(vm, private_key)
# Authoritative fail-closed egress-boundary check, before the
# agent runs: prove the VM cannot reach the host directly.
isolation_probe.verify_isolation(private_key, slot.guest_ip)
bottle = FirecrackerBottle(
plan.container_name,
private_key=private_key,
guest_ip=slot.guest_ip,
guest_env=_agent_guest_env(plan, slot.host_ip),
agent_command=plan.agent_command,
agent_prompt_mode=plan.agent_prompt_mode,
agent_provider_template=plan.agent_provider_template,
terminal_title=(
f"{plan.spec.label} ({plan.spec.agent_name})"
if plan.spec.label else plan.spec.agent_name
),
terminal_color=plan.spec.color,
agent_workdir=plan.workspace_plan.workdir,
)
bottle.prompt_path = provision(plan, bottle)
yield bottle
finally:
teardown()
def _mint_certs(plan: FirecrackerBottlePlan) -> FirecrackerBottlePlan:
egress_ca_host, egress_ca_cert_only = egress_tls_init(egress_state_dir(plan.slug))
egress_plan = dataclasses.replace(
plan.egress_plan,
mitmproxy_ca_host_path=egress_ca_host,
mitmproxy_ca_cert_only_host_path=egress_ca_cert_only,
)
return dataclasses.replace(plan, egress_plan=egress_plan)
def _build_agent_image(plan: FirecrackerBottlePlan) -> FirecrackerBottlePlan:
_docker_build(SIDECAR_BUNDLE_IMAGE, _REPO_DIR, dockerfile=SIDECAR_BUNDLE_DOCKERFILE)
committed = read_committed_image(plan.slug)
if committed and _image_exists(committed):
info(f"using committed image {committed!r}")
return dataclasses.replace(
plan,
agent_provision=dataclasses.replace(plan.agent_provision, image=committed),
)
_docker_build(plan.image, _REPO_DIR, dockerfile=plan.dockerfile_path)
return plan
def _provision_git_gate_keys(plan: FirecrackerBottlePlan) -> FirecrackerBottlePlan:
if not plan.git_gate_plan.upstreams:
return plan
git_gate_plan = provision_git_gate_dynamic_keys(
plan.manifest.bottle, plan.git_gate_plan, git_gate_state_dir(plan.slug),
)
return dataclasses.replace(plan, git_gate_plan=git_gate_plan)
def _stamp_agent_urls(
plan: FirecrackerBottlePlan, host_ip: str,
) -> FirecrackerBottlePlan:
proxy_url = f"http://{host_ip}:{EGRESS_PORT}"
supervise_url = (
f"http://{host_ip}:{SUPERVISE_PORT}/" if plan.supervise_plan is not None else ""
)
git_gate_url = (
f"http://{host_ip}:{_GIT_HTTP_PORT}" if plan.git_gate_plan.upstreams else ""
)
return dataclasses.replace(
plan,
agent_proxy_url=proxy_url,
agent_git_gate_url=git_gate_url,
agent_supervise_url=supervise_url,
)
# --- sidecar bundle (Docker) ----------------------------------------
def _start_sidecar_bundle(
plan: FirecrackerBottlePlan, sidecar_name: str, host_ip: str,
) -> None:
argv = ["docker", "run", "--name", sidecar_name, "--detach", "--rm",
"-e", f"BOT_BOTTLE_SIDECAR_DAEMONS={','.join(_sidecar_daemons(plan))}"]
for entry in _sidecar_env_entries(plan):
argv += ["-e", entry]
for host_path, container_path, read_only in _sidecar_mounts(plan):
argv += ["-v", f"{host_path}:{container_path}{':ro' if read_only else ''}"]
# Publish on the slot's host TAP IP at fixed ports — each bottle
# has a distinct host_ip, so fixed ports never collide, and the VM
# reaches them at a stable, well-known address (its only route out).
for port in _sidecar_ports(plan):
argv += ["-p", f"{host_ip}:{port}:{port}"]
argv.append(SIDECAR_BUNDLE_IMAGE)
effective_env = {**dict(os.environ), **plan.agent_provision.provisioned_env}
token_values = egress_resolve_token_values(
plan.egress_plan.token_env_map, effective_env,
)
env = {**os.environ, **token_values}
info(f"docker run sidecar bundle {sidecar_name} (published on {host_ip})")
result = subprocess.run(argv, capture_output=True, text=True, env=env, check=False)
if result.returncode != 0:
die(f"docker run for sidecar bundle {sidecar_name} failed: "
f"{(result.stderr or '').strip() or '<no stderr>'}")
def _sidecar_daemons(plan: FirecrackerBottlePlan) -> tuple[str, ...]:
daemons = ["egress"]
if plan.git_gate_plan.upstreams:
daemons += ["git-gate", "git-http"]
if plan.supervise_plan is not None:
daemons.append("supervise")
return tuple(daemons)
def _sidecar_ports(plan: FirecrackerBottlePlan) -> tuple[int, ...]:
ports = [EGRESS_PORT]
if plan.git_gate_plan.upstreams:
ports.append(_GIT_HTTP_PORT)
if plan.supervise_plan is not None:
ports.append(SUPERVISE_PORT)
return tuple(ports)
def _sidecar_env_entries(plan: FirecrackerBottlePlan) -> tuple[str, ...]:
env: list[str] = list(egress_sidecar_env_entries(plan.egress_plan))
if plan.git_gate_plan.upstreams:
env.append(f"BOT_BOTTLE_GIT_GATE_READY_FILE={_GIT_GATE_READY_FILE}")
if plan.supervise_plan is not None:
env += [
f"SUPERVISE_BOTTLE_SLUG={plan.slug}",
f"SUPERVISE_DB_PATH={DB_PATH_IN_CONTAINER}",
f"SUPERVISE_PORT={SUPERVISE_PORT}",
]
return tuple(env)
def _sidecar_mounts(
plan: FirecrackerBottlePlan,
) -> tuple[tuple[str, str, bool], ...]:
mounts: list[tuple[str, str, bool]] = []
ep = plan.egress_plan
mounts.append((str(ep.mitmproxy_ca_host_path.parent),
str(Path(EGRESS_CA_IN_CONTAINER).parent), False))
if ep.routes:
mounts.append((str(ep.routes_path.parent),
str(Path(EGRESS_ROUTES_IN_CONTAINER).parent), True))
sp = plan.supervise_plan
if sp is not None:
mounts.append((str(sp.db_path.parent),
str(Path(DB_PATH_IN_CONTAINER).parent), False))
return tuple(mounts)
def _stage_git_gate(plan: FirecrackerBottlePlan, sidecar_name: str) -> None:
gp = plan.git_gate_plan
if not gp.upstreams:
return
_docker_exec(sidecar_name, [
"mkdir", "-p",
str(Path(GIT_GATE_HOOK_IN_CONTAINER).parent),
GIT_GATE_CREDS_DIR_IN_CONTAINER, "/git",
str(Path(_GIT_GATE_READY_FILE).parent),
])
for host_path, container_path in _git_gate_files(plan):
_docker_cp(host_path, f"{sidecar_name}:{container_path}")
_docker_exec(sidecar_name, [
"sh", "-c",
f"chmod 755 {GIT_GATE_ENTRYPOINT_IN_CONTAINER} "
f"{GIT_GATE_HOOK_IN_CONTAINER} {GIT_GATE_ACCESS_HOOK_IN_CONTAINER} && "
f"chmod 600 {GIT_GATE_CREDS_DIR_IN_CONTAINER}/* && "
f"touch {_GIT_GATE_READY_FILE}",
])
def _git_gate_files(plan: FirecrackerBottlePlan) -> tuple[tuple[str, str], ...]:
gp = plan.git_gate_plan
files: list[tuple[str, str]] = [
(str(gp.entrypoint_script), GIT_GATE_ENTRYPOINT_IN_CONTAINER),
(str(gp.hook_script), GIT_GATE_HOOK_IN_CONTAINER),
(str(gp.access_hook_script), GIT_GATE_ACCESS_HOOK_IN_CONTAINER),
]
for upstream in gp.upstreams:
files.append((expand_tilde(upstream.identity_file),
f"{GIT_GATE_CREDS_DIR_IN_CONTAINER}/{upstream.name}-key"))
if upstream.known_hosts_file:
files.append((str(upstream.known_hosts_file),
f"{GIT_GATE_CREDS_DIR_IN_CONTAINER}/{upstream.name}-known_hosts"))
return tuple(files)
# --- agent guest env -------------------------------------------------
def _agent_guest_env(plan: FirecrackerBottlePlan, host_ip: str) -> dict[str, str]:
"""Env injected into every agent/exec call over SSH. The VM has no
baked process env (it just runs init), so the proxy/CA/git/supervise
wiring is applied per-invocation."""
proxy_url = f"http://{host_ip}:{EGRESS_PORT}"
no_proxy = f"localhost,127.0.0.1,{host_ip}"
env: dict[str, str] = {
"HTTPS_PROXY": proxy_url, "HTTP_PROXY": proxy_url,
"https_proxy": proxy_url, "http_proxy": proxy_url,
"NO_PROXY": no_proxy, "no_proxy": no_proxy,
"NODE_EXTRA_CA_CERTS": AGENT_CA_PATH,
"SSL_CERT_FILE": AGENT_CA_BUNDLE,
"REQUESTS_CA_BUNDLE": AGENT_CA_BUNDLE,
}
if plan.agent_git_gate_url:
env["GIT_GATE_URL"] = plan.agent_git_gate_url
if plan.agent_supervise_url:
env["MCP_SUPERVISE_URL"] = plan.agent_supervise_url
for entry in egress_agent_env_entries(plan.egress_plan):
key, _, value = entry.partition("=")
env[key] = value
env.update(plan.agent_provision.guest_env)
# Forwarded (bare-name) env: resolve host values now, since the VM
# can't inherit them from a `docker run --env NAME`.
for name in plan.forwarded_env:
value = os.environ.get(name)
if value is not None:
env[name] = value
return env
# --- docker helpers --------------------------------------------------
def _docker_build(ref: str, context: str, *, dockerfile: str = "") -> None:
info(f"docker build {ref}")
args = ["docker", "build", "-t", ref]
if dockerfile:
if not os.path.isabs(dockerfile):
dockerfile = os.path.join(context, dockerfile)
args += ["-f", dockerfile]
args.append(context)
result = subprocess.run(args, check=False)
if result.returncode != 0:
die(f"docker build for {ref!r} failed")
def _image_exists(ref: str) -> bool:
return subprocess.run(
["docker", "image", "inspect", ref],
stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL, check=False,
).returncode == 0
def _force_remove_container(name: str) -> None:
subprocess.run(
["docker", "rm", "-f", name],
stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL, check=False,
)
def _docker_exec(name: str, argv: list[str]) -> None:
result = subprocess.run(
["docker", "exec", name, *argv], capture_output=True, text=True, check=False,
)
if result.returncode != 0:
die(f"docker exec in {name} failed: "
f"{(result.stderr or '').strip() or '<no stderr>'}")
def _docker_cp(host_path: str, dest: str) -> None:
result = subprocess.run(
["docker", "cp", host_path, dest], capture_output=True, text=True, check=False,
)
if result.returncode != 0:
die(f"docker cp {host_path} -> {dest} failed: "
f"{(result.stderr or '').strip() or '<no stderr>'}")
-318
View File
@@ -1,318 +0,0 @@
"""Firecracker network pool: constants, IP math, allocation, and the
config renderers (shell command + NixOS module) shown to operators.
The Firecracker backend needs a privileged one-time network setup:
a pool of point-to-point TAP devices (owned by the invoking user, so
`./cli.py start` never needs root) and a dedicated nftables table that
isolates every VM. This module is the single source of truth for the
pool parameters — the shell script (`scripts/firecracker-netpool.sh`),
the printed NixOS snippet, and the backend's fail-closed preflight all
derive from the constants here so they can't drift.
Topology (per slot i):
* TAP ``bbfc{i}`` — no shared bridge, so no docker0 / virbr0 / cni0
/ br-* collisions.
* a /31 host<->guest link: host = base + 2i (the gateway the VM
routes through), guest = base + 2i + 1 (the VM's address).
* isolation via ``table inet bot_bottle_fc``: a VM reaches only its
own sidecar (DNAT'd from the host TAP IP) and nothing else.
The default IP block is ``10.243.0.0/16`` — an intentionally obscure
corner of RFC-1918 private space. RFC-1918 is the range *designated*
for private links; we pick a high, unusual /16 to steer clear of the
common occupants (Docker's 172.17-31, libvirt's 192.168.122, k8s
10.42/10.244, and typical home LANs on 192.168.0/1.x or 10.0.0.x).
We deliberately avoid ``100.64.0.0/10`` (RFC-6598 CGNAT) because
Tailscale hands out node addresses from exactly that range. No default
is collision-proof, so `overlapping_routes()` is the real guard —
`backend setup`/`status` and the launch preflight call it to catch
a base that clashes with something already on the host.
"""
from __future__ import annotations
import fcntl
import ipaddress
import os
import subprocess
from dataclasses import dataclass
from pathlib import Path
from typing import IO
from ...log import die
# Interface names are capped at 15 chars (IFNAMSIZ-1); "bbfc" + a small
# index stays well under that and is distinctive enough to grep for.
IFACE_PREFIX = os.environ.get("BOT_BOTTLE_FC_IFACE_PREFIX", "bbfc")
NFT_TABLE = "bot_bottle_fc"
def pool_size() -> int:
return int(os.environ.get("BOT_BOTTLE_FC_POOL_SIZE", "8"))
def ip_base() -> str:
return os.environ.get("BOT_BOTTLE_FC_IP_BASE", "10.243.0.0")
# Sidecar ports the VM reaches at its host-side TAP IP. Kept in sync
# with the backend constants (egress 9099, supervise 9100, git-http
# 9420); rendered into the setup output for operator visibility.
SIDECAR_PORTS = (9099, 9100, 9420)
@dataclass(frozen=True)
class Slot:
"""One pool slot: a TAP device and its host/guest /31 addresses."""
index: int
iface: str
host_ip: str
guest_ip: str
@property
def guest_cidr(self) -> str:
"""Guest address with the /31 prefix, for the kernel `ip=` arg."""
return f"{self.guest_ip}/31"
def slot(index: int) -> Slot:
base = int(ipaddress.IPv4Address(ip_base()))
return Slot(
index=index,
iface=f"{IFACE_PREFIX}{index}",
host_ip=str(ipaddress.IPv4Address(base + 2 * index)),
guest_ip=str(ipaddress.IPv4Address(base + 2 * index + 1)),
)
def all_slots() -> list[Slot]:
return [slot(i) for i in range(pool_size())]
# --- fail-closed verification ---------------------------------------
def _run_ok(argv: list[str]) -> bool:
"""Run a probe command, treating a missing binary as failure
(rather than crashing) so callers can stay fail-closed."""
try:
return subprocess.run(
argv, stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL,
check=False,
).returncode == 0
except FileNotFoundError:
return False
def nft_table_present() -> bool:
"""True iff the isolation table exists in the active nftables
backend. The backend's preflight treats absence as fatal — the VM
must not boot without its egress boundary in place.
Listing may require root; when it can't be confirmed here the
launch path falls back to an empirical post-boot isolation probe.
Returns False (fail-closed) if `nft` is unavailable."""
return _run_ok(["nft", "list", "table", "inet", NFT_TABLE])
def tap_present(iface: str) -> bool:
# `ip link show` is unprivileged, so the TAP-pool check is reliable
# for the non-root launcher.
return _run_ok(["ip", "link", "show", iface])
def missing_taps() -> list[str]:
"""Pool TAPs that the one-time setup has not created yet."""
return [s.iface for s in all_slots() if not tap_present(s.iface)]
@dataclass(frozen=True)
class RouteConflict:
"""An existing host route whose destination overlaps the pool range
but isn't one of our own bbfc TAPs — i.e. the chosen IP base clashes
with something already on the host (a Tailscale CGNAT peer, a
docker/libvirt bridge, the LAN…)."""
dst: str
dev: str
def _pool_span() -> tuple[int, int]:
"""Inclusive [first, last] integer bounds of every address the pool
occupies: base .. base + 2*pool_size - 1."""
base = int(ipaddress.IPv4Address(ip_base()))
return base, base + 2 * pool_size() - 1
def overlapping_routes() -> list[RouteConflict]:
"""Existing routes whose destination overlaps the pool's address
range, excluding our own `bbfc*` TAP routes and the default route.
A non-empty result means `BOT_BOTTLE_FC_IP_BASE` collides with
something already configured on this host, so the pool would shadow
(or be shadowed by) it. Empty on success — or when `ip` is missing
or unparseable, since this is an advisory guard, not the fail-closed
isolation check (that's the nft table + post-boot probe)."""
import json
try:
proc = subprocess.run(
["ip", "-json", "route", "show", "table", "all"],
capture_output=True, text=True, check=False,
)
except FileNotFoundError:
return []
if proc.returncode != 0 or not proc.stdout.strip():
return []
try:
routes = json.loads(proc.stdout)
except json.JSONDecodeError:
return []
lo, hi = _pool_span()
conflicts: list[RouteConflict] = []
for r in routes:
if not isinstance(r, dict):
continue
dst = r.get("dst")
dev = r.get("dev", "")
if not isinstance(dst, str) or dst in ("", "default"):
continue
if isinstance(dev, str) and dev.startswith(IFACE_PREFIX):
continue # our own pool link
try:
net = ipaddress.ip_network(dst, strict=False)
except ValueError:
continue
if net.version != 4:
continue
r_lo = int(net.network_address)
r_hi = int(net.broadcast_address)
if r_lo <= hi and lo <= r_hi: # ranges intersect
conflicts.append(RouteConflict(dst=dst, dev=str(dev)))
return conflicts
# --- allocation ------------------------------------------------------
def _lock_dir() -> Path:
d = Path.home() / ".cache" / "bot-bottle" / "firecracker" / "pool"
d.mkdir(parents=True, exist_ok=True)
return d
def allocate(slug: str) -> tuple[Slot, IO[str]]:
"""Claim a free pool slot for one bottle. Returns the slot and the
held lock file — the caller keeps it open for the VM's lifetime and
closes it on teardown; the flock auto-releases if the launcher
crashes, so a slot is never leaked. Dies when the pool is
exhausted.
A per-slot `flock` (rather than inspecting running processes) makes
allocation race-free across concurrent launches without a central
registry: the first launcher to grab the lock owns the slot."""
del slug # logged by the caller; allocation is purely lock-driven
for s in all_slots():
lock_path = _lock_dir() / f"{s.iface}.lock"
handle = open(lock_path, "w", encoding="utf-8")
try:
fcntl.flock(handle, fcntl.LOCK_EX | fcntl.LOCK_NB)
except OSError:
handle.close()
continue
return s, handle
die(f"Firecracker TAP pool exhausted ({pool_size()} slots, all in "
f"use). Stop a running bottle or raise BOT_BOTTLE_FC_POOL_SIZE "
f"and re-run `./cli.py backend setup --backend=firecracker`.")
raise AssertionError("unreachable")
# --- config renderers (shown by `./cli.py backend setup`) -----------
def render_shell_setup() -> str:
"""The imperative command for non-NixOS hosts."""
env = _nondefault_env()
prefix = f"{env} " if env else ""
return f"sudo {prefix}./scripts/firecracker-netpool.sh up"
def render_nixos_module() -> str:
"""A paste-ready NixOS snippet, derived from the live constants.
Uses systemd-networkd tap netdevs (so TAPs are only reconfigured
when this config changes, not torn down on every rebuild — which
would drop running VMs) and `networking.nftables.tables.<name>`
(an independent table that coexists with Docker's iptables rules
rather than replacing the whole ruleset)."""
slots = all_slots()
owner = os.environ.get("USER", "youruser")
netdevs = "\n".join(
f''' "10-{s.iface}" = {{
netdevConfig = {{ Name = "{s.iface}"; Kind = "tap"; }};
tapConfig = {{ User = "{owner}"; }};
}};'''
for s in slots
)
networks = "\n".join(
f''' "10-{s.iface}" = {{
matchConfig.Name = "{s.iface}";
address = [ "{s.host_ip}/31" ];
networkConfig.ConfigureWithoutCarrier = true;
}};'''
for s in slots
)
return f"""# bot-bottle Firecracker network pool ({len(slots)} slots, base {ip_base()}).
# Generated by `./cli.py backend setup --backend=firecracker`; owner user = {owner!r}.
{{ ... }}:
{{
boot.kernel.sysctl."net.ipv4.ip_forward" = 1;
systemd.network.enable = true;
systemd.network.netdevs = {{
{netdevs}
}};
systemd.network.networks = {{
{networks}
}};
# Independent table — does not touch Docker/ufw/firewalld rules.
# A bottle VM (traffic from a bbfc* TAP) may reach only its own
# sidecar (DNAT'd from the host TAP IP); everything else is dropped,
# so there is no egress except through the sidecar proxy.
networking.nftables.enable = true;
networking.nftables.tables."{NFT_TABLE}" = {{
family = "inet";
content = ''
chain forward {{
type filter hook forward priority -10; policy accept;
iifname != "{IFACE_PREFIX}*" return
ct state established,related accept
ct status dnat accept
drop
}}
chain input {{
type filter hook input priority -10; policy accept;
iifname != "{IFACE_PREFIX}*" return
ct state established,related accept
drop
}}
'';
}};
}}
"""
def _nondefault_env() -> str:
"""Render any non-default pool env overrides so the printed shell
command reproduces the operator's current settings."""
pairs = []
if os.environ.get("BOT_BOTTLE_FC_POOL_SIZE"):
pairs.append(f"BOT_BOTTLE_FC_POOL_SIZE={pool_size()}")
if os.environ.get("BOT_BOTTLE_FC_IP_BASE"):
pairs.append(f"BOT_BOTTLE_FC_IP_BASE={ip_base()}")
if os.environ.get("BOT_BOTTLE_FC_IFACE_PREFIX"):
pairs.append(f"BOT_BOTTLE_FC_IFACE_PREFIX={IFACE_PREFIX}")
return " ".join(pairs)
@@ -1,47 +0,0 @@
"""Prepare step for the Firecracker backend."""
from __future__ import annotations
from pathlib import Path
from ...agent_provider import AgentProvisionPlan
from ...egress import EgressPlan
from ...env import ResolvedEnv
from ...git_gate import GitGatePlan
from ...manifest import Manifest
from ...supervise import SupervisePlan
from .. import BottleSpec
from . import util
from .bottle_plan import FirecrackerBottlePlan
def preflight() -> None:
util.require_firecracker()
def build_guest_env(resolved_env: ResolvedEnv) -> dict[str, str]:
return dict(resolved_env.literals)
def resolve_plan(
spec: BottleSpec,
manifest: Manifest,
slug: str,
resolved_env: ResolvedEnv,
agent_provision_plan: AgentProvisionPlan,
egress_plan: EgressPlan,
supervise_plan: SupervisePlan | None,
git_gate_plan: GitGatePlan,
stage_dir: Path,
) -> FirecrackerBottlePlan:
return FirecrackerBottlePlan(
spec=spec,
manifest=manifest,
stage_dir=stage_dir,
slug=slug,
forwarded_env=dict(resolved_env.forwarded),
git_gate_plan=git_gate_plan,
egress_plan=egress_plan,
supervise_plan=supervise_plan,
agent_provision=agent_provision_plan,
)
-165
View File
@@ -1,165 +0,0 @@
"""Host setup + status for the Firecracker backend.
`setup()` prints the host-appropriate config for the privileged,
one-time network pool (TAP devices + isolation nftables table) the
backend needs. On NixOS it points at the flake module (and prints a
paste-able fallback); elsewhere it prints the sudo command for the
bundled setup script. `status()` reports what's present, including
whether the pool range collides with an existing route.
Called through `FirecrackerBottleBackend.setup` / `.status`, which the
generic `./cli.py backend {setup,status}` command dispatches to.
"""
from __future__ import annotations
import os
import shutil
import sys
from pathlib import Path
from . import netpool
from . import util
_FC_RELEASES = "https://github.com/firecracker-microvm/firecracker/releases"
def _owner() -> str:
return os.environ.get("USER", "youruser")
def _print_prereqs() -> None:
"""The firecracker binary + KVM + guest artifacts, shown before the
privileged network-pool step so operators see the full picture."""
fc = shutil.which("firecracker")
if fc:
sys.stderr.write(f"1) firecracker binary: found ({fc}).\n")
else:
sys.stderr.write(
"1) firecracker binary: NOT found on PATH. Install a release binary "
"and put it on PATH:\n"
f" {_FC_RELEASES}\n"
" e.g.: download firecracker-vX.Y.Z-$(uname -m).tgz, extract, and\n"
" install -m755 release-*/firecracker-* ~/.local/bin/firecracker\n"
" (NixOS: not packaged as a user binary — fetch the release, pin\n"
" the version, and add it to PATH.)\n"
)
if util.is_host_capable():
sys.stderr.write(" KVM: /dev/kvm present.\n")
else:
sys.stderr.write(
" KVM: /dev/kvm missing/unusable — load kvm-intel/kvm-amd, enable\n"
" virtualization in firmware, and add your user to the `kvm` group.\n"
)
sys.stderr.write(
" Guest artifacts: a kernel (BOT_BOTTLE_FC_KERNEL) and static dropbear\n"
" (BOT_BOTTLE_FC_DROPBEAR) must be cached, and `mke2fs` (e2fsprogs) is\n"
" needed to build the rootfs.\n\n"
)
def _is_nixos() -> bool:
if Path("/etc/NIXOS").exists():
return True
try:
return "ID=nixos" in Path("/etc/os-release").read_text()
except OSError:
return False
def _warn_overlaps() -> None:
"""Warn if the chosen pool range collides with an existing host route
(Tailscale CGNAT peer, a docker/libvirt bridge, the LAN…)."""
conflicts = netpool.overlapping_routes()
if not conflicts:
return
detail = "\n".join(f" {c.dst} dev {c.dev}" for c in conflicts)
sys.stderr.write(
f"WARNING: pool range (base {netpool.ip_base()}, "
f"{netpool.pool_size()} slots) overlaps existing routes:\n"
f"{detail}\n"
"Set BOT_BOTTLE_FC_IP_BASE to a free range before setup, or the "
"pool may shadow / be shadowed by the above.\n\n"
)
def setup() -> int:
sys.stderr.write("Firecracker backend — one-time host setup.\n\n")
_print_prereqs()
slots = netpool.all_slots()
sys.stderr.write(
f"2) network pool: {len(slots)} slots "
f"({slots[0].iface}..{slots[-1].iface}), base {netpool.ip_base()}"
f"TAP devices + nft isolation table, privileged (needs root once).\n\n"
)
_warn_overlaps()
if _is_nixos():
sys.stderr.write(
"Detected NixOS. Preferred: consume the flake module (versioned, "
"no copy-paste drift):\n\n"
" # flake inputs (point at wherever you host bot-bottle):\n"
" inputs.bot-bottle.url = \"git+ssh://<your-bot-bottle-remote>\";\n"
" # host module:\n"
" imports = [ inputs.bot-bottle.nixosModules.firecracker-netpool ];\n"
f" services.bot-bottle-firecracker = {{ enable = true; owner = \"{_owner()}\"; }};\n\n"
"Then `nixos-rebuild switch`. Channel (non-flake) users can "
"`imports = [ <bot-bottle>/nix/firecracker-netpool.nix ];` instead.\n\n"
"Fallback — paste this generated module directly:\n\n"
)
sys.stdout.write(netpool.render_nixos_module())
else:
sys.stderr.write("Run the one-time setup as root:\n\n")
sys.stdout.write(netpool.render_shell_setup() + "\n")
sys.stderr.write(
"\n(On NixOS, use the declarative module instead — this host "
"was not detected as NixOS.)\n"
)
return 0
def teardown() -> int:
slots = netpool.all_slots()
sys.stderr.write(
f"Undo the Firecracker network pool ({len(slots)} slots, base "
f"{netpool.ip_base()}) — a privileged, one-time operation.\n\n"
)
if _is_nixos():
sys.stderr.write(
"On NixOS: set `services.bot-bottle-firecracker.enable = false;` "
"(or drop the module import) and `nixos-rebuild switch`. The TAP "
"netdevs and nft table are removed declaratively.\n\n"
"To tear down imperatively before a rebuild (does not persist):\n\n"
)
else:
sys.stderr.write("Run the teardown as root:\n\n")
sys.stdout.write("sudo ./scripts/firecracker-netpool.sh down\n")
return 0
def status() -> int:
ok = True
if netpool.nft_table_present():
sys.stderr.write(f"nft table inet {netpool.NFT_TABLE}: present\n")
else:
sys.stderr.write(f"nft table inet {netpool.NFT_TABLE}: MISSING\n")
ok = False
missing = netpool.missing_taps()
total = netpool.pool_size()
if missing:
sys.stderr.write(f"TAP pool: {total - len(missing)}/{total} present "
f"(missing: {', '.join(missing)})\n")
ok = False
else:
sys.stderr.write(f"TAP pool: {total}/{total} present\n")
conflicts = netpool.overlapping_routes()
if conflicts:
detail = ", ".join(f"{c.dst} dev {c.dev}" for c in conflicts)
sys.stderr.write(f"range overlap: base {netpool.ip_base()} CLASHES "
f"with {detail}\n")
ok = False
else:
sys.stderr.write(f"range overlap: none (base {netpool.ip_base()})\n")
if not ok:
sys.stderr.write("\nRun: ./cli.py backend setup --backend=firecracker\n")
return 0 if ok else 1
-311
View File
@@ -1,311 +0,0 @@
"""Host-side primitives for the Firecracker backend.
Covers the pieces that don't need root at launch time: locating the
firecracker binary / guest kernel / injected dropbear, the fail-closed
preflight (KVM + kernel + isolation table + TAP pool must all be
present before a VM boots), the rootless rootfs pipeline
(`docker export` -> `mke2fs -d`, no mount), and per-bottle SSH key
generation.
The privileged network setup (TAP pool + nft table) is a one-time
operator step — see `netpool.py`, `scripts/firecracker-netpool.sh`,
and `./cli.py backend setup --backend=firecracker`.
"""
from __future__ import annotations
import os
import platform
import shutil
import subprocess
from pathlib import Path
from ...log import die, info, warn
from . import netpool
# Guest agent images are Debian-family with USER node; the VM is
# reached over SSH as root (init drops the pubkey into both root's and
# node's authorized_keys) and commands `runuser` down to node.
GUEST_SSH_USER = "root"
# `/dev/kvm` must exist and be openable by the invoking user.
_KVM_DEVICE = "/dev/kvm"
def cache_dir() -> Path:
d = Path(
os.environ.get(
"BOT_BOTTLE_FC_CACHE",
str(Path.home() / ".cache" / "bot-bottle" / "firecracker"),
)
)
d.mkdir(parents=True, exist_ok=True)
return d
def kernel_path() -> Path:
return Path(os.environ.get("BOT_BOTTLE_FC_KERNEL", str(cache_dir() / "vmlinux")))
def dropbear_path() -> Path:
"""The static dropbear injected into every guest rootfs as its SSH
server. Must be statically linked — the guest has none of the
host's shared libraries."""
return Path(
os.environ.get("BOT_BOTTLE_FC_DROPBEAR", str(cache_dir() / "dropbear"))
)
# --- availability + preflight ---------------------------------------
def is_linux() -> bool:
return platform.system() == "Linux"
def is_host_capable() -> bool:
"""Whether this host *could* run firecracker — Linux with KVM —
regardless of whether the `firecracker` binary is installed. Used
for default-backend selection so a KVM Linux host that hasn't
installed firecracker yet still selects it and gets an install
pointer at launch (see `require_firecracker`), rather than silently
falling back to docker."""
return is_linux() and os.path.exists(_KVM_DEVICE)
def is_available() -> bool:
"""Cheap capability probe used by cross-backend enumeration —
firecracker on PATH, on Linux, with KVM. Does not check the
(operator-provisioned) kernel / pool, so an available-but-unset
host still shows up and gets an actionable error at launch."""
return is_host_capable() and shutil.which("firecracker") is not None
def require_firecracker() -> None:
"""Fail-closed preflight. Every check that gates the security
boundary (the isolation table, the TAP pool) errors rather than
booting a VM without it."""
if not is_linux():
die("firecracker backend is only supported on Linux (KVM). "
"On macOS use --backend=macos-container.")
if shutil.which("firecracker") is None:
info("Firecracker is required but was not found on PATH.")
info("Install: https://github.com/firecracker-microvm/firecracker/releases")
die("firecracker not found on PATH")
_require_kvm()
if not kernel_path().is_file():
die(f"guest kernel not found at {kernel_path()}. Set "
"BOT_BOTTLE_FC_KERNEL or place a vmlinux there.")
if not dropbear_path().is_file():
die(f"static dropbear not found at {dropbear_path()}. Set "
"BOT_BOTTLE_FC_DROPBEAR or cache one there.")
if shutil.which("mke2fs") is None:
die("mke2fs (e2fsprogs) not found — required to build guest rootfs")
_require_network_pool()
def _require_kvm() -> None:
if not os.path.exists(_KVM_DEVICE):
die(f"{_KVM_DEVICE} is missing. Enable KVM (load kvm-intel/kvm-amd; "
"confirm virtualization is on in firmware).")
if not os.access(_KVM_DEVICE, os.R_OK | os.W_OK):
die(f"{_KVM_DEVICE} exists but is not accessible. Add your user to "
"the `kvm` group and re-login.")
def _require_network_pool() -> None:
"""Fail-closed on the parts we can check unprivileged; defer the
rest to the post-boot isolation probe.
The TAP pool is verified here (`ip link show` is unprivileged). The
nft table can only be *confirmed present* here when `nft` is
queryable — which usually needs root — so a missing/absent nft is
not treated as fatal at this stage: the authoritative check is the
empirical isolation probe run after boot, before the agent starts
(see isolation_probe.verify_isolation). What we must never do is
boot without the TAP pool."""
conflicts = netpool.overlapping_routes()
if conflicts:
detail = "; ".join(f"{c.dst} dev {c.dev}" for c in conflicts)
warn(f"Firecracker pool range ({netpool.ip_base()}, "
f"{netpool.pool_size()} slots) overlaps existing routes: "
f"{detail}. This can shadow or be shadowed by that route; "
f"set BOT_BOTTLE_FC_IP_BASE to a free range and re-run "
f"./cli.py backend setup --backend=firecracker.")
missing = netpool.missing_taps()
if missing:
die(f"network pool incomplete — missing TAP devices: "
f"{', '.join(missing)}.\n ./cli.py backend setup --backend=firecracker")
if shutil.which("nft") is not None and not netpool.nft_table_present():
# nft is queryable and says the table is absent — that's a
# definite, catchable misconfiguration; fail early.
warn(f"isolation table `inet {netpool.NFT_TABLE}` not found via nft. "
"If this is a permissions issue it will be re-checked "
"empirically after boot; otherwise run: "
"./cli.py backend setup --backend=firecracker")
# --- rootfs pipeline (rootless) -------------------------------------
def docker_image_id(ref: str) -> str:
"""The image's content digest, used as the rootfs cache key."""
result = subprocess.run(
["docker", "image", "inspect", "--format", "{{.Id}}", ref],
capture_output=True, text=True, check=False,
)
if result.returncode != 0 or not result.stdout.strip():
die(f"docker image inspect for {ref!r} failed: "
f"{(result.stderr or '').strip() or '<no output>'}")
return result.stdout.strip().replace("sha256:", "")[:16]
def build_base_rootfs_dir(image_ref: str) -> Path:
"""Export the agent image's filesystem and inject the guest init +
static dropbear. Cached by image digest — the per-bottle bits
(authorized_keys, IP) are passed at boot via the kernel cmdline, so
this tree carries nothing bottle-specific and is safely shared.
Returns the prepared directory (read as the `mke2fs -d` source)."""
digest = docker_image_id(image_ref)
base = cache_dir() / "rootfs" / digest
ready = base / ".bb-ready"
if ready.is_file():
return base
if base.exists():
shutil.rmtree(base, ignore_errors=True)
base.mkdir(parents=True)
info(f"exporting {image_ref} rootfs -> {base}")
cid = subprocess.run(
["docker", "create", image_ref, "sleep", "infinity"],
capture_output=True, text=True, check=False,
)
if cid.returncode != 0 or not cid.stdout.strip():
die(f"docker create {image_ref!r} failed: {cid.stderr.strip()}")
container = cid.stdout.strip()
try:
export = subprocess.Popen(
["docker", "export", container], stdout=subprocess.PIPE,
)
untar = subprocess.run(
["tar", "-x", "-C", str(base)], stdin=export.stdout, check=False,
)
export.wait()
if export.returncode != 0 or untar.returncode != 0:
die(f"exporting rootfs for {image_ref!r} failed")
finally:
subprocess.run(
["docker", "rm", "-f", container],
stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL, check=False,
)
_inject_guest_boot(base)
ready.write_text("ok\n")
return base
def _inject_guest_boot(rootfs: Path) -> None:
"""Drop the static dropbear and the PID-1 init into the rootfs."""
shutil.copy2(dropbear_path(), rootfs / "bb-dropbear")
os.chmod(rootfs / "bb-dropbear", 0o755)
init = rootfs / "bb-init"
init.write_text(_GUEST_INIT)
os.chmod(init, 0o755)
def build_rootfs_ext4(base_dir: Path, out_path: Path, *, slack_mib: int = 1024) -> None:
"""Build a fresh, writable ext4 for one bottle from the cached base
dir. Rootless: `mke2fs -d` populates the image from a directory
without mounting. Each call produces an independent disk, so the
shared base dir stays untouched and concurrent bottles don't race."""
used_mib = _dir_size_mib(base_dir)
size_mib = used_mib + slack_mib
out_path.parent.mkdir(parents=True, exist_ok=True)
result = subprocess.run(
["mke2fs", "-q", "-t", "ext4", "-d", str(base_dir), "-F",
str(out_path), f"{size_mib}M"],
capture_output=True, text=True, check=False,
)
if result.returncode != 0:
die(f"mke2fs for {out_path} failed: {result.stderr.strip()}")
def _dir_size_mib(path: Path) -> int:
result = subprocess.run(
["du", "-sm", str(path)], capture_output=True, text=True, check=False,
)
try:
return int(result.stdout.split()[0])
except (ValueError, IndexError):
return 2048
# --- per-bottle SSH keys --------------------------------------------
def generate_keypair(dest_dir: Path) -> tuple[Path, str]:
"""Mint a per-bottle ed25519 keypair. Returns (private_key_path,
public_key_line). The public line is injected into the guest via
the boot cmdline; the private key authenticates host->guest SSH."""
dest_dir.mkdir(parents=True, exist_ok=True)
key = dest_dir / "bottle_id_ed25519"
if key.exists():
key.unlink()
(dest_dir / "bottle_id_ed25519.pub").unlink(missing_ok=True)
subprocess.run(
["ssh-keygen", "-t", "ed25519", "-N", "", "-q", "-f", str(key),
"-C", "bot-bottle-firecracker"],
check=True,
)
pub = (dest_dir / "bottle_id_ed25519.pub").read_text().strip()
return key, pub
def ssh_base_argv(private_key: Path, guest_ip: str) -> list[str]:
"""Common SSH options for host->guest control. The VM is ephemeral
and per-bottle, so host-key TOFU is meaningless — pin no known_hosts
and skip the check rather than accumulate churn."""
return [
"ssh",
"-i", str(private_key),
"-o", "StrictHostKeyChecking=no",
"-o", "UserKnownHostsFile=/dev/null",
"-o", "LogLevel=ERROR",
"-o", "ConnectTimeout=5",
f"{GUEST_SSH_USER}@{guest_ip}",
]
# PID-1 init injected into every guest. Kept dependency-light: relies
# only on coreutils + a POSIX shell (present in the Debian-family agent
# images). The kernel `ip=` cmdline configures eth0 before init runs,
# so no iproute2 is needed. The per-bottle SSH pubkey arrives base64 on
# the cmdline; dropbear generates ephemeral host keys with -R.
_GUEST_INIT = r"""#!/bin/sh
# bot-bottle Firecracker guest init (PID 1).
mount -t proc proc /proc 2>/dev/null
mount -t sysfs sys /sys 2>/dev/null
mount -t devtmpfs dev /dev 2>/dev/null
mkdir -p /dev/pts && mount -t devpts devpts /dev/pts 2>/dev/null
mount -o remount,rw / 2>/dev/null
# Install the per-bottle SSH pubkey from the kernel cmdline.
KEY=$(sed -n 's/.*bb_pubkey=\([^ ]*\).*/\1/p' /proc/cmdline | base64 -d 2>/dev/null)
if [ -n "$KEY" ]; then
for home in /root /home/node; do
mkdir -p "$home/.ssh"
printf '%s\n' "$KEY" > "$home/.ssh/authorized_keys"
chmod 700 "$home/.ssh"
chmod 600 "$home/.ssh/authorized_keys"
done
chown -R node:node /home/node/.ssh 2>/dev/null || true
fi
mkdir -p /etc/dropbear /run
# -R: generate host keys on demand. -E: log to stderr (guest console).
/bb-dropbear -R -E -p 22 2>/dev/null &
# Reap zombies as PID 1. dropbear is always a child, so `wait` blocks
# rather than busy-looping.
while : ; do wait ; done
"""
+4 -4
View File
@@ -90,11 +90,11 @@ def get_freezer(backend_name: str) -> Freezer:
if resolved == "macos-container":
from .macos_container.freezer import MacosContainerFreezer
return MacosContainerFreezer()
if resolved == "firecracker":
from .firecracker.freezer import FirecrackerFreezer
return FirecrackerFreezer()
if resolved == "smolmachines":
from .smolmachines.freezer import SmolmachinesFreezer
return SmolmachinesFreezer()
die(
f"commit is only supported for docker, macos-container, and "
f"firecracker; backend {backend_name!r} has no freezer"
f"smolmachines; backend {backend_name!r} has no freezer"
)
raise AssertionError("unreachable")
@@ -36,21 +36,6 @@ class MacosContainerBottleBackend(
def is_available(cls) -> bool:
return _container.is_available()
@classmethod
def setup(cls) -> int:
from . import setup as _setup
return _setup.setup()
@classmethod
def status(cls) -> int:
from . import setup as _setup
return _setup.status()
@classmethod
def teardown(cls) -> int:
from . import setup as _setup
return _setup.teardown()
def _preflight(self) -> None:
_resolve_plan.preflight()
+4 -30
View File
@@ -28,12 +28,9 @@ from ...egress import (
egress_resolve_token_values,
egress_sidecar_env_entries,
)
from ...git_gate import (
provision_git_gate_dynamic_keys,
revoke_git_gate_provisioned_keys,
)
from ...git_gate import revoke_git_gate_provisioned_keys
from ...log import die, info, warn
from ...supervise import DB_PATH_IN_CONTAINER, SUPERVISE_PORT
from ...supervise import QUEUE_DIR_IN_CONTAINER, SUPERVISE_PORT
from ...util import expand_tilde
from ..docker.egress import EGRESS_CA_IN_CONTAINER, EGRESS_PORT
from ..docker.git_gate import (
@@ -101,8 +98,6 @@ def launch(
egress_network = egress_network_name(plan.slug)
_create_networks(internal_network, egress_network, stack)
plan = _provision_git_gate_keys(plan)
sidecar_name = sidecar_container_name(plan.slug)
container_mod.force_remove_container(sidecar_name)
_start_sidecar_bundle(plan, sidecar_name, internal_network, egress_network)
@@ -246,19 +241,6 @@ def _stamp_agent_urls(
)
def _provision_git_gate_keys(
plan: MacosContainerBottlePlan,
) -> MacosContainerBottlePlan:
if not plan.git_gate_plan.upstreams:
return plan
git_gate_plan = provision_git_gate_dynamic_keys(
plan.manifest.bottle,
plan.git_gate_plan,
git_gate_state_dir(plan.slug),
)
return dataclasses.replace(plan, git_gate_plan=git_gate_plan)
def _stage_git_gate(plan: MacosContainerBottlePlan, sidecar_name: str) -> None:
gp = plan.git_gate_plan
if not gp.upstreams:
@@ -379,7 +361,7 @@ def _sidecar_env_entries(plan: MacosContainerBottlePlan) -> tuple[str, ...]:
if plan.supervise_plan is not None:
env += [
f"SUPERVISE_BOTTLE_SLUG={plan.slug}",
f"SUPERVISE_DB_PATH={DB_PATH_IN_CONTAINER}",
f"SUPERVISE_QUEUE_DIR={QUEUE_DIR_IN_CONTAINER}",
f"SUPERVISE_PORT={SUPERVISE_PORT}",
]
return tuple(env)
@@ -405,15 +387,7 @@ def _sidecar_mounts(
sp = plan.supervise_plan
if sp is not None:
# `container run --mount type=bind` only accepts directory
# sources (a file source fails with "is not a directory") —
# mount db_path's dedicated parent dir instead of the file
# itself, same as the CA/routes mounts above.
mounts.append((
str(sp.db_path.parent),
str(Path(DB_PATH_IN_CONTAINER).parent),
False,
))
mounts.append((str(sp.queue_dir), QUEUE_DIR_IN_CONTAINER, False))
return tuple(mounts)
@@ -1,79 +0,0 @@
"""Host setup + status for the macOS Apple Container backend.
Like Docker, this backend needs no privileged network-pool provisioning
— it wants Apple's `container` CLI installed and its system service
running. `setup()` points at the install/`container system start` steps;
`status()` reports readiness. Reached via
`MacosContainerBottleBackend.setup` / `.status`, dispatched from the
generic `./cli.py backend {setup,status}`.
"""
from __future__ import annotations
import shutil
import subprocess
import sys
from . import util as _container
def _service_running() -> bool:
if shutil.which("container") is None:
return False
return subprocess.run(
["container", "system", "status"],
stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL, check=False,
).returncode == 0
def setup() -> int:
if not _container.is_macos():
sys.stderr.write("macos-container backend requires macOS.\n")
return 1
if shutil.which("container") is None:
sys.stderr.write("Apple Container is required but was not found on PATH.\n")
sys.stderr.write("Install: https://github.com/apple/container/releases\n")
return 1
if not _service_running():
sys.stderr.write(
"Apple Container is installed but its system service isn't "
"running. Start it with: container system start\n"
)
return 1
sys.stderr.write(
"macos-container backend: ready — no privileged host setup required.\n"
)
return 0
def teardown() -> int:
sys.stderr.write(
"macos-container backend: nothing to undo — it provisions no "
"privileged host state. The Apple Container CLI and its system "
"service are left as-is (stop the service yourself with "
"`container system stop` if you want).\n"
)
return 0
def status() -> int:
ok = True
if _container.is_macos():
sys.stderr.write("host: macOS\n")
else:
sys.stderr.write("host: NOT macOS (backend unsupported here)\n")
ok = False
if shutil.which("container") is not None:
sys.stderr.write("container CLI on PATH: yes\n")
else:
sys.stderr.write("container CLI on PATH: NO\n")
ok = False
if ok:
sys.stderr.write(
f"container system service: {'running' if _service_running() else 'NOT running'}\n"
)
if not _service_running():
ok = False
if not ok:
sys.stderr.write("\nRun: ./cli.py backend setup --backend=macos-container\n")
return 0 if ok else 1
@@ -68,11 +68,6 @@ def build_image(ref: str, context: str, *, dockerfile: str = "") -> None:
_ensure_builder_dns()
args = [_CONTAINER, "build", "-t", ref, "--dns", dns_server()]
if dockerfile:
# `container build` resolves -f relative to the current working
# directory, not the build context. Anchor a relative Dockerfile to
# the context so builds work from any cwd.
if not os.path.isabs(dockerfile):
dockerfile = os.path.join(context, dockerfile)
args.extend(["-f", dockerfile])
args.append(context)
subprocess.run(args, check=True)
+4 -3
View File
@@ -1,8 +1,9 @@
"""Shared print helpers for BottlePlan.print implementations.
Lifts the multi-value label printer out of DockerBottlePlan so every
backend (and any future backend) renders the same two-column
scannable preflight without duplicating the indent math."""
Lifts the multi-value label printer out of DockerBottlePlan so the
smolmachines backend (and any future backend) renders the same
two-column scannable preflight without duplicating the indent
math."""
from __future__ import annotations
+2 -3
View File
@@ -1,7 +1,7 @@
"""Shared helpers used across backends' resolve_plan steps.
"""Shared helpers used by both backends' resolve_plan steps.
Each helper owns one well-defined step of the per-bottle plan
resolution so the backends don't repeat the same logic.
resolution so docker and smolmachines don't repeat the same logic.
Backend-specific steps (container names, env-file, per-bottle
Dockerfile overrides, subnet allocation) stay in the backend's own
resolve_plan.py.
@@ -63,7 +63,6 @@ def write_launch_metadata(
backend=backend,
label=spec.label,
color=spec.color,
bottle_names=spec.bottle_names,
))
@@ -0,0 +1,15 @@
"""smolmachines bottle backend (PRD 0023).
Selectable via `BOT_BOTTLE_BACKEND=smolmachines`. Runs each
bottle inside a per-agent microVM (libkrun / Hypervisor.framework
on macOS) with a userspace gvproxy gateway as the egress
primitive. The sidecar bundle (PRD 0024) runs as a host-side
docker container reached only through gvproxy's port-forward list.
Chunk 1 (this commit) ships the backend skeleton + Smolfile +
gvproxy renderers + preflight check. VM lifecycle, sidecar
bringup, and provisioning land in later chunks."""
from .backend import SmolmachinesBottleBackend # noqa: F401
__all__ = ["SmolmachinesBottleBackend"]
@@ -1,10 +1,11 @@
"""FirecrackerBottleBackend — Linux microVM implementation.
"""SmolmachinesBottleBackend — the smolmachines implementation of
BottleBackend (PRD 0023).
Replaces smolmachines on Linux (issue #342): mature KVM-based
isolation via Firecracker, SSH control over a point-to-point TAP, and a
fail-closed nftables egress boundary. Selected by
`BOT_BOTTLE_BACKEND=firecracker` or `--backend=firecracker`.
"""
Per PRD 0050 the per-provider provisioning steps (prompt, skills,
the declarative provision-plan apply, supervise MCP registration)
live on the `AgentProvider` plugin under `bot_bottle/contrib/`. The
smolmachines backend only owns the steps that are about backend
infrastructure: CA install (no-op for now), workspace, git copy-in."""
from __future__ import annotations
@@ -16,50 +17,34 @@ from ...agent_provider import AgentProvisionPlan
from ...egress import EgressPlan
from ...env import ResolvedEnv
from ...git_gate import GitGatePlan
from ...manifest import Manifest
from ...supervise import SupervisePlan
from ...manifest import Manifest
from .. import ActiveAgent, BottleBackend, BottleSpec
from . import cleanup as _cleanup
from . import enumerate as _enumerate
from . import launch as _launch
from . import resolve_plan as _resolve_plan
from . import util as _util
from .bottle import FirecrackerBottle
from .bottle_cleanup_plan import FirecrackerBottleCleanupPlan
from .bottle_plan import FirecrackerBottlePlan
from . import smolvm as _smolvm
from .bottle import SmolmachinesBottle
from .bottle_cleanup_plan import SmolmachinesBottleCleanupPlan
from .bottle_plan import SmolmachinesBottlePlan
class FirecrackerBottleBackend(
BottleBackend["FirecrackerBottlePlan", "FirecrackerBottleCleanupPlan"]
class SmolmachinesBottleBackend(
BottleBackend["SmolmachinesBottlePlan", "SmolmachinesBottleCleanupPlan"]
):
name = "firecracker"
"""smolmachines backend. Selected by
`BOT_BOTTLE_BACKEND=smolmachines`."""
name = "smolmachines"
@classmethod
def is_available(cls) -> bool:
return _util.is_available()
@classmethod
def is_host_capable(cls) -> bool:
"""Linux + KVM, regardless of whether the `firecracker` binary
is installed. Drives default-backend selection so a capable host
without the binary still lands on firecracker and gets an
install pointer at launch."""
return _util.is_host_capable()
@classmethod
def setup(cls) -> int:
from . import setup as _setup
return _setup.setup()
@classmethod
def status(cls) -> int:
from . import setup as _setup
return _setup.status()
@classmethod
def teardown(cls) -> int:
from . import setup as _setup
return _setup.teardown()
"""`smolvm` on PATH. The backend additionally needs macOS
for libkrun + TSI, but `enumerate_active` / `cleanup` are
host-shell ops that gracefully no-op on Linux too the
runtime check happens at `prepare`."""
return _smolvm.is_available()
def _preflight(self) -> None:
_resolve_plan.preflight()
@@ -79,7 +64,7 @@ class FirecrackerBottleBackend(
git_gate_plan: GitGatePlan,
supervise_plan: SupervisePlan | None,
stage_dir: Path,
) -> FirecrackerBottlePlan:
) -> SmolmachinesBottlePlan:
return _resolve_plan.resolve_plan(
spec,
manifest=manifest,
@@ -94,19 +79,23 @@ class FirecrackerBottleBackend(
@contextmanager
def launch(
self, plan: FirecrackerBottlePlan
) -> Generator[FirecrackerBottle, None, None]:
self, plan: SmolmachinesBottlePlan
) -> Generator[SmolmachinesBottle, None, None]:
with _launch.launch(plan, provision=self.provision) as bottle:
yield bottle
def prepare_cleanup(self) -> FirecrackerBottleCleanupPlan:
def supervise_mcp_url(self, plan: SmolmachinesBottlePlan) -> str:
"""The smolmachines guest reaches the supervise sidecar via a
host-published random port the launch step pinned earlier
(`http://<loopback_ip>:<random_port>/`). `agent_supervise_url`
on the plan is "" when the bottle has no sidecar."""
return plan.agent_supervise_url
def prepare_cleanup(self) -> SmolmachinesBottleCleanupPlan:
return _cleanup.prepare_cleanup()
def cleanup(self, plan: FirecrackerBottleCleanupPlan) -> None:
def cleanup(self, plan: SmolmachinesBottleCleanupPlan) -> None:
_cleanup.cleanup(plan)
def enumerate_active(self) -> Sequence[ActiveAgent]:
return _enumerate.enumerate_active()
def supervise_mcp_url(self, plan: FirecrackerBottlePlan) -> str:
return plan.agent_supervise_url
+203
View File
@@ -0,0 +1,203 @@
"""SmolmachinesBottle — running-instance handle (PRD 0023 chunk 2d).
Routes `exec_agent` / `exec` / `cp_in` through `smolvm machine
exec` / `smolvm machine cp`. The handle is yielded by `launch`
and torn down via the surrounding ExitStack on context exit;
`close` is a no-op idempotent alias so the BottleBackend ABC's
context-manager contract is satisfied.
User context: `smolvm machine exec` runs commands as root in the
VM, but the agent image's USER is `node` and agent CLIs may refuse
to run as root in bypass modes. Both
`exec_agent` and `exec` switch to the requested user (default
`node`) via `runuser -u <user> --` and set `HOME` / `USER`
through `smolvm -e` — avoiding `runuser -l`'s login-shell wiring
(PAM session setup, /etc/profile sourcing) which can hang on a
minimal Debian VM with no PAM session config."""
from __future__ import annotations
import subprocess
import sys
import time
import shlex
from typing import Mapping, cast
from ...agent_provider import PromptMode, prompt_args
from .. import Bottle, ExecResult
from ..terminal import exec_shell_script
from . import pty_resize as _pty_resize
from . import smolvm as _smolvm
# Absolute path to the pty_resize wrapper. Invoke as
# `python <path>` rather than `python -m <dotted-path>` so the
# wrapper runs regardless of cwd / sys.path — it has no
# bot_bottle.* imports, so it's self-contained.
_PTY_RESIZE_SCRIPT = _pty_resize.__file__
# Per-user env the agent image's USER (node) expects. Some providers
# write session state under the user's home directory;
# bare `runuser -u` inherits root's HOME=/root, which claude
# can't write to. Set HOME / USER explicitly through smolvm -e
# so the child process sees them.
_HOME_FOR = {
"node": "/home/node",
"root": "/root",
}
def _env_assignments_for(user: str, env: Mapping[str, str]) -> list[str]:
home = _HOME_FOR.get(user, f"/home/{user}")
out = [f"HOME={home}", f"USER={user}"]
for k, v in env.items():
out.append(f"{k}={v}")
return out
class SmolmachinesBottle(Bottle):
"""Handle returned by `SmolmachinesBottleBackend.launch`. The
underlying VM lifecycle (create / start / stop / delete) lives
on the launch ExitStack — this class only routes runtime
operations to the right `smolvm machine ...` subcommand."""
def __init__(
self,
machine_name: str,
*,
prompt_path: str | None = None,
guest_env: Mapping[str, str] | None = None,
agent_command: str = "claude",
agent_prompt_mode: PromptMode = "append_file",
agent_provider_template: str = "claude",
terminal_title: str = "",
terminal_color: str = "",
agent_workdir: str = "/home/node",
) -> None:
self.name = machine_name
# In-VM path to the agent's prompt file. None when the
# agent declared no prompt (file still exists; we just
# don't pass --append-system-prompt-file).
self.prompt_path = prompt_path
# Env vars the agent process needs (HTTPS_PROXY,
# CLAUDE_CODE_OAUTH_TOKEN, manifest-declared bottle env, …).
# Forwarded on every `smolvm machine exec` via `-e K=V`
# because exec doesn't inherit from machine_create's env.
self._guest_env = dict(guest_env or {})
self._agent_prompt_mode = agent_prompt_mode
self.agent_command = agent_command
self.terminal_title = terminal_title
self.terminal_color = terminal_color
self.agent_provider_template = agent_provider_template
self.agent_workdir = agent_workdir
def agent_argv(
self, argv: list[str], *, tty: bool = True,
) -> list[str]:
flags = ["smolvm", "machine", "exec", "--name", self.name]
if tty:
flags += ["-i", "-t"]
agent_tail = ["env", *_env_assignments_for("node", self._guest_env)]
if self.agent_workdir and self.agent_workdir != _HOME_FOR["node"]:
agent_tail += [
"sh", "-lc",
f"cd {shlex.quote(self.agent_workdir)} && exec \"$@\"",
"bot-bottle-agent",
]
agent_tail.append(self.agent_command)
provider_prompt_args = prompt_args(
cast(PromptMode, self._agent_prompt_mode), self.prompt_path, argv=argv,
)
if cast(PromptMode, self._agent_prompt_mode) == "read_prompt_file":
agent_tail += argv
agent_tail += provider_prompt_args
else:
agent_tail += provider_prompt_args
agent_tail += argv
flags += ["--", "runuser", "-u", "node", "--", *agent_tail]
if not tty:
# No PTY allocated — no SIGWINCH to forward, no resize
# bridge needed. Skip the wrapper so non-interactive
# exec paths (e.g., provisioning shell-outs that
# happen to go through this method) stay light.
return flags
return [
sys.executable, _PTY_RESIZE_SCRIPT,
self.name, "--", *flags,
]
def exec_agent(self, argv: list[str], *, tty: bool = True) -> int:
"""Run the selected agent interactively inside the VM as the `node`
user. Inherits the operator's terminal (stdin / stdout /
stderr) so the session feels native. Blocks until the agent
exits; returns the in-VM exit code.
We bypass the captured-output `machine_exec` helper here
because that one wraps stdout/stderr in pipes — fine for
scripted exec, wrong for an interactive shell. Drop down
to `subprocess.run` with the TTY inherited.
UID switches via `runuser -u node --` (not `-l`) so we
avoid login-shell wiring. HOME / USER come from `smolvm
-e` instead, which sets them on the process env."""
agent_argv = self.agent_argv(argv, tty=tty)
script = exec_shell_script(agent_argv, self.terminal_title, self.terminal_color) if tty else None
if script is None:
return subprocess.run(agent_argv, check=False).returncode
# Use sh -c (not -lc) so the script inherits PATH from the calling
# process. sh -l sources login-shell init files (e.g. /etc/profile)
# which may NOT include smolvm's location when it was installed via
# homebrew. The calling process (./cli.py) already has smolvm on PATH
# (provision steps succeed), so -c is sufficient.
return subprocess.run(["sh", "-c", script], check=False).returncode
# smolvm/libkrun can SIGKILL an otherwise-normal exec during
# early-VM provisioning. Retry once after a short settle so
# callers (provision_ca, etc.) don't have to handle it themselves.
_SIGKILL_EXIT = 128 + 9
def exec(self, script: str, *, user: str = "node") -> ExecResult:
"""Run a POSIX shell script as `user` (default `node`) and
capture the result. Matches the docker backend's `exec`,
which defaults to the image's USER (also node) — so test
helpers / provision shell-outs run with the same identity
on both backends. Pass `user="root"` for tests that need
root.
`runuser -u <user> -- env ... /bin/sh -c <script>` switches UID
without invoking a login shell, then sets HOME / USER and the
bottle env in the child process.
Retries once on SIGKILL (exit 137) — libkrun occasionally
kills short-lived execs during VM bring-up."""
r = self._exec_raw(script, user=user)
if r.returncode == self._SIGKILL_EXIT:
time.sleep(1.0)
r = self._exec_raw(script, user=user)
return r
def _exec_raw(self, script: str, *, user: str = "node") -> ExecResult:
argv = [
"--", "runuser", "-u", user, "--",
"env", *_env_assignments_for(user, self._guest_env),
"/bin/sh", "-c", script,
]
r = subprocess.run(
["smolvm", "machine", "exec", "--name", self.name] + argv,
capture_output=True, text=True, check=False,
)
return ExecResult(
returncode=r.returncode,
stdout=r.stdout or "",
stderr=r.stderr or "",
)
def cp_in(self, host_path: str, container_path: str) -> None:
"""Copy a host path into the guest at `container_path`."""
_smolvm.machine_cp(host_path, f"{self.name}:{container_path}")
def close(self) -> None:
# Real teardown lives on the launch ExitStack; this is just
# the idempotent alias the BottleBackend ABC expects.
pass
@@ -0,0 +1,55 @@
"""SmolmachinesBottleCleanupPlan — concrete BottleCleanupPlan (issue #77).
Tracks the resources `SmolmachinesBottleBackend.cleanup` will
remove:
- machines: smolvm machines whose name starts with
`bot-bottle-` (running or stopped). Stopped +
deleted via `smolvm machine stop` + `machine delete -f`.
- bundles: docker containers `bot-bottle-sidecars-<slug>`
left over from a smolmachines bottle (the bundle's
port-forwards stay published on lo0 aliases until
the container is gone). Removed via `docker rm -f`.
- networks: docker networks `bot-bottle-bundle-<slug>`
attached to the bundles. Removed via
`docker network rm`.
Smolmachines state dirs live under the same `~/.bot-bottle/state/`
path the docker backend uses; the docker backend's
`prepare_cleanup` already enumerates orphan state dirs and is the
single source of truth for that bucket (consults
`enumerate_active_bottles()` so it doesn't reap a live
smolmachines bottle's dir)."""
from __future__ import annotations
import sys
from dataclasses import dataclass
from ...log import info
from .. import BottleCleanupPlan
@dataclass(frozen=True)
class SmolmachinesBottleCleanupPlan(BottleCleanupPlan):
"""Resources SmolmachinesBottleBackend.cleanup will remove.
Produced by `prepare_cleanup`; sorted so the y/N output is
stable."""
machines: tuple[str, ...] = ()
bundles: tuple[str, ...] = ()
networks: tuple[str, ...] = ()
@property
def empty(self) -> bool:
return not self.machines and not self.bundles and not self.networks
def print(self) -> None:
print(file=sys.stderr)
for name in self.machines:
info(f"smolvm machine: {name}")
for name in self.bundles:
info(f"bundle container:{name}")
for name in self.networks:
info(f"bundle network: {name}")
print(file=sys.stderr)
@@ -0,0 +1,109 @@
"""SmolmachinesBottlePlan — concrete BottlePlan for the smolmachines
backend (PRD 0023).
Slug + bundle docker subnet / gateway / pinned IP + smolvm
machine name + agent `.smolmachine` artifact + per-bottle guest
env. Provisioning fields (CA cert path, prompt path, etc.) land
in chunk 4."""
from __future__ import annotations
from dataclasses import dataclass
from pathlib import Path
from ...agent_provider import PromptMode
from .. import BottlePlan
@dataclass(frozen=True)
class SmolmachinesBottlePlan(BottlePlan):
"""Resolved fields the launch step needs to bring up the bottle.
Inherits `spec`, `stage_dir`, `git_gate_plan`, `egress_plan`,
`supervise_plan`, and `agent_provision` from BottlePlan."""
slug: str
# Per-bottle docker subnet for the sidecar bundle container.
# The bundle runs at `bundle_ip` (always `.2`); the gateway is
# at `.1`. smolvm's TSI allowlist is set to `bundle_ip/32`.
bundle_subnet: str
bundle_gateway: str
bundle_ip: str
# In-guest env vars (HTTPS_PROXY etc) — IP-literal URLs since
# the guest has no DNS resolver inside the TSI allowlist.
# Passed to `smolvm machine create` as `-e K=V` flags.
# Smolfile-rendering is gone (smolvm 0.8.0's
# `--smolfile` is mutually exclusive with `--from`, and
# `--from` is the path that avoids the registry-pull race).
guest_env: dict[str, str]
# Inner Plans for the sidecar bundle daemons. The same shape the
# docker backend uses — same `.prepare()` calls produced
# them — but our launch step doesn't populate the
# docker-specific network fields (internal_network,
# egress_network) because the smolmachines bundle isn't on
# docker's `--internal` + egress bridge topology; it's on a
# per-bottle bridge with a pinned IP. The unused fields stay
# at their dataclass defaults.
# Agent-side endpoints. On Docker Desktop the docker bridge
# IPs aren't reachable from the smolvm guest (TSI uses macOS
# networking; docker container IPs live in the daemon's VM),
# so the agent dials the bundle via host loopback +
# docker-published random ports. Empty at prepare time;
# launch populates these after bundle bringup via
# `dataclasses.replace`. Format: a `host:port` for git-gate
# (insteadOf URL prefix) + full URLs for proxy / supervise.
agent_proxy_url: str = ""
agent_git_gate_host: str = ""
agent_supervise_url: str = ""
@property
def machine_name(self) -> str:
"""smolvm machine name. `machine_create` boots from a packed
`.smolmachine` artifact (pre-baked at prepare time via
`smolvm pack create`); using `--from` instead of `--image`
avoids the registry-pull race we hit when machine_start tried
to fetch on-demand and the libkrun agent's network attempt
got refused by macOS."""
return self.agent_provision.instance_name
@property
def agent_image(self) -> str:
"""Agent image ref (docker tag). `launch` runs the
build → save → registry push → smolvm pack pipeline against
this and feeds the resulting `.smolmachine` artifact to
`machine_create --from`. The pipeline runs at launch time
(not prepare time) so the docker build output doesn't garble
the dashboard's preflight modal."""
return self.agent_provision.image
@property
def prompt_file(self) -> Path:
"""Path to the agent's prompt file on the host. Always written
(mode 0o600) so the in-VM path always exists; the file is
empty when the agent has no prompt — claude-code reads it
via --append-system-prompt-file only when non-empty."""
return self.agent_provision.prompt_file
@property
def git_gate_insteadof_host(self) -> str:
return self.agent_git_gate_host
@property
def git_gate_insteadof_scheme(self) -> str:
return "http"
@property
def agent_command(self) -> str:
return self.agent_provision.command
@property
def agent_prompt_mode(self) -> PromptMode:
return self.agent_provision.prompt_mode
@property
def agent_provider_template(self) -> str:
return self.agent_provision.template
@property
def agent_dockerfile_path(self) -> str:
return self.agent_provision.dockerfile
+159
View File
@@ -0,0 +1,159 @@
"""Cleanup + active-listing for the smolmachines backend (issue #77).
`prepare_cleanup` enumerates leftover smolmachines resources:
- smolvm machines (`smolvm machine ls --json`) whose name starts
with `bot-bottle-`.
- bundle docker containers (`bot-bottle-sidecars-<slug>`).
- bundle docker networks (`bot-bottle-bundle-<slug>`).
State dirs live under `~/.bot-bottle/state/<identity>/` —
shared layout with the docker backend, which has the single
orphan-state-dir enumerator (it already consults
`enumerate_active_agents()` so a live smolmachines bottle's dir
is preserved).
`cleanup` removes everything in the plan: stop + delete each VM,
force-rm each container, rm each network. Each step is
best-effort — a failure on one resource doesn't block the others."""
from __future__ import annotations
import json
import subprocess
from ...log import info, warn
from . import sidecar_bundle as _bundle
from . import smolvm as _smolvm
from .bottle_cleanup_plan import SmolmachinesBottleCleanupPlan
# Both names start with the same prefix the launcher uses.
_VM_PREFIX = "bot-bottle-"
_BUNDLE_PREFIX = _bundle.bundle_container_name("") # `bot-bottle-sidecars-`
_NETWORK_PREFIX = _bundle.bundle_network_name("") # `bot-bottle-bundle-`
def prepare_cleanup() -> SmolmachinesBottleCleanupPlan:
"""Enumerate every smolmachines-owned resource on the host.
No side effects. Returns an empty plan when smolvm isn't on
PATH (no machines to reap) — `cleanup` is a no-op in that
case too."""
machines = _list_bot_bottle_machines()
bundles = _list_bundle_containers()
networks = _list_bundle_networks()
return SmolmachinesBottleCleanupPlan(
machines=tuple(sorted(machines)),
bundles=tuple(sorted(bundles)),
networks=tuple(sorted(networks)),
)
def cleanup(plan: SmolmachinesBottleCleanupPlan) -> None:
"""Remove everything in the plan. Order matters: stop VMs
first (they hold ports on lo0 aliases via libkrun), then the
bundle containers (which hold the host port-forwards), then
the networks (which docker won't reap until the containers
are gone)."""
for name in plan.machines:
info(f"stopping smolvm machine {name}")
subprocess.run(
["smolvm", "machine", "stop", "--name", name],
stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL,
check=False,
)
info(f"deleting smolvm machine {name}")
r = subprocess.run(
["smolvm", "machine", "delete", "-f", name],
capture_output=True, text=True, check=False,
)
if r.returncode != 0:
warn(
f"smolvm machine delete -f {name} failed: "
f"{(r.stderr or '').strip()}"
)
for name in plan.bundles:
info(f"removing bundle container {name}")
subprocess.run(
["docker", "rm", "-f", name],
stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL,
check=False,
)
for name in plan.networks:
info(f"removing bundle network {name}")
r = subprocess.run(
["docker", "network", "rm", name],
capture_output=True, text=True, check=False,
)
if r.returncode != 0 and "no such network" not in (r.stderr or "").lower():
warn(
f"docker network rm {name} failed: "
f"{(r.stderr or '').strip()}"
)
def _list_bot_bottle_machines() -> list[str]:
"""All smolvm machines named `bot-bottle-*`, regardless of
state (running / stopped / created). Empty when smolvm isn't
installed."""
if not _smolvm.is_available():
return []
r = subprocess.run(
["smolvm", "machine", "ls", "--json"],
capture_output=True, text=True, check=False,
)
if r.returncode != 0:
return []
try:
machines = json.loads(r.stdout or "[]")
except json.JSONDecodeError:
return []
return [
m["name"] for m in machines
if isinstance(m, dict)
and m.get("name", "").startswith(_VM_PREFIX)
]
def _list_bundle_containers() -> list[str]:
"""All docker containers named `bot-bottle-sidecars-*`,
running or stopped. Empty when docker isn't installed."""
# Late import: `backend/__init__` imports this module
# transitively via the smolmachines backend.
from .. import has_backend
if not has_backend("docker"):
return []
r = subprocess.run(
["docker", "ps", "-a",
"--filter", f"name=^{_BUNDLE_PREFIX}",
"--format", "{{.Names}}"],
capture_output=True, text=True, check=False,
)
if r.returncode != 0:
return []
return [
line for line in (r.stdout or "").splitlines()
if line and line.startswith(_BUNDLE_PREFIX)
]
def _list_bundle_networks() -> list[str]:
"""All docker networks named `bot-bottle-bundle-*`. Empty
when docker isn't installed."""
from .. import has_backend
if not has_backend("docker"):
return []
r = subprocess.run(
["docker", "network", "ls",
"--filter", f"name={_NETWORK_PREFIX}",
"--format", "{{.Name}}"],
capture_output=True, text=True, check=False,
)
if r.returncode != 0:
return []
return [
line for line in (r.stdout or "").splitlines()
if line and line.startswith(_NETWORK_PREFIX)
]
@@ -0,0 +1,21 @@
"""Egress apply for the smolmachines backend.
The smolmachines sidecar bundle runs as a host-side Docker container,
so egress signalling is identical to the docker backend.
"""
from __future__ import annotations
from ..docker.egress_apply import ( # noqa: F401
DockerEgressApplicator,
EgressApplyError,
applicator,
fetch_current_routes,
)
__all__ = [
"DockerEgressApplicator",
"EgressApplyError",
"applicator",
"fetch_current_routes",
]
@@ -0,0 +1,123 @@
"""Active-agent enumeration for the smolmachines backend (PRD
0023 chunk 4 follow-up + issue #77).
Returns a list of `ActiveAgent` records — same shape the docker
backend produces — so CLI `list active` and the dashboard agents
pane render both backends through one code path.
A smolmachines agent is "active" when its smolvm guest is
running. We cross-reference against the per-bottle sidecar
bundle container to populate the `services` field (which daemons
are up in the bundle); without a bundle we still surface the VM
so the operator can see + clean it up.
The cross-backend caller gates on `has_backend("smolmachines")`
and `has_backend("docker")`, so this module assumes both are
available when called. Both subprocess calls below still
tolerate "command not on PATH" defensively, but the gate is the
intended access pattern."""
from __future__ import annotations
import json
import subprocess
from .. import ActiveAgent
from ...bottle_state import read_metadata
from . import sidecar_bundle as _bundle
# Smolvm VM names produced by prepare are `bot-bottle-<slug>`,
# matching the bundle container name pattern. We use the prefix
# both as a filter and to strip back to the slug.
_VM_NAME_PREFIX = "bot-bottle-"
def enumerate_active() -> list[ActiveAgent]:
"""All currently-running smolmachines-backed agents. Empty
list when no matching VMs are running. Caller is responsible
for gating on `has_backend('smolmachines')` if needed; if
smolvm is missing the `smolvm machine ls` call below returns
nothing silently."""
result = subprocess.run(
["smolvm", "machine", "ls", "--json"],
capture_output=True, text=True, check=False,
)
if result.returncode != 0:
return []
try:
machines = json.loads(result.stdout or "[]")
except json.JSONDecodeError:
return []
services_by_slug = _query_bundle_services()
out: list[ActiveAgent] = []
for m in machines:
name = m.get("name") or ""
state = m.get("state") or ""
if state != "running" or not name.startswith(_VM_NAME_PREFIX):
continue
slug = name[len(_VM_NAME_PREFIX):]
metadata = read_metadata(slug)
out.append(ActiveAgent(
backend_name="smolmachines",
slug=slug,
agent_name=metadata.agent_name if metadata else "?",
started_at=metadata.started_at if metadata else "",
services=services_by_slug.get(slug, ()),
label=metadata.label if metadata else "",
color=metadata.color if metadata else "",
))
return out
def _query_bundle_services() -> dict[str, tuple[str, ...]]:
"""`{slug: ('egress', ...)}` from each running bundle container's
`BOT_BOTTLE_SIDECAR_DAEMONS` env var.
Smolmachines bundles all run the PRD-0024 image with the
same daemon set declared via env, so one inspect per bundle
gets us the picture without exec'ing into the container.
Returns an empty mapping when the docker backend isn't
available — the bundle services field on each ActiveAgent
just shows up empty, matching the docker backend's "starting"
state."""
# Late import: `has_backend` lives on the backend package's
# __init__, which imports this module transitively. Pulling
# the name in at call time sidesteps the cycle.
from .. import has_backend
if not has_backend("docker"):
return {}
ps = subprocess.run(
["docker", "ps",
"--filter", "name=" + _bundle.bundle_container_name(""),
"--format", "{{.Names}}"],
capture_output=True, text=True, check=False,
)
if ps.returncode != 0:
return {}
out: dict[str, tuple[str, ...]] = {}
for line in (ps.stdout or "").splitlines():
name = line.strip()
if not name:
continue
slug = name.removeprefix(_bundle.bundle_container_name(""))
if not slug:
continue
inspect = subprocess.run(
["docker", "inspect", name, "--format", "{{json .Config.Env}}"],
capture_output=True, text=True, check=False,
)
if inspect.returncode != 0:
continue
try:
env_list = json.loads(inspect.stdout or "[]")
except json.JSONDecodeError:
continue
for entry in env_list:
key, _, value = entry.partition("=")
if key == "BOT_BOTTLE_SIDECAR_DAEMONS":
out[slug] = tuple(sorted(
d for d in value.split(",") if d
))
break
return out
+145
View File
@@ -0,0 +1,145 @@
"""SmolmachinesFreezer — snapshot a smolmachines bottle.
`smolvm pack create --from-vm` requires the VM to be stopped, and smolvm
removes VMs when stopped (same issue as Apple Container). Instead, exec
into the running VM as root to write a gzip-compressed tar of the root
filesystem to /var/tmp, then copy it to the host with `smolvm machine cp`,
build a Docker image from the archive, convert it to a smolmachine artifact
via the existing registry pipeline, and record the sidecar path. The VM
stays running throughout."""
from __future__ import annotations
import tempfile
from pathlib import Path
from .. import ActiveAgent
from ..freeze import Freezer
from ..docker import util as docker_mod
from .local_registry import crane_push_tarball, ephemeral_registry
from .smolvm import machine_cp, machine_exec, pack_create
from ...bottle_state import bottle_state_dir
from ...log import die, info
# Temp file written inside the VM during commit. Lives in /var/tmp
# (on-disk, unlike tmpfs /tmp) to survive for machine_cp.
_VM_COMMIT_TAR = "/var/tmp/.bot-bottle-commit.tar.gz"
class SmolmachinesFreezer(Freezer):
"""Freezes a smolmachines bottle via exec-tar + Docker image + smolmachine pack.
The VM is NOT stopped. We exec into the running VM to write a compressed
tar of the root filesystem to /var/tmp, copy it to the host with
machine_cp, build a Docker image (Docker's ADD decompresses .tar.gz
automatically), then run the same image→registry→pack_create pipeline
that _ensure_smolmachine uses for fresh builds."""
backend_name = "smolmachines"
def _freeze(self, agent: ActiveAgent) -> str:
machine = f"bot-bottle-{agent.slug}"
image_ref = f"bot-bottle-committed-{agent.slug}:latest"
output_dir = bottle_state_dir(agent.slug)
output_dir.mkdir(parents=True, exist_ok=True)
binary = output_dir / "committed-smolmachine"
sidecar = output_dir / "committed-smolmachine.smolmachine"
_snapshot_running_vm(machine, image_ref, binary)
return str(sidecar)
def _export_hint(self, slug: str, image_ref: str) -> None:
info(f"to export for migration: cp {image_ref} {slug}.smolmachine")
def _snapshot_running_vm(machine: str, image_ref: str, binary: Path) -> None:
"""Exec-tar the running VM, build a Docker image, and pack to a smolmachine.
binary: destination for the launcher (sibling .smolmachine is the artifact
that machine_create --from consumes, same convention as pack_create).
"""
with tempfile.TemporaryDirectory(prefix="bot-bottle-vm-commit.") as tmp:
tmp_path = Path(tmp)
# Use .tar.gz — Docker ADD decompresses automatically and the
# compressed archive fits in the VM's /var/tmp more easily.
rootfs_tar_gz = tmp_path / "rootfs.tar.gz"
dockerfile = tmp_path / "Dockerfile"
_exec_tar_to_file(machine, rootfs_tar_gz)
dockerfile.write_text(
"FROM scratch\n"
"ADD rootfs.tar.gz /\n"
"USER node\n"
"WORKDIR /home/node\n"
)
docker_mod.build_image(image_ref, str(tmp_path), dockerfile=str(dockerfile))
image_tarball = binary.parent / "committed.image.tar"
docker_mod.save(image_ref, str(image_tarball))
try:
with ephemeral_registry() as handle:
digest = docker_mod.image_id(image_ref).split(":", 1)[-1][:16]
push_ref = f"{handle.push_endpoint}/bot-bottle-committed:{digest}"
pack_ref = f"{handle.pull_endpoint}/bot-bottle-committed:{digest}"
crane_push_tarball(handle, str(image_tarball), push_ref)
pack_create(pack_ref, binary)
finally:
image_tarball.unlink(missing_ok=True)
def _exec_tar_to_file(machine: str, dest: Path) -> None:
"""Snapshot the running VM's root filesystem to dest (.tar.gz).
Writes a gzip-compressed tar to _VM_COMMIT_TAR inside the VM via
machine_exec (same mechanism as provisioning), then copies it to the
host with machine_cp. This avoids binary-stdout piping through the
smolvm exec channel, which does not reliably handle large binary output.
A connectivity probe (machine_exec true) runs first so a concurrent-exec
limitation (smolvm may reject a second exec while -i -t is active) is
reported clearly rather than as a silent failure."""
# Connectivity probe — if smolvm rejects concurrent exec while an
# interactive session is running, fail clearly here.
probe = machine_exec(machine, ["true"])
if probe.returncode != 0:
die(
f"smolvm exec is not available for {machine!r} "
f"(exit {probe.returncode}: {probe.stderr.strip() or probe.stdout.strip() or '<no output>'}). "
f"If an interactive session is active, smolvm may not support concurrent exec."
)
# Create the compressed tar inside the VM.
# tar exits 1 when files change during archiving (normal for a live
# filesystem); only treat exit > 1 as fatal.
tar_result = machine_exec(
machine,
[
"tar", "--create", "--gzip",
"--exclude=./proc",
"--exclude=./sys",
"--exclude=./dev",
"--exclude=./run",
# /tmp and /var/tmp are ephemeral. Their stale contents
# (e.g. /tmp/claude-<uid>) have uid remapped by smolvm's
# pack process, causing Claude Code to refuse to use them
# on resume. Exclude both; _init_vm recreates them with
# mkdir -p + correct ownership on every boot.
"--exclude=./tmp",
"--exclude=./var/tmp",
f"--file={_VM_COMMIT_TAR}",
"--directory=/",
".",
],
)
if tar_result.returncode > 1:
die(
f"smolvm exec tar {machine!r} failed (exit {tar_result.returncode}): "
f"{tar_result.stderr.strip() or tar_result.stdout.strip() or '<no output>'}"
)
# Copy from VM to host, then clean up.
try:
machine_cp(f"{machine}:{_VM_COMMIT_TAR}", str(dest))
finally:
machine_exec(machine, ["rm", "-f", _VM_COMMIT_TAR])
+464
View File
@@ -0,0 +1,464 @@
"""End-to-end launch flow for the smolmachines backend
(PRD 0023 chunks 2d + 4b).
Brings up the per-bottle docker bridge + sidecar bundle (with
real daemons + their config files), creates + starts the smolvm
guest pointed at the bundle's pinned IP via TSI's
`--allow-cidr <bundle-ip>/32` allowlist, yields a
`SmolmachinesBottle` handle, tears everything down on context
exit.
The bundle's daemons consume the inner Plans the docker backend
already produces: egress reads routes + CAs from the EgressPlan.
Git-gate + supervise plumb through the same plans the docker
backend uses, minus the docker-network fields that don't apply here."""
from __future__ import annotations
import dataclasses
import os
from contextlib import ExitStack, contextmanager
from pathlib import Path
from typing import Callable, Generator
from ...egress import (
EGRESS_ROUTES_IN_CONTAINER,
egress_agent_env_entries,
egress_resolve_token_values,
egress_sidecar_env_entries,
)
from ...supervise import QUEUE_DIR_IN_CONTAINER, SUPERVISE_PORT
from ...util import expand_tilde
from ..docker import util as docker_mod
from ..docker.egress import (
EGRESS_CA_IN_CONTAINER,
EGRESS_PORT as _EGRESS_PORT,
egress_tls_init,
)
from ..docker.git_gate import (
GIT_GATE_ACCESS_HOOK_IN_CONTAINER,
GIT_GATE_CREDS_DIR_IN_CONTAINER,
GIT_GATE_ENTRYPOINT_IN_CONTAINER,
GIT_GATE_HOOK_IN_CONTAINER,
)
from ...git_gate import revoke_git_gate_provisioned_keys
from ...log import info, warn
from ...bottle_state import (
egress_state_dir,
git_gate_state_dir,
read_committed_image,
)
from . import loopback_alias as _loopback
from . import sidecar_bundle as _bundle
from . import smolvm as _smolvm
from .bottle import SmolmachinesBottle
from .bottle_plan import SmolmachinesBottlePlan
from .local_registry import crane_push_tarball, ephemeral_registry
# Repo root, used as the `docker build` context for the agent image.
_REPO_DIR = str(Path(__file__).resolve().parent.parent.parent.parent)
# Per-host cache for `smolvm pack create` outputs. Keyed by the
# docker image ID so a Dockerfile change automatically invalidates
# the cache. `pack create` is idempotent on the smolvm side but
# takes several seconds even on a no-op rebuild.
_SMOLMACHINE_CACHE_DIR = Path.home() / ".cache" / "bot-bottle" / "smolmachines"
# Container-internal listening ports for each bundle daemon. The
# bundle publishes each one on a random host loopback port (see
# `_bundle.start_bundle`), and `_bundle.bundle_host_port` looks
# them up post-start.
_GIT_HTTP_PORT = 9420
_SUPERVISE_PORT = SUPERVISE_PORT
@contextmanager
def launch(
plan: SmolmachinesBottlePlan,
*,
provision: Callable[[SmolmachinesBottlePlan, "SmolmachinesBottle"], str | None],
) -> Generator[SmolmachinesBottle, None, None]:
"""Build + run the bottle and yield a handle; tear everything
down on exit. Errors during bringup unwind any partial state
via the ExitStack."""
stack = ExitStack()
try:
loopback_ip, network = _allocate_resources(plan, stack)
plan = _mint_certs(plan)
plan = _start_bundle(plan, network, loopback_ip, stack)
plan = _discover_urls(plan, loopback_ip)
agent_from_path = _agent_from_path(plan)
_launch_vm(plan, agent_from_path, loopback_ip, stack)
_init_vm(plan)
bottle = SmolmachinesBottle(
plan.machine_name,
prompt_path=None,
guest_env=plan.guest_env,
agent_command=plan.agent_command,
agent_prompt_mode=plan.agent_prompt_mode,
agent_provider_template=plan.agent_provider_template,
terminal_title=f"{plan.spec.label} ({plan.spec.agent_name})" if plan.spec.label else plan.spec.agent_name,
terminal_color=plan.spec.color,
agent_workdir=plan.workspace_plan.workdir,
)
bottle.prompt_path = provision(plan, bottle)
yield bottle
finally:
_teardown_smolmachines(stack, plan)
def _teardown_smolmachines(
stack: ExitStack,
plan: SmolmachinesBottlePlan,
) -> None:
"""Unwind the ExitStack, then revoke any provisioned deploy keys.
ExitStack errors are caught and logged (non-fatal) so that key
revocation always runs. Revocation errors propagate — a stranded
deploy key is a security concern the operator must address."""
teardown_exc: BaseException | None = None
try:
stack.close()
except BaseException as exc: # noqa: W0718 — teardown must not fail
teardown_exc = exc
warn(f"smolmachines teardown failed: {exc!r}")
bottle = plan.manifest.bottle
revoke_git_gate_provisioned_keys(bottle, git_gate_state_dir(plan.slug))
if teardown_exc is not None:
raise teardown_exc
def _allocate_resources(
plan: SmolmachinesBottlePlan,
stack: ExitStack,
) -> tuple[str, str]:
"""Reserve a loopback alias and create the per-bottle docker bridge.
macOS only routes 127.0.0.1 by default; the per-bottle alias
scopes TSI's allowlist to this bottle's published ports so the
agent can't reach other bottles' or host services' ports on
loopback. No-op on Linux."""
_loopback.ensure_pool()
loopback_ip = _loopback.allocate(plan.slug)
network = _bundle.bundle_network_name(plan.slug)
_bundle.create_bundle_network(network, plan.bundle_subnet, plan.bundle_gateway)
stack.callback(_bundle.remove_bundle_network, network)
return loopback_ip, network
def _mint_certs(plan: SmolmachinesBottlePlan) -> SmolmachinesBottlePlan:
"""Mint the egress MITM CA and return the plan with CA paths filled."""
egress_ca_host, egress_ca_cert_only = egress_tls_init(
egress_state_dir(plan.slug),
)
egress_plan = dataclasses.replace(
plan.egress_plan,
mitmproxy_ca_host_path=egress_ca_host,
mitmproxy_ca_cert_only_host_path=egress_ca_cert_only,
)
return dataclasses.replace(plan, egress_plan=egress_plan)
def _start_bundle(
plan: SmolmachinesBottlePlan,
network: str,
loopback_ip: str,
stack: ExitStack,
) -> SmolmachinesBottlePlan:
"""Build the BundleLaunchSpec, resolve token env, start the
sidecar bundle container, and register teardown."""
bundle_spec = _bundle_launch_spec(plan, network, loopback_ip)
token_env = _resolve_token_env(plan, dict(os.environ))
_bundle.ensure_bundle_image(bundle_spec.image)
_bundle.start_bundle(bundle_spec, env={**os.environ, **token_env})
stack.callback(_bundle.stop_bundle, plan.slug)
return plan
def _discover_urls(
plan: SmolmachinesBottlePlan,
loopback_ip: str,
) -> SmolmachinesBottlePlan:
"""Discover host-side ports for published container ports and
return the plan with URLs + guest_env stamped in.
Docker container IPs (192.168.x.x in the daemon's bridge)
aren't reachable from the smolvm guest on macOS — TSI uses
macOS networking, and macOS sees the daemon's bridge via the
published-port loopback forward only.
NO_PROXY includes the per-bottle loopback alias so the
supervise + git-gate URLs bypass HTTPS_PROXY."""
agent_facing_host_port = _bundle.bundle_host_port(
plan.slug, _EGRESS_PORT, host_ip=loopback_ip,
)
agent_proxy_url = f"http://{loopback_ip}:{agent_facing_host_port}"
agent_git_gate_host = ""
if plan.git_gate_plan.upstreams:
git_gate_host_port = _bundle.bundle_host_port(
plan.slug, _GIT_HTTP_PORT, host_ip=loopback_ip,
)
agent_git_gate_host = f"{loopback_ip}:{git_gate_host_port}"
agent_supervise_url = ""
if plan.supervise_plan is not None:
supervise_host_port = _bundle.bundle_host_port(
plan.slug, _SUPERVISE_PORT, host_ip=loopback_ip,
)
agent_supervise_url = f"http://{loopback_ip}:{supervise_host_port}/"
existing_no_proxy = plan.guest_env.get("NO_PROXY", "localhost,127.0.0.1")
no_proxy = f"{existing_no_proxy},{loopback_ip}"
guest_env = {
**plan.guest_env,
"HTTPS_PROXY": agent_proxy_url,
"HTTP_PROXY": agent_proxy_url,
"https_proxy": agent_proxy_url,
"http_proxy": agent_proxy_url,
"NO_PROXY": no_proxy,
"no_proxy": no_proxy,
}
if agent_git_gate_host:
guest_env["GIT_GATE_URL"] = f"http://{agent_git_gate_host}"
if agent_supervise_url:
guest_env["MCP_SUPERVISE_URL"] = agent_supervise_url
for entry in egress_agent_env_entries(plan.egress_plan):
name, value = entry.split("=", 1)
guest_env[name] = value
return dataclasses.replace(
plan,
guest_env=guest_env,
agent_proxy_url=agent_proxy_url,
agent_git_gate_host=agent_git_gate_host,
agent_supervise_url=agent_supervise_url,
)
def _launch_vm(
plan: SmolmachinesBottlePlan,
agent_from_path: Path,
loopback_ip: str,
stack: ExitStack,
) -> None:
"""Create, patch, and start the smolvm VM; register teardown.
--allow-cidr is the per-bottle loopback alias so the guest can
only reach this bottle's bundle ports. force_allowlist patches
smolvm 0.8.0's silent-drop of --allow-cidr when combined with
--from. Smolfile isn't usable here — smolvm 0.8.0 makes --from
and --smolfile mutually exclusive."""
_smolvm.machine_create(
plan.machine_name,
from_path=agent_from_path,
allow_cidrs=[f"{loopback_ip}/32"],
env=plan.guest_env,
)
stack.callback(_smolvm.machine_delete, plan.machine_name)
# Workaround smolvm 0.8.0: `--allow-cidr` is silently dropped
# when combined with `--from`. Patch the persisted state DB
# before start so the booted VM's TSI actually enforces.
_loopback.force_allowlist(plan.machine_name, [f"{loopback_ip}/32"])
_smolvm.machine_start(plan.machine_name)
stack.callback(_smolvm.machine_stop, plan.machine_name)
def _init_vm(plan: SmolmachinesBottlePlan) -> None:
"""Repair filesystem ownership and wait for exec channel readiness.
Ownership repair: smolvm's pack process remaps files to the host
invoker's uid (501 on macOS). /home/node must be node:node so
Claude Code can write ~/.claude.json; /tmp + /var/tmp need root
mode 1777 so non-root processes can create per-uid scratch dirs.
All folded into one sh -c to avoid back-to-back exec calls
immediately after machine_start (libkrun exec-channel race).
mkdir -p guards: when booting from a committed snapshot, /tmp and
/var/tmp are excluded from the archive (they're ephemeral and their
stale contents would have wrong uid after smolvm's uid remap). The
directories must be created before chown/chmod can set permissions.
wait_exec_ready polls until the exec channel is ready for the
subsequent provision calls, replacing the empirical sleep."""
_smolvm.machine_exec(plan.machine_name, [
"sh", "-c",
"mkdir -p /tmp /var/tmp && "
"chown -R node:node /home/node && "
"chown root:root /tmp /var/tmp && "
"chmod 1777 /tmp /var/tmp",
])
_smolvm.wait_exec_ready(plan.machine_name)
def _bundle_launch_spec(
plan: SmolmachinesBottlePlan, network: str, loopback_ip: str,
) -> _bundle.BundleLaunchSpec:
"""Build a BundleLaunchSpec from the resolved inner Plans.
Daemons in the CSV:
- egress is always present.
- git-gate + git-http are conditional on plan.git_gate_plan.upstreams.
- supervise is conditional on plan.supervise_plan.
Env + volumes are the union of the sidecar daemons' needs, with
daemon-private values only (HTTPS_PROXY is scoped to the
egress process by egress_entrypoint.sh — see PRD 0024's bundle
bind-address PR)."""
daemons: list[str] = ["egress"]
env: list[str] = []
volumes: list[tuple[str, str, bool]] = []
# --- egress -----------------------------------------------
ep = plan.egress_plan
volumes.append((str(ep.mitmproxy_ca_host_path), EGRESS_CA_IN_CONTAINER, True))
if ep.routes:
volumes.append((str(ep.routes_path.parent), str(Path(EGRESS_ROUTES_IN_CONTAINER).parent), True))
env.extend(egress_sidecar_env_entries(ep))
# --- git-gate ---------------------------------------------
gp = plan.git_gate_plan
if gp.upstreams:
daemons += ["git-gate", "git-http"]
volumes += [
(str(gp.entrypoint_script), GIT_GATE_ENTRYPOINT_IN_CONTAINER, True),
(str(gp.hook_script), GIT_GATE_HOOK_IN_CONTAINER, True),
(str(gp.access_hook_script), GIT_GATE_ACCESS_HOOK_IN_CONTAINER, True),
]
for u in gp.upstreams:
keypath = expand_tilde(u.identity_file)
volumes.append((
keypath,
f"{GIT_GATE_CREDS_DIR_IN_CONTAINER}/{u.name}-key",
True,
))
if u.known_hosts_file:
volumes.append((
str(u.known_hosts_file),
f"{GIT_GATE_CREDS_DIR_IN_CONTAINER}/{u.name}-known_hosts",
True,
))
# --- supervise --------------------------------------------
sp = plan.supervise_plan
if sp is not None:
daemons.append("supervise")
env += [
f"SUPERVISE_BOTTLE_SLUG={plan.slug}",
f"SUPERVISE_QUEUE_DIR={QUEUE_DIR_IN_CONTAINER}",
f"SUPERVISE_PORT={SUPERVISE_PORT}",
]
volumes.append((str(sp.queue_dir), QUEUE_DIR_IN_CONTAINER, False))
# Container ports the agent reaches from the smolvm guest —
# published on host loopback so the guest can dial via TSI +
# macOS networking. Egress is always the agent's HTTP/HTTPS proxy.
ports_to_publish: list[int] = [_EGRESS_PORT]
if gp.upstreams:
ports_to_publish.append(_GIT_HTTP_PORT)
if sp is not None:
ports_to_publish.append(_SUPERVISE_PORT)
return _bundle.BundleLaunchSpec(
slug=plan.slug,
network_name=network,
subnet=plan.bundle_subnet,
gateway=plan.bundle_gateway,
bundle_ip=plan.bundle_ip,
daemons_csv=",".join(daemons),
environment=tuple(env),
volumes=tuple(volumes),
ports_to_publish=tuple(ports_to_publish),
publish_host_ip=loopback_ip,
)
def _resolve_token_env(
plan: SmolmachinesBottlePlan, host_env: dict[str, str],
) -> dict[str, str]:
"""Resolve the egress token env-var values from the host's
environ so they reach the bundle's process env via docker's
`-e NAME` inheritance. Empty when no routes declare auth."""
effective_env = {**host_env, **plan.agent_provision.provisioned_env}
return egress_resolve_token_values(plan.egress_plan.token_env_map, effective_env)
def _agent_from_path(plan: SmolmachinesBottlePlan) -> Path:
"""Return the `.smolmachine` artifact used for `machine create --from`.
Prefer a committed VM artifact when one is recorded and still
present. If the file was removed, fall back to the normal image
build + pack cache path.
"""
committed = read_committed_image(plan.slug)
if committed:
committed_path = Path(committed)
if committed_path.is_file():
info(f"using committed smolmachine {str(committed_path)!r}")
return committed_path
# Build the agent image and pack it into a `.smolmachine`
# artifact (or hit the per-Dockerfile-digest cache). Runs here,
# not in prepare, so the docker-build output doesn't garble the
# dashboard's preflight modal.
return _ensure_smolmachine(
plan.agent_image,
dockerfile=plan.agent_dockerfile_path,
)
def _ensure_smolmachine(image_ref: str, *, dockerfile: str = "") -> Path:
"""Build the agent docker image and convert it into a
`.smolmachine` artifact, caching the result under
`~/.cache/bot-bottle/smolmachines/` keyed by the docker image
ID (so a Dockerfile change automatically invalidates the cache).
Returns the `.smolmachine.smolmachine` sidecar path — that's
the file `machine create --from` consumes (pack create produces
a launcher binary at `.smolmachine` plus the sidecar alongside
it; the sidecar is the actual artifact).
Conversion path: `docker build` (the existing layer cache
makes no-change rebuilds cheap) → `docker save` to a tarball
→ spin up an ephemeral registry on a private docker network →
`crane push --insecure` from a one-shot container on the same
network → `smolvm pack create --image localhost:<host port>/...`
→ tear down the registry + network. The crane push detour
sidesteps the Docker-Desktop daemon's HTTPS preference for
non-loopback registries — see the `local_registry` module
docstring for the gory details.
Each pack-create costs several seconds even on a hot cache,
so we skip the whole pipeline when the cached sidecar is
already on disk for this image ID."""
_SMOLMACHINE_CACHE_DIR.mkdir(parents=True, exist_ok=True)
docker_mod.build_image(image_ref, _REPO_DIR, dockerfile=dockerfile)
# `sha256:abcd...` -> `abcd...` first 16 chars: short enough to
# keep filenames manageable, long enough to make collisions
# astronomically unlikely.
digest = docker_mod.image_id(image_ref).split(":", 1)[-1][:16]
binary = _SMOLMACHINE_CACHE_DIR / f"{digest}.smolmachine"
sidecar = _SMOLMACHINE_CACHE_DIR / f"{digest}.smolmachine.smolmachine"
if sidecar.is_file():
return sidecar
tarball = _SMOLMACHINE_CACHE_DIR / f"{digest}.image.tar"
docker_mod.save(image_ref, str(tarball))
try:
with ephemeral_registry() as handle:
push_ref = f"{handle.push_endpoint}/bot-bottle:{digest}"
pack_ref = f"{handle.pull_endpoint}/bot-bottle:{digest}"
crane_push_tarball(handle, str(tarball), push_ref)
_smolvm.pack_create(pack_ref, binary)
finally:
# Tarball is ~500MB-1GB for the agent image; reclaim once
# the smolmachine artifact exists. The artifact itself is
# the long-lived cache entry.
tarball.unlink(missing_ok=True)
return sidecar
@@ -0,0 +1,236 @@
"""Ephemeral local OCI registry for the smolmachines agent-image
conversion path (PRD 0023 chunk 4c).
`smolvm pack create --image <ref>` only accepts OCI registry refs
it can't read the local docker daemon's image cache, an OCI
layout directory, or a `docker save` tarball. To convert the
agent's Dockerfile-built image into a `.smolmachine` artifact we
spin up a short-lived `registry:2.8.3` container alongside a
`crane` helper container on a private docker network, push via
`crane push --insecure <tarball> <registry-container>:5000/...`,
and let smolvm pull from the registry's published host port. The
network + both containers are torn down after the pack completes.
Why this two-container dance instead of plain `docker push`:
- Docker Desktop's daemon runs in its own Linux VM, so its
`localhost` is not the host's loopback. A registry bound to
the host's 127.0.0.1 is unreachable from the daemon side.
- `host.docker.internal` is reachable from the daemon but isn't
in Docker's default insecure-registries CIDRs (only `::1/128`
and `127.0.0.0/8` are), so `docker push` to it tries HTTPS,
hits a plain-HTTP registry, and dies with
`http: server gave HTTP response to HTTPS client`. Adding
`host.docker.internal` to daemon.json works but is a one-time
manual step the user has to do in Docker Desktop's UI.
- Going through a docker network sidesteps the host-vs-daemon
loopback mismatch (crane and registry containers see each
other on the network) AND the HTTPS preference (crane has an
`--insecure` flag that forces plain HTTP).
The registry is also published on a random host port so smolvm
a host process can pull from `localhost:<port>` via Docker's
port-forward. smolvm's bundled crane auto-falls-back to HTTP for
localhost addresses, so no insecure-registries config is needed
on that side either."""
from __future__ import annotations
import os
import socket
import subprocess
import time
import uuid
from contextlib import contextmanager
from dataclasses import dataclass
from typing import Generator
from ...log import die
# registry:2.8.3, pinned by digest. Same env-override pattern as the
# sidecar image pin in bot_bottle/backend/docker/sidecar_bundle.py.
REGISTRY_IMAGE = os.environ.get(
"BOT_BOTTLE_REGISTRY_IMAGE",
"registry@sha256:a3d8aaa63ed8681a604f1dea0aa03f100d5895b6a58ace528858a7b332415373",
)
# gcr.io/go-containerregistry/crane:latest, pinned by digest. ~10MB,
# stable upstream from Google; we only invoke `crane push --insecure`
# against a localhost-equivalent registry, so the trust surface is
# narrow.
CRANE_IMAGE = os.environ.get(
"BOT_BOTTLE_CRANE_IMAGE",
(
"gcr.io/go-containerregistry/crane@sha256:"
"0ae17ecb34315aa7cbff28f6eddee3b7adae0b2f90101260d990804db1eb0084"
),
)
# Internal port the registry binds to inside its container — fixed
# by the registry:2 image. The host-side mapping is random.
_REGISTRY_CONTAINER_PORT = "5000"
# How long to wait for the registry's HTTP layer to bind before
# giving up. Two seconds is empirically enough; 10s leaves headroom
# for slow CI runners without making the failure mode chatty.
_READY_TIMEOUT_S = 10.0
@dataclass(frozen=True)
class RegistryHandle:
"""Everything callers need to push to + pull from the ephemeral
registry.
`network` is the per-session docker network a `crane push`
container has to join it to reach the registry by name.
`push_endpoint` is the `<host>:<port>` form to embed in image
refs given to the crane push container (resolves via docker
network DNS). `pull_endpoint` is the `<host>:<port>` form a
host process (smolvm) uses; the registry's host port mapping
backs this."""
network: str
push_endpoint: str
pull_endpoint: str
@contextmanager
def ephemeral_registry() -> Generator[RegistryHandle, None, None]:
"""Bring up a per-session docker network + a `registry:2.8.3`
container on it (published on a random host port), yield a
`RegistryHandle`, force-remove both on exit.
The container is started with `--rm` so a clean exit cleans up
on its own; the `finally` block force-removes on abnormal exit
(the calling process crashes between yield and close)."""
session_id = uuid.uuid4().hex[:12]
network = f"bot-bottle-registry-net-{session_id}"
registry_name = f"bot-bottle-registry-{session_id}"
subprocess.run(
["docker", "network", "create", network],
check=True,
capture_output=True,
)
try:
subprocess.run(
[
"docker", "run", "-d", "--rm",
"--name", registry_name,
"--network", network,
# `-p :5000` (no IP prefix) binds the container's
# port 5000 on a random host port across all
# interfaces. The host side reaches the registry
# via this port — smolvm's `pack create` pulls from
# `localhost:<port>` and the docker port-forward
# routes there.
"-p", _REGISTRY_CONTAINER_PORT,
REGISTRY_IMAGE,
],
check=True,
capture_output=True,
)
try:
port = _host_port(registry_name)
_wait_ready(port)
yield RegistryHandle(
network=network,
push_endpoint=f"{registry_name}:{_REGISTRY_CONTAINER_PORT}",
pull_endpoint=f"localhost:{port}",
)
finally:
subprocess.run(
["docker", "rm", "-f", registry_name],
check=False,
capture_output=True,
)
finally:
subprocess.run(
["docker", "network", "rm", network],
check=False,
capture_output=True,
)
def crane_push_tarball(handle: RegistryHandle, tarball_path: str, ref: str) -> None:
"""Run `crane push --insecure <tarball> <ref>` inside a one-shot
container on the registry's docker network. `ref` should
reference the registry by `handle.push_endpoint` so the crane
container resolves it via docker network DNS.
Doesn't go through `docker push` to avoid the Docker-Desktop
daemon's HTTPS preference for non-loopback hostnames — crane's
`--insecure` flag forces plain HTTP, which is what the
registry container speaks."""
r = subprocess.run(
[
"docker", "run", "--rm",
"--network", handle.network,
"-v", f"{tarball_path}:/img.tar:ro",
CRANE_IMAGE,
"push", "--insecure", "/img.tar", ref,
],
capture_output=True,
text=True,
check=False,
)
if r.returncode != 0:
die(
f"crane push of {tarball_path!r} to {ref!r} failed: "
f"{(r.stderr or r.stdout or '').strip() or '<no output>'}"
)
def _host_port(name: str) -> int:
"""Resolve the host-side port docker mapped to the registry's
container port. `docker port <name> 5000/tcp` returns one or
more `host:port` lines (one per address family) we take the
first."""
r = subprocess.run(
["docker", "port", name, f"{_REGISTRY_CONTAINER_PORT}/tcp"],
capture_output=True,
text=True,
check=False,
)
if r.returncode != 0:
die(
f"docker port {name} {_REGISTRY_CONTAINER_PORT}/tcp failed: "
f"{(r.stderr or '').strip() or '<no stderr>'}"
)
# `0.0.0.0:54321\n[::]:54321\n` — split on the last colon to
# handle either IPv4 or IPv6 host syntax.
line = (r.stdout or "").splitlines()[0].strip()
_, _, port_str = line.rpartition(":")
try:
return int(port_str)
except ValueError:
die(f"unexpected `docker port` output: {line!r}")
def _wait_ready(port: int) -> None:
"""Block until the registry's HTTP layer accepts a TCP
connection on `127.0.0.1:<port>`, or `_READY_TIMEOUT_S`
elapses.
A successful TCP connect is sufficient registry:2.8.3 binds
after it's ready to serve `/v2/` requests, so the push that
follows will land on a working server. We probe loopback
specifically (not via the docker network) because this helper
runs on the host."""
deadline = time.monotonic() + _READY_TIMEOUT_S
last_err: Exception | None = None
while time.monotonic() < deadline:
try:
with socket.create_connection(("127.0.0.1", port), timeout=0.5):
return
except OSError as e:
last_err = e
time.sleep(0.1)
die(
f"local registry on 127.0.0.1:{port} did not accept "
f"connections within {_READY_TIMEOUT_S:.0f}s "
f"(last error: {last_err})"
)
@@ -0,0 +1,272 @@
"""Per-bottle loopback alias allocation + TSI allowlist
enforcement (PRD 0023, follow-up to PR #74).
After the pivot to host-loopback port-forwards, the smolmachines
TSI allowlist was `127.0.0.1/32` which meant the agent VM could
reach **any** service bound to macOS's loopback, not just the
bundle's published ports. Real downgrade from the docker
backend's `--internal` network isolation.
This module narrows the allowlist by allocating each bottle a
unique loopback alias (`127.0.0.16` .. `127.0.0.31`). The
bundle's port-forwards bind to that alias, and the alias's /32
is what TSI allows.
**Smolvm 0.8.0 quirk + workaround.** `smolvm machine create
--from <smolmachine> --net --allow-cidr X/32` silently drops the
flag verified empirically that the agent process's allowlist
ends up `null` in smolvm's persistent state DB (`~/Library/
Application Support/smolvm/server/smolvm.db`, `vms` table,
`data` BLOB), and the booted VM reaches all of `127.0.0.0/8`
regardless of what we passed. Workaround: after machine_create,
open the SQLite DB and patch the row's `allowed_cidrs` field
directly. Smolvm reads the DB at machine_start, so the patched
value takes effect on boot. Tested: enforcement is real the
guest's connect to a non-allowlisted IP fails with `Permission
denied`. Other paths we tried (machine update, stop-edit-
agent.config.json-restart, --smolfile, --image localhost:N/...)
were dead ends.
macOS only configures `127.0.0.1` on `lo0` by default; the
additional aliases require `sudo ifconfig lo0 alias`. We lazily
sudo-add the missing pool on first use per boot the aliases
persist on `lo0` until reboot, so subsequent launches don't
prompt.
Linux native daemons share the host's network namespace; the
whole `127.0.0.0/8` is reachable by default and aliases are
unnecessary. The pool logic detects native-Linux and skips sudo
entirely; the DB patch is also gated on macOS.
Allocation is coordinated by inspecting running bundle
containers' published host IPs — each bottle's bundle owns the
alias appearing in its port bindings. The lowest-numbered free
alias gets handed to a new bottle."""
from __future__ import annotations
import fcntl
import json
import platform
import re
import sqlite3
import subprocess
from pathlib import Path
from typing import Iterable
from ...log import die, info
# smolvm's persistent VM state on macOS — a SQLite DB whose `vms`
# table holds one JSON BLOB per machine. The Linux path is
# different, but smolmachines is macOS-only in v1 (PRD 0023) so
# we hard-code this. If the file moves under us we'll see a
# clear FileNotFoundError; not worth defensive cross-platform
# detection until the backend actually needs Linux.
_SMOLVM_DB_PATH = (
Path.home()
/ "Library"
/ "Application Support"
/ "smolvm"
/ "server"
/ "smolvm.db"
)
# Sixteen aliases by default. Tunable for hosts that want more
# concurrent bottles (each bottle reserves one alias for its
# bundle bringup). The range is chosen to avoid the reserved
# 127.0.0.1/2/3 ports (1 is the default, 2 is sometimes used by
# CUPS, 3 by other macOS services) and stay well clear of
# 127.0.0.53 (systemd-resolved) and 127.0.0.54 (libvirt).
_POOL_START = 16
_POOL_END = 31 # inclusive
# File lock that serialises concurrent allocate() calls so two
# simultaneous launches can't read the same docker state and claim
# the same alias. Narrowed to the allocate() call itself; docker run
# runs after the lock is released. Once the container is running it
# appears in docker state and future allocate() calls will see it.
_ALLOC_LOCK_PATH = Path.home() / ".cache" / "bot-bottle" / "smolmachines.lock"
# Loopback aliases pool: 127.0.0.<start>..127.0.0.<end>.
def _pool_addresses() -> list[str]:
return [f"127.0.0.{i}" for i in range(_POOL_START, _POOL_END + 1)]
def _is_macos() -> bool:
return platform.system() == "Darwin"
def ensure_pool() -> None:
"""Make sure each address in the pool is up on `lo0`. Lazily
runs `sudo ifconfig lo0 alias <ip>/32 up` for missing entries
(sudo prompts once, then the aliases persist on lo0 until
reboot). No-op on non-macOS hosts."""
if not _is_macos():
return
missing = [ip for ip in _pool_addresses() if not _alias_present(ip)]
if not missing:
return
info(
f"smolmachines needs {len(missing)} loopback alias(es) on lo0 "
f"({', '.join(missing[:3])}{', ...' if len(missing) > 3 else ''}) "
f"to scope per-bottle TSI allowlists. sudo will prompt once; "
f"aliases persist until reboot."
)
for ip in missing:
result = subprocess.run(
["sudo", "-p", "bot-bottle (loopback alias): ",
"ifconfig", "lo0", "alias", f"{ip}/32", "up"],
check=False,
)
if result.returncode != 0:
die(
f"sudo ifconfig lo0 alias {ip} failed (exit "
f"{result.returncode}). Re-run with sudo available, "
f"or add manually: sudo ifconfig lo0 alias {ip}/32 up"
)
def force_allowlist(machine_name: str, allowed_cidrs: list[str]) -> None:
"""Patch smolvm's persistent VM-state DB to set the machine's
`allowed_cidrs` to the given list. Workaround for smolvm
0.8.0's silent-drop of `--allow-cidr` when used with `--from`.
Must run AFTER `smolvm machine create` (the row has to
exist) and BEFORE `smolvm machine start` (smolvm reads the
row on start; in-flight VMs don't pick up changes). Once
smolvm honors the CLI flag upstream this whole function is
redundant flag-respecting create + remove this call from
launch.
No-op on non-macOS the DB path differs and the Linux
smolmachines code path isn't exercised in v1."""
if not _is_macos():
return
if not _SMOLVM_DB_PATH.is_file():
die(
f"smolvm state DB not found at {_SMOLVM_DB_PATH}. "
f"smolvm 0.8.0 expected? `smolvm --version` to check."
)
con = sqlite3.connect(str(_SMOLVM_DB_PATH))
try:
cur = con.cursor()
row = cur.execute(
"SELECT data FROM vms WHERE name = ?", (machine_name,),
).fetchone()
if row is None:
die(
f"smolvm DB has no row for machine {machine_name!r}"
f"machine_create must run before force_allowlist."
)
cfg = json.loads(row[0])
cfg["allowed_cidrs"] = list(allowed_cidrs)
# Write as BLOB (the column type smolvm uses) — passing a
# plain str makes sqlite store it as Text and smolvm then
# fails to read it.
cur.execute(
"UPDATE vms SET data = ? WHERE name = ?",
(sqlite3.Binary(json.dumps(cfg).encode()), machine_name),
)
con.commit()
finally:
con.close()
def allocate(_slug: str) -> str:
"""Pick the lowest-numbered alias from the pool not already
in use by a running smolmachines bundle. Bails when the pool
is exhausted the caller should report the limit to the
operator. `_slug` is logged for traceability; not otherwise
used (no on-disk reservation, allocation is purely
docker-state-driven).
On non-macOS the whole `127.0.0.0/8` is loopback by default;
`127.0.0.1` is fine to share and we skip the alias dance.
This still returns a deterministic address so launch.py's
callers don't have to branch on platform.
An exclusive file lock serialises concurrent calls so two
simultaneous launches don't read the same docker state and
claim the same alias."""
if not _is_macos():
return "127.0.0.1"
_ALLOC_LOCK_PATH.parent.mkdir(parents=True, exist_ok=True)
with open(_ALLOC_LOCK_PATH, "w", encoding="utf-8") as lf:
fcntl.flock(lf, fcntl.LOCK_EX)
return _allocate_locked()
def _allocate_locked() -> str:
in_use = _aliases_in_use()
for ip in _pool_addresses():
if ip not in in_use:
return ip
die(
f"smolmachines loopback alias pool exhausted "
f"({_POOL_END - _POOL_START + 1} aliases, all in use). "
f"Stop a running bottle (`smolvm machine ls --json`) or "
f"raise _POOL_END in loopback_alias.py."
)
def _alias_present(ip: str) -> bool:
"""True iff `ifconfig lo0` shows `<ip>` as an inet address.
Exact-match `127.0.0.1` shouldn't match `127.0.0.16`."""
result = subprocess.run(
["/sbin/ifconfig", "lo0"],
capture_output=True, text=True, check=False,
)
if result.returncode != 0:
return False
pattern = re.compile(rf"\binet {re.escape(ip)}\b")
return bool(pattern.search(result.stdout or ""))
def _aliases_in_use() -> set[str]:
"""Aliases already bound by another smolmachines bundle's
published-port mappings. We inspect every container whose
name matches the smolmachines bundle prefix and pull the
`HostIp` out of its port bindings."""
result = subprocess.run(
["docker", "ps", "--format", "{{.Names}}",
"--filter", "name=bot-bottle-sidecars-"],
capture_output=True, text=True, check=False,
)
if result.returncode != 0:
return set()
names = [n.strip() for n in (result.stdout or "").splitlines() if n.strip()]
in_use: set[str] = set()
for name in names:
in_use.update(_host_ips_for_container(name))
return in_use
def _host_ips_for_container(name: str) -> Iterable[str]:
"""Yield the `HostIp` values across all port bindings on
container `name`. A bundle binds three or four ports and
they all share the same HostIp, so callers can take any."""
result = subprocess.run(
["docker", "inspect", name,
"--format", "{{json .HostConfig.PortBindings}}"],
capture_output=True, text=True, check=False,
)
if result.returncode != 0:
return ()
try:
bindings = json.loads(result.stdout or "{}")
except json.JSONDecodeError:
return ()
seen: set[str] = set()
for _port, mappings in (bindings or {}).items():
for m in mappings or []:
host_ip = m.get("HostIp") or ""
if host_ip:
seen.add(host_ip)
return seen
__all__ = ["allocate", "ensure_pool", "force_allowlist"]
@@ -0,0 +1,12 @@
"""Backend-infrastructure provisioners for the smolmachines backend.
Per PRD 0050 the per-provider provisioning steps (prompt, skills,
declarative provision-plan apply, supervise MCP registration) live on
the `AgentProvider` plugin under `bot_bottle/contrib/`. CA and git
provisioning also moved to the AgentProvider ABC (with Debian/node
defaults); user plugins override them for non-standard images.
No modules remain in this subpackage. Workspace copying now runs
through `BottleBackend.provision_workspace` against the running
bottle for every backend.
"""
@@ -0,0 +1,151 @@
"""Host-side SIGWINCH → in-VM PTY resize bridge (issue #82).
smolvm 0.8.0 `machine exec -t` allocates an in-VM PTY but never
forwards the host terminal's window size (TIOCSWINSZ) to it. The
PTY's initial size is `0 0`, and any host-side resize during the
session goes unnoticed the in-VM claude TUI keeps rendering for
whatever (typically tiny) box it last saw, ignoring the operator's
tmux pane resize. `docker exec -it` does this forwarding
automatically; smolvm doesn't.
This module wraps `smolvm machine exec` with a thin parent
process that:
1. Spawns the original argv as a child (it gets the inherited
TTY, so claude's stdin/stdout/stderr work unchanged).
2. On startup + every host SIGWINCH, reads the host terminal
size via TIOCGWINSZ on stdin (or stderr if stdin isn't a
TTY tmux respawn-pane gives us a TTY on stdout/stderr)
and pushes it into the VM with a side-channel
`smolvm machine exec -- sh -c 'for f in /dev/pts/*; do
stty -F $f cols X rows Y; done'`. The kernel delivers
SIGWINCH to the foreground process group on the slave end
automatically, so claude picks up the new size without
extra signalling.
3. Waits on the child and exits with its returncode.
The dashboard's tmux pane respawn calls `bottle.agent_argv`
which now prepends `[sys.executable, -m, ..., <machine>, --, ...]`
to the smolvm argv. Foreground handoff (curses endwin
subprocess.run) goes through the same path so behavior is
identical.
Removable once smolvm grows native SIGWINCH forwarding (upstream
follow-up tracked separately)."""
from __future__ import annotations
import fcntl
import signal
import struct
import subprocess
import sys
import termios
import threading
from types import FrameType
# How long to wait after the main exec starts before pushing the
# initial size. Concurrent `smolvm machine exec` invocations race
# libkrun's per-exec OCI config write during the main exec's
# bringup window; the side-channel firing immediately corrupts
# `config.json` and the main exec dies with SIGKILL (rc=137) or
# libkrun's "parse error: trailing garbage" depending on
# scheduling. Two seconds is well past the bringup window on a
# warm VM, well under the operator's "this is unresponsive"
# threshold, and short enough that claude's initial render
# almost always fires after the size has been set.
_STARTUP_SYNC_DELAY_SEC = 2.0
def _read_winsize() -> tuple[int, int] | None:
"""Return `(rows, cols)` from whichever of stdin / stdout /
stderr is a TTY, or None if none are. Different invocation
surfaces give us different TTYs:
- foreground handoff (curses endwin subprocess.run): all
three are the operator's terminal.
- tmux respawn-pane: tmux sets all three to the pane's PTY.
- non-TTY (someone piped stdin in tests): none are; the
sync just no-ops, which is the right behavior."""
for stream in (sys.stdin, sys.stdout, sys.stderr):
try:
fd = stream.fileno()
data = fcntl.ioctl(fd, termios.TIOCGWINSZ, b"\x00" * 8)
except OSError:
continue
rows, cols, _, _ = struct.unpack("hhhh", data)
if rows > 0 and cols > 0:
return rows, cols
return None
def _push_size(machine: str, rows: int, cols: int) -> None:
"""Side-channel `smolvm machine exec` that sets the size of
every PTY in the VM. The shell `for` loop covers the case of
multiple concurrent interactive sessions (rare but cheap to
handle); `stty -F` returns silently on PTYs that don't apply.
Best-effort: swallow failures. A failed resize doesn't break
the session it just leaves the in-VM PTY at its old size.
`stdin=DEVNULL` is load-bearing: under tmux, inheriting the
pane PTY here means two concurrent smolvm processes (this one
and the agent session the wrapper is shepherding) share the
PTY's foreground-process-group / input plumbing, and smolvm
bails with an internal config-parse error or SIGKILL within
~100ms of the side-channel firing. Outside tmux the same
pattern survived, presumably because iTerm's PTY plumbing is
more forgiving than tmux's, but the DEVNULL is the right
default either way the side-channel never needs stdin."""
subprocess.run(
["smolvm", "machine", "exec", "--name", machine, "--",
"sh", "-c",
f"for f in /dev/pts/*; do "
f"stty -F \"$f\" cols {cols} rows {rows} 2>/dev/null; "
f"done"],
stdin=subprocess.DEVNULL,
stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL,
check=False,
)
def main(argv: list[str]) -> int:
"""Entry point. `argv` shape: `<machine> -- <smolvm-argv...>`.
We don't use argparse — the `--` separator is the contract and
everything past it is forwarded verbatim. Keeps the wrapper
transparent for callers building argv programmatically."""
if len(argv) < 3 or argv[1] != "--":
sys.stderr.write(
"usage: python -m bot_bottle.backend.smolmachines.pty_resize "
"<machine> -- <smolvm-argv...>\n"
)
return 2
machine = argv[0]
inner = argv[2:]
def sync(_signum: int | None = None, _frame: FrameType | None = None) -> None:
size = _read_winsize()
if size is None:
return
_push_size(machine, *size)
signal.signal(signal.SIGWINCH, sync) # type: ignore[arg-type]
proc = subprocess.Popen(inner)
# Initial sync is deferred — see _STARTUP_SYNC_DELAY_SEC.
# daemon=True so the timer doesn't block exit when the child
# finishes before the delay elapses.
timer = threading.Timer(_STARTUP_SYNC_DELAY_SEC, sync)
timer.daemon = True
timer.start()
while True:
try:
return proc.wait()
except KeyboardInterrupt:
proc.send_signal(signal.SIGINT)
if __name__ == "__main__":
sys.exit(main(sys.argv[1:]))
@@ -0,0 +1,83 @@
"""smolmachines `_resolve_plan` (PRD 0023 chunks 2d + 4c).
Resolves the per-bottle docker subnet + bundle IP and assembles
the guest env. The agent's docker image build → smolmachine
pack pipeline runs in `launch.launch`, not here, so the
dashboard's preflight modal isn't garbled by docker-build output
before the operator has confirmed.
No VM bringup that's `launch.launch`'s job."""
from __future__ import annotations
from pathlib import Path
from .. import BottleSpec
from ...manifest import Manifest
from ...env import ResolvedEnv
from ...agent_provider import AgentProvisionPlan
from ...egress import EgressPlan
from ...supervise import SupervisePlan
from ...git_gate import GitGatePlan
from .bottle_plan import SmolmachinesBottlePlan
from .util import smolmachines_bundle_subnet, smolmachines_preflight
def preflight() -> None:
smolmachines_preflight()
def build_guest_env(resolved_env: ResolvedEnv) -> dict[str, str]:
# Agent's env: resolve through resolve_env() so ?prompt entries
# are prompted and ${HOST_VAR} entries are interpolated — matching
# the Docker backend's contract. Forwarded (secret/interpolated)
# values still reach the guest as -e K=V smolvm flags because
# smolvm 0.8.0 has no env-file or stdin injection path; this is
# the known argv-exposure gap documented in PRD 0038.
# HTTPS_PROXY / GIT_GATE_URL / MCP_SUPERVISE_URL are populated
# in launch.py after bundle bringup.
return {
**resolved_env.literals,
**resolved_env.forwarded,
"NO_PROXY": "localhost,127.0.0.1",
"NODE_EXTRA_CA_CERTS": "/etc/ssl/certs/ca-certificates.crt",
"SSL_CERT_FILE": "/etc/ssl/certs/ca-certificates.crt",
"REQUESTS_CA_BUNDLE": "/etc/ssl/certs/ca-certificates.crt",
}
def resolve_plan(
spec: BottleSpec,
manifest: Manifest,
slug: str,
resolved_env: ResolvedEnv,
agent_provision_plan: AgentProvisionPlan,
egress_plan: EgressPlan,
supervise_plan: SupervisePlan | None,
git_gate_plan: GitGatePlan,
stage_dir: Path,
) -> SmolmachinesBottlePlan:
"""Materialize the smolmachines plan. The bundle's docker
subnet + pinned IP are derived from the slug; the agent's
`.smolmachine` artifact is built (or cache-hit) here so
launch's `machine create --from` boots without a registry
pull. Per-bottle guest env + the TSI allow_cidrs land on the
plan for launch to pass straight through to
`machine create` flags."""
# ==== smolmachines specific setup ====
subnet, gateway, bundle_ip = smolmachines_bundle_subnet(slug)
return SmolmachinesBottlePlan(
spec=spec,
manifest=manifest,
stage_dir=stage_dir,
slug=slug,
bundle_subnet=subnet,
bundle_gateway=gateway,
bundle_ip=bundle_ip,
guest_env=agent_provision_plan.guest_env,
git_gate_plan=git_gate_plan,
egress_plan=egress_plan,
supervise_plan=supervise_plan,
agent_provision=agent_provision_plan,
)
@@ -0,0 +1,242 @@
"""Per-bottle sidecar bundle bringup for the smolmachines backend
(PRD 0023).
Two docker resources per bottle live here:
- **A dedicated bridge network**, subnet derived from the slug.
The bundle container gets a pinned IP at `<subnet>.2` so the
smolvm guest's TSI allowlist (`<bundle-ip>/32`) has a stable
target. Without pinning, we'd have to inspect the container's
assigned IP after start and feed it back into the Smolfile
a race we can sidestep with `--ip`.
- **The bundle container itself**, running the PRD 0024 bundle
image (`bot-bottle-sidecars:latest` by default). Same
image, same daemons, same daemon-private env / bind-mounts
as the docker backend.
This module ships the lifecycle primitives only create
network, start bundle, stop bundle, remove network wrapped
around `subprocess.run(["docker", ...])`. Wiring them into the
launch flow + populating the `BundleLaunchSpec` from the inner
Plans (EgressPlan, ) lands in chunk 2d."""
from __future__ import annotations
import subprocess
from dataclasses import dataclass, field
from pathlib import Path
from typing import Sequence
from ...log import die, warn
from ..docker import util as docker_mod
from ..docker.sidecar_bundle import (
SIDECAR_BUNDLE_DOCKERFILE,
SIDECAR_BUNDLE_IMAGE,
)
_REPO_DIR = str(Path(__file__).resolve().parent.parent.parent.parent)
def bundle_network_name(slug: str) -> str:
"""`bot-bottle-bundle-<slug>` — distinct from the docker
backend's `bot-bottle-net-<slug>` so a smolmachines bottle
and a docker bottle for the same agent don't collide on
network name."""
return f"bot-bottle-bundle-{slug}"
def bundle_container_name(slug: str) -> str:
"""`bot-bottle-sidecars-<slug>` — same name shape the docker
backend uses for the bundle (PRD 0024 chunk 5). The dashboard's
prefix-based discovery covers both backends with one filter."""
return f"bot-bottle-sidecars-{slug}"
@dataclass(frozen=True)
class BundleLaunchSpec:
"""Everything `start_bundle` needs to bring up one bundle
container. Populated by chunk-2d's launch flow from the inner
Plans the prepare step already produces."""
slug: str
network_name: str
subnet: str
gateway: str
bundle_ip: str
image: str = SIDECAR_BUNDLE_IMAGE
# Daemon subset CSV for BOT_BOTTLE_SIDECAR_DAEMONS. The
# supervisor inside the bundle reads it to skip
# bottle-irrelevant daemons (e.g. supervise=False bottles).
daemons_csv: str = "egress"
# Plain "KEY=VALUE" strings + "KEY" bare names (the bare-name
# form inherits the value from the docker-run subprocess env,
# matching the docker backend's compose-up secret-forwarding
# pattern).
environment: Sequence[str] = field(default_factory=tuple)
# (host_path, container_path, read_only) bind mounts.
volumes: Sequence[tuple[str, str, bool]] = field(default_factory=tuple)
# Container ports to publish on `publish_host_ip`, random
# host-side port per entry. The smolvm guest's TSI talks via
# macOS networking, so docker container IPs (192.168.x.x in
# the daemon's bridge) aren't directly reachable from the
# guest — host-loopback port-forwards are. Egress's port
# is bundle-internal and never published.
ports_to_publish: Sequence[int] = field(default_factory=tuple)
# Loopback IP to bind published ports against. Per-bottle
# loopback aliases (`127.0.0.16` etc., added via sudo
# ifconfig lo0 alias) narrow the TSI allowlist so a bottle
# can't reach other bottles' (or other host services') ports
# via 127.0.0.1.
publish_host_ip: str = "127.0.0.1"
def ensure_bundle_image(image: str = SIDECAR_BUNDLE_IMAGE) -> None:
"""Build the sidecar bundle image before `docker run`.
The Docker backend gets this for free from compose's `build:`
stanza. smolmachines starts the bundle with plain `docker run`,
so without an explicit build a first launch tries to pull the
local-only `bot-bottle-sidecars:latest` tag from a registry.
"""
docker_mod.build_image(
image,
_REPO_DIR,
dockerfile=SIDECAR_BUNDLE_DOCKERFILE,
)
def create_bundle_network(network_name: str, subnet: str, gateway: str) -> None:
"""`docker network create` with an explicit subnet + gateway
so the bundle's `--ip` lands on the address the Smolfile's
TSI allowlist points at. Idempotent on the caller's side —
`start_bundle` catches the "network exists" error and treats
it as success (chunk-2d teardown is paired with each create).
"""
result = subprocess.run(
["docker", "network", "create",
"--subnet", subnet, "--gateway", gateway,
network_name],
capture_output=True, text=True, check=False,
)
if result.returncode != 0:
# Already-exists is fine on a resume path; everything else
# is fatal — the bundle won't have an addressable network.
if "already exists" in (result.stderr or "").lower():
return
die(
f"docker network create {network_name} failed: "
f"{(result.stderr or '').strip()}"
)
def remove_bundle_network(network_name: str) -> None:
"""Idempotent: a missing network returns success."""
result = subprocess.run(
["docker", "network", "rm", network_name],
capture_output=True, text=True, check=False,
)
if result.returncode == 0:
return
if "no such network" in (result.stderr or "").lower():
return
# Network with attached containers is the common non-fatal
# case during a partial teardown — warn but don't die.
warn(
f"docker network rm {network_name} failed: "
f"{(result.stderr or '').strip()}"
)
def start_bundle(spec: BundleLaunchSpec, *,
env: dict[str, str] | None = None) -> None:
"""Bring the bundle container up on the per-bottle bridge with
the pinned IP. Argv is built deterministically from `spec`;
`env` is the host subprocess env (forwarded values for any
bare-name entries in `spec.environment`)."""
container = bundle_container_name(spec.slug)
argv = [
"docker", "run",
"--name", container,
"--detach",
"--rm",
"--network", spec.network_name,
"--ip", spec.bundle_ip,
"-e", f"BOT_BOTTLE_SIDECAR_DAEMONS={spec.daemons_csv}",
]
for entry in spec.environment:
argv += ["-e", entry]
for host_path, container_path, read_only in spec.volumes:
suffix = ":ro" if read_only else ""
argv += ["-v", f"{host_path}:{container_path}{suffix}"]
# Loopback-only host port-forwards — the smolvm guest's TSI
# uses macOS networking, and macOS loopback is the only host
# surface that round-trips into Docker Desktop's daemon VM.
# Binds to the per-bottle alias so TSI's IP-only allowlist
# narrows reachability to this bottle's bundle only.
for port in spec.ports_to_publish:
argv += ["-p", f"{spec.publish_host_ip}::{port}"]
argv.append(spec.image)
result = subprocess.run(
argv, capture_output=True, text=True,
env=dict(env) if env is not None else None, check=False,
)
if result.returncode != 0:
die(
f"docker run for bundle {container} failed: "
f"{(result.stderr or '').strip()}"
)
def bundle_host_port(
slug: str, container_port: int, *, host_ip: str = "127.0.0.1",
) -> int:
"""`docker port <bundle> <container_port>/tcp` → the random
host-side port docker assigned for the binding on `host_ip`.
Called after `start_bundle` on each container port listed in
`BundleLaunchSpec.ports_to_publish` so the launch step can
build the agent's HTTPS_PROXY / GIT_GATE / SUPERVISE URLs in
`<host_ip>:<host port>` form."""
container = bundle_container_name(slug)
result = subprocess.run(
["docker", "port", container, f"{container_port}/tcp"],
capture_output=True, text=True, check=False,
)
if result.returncode != 0:
die(
f"docker port {container} {container_port}/tcp failed: "
f"{(result.stderr or '').strip() or '<no stderr>'}"
)
# Each line looks like `127.0.0.16:54321` — one per address
# family / host IP. Match on the expected host_ip prefix so
# bottles bound to per-bottle aliases pick the right line.
for raw in (result.stdout or "").splitlines():
line = raw.strip()
if line.startswith(f"{host_ip}:"):
_, _, port_str = line.rpartition(":")
try:
return int(port_str)
except ValueError:
die(f"unexpected `docker port` output: {line!r}")
die(
f"no port mapping on {host_ip} for {container} "
f"{container_port}/tcp; got: {(result.stdout or '').strip()!r}"
)
def stop_bundle(slug: str) -> None:
"""Idempotent: a missing container returns success."""
container = bundle_container_name(slug)
result = subprocess.run(
["docker", "rm", "-f", container],
capture_output=True, text=True, check=False,
)
if result.returncode == 0:
return
if "no such container" in (result.stderr or "").lower():
return
warn(
f"docker rm -f {container} failed: "
f"{(result.stderr or '').strip()}"
)
+273
View File
@@ -0,0 +1,273 @@
"""Thin subprocess wrapper around the `smolvm` CLI (PRD 0023).
One thin Python function per smolvm subcommand the launch flow
needs. Two design choices worth flagging:
- **No daemon, no SDK.** smolvm 0.8.0 ships a `smolvm serve`
HTTP API as the long-term-clean integration target. The
project's stdlib-first ethos + the lower-overhead CLI calls
push v1 to shell out via `subprocess.run`. If a future
smolvm release makes `serve` mandatory (or significantly
faster), revisit.
- **Two return shapes.** `SmolvmRunResult` (returncode + stdout
+ stderr captured) is returned by `machine_exec` because the
caller cares about the in-VM command's exit status, and by
test helpers that introspect output. The other calls
(`machine_start`, `machine_stop`, `pack_create`, etc.) raise
`SmolvmError` on non-zero exit failure to start a VM is
fatal to the launch flow, not something callers want to
branch on.
The wrapper is unit-tested with `subprocess.run` mocked; the
integration smoke test (chunk 2d) exercises against a real
smolvm binary."""
from __future__ import annotations
import json
import shutil
import subprocess
import time
from dataclasses import dataclass
from pathlib import Path
from typing import Mapping, Sequence
_SMOLVM = "smolvm"
@dataclass(frozen=True)
class SmolvmRunResult:
"""Captured result of an in-VM command. Mirrors the structure
`Bottle.exec` returns so callers can hand it straight through."""
returncode: int
stdout: str
stderr: str
class SmolvmError(RuntimeError):
"""Raised when a smolvm subprocess returns non-zero on a path
where the caller has no useful branch to take (start failed,
pack failed, etc.). Carries the captured stderr for the
operator-facing log line."""
def __init__(self, argv: Sequence[str], result: subprocess.CompletedProcess[str]):
self.argv = list(argv)
self.returncode = result.returncode
self.stdout = result.stdout
self.stderr = result.stderr
cmd = " ".join(self.argv)
super().__init__(
f"{cmd!r} failed (exit {result.returncode}): "
f"{(result.stderr or '').strip() or '<no stderr>'}"
)
def _smolvm(*args: str, env: Mapping[str, str] | None = None,
check: bool = True) -> subprocess.CompletedProcess[str]:
"""One subprocess call into the smolvm CLI. `check=True`
raises SmolvmError on non-zero; `check=False` returns the
CompletedProcess for the caller to inspect."""
argv = [_SMOLVM, *args]
result = subprocess.run(
argv,
capture_output=True,
text=True,
env=dict(env) if env is not None else None,
check=False,
)
if check and result.returncode != 0:
raise SmolvmError(argv, result)
return result
# --- Pack ----------------------------------------------------------------
def pack_create(image: str, output: Path) -> None:
"""`smolvm pack create --image <image> -o <output>`. Converts
an OCI image into a self-contained `.smolmachine` artifact
smolvm can boot via `machine create --from`. Idempotent on the
smolvm side re-running with the same image+output rebuilds
from layer cache."""
_smolvm("pack", "create", "--image", image, "-o", str(output))
def pack_create_from_vm(name: str, output: Path) -> None:
"""`smolvm pack create --from-vm <name> -o <output>`.
Snapshots an existing persistent VM into a pack artifact. As
with `pack_create`, smolvm writes a launcher at `output` and the
bootable sidecar at `output.smolmachine`.
"""
_smolvm("pack", "create", "--from-vm", name, "-o", str(output))
# --- Machine lifecycle ---------------------------------------------------
def machine_create(
name: str,
*,
image: str | None = None,
from_path: Path | None = None,
allow_cidrs: Sequence[str] = (),
env: Mapping[str, str] | None = None,
) -> None:
"""`smolvm machine create NAME [--image IMG | --from PATH]
[--allow-cidr CIDR ...] [-e K=V ...]`. NAME is positional
(the CLI's exception to the `--name` pattern other
subcommands use).
`image` (registry ref like `alpine:latest`) and `from_path`
(a `.smolmachine` artifact) are mutually exclusive one or
the other tells smolvm what to boot. The wrapper doesn't
enforce exclusivity; smolvm errors clearly enough.
`allow_cidrs` and `env` are passed as CLI flags instead of a
Smolfile because `--from` and `--smolfile` are themselves
mutually exclusive in smolvm 0.8.0 and we want `--from`'s
no-pull-at-start property. The flag form gives the same
result without the Smolfile complication.
`--net` is sent explicitly when `allow_cidrs` is non-empty.
smolvm 0.8.0's docs say `--allow-cidr` implies `--net`, but
empirically the implication only fires when no `--from` is
set `--from PATH --allow-cidr X/32` silently produces a
machine with `network: false` and no routes in the guest, so
the agent can't reach the bundle's pinned IP."""
args: list[str] = ["machine", "create"]
if image is not None:
args += ["--image", image]
if from_path is not None:
args += ["--from", str(from_path)]
if allow_cidrs:
args.append("--net")
for cidr in allow_cidrs:
args += ["--allow-cidr", cidr]
if env:
for k, v in env.items():
args += ["-e", f"{k}={v}"]
args.append(name)
_smolvm(*args)
def machine_is_running(name: str) -> bool:
"""Return True if the named VM is in the 'running' state."""
result = _smolvm("machine", "ls", "--json", check=False)
if result.returncode != 0:
return False
try:
machines = json.loads(result.stdout or "[]")
except ValueError:
return False
return any(
isinstance(m, dict) and m.get("name") == name and m.get("state") == "running"
for m in machines
)
def machine_start(name: str) -> None:
"""`smolvm machine start --name NAME`."""
_smolvm("machine", "start", "--name", name)
def machine_stop(name: str) -> None:
"""`smolvm machine stop --name NAME`. Idempotent against
already-stopped machines: smolvm prints a notice and exits 0
in that case, so no special handling here."""
_smolvm("machine", "stop", "--name", name)
def machine_delete(name: str) -> None:
"""`smolvm machine delete -f NAME`. NAME is positional. `-f`
skips the interactive confirmation required for
non-interactive teardown."""
_smolvm("machine", "delete", "-f", name)
def machine_exec(
name: str,
argv: Sequence[str],
*,
env: Mapping[str, str] | None = None,
workdir: str | None = None,
timeout: str | None = None,
) -> SmolvmRunResult:
"""`smolvm machine exec --name NAME [-w DIR] [--timeout DUR]
[-e K=V ...] -- ARGV...`. Returns the captured result rather
than raising callers (including `Bottle.exec`) care about
the in-VM command's exit code, not just whether smolvm ran.
`env` here is in-VM env vars (`-e K=V`), not the host
subprocess env smolvm's own argv carries them through the
VMM."""
flags: list[str] = ["machine", "exec", "--name", name]
if workdir is not None:
flags += ["-w", workdir]
if timeout is not None:
flags += ["--timeout", timeout]
if env:
for k, v in env.items():
flags += ["-e", f"{k}={v}"]
# `--` separator before the command. smolvm's CLI requires it
# so its own flag parser doesn't grab argv items that look
# like flags.
flags.append("--")
flags += list(argv)
result = _smolvm(*flags, check=False)
return SmolvmRunResult(
returncode=result.returncode,
stdout=result.stdout or "",
stderr=result.stderr or "",
)
def wait_exec_ready(name: str, *, timeout: float = 5.0) -> None:
"""Poll `machine exec true` until exit 0 or `timeout` elapses.
Replaces `time.sleep(1.5)` after `machine_start`: libkrun's exec
channel needs a brief warm-up before back-to-back exec calls are
safe. Polling exits as soon as the channel is ready and fails
loudly if the VM never responds."""
deadline = time.monotonic() + timeout
delay = 0.1
while time.monotonic() < deadline:
r = machine_exec(name, ["true"])
if r.returncode == 0:
return
remaining = deadline - time.monotonic()
if remaining <= 0:
break
time.sleep(min(delay, remaining))
delay = min(delay * 2, 0.5)
argv = ["smolvm", "machine", "exec", "--name", name, "--", "true"]
raise SmolvmError(
argv,
subprocess.CompletedProcess(
args=argv, returncode=-1, stdout="",
stderr=f"exec channel not ready after {timeout:.0f}s — VM may have failed to boot.",
),
)
def machine_cp(src: str, dst: str) -> None:
"""`smolvm machine cp SRC DST`. Path syntax: `machine:path` to
reference a path inside the VM, bare path for the host. Both
SRC and DST are positional; either side can be machine: or
bare. Empty path is a no-op (returns immediately without
invoking smolvm)."""
if not src or not dst:
return
_smolvm("machine", "cp", src, dst)
# --- Discovery -----------------------------------------------------------
def is_available() -> bool:
"""True iff `smolvm` is on PATH. Used by the integration test
suite's skip-guards."""
return shutil.which(_SMOLVM) is not None
+50
View File
@@ -0,0 +1,50 @@
"""Slug / preflight / subnet helpers for the smolmachines backend
(PRD 0023). Kept in its own module so the renderers can be
unit-tested without importing the docker subprocess paths."""
from __future__ import annotations
import hashlib
import shutil
from ...log import die
def smolmachines_preflight() -> None:
"""Ensure `smolvm` is on PATH before the launch flow runs.
Called from `_resolve_plan`; gives the operator a clear
install pointer rather than a cryptic FileNotFoundError
later. `gvproxy` is no longer required see the PRD's design
pivot section."""
if shutil.which("smolvm") is not None:
return
die(
"BOT_BOTTLE_BACKEND=smolmachines requires `smolvm` on "
"PATH. Install with: "
"curl -sSL https://smolmachines.com/install.sh | sh. "
"To use the legacy Docker backend instead, set "
"BOT_BOTTLE_BACKEND=docker or pass --backend=docker."
)
def smolmachines_bundle_subnet(slug: str) -> tuple[str, str, str]:
"""Derive a per-bottle docker subnet + gateway IP + bundle IP
from the slug.
Returns `(subnet_cidr, gateway_ip, bundle_ip)`. The third
octet comes from SHA-256 of the slug mod 254 (skipping 17 to
avoid the docker-default bridge), so parallel bottles get
distinct /24s and `resume` reuses the same /24. The bundle
container always lands at `.2`; gateway is `.1`; the smolvm
Smolfile's `allow_cidrs` is `<bundle_ip>/32`."""
digest = hashlib.sha256(slug.encode("utf-8")).digest()
octet = (digest[0] % 254) + 1
# Skip the docker-default bridge to dodge the most common
# collision (operators with `docker0` at 172.17.x.x or a
# 192.168.17.x VPN client).
if octet == 17:
octet = 18
subnet = f"192.168.{octet}.0/24"
gateway = f"192.168.{octet}.1"
bundle_ip = f"192.168.{octet}.2"
return subnet, gateway, bundle_ip
+22 -24
View File
@@ -1,7 +1,8 @@
"""Per-bottle persistent state.
"""Per-bottle persistent state (PRD 0016).
Holds optional per-bottle Dockerfile overrides, the transcript snapshot
the state-preservation helper saves before teardown, and the launch metadata that lets
Holds the per-bottle Dockerfile override that capability-block
remediation writes, the transcript snapshot the state-preservation
helper saves before teardown, and the launch metadata that lets
`cli.py resume <identity>` reconstruct a bottle's spec. State
lives at:
@@ -60,7 +61,7 @@ _METADATA_NAME = "metadata.json"
_LIVE_CONFIG_SUBDIR = "live-config"
LIVE_CONFIG_ROUTES_NAME = "routes.yaml"
LIVE_CONFIG_ALLOWLIST_NAME = "allowlist"
# Empty marker file. Session preservation writes it before teardown so
# Empty marker file. capability_apply writes it before teardown so
# cli.py's session-end cleanup knows to preserve the state dir for
# `cli.py resume <identity>`. Absent = clean up.
_PRESERVE_MARKER = ".preserve"
@@ -105,16 +106,12 @@ class BottleMetadata:
# written before chunk 3 (resume / inspect should fall back to
# deriving from identity in that case).
compose_project: str = ""
# PRD 0040: backend name ("docker", "firecracker", "macos-container").
# Empty string for state dirs written before PRD 0040; callers default
# to "docker" for backward compatibility.
# PRD 0040: backend name ("docker" or "smolmachines"). Empty string
# for state dirs written before PRD 0040; callers default to "docker"
# for backward compatibility.
backend: str = ""
label: str = ""
color: str = ""
# Ordered bottle names selected at launch (issue #269). Empty tuple
# for state dirs written before this change; resume falls back to
# the agent's `bottle:` field in that case.
bottle_names: tuple[str, ...] = ()
def metadata_path(identity: str) -> Path:
@@ -142,10 +139,6 @@ def read_metadata(identity: str) -> BottleMetadata | None:
if not isinstance(raw, dict):
return None
raw_typed = cast(dict[str, object], raw)
raw_bottle_names = raw_typed.get("bottle_names", [])
bottle_names: tuple[str, ...] = ()
if isinstance(raw_bottle_names, list):
bottle_names = tuple(str(n) for n in raw_bottle_names if isinstance(n, str))
return BottleMetadata(
identity=str(raw_typed.get("identity", identity)),
agent_name=str(raw_typed.get("agent_name", "")),
@@ -156,7 +149,6 @@ def read_metadata(identity: str) -> BottleMetadata | None:
backend=str(raw_typed.get("backend", "")),
label=str(raw_typed.get("label", "")),
color=str(raw_typed.get("color", "")),
bottle_names=bottle_names,
)
@@ -172,7 +164,8 @@ def per_bottle_dockerfile_path(identity: str) -> Path:
def per_bottle_dockerfile(identity: str) -> str | None:
"""Return the per-bottle Dockerfile content if present, else
None. None means: use the provider or manifest Dockerfile."""
None. None means: use the repo's Dockerfile (the original
pre-capability-block behavior)."""
p = per_bottle_dockerfile_path(identity)
if p.is_file():
return p.read_text()
@@ -256,7 +249,9 @@ def write_live_config(
def transcript_snapshot_dir(identity: str) -> Path:
"""Where agent session snapshots are kept for resume flows."""
"""Where capability_apply stashes the agent's transcript before
teardown, so the next `cli.py start <agent>` can offer to
resume from it."""
return bottle_state_dir(identity) / _TRANSCRIPT_SUBDIR
@@ -283,9 +278,11 @@ def git_gate_state_dir(identity: str) -> Path:
def supervise_state_dir(identity: str) -> Path:
"""State subdir reserved for supervise sidecar bind-mount sources.
Runtime queue/audit rows live in the host-level bot-bottle SQLite
database, so they survive state-dir cleanup."""
"""State subdir for the supervise sidecar's current-config dir
(bind-mounted into the agent at /etc/bot-bottle/current-config).
The queue dir is intentionally NOT under here it lives at
~/.bot-bottle/queue/<slug>/ alongside the audit logs, so it
survives state-dir cleanup."""
return bottle_state_dir(identity) / _SUPERVISE_SUBDIR
@@ -304,8 +301,9 @@ def preserve_marker_path(identity: str) -> Path:
def mark_preserved(identity: str) -> Path:
"""Mark this bottle's state for preservation across session
teardown so cli.py's session-end cleanup leaves the state dir
intact for a subsequent `cli.py resume`."""
teardown. Written by capability_apply.apply_capability_change so
cli.py's session-end cleanup leaves the state dir intact for a
subsequent `cli.py resume`."""
path = preserve_marker_path(identity)
path.parent.mkdir(parents=True, exist_ok=True)
path.touch()
@@ -318,7 +316,7 @@ def is_preserved(identity: str) -> bool:
def clear_preserve_marker(identity: str) -> None:
"""Idempotent removal. Called at fresh launch (start or resume)
so a marker left from a prior preserved session doesn't keep
so a marker left from a prior capability-block doesn't keep
state alive past the next normal session-end."""
try:
preserve_marker_path(identity).unlink()
+3 -22
View File
@@ -1,19 +1,16 @@
"""Main CLI dispatcher.
Commands: backend, cleanup, commit, edit, info, init, list, resume, start, supervise
Commands: cleanup, commit, edit, info, init, list, resume, start, supervise
"""
from __future__ import annotations
import sys
from ..errors import MissingEnvVarError
from ..log import Die, die, error
from ..manifest import ManifestError
from ..store_manager import StoreManager
from ._common import PROG
from . import list as _list_mod
from .backend import cmd_backend
from .cleanup import cmd_cleanup
from .commit import cmd_commit
from .edit import cmd_edit
@@ -26,7 +23,6 @@ from .supervise import cmd_supervise
cmd_list = _list_mod.cmd_list
COMMANDS = {
"backend": cmd_backend,
"cleanup": cmd_cleanup,
"commit": cmd_commit,
"edit": cmd_edit,
@@ -42,7 +38,6 @@ COMMANDS = {
def usage() -> None:
sys.stderr.write(f"usage: {PROG} <command> [args...]\n\n")
sys.stderr.write("Commands:\n")
sys.stderr.write(" backend set up / check / undo a backend's host prerequisites (setup|status|teardown)\n")
sys.stderr.write(" cleanup stop and remove all active bot-bottle containers\n")
sys.stderr.write(" commit snapshot a running bottle's container state to a Docker image\n")
sys.stderr.write(" edit open an agent in vim for editing\n")
@@ -79,25 +74,11 @@ def main(argv: list[str] | None = None) -> int:
if handler is None:
usage()
die(f"unknown command: {command}")
mgr = StoreManager.instance()
if not mgr.is_migrated():
sys.stderr.write("bot-bottle: database schema is out of date\n")
sys.stderr.write("Migrate now? [y/N] ")
sys.stderr.flush()
try:
answer = sys.stdin.readline().strip().lower()
except EOFError:
answer = ""
if answer != "y":
error("migration required — re-run and confirm to migrate")
return 1
mgr.migrate()
try:
return handler(rest) or 0
except MissingEnvVarError as e:
error(str(e))
return 1
except ManifestError as e:
# Manifest/config problems surface as a catchable exception;
# print the reason and exit non-zero (same UX die() used to give).
error(str(e))
return 1
except Die as e:
-46
View File
@@ -1,46 +0,0 @@
"""`backend` CLI command — generic host setup/status across backends.
`./cli.py backend setup [--backend=NAME]` provisions (or points at how
to provision) the chosen backend's one-time host prerequisites.
`./cli.py backend status [--backend=NAME]` reports readiness.
`./cli.py backend teardown [--backend=NAME]` undoes setup (uninstall).
All dispatch to the backend's `setup()` / `status()` / `teardown()`
classmethods, so there are no backend-specific commands swapping
backends is just a different `--backend` (or `$BOT_BOTTLE_BACKEND`, or
the host default).
"""
from __future__ import annotations
import argparse
from ..backend import get_bottle_backend, known_backend_names
from ._common import PROG
def cmd_backend(args: list[str]) -> int:
parser = argparse.ArgumentParser(
prog=f"{PROG} backend",
description="Set up or check a backend's host prerequisites.",
)
parser.add_argument(
"action",
choices=("setup", "status", "teardown"),
help="setup: provision/print host prerequisites; status: report "
"readiness; teardown: undo setup (uninstall)",
)
parser.add_argument(
"--backend",
choices=known_backend_names(),
default=None,
help="backend to target (default: $BOT_BOTTLE_BACKEND or the host default)",
)
ns = parser.parse_args(args)
backend = get_bottle_backend(ns.backend)
if ns.action == "setup":
return backend.setup()
if ns.action == "teardown":
return backend.teardown()
return backend.status()
+8 -7
View File
@@ -1,20 +1,21 @@
"""cleanup: stop and remove all orphaned bot-bottle resources.
Walks every registered backend (docker, firecracker, macos-container)
so a single `./cli.py cleanup` reaps every backend's leftovers — a
firecracker bottle's sidecars won't survive a docker-only cleanup pass
(issue addressed alongside #77).
Walks every registered backend (docker + smolmachines) so a single
`./cli.py cleanup` reaps both backends' leftovers — orphaned
smolvm machines won't survive a docker-only cleanup pass (issue
addressed alongside #77).
Each backend's `prepare_cleanup` enumerates its own resources;
docker's `_list_orphan_state_dirs` consults
`enumerate_active_agents()` for the union of live identities so
state dirs of running non-docker bottles aren't reaped. State
state dirs of running smolmachines bottles aren't reaped. State
dirs are shared layout, so docker is the single owner of that
bucket.
State dirs with `.preserve` are intentionally never touched they
hold preserved sessions the operator may want to `resume`. Manual
`rm -rf ~/.bot-bottle/state/<identity>` is the path for those.
hold capability-block rebuilds or crash snapshots the operator may
want to `resume`. Manual `rm -rf ~/.bot-bottle/state/<identity>`
is the path for those.
"""
from __future__ import annotations
+4 -4
View File
@@ -2,10 +2,10 @@
Docker bottles are committed to a local Docker image. Macos-container
bottles are exported and rebuilt as a local Apple Container image.
Firecracker bottles stream the guest rootfs out over SSH and rebuild a
local Docker image. The resulting reference is stored in per-bottle
state so the next `./cli.py resume <slug>` boots from the snapshot
instead of rebuilding from the Dockerfile.
Smolmachines bottles are packed from the running VM into a
`.smolmachine` artifact. The resulting reference is stored in
per-bottle state so the next `./cli.py resume <slug>` boots from the
snapshot instead of rebuilding from the Dockerfile.
"""
from __future__ import annotations
+2 -3
View File
@@ -45,9 +45,8 @@ def cmd_list(argv: list[str]) -> int:
print(name)
return 0
# `active` enumerates every backend (docker, firecracker,
# macos-container) so non-docker bottles aren't hidden behind
# the env var.
# `active` enumerates every backend (docker + smolmachines)
# so smolmachines bottles aren't hidden behind the env var.
active = enumerate_active_agents()
if not active:
print("no active bot-bottle bottles", file=sys.stderr)
+7 -5
View File
@@ -4,12 +4,13 @@ Reads ~/.bot-bottle/state/<identity>/metadata.json to recover the
(agent_name, cwd, copy_cwd) the bottle was originally started with,
then runs the same launch core as `start` but pinned to the
recorded identity so the new bottle picks up any per-bottle Dockerfile
override and transcript snapshot under the same state dir.
(from capability-block apply) and transcript snapshot under the same
state dir.
Use case: an interrupted or preserved bottle needs to be relaunched;
the operator runs
Use case: an agent calls capability-block, the dashboard approves
and tears down the bottle, the operator runs
./cli.py resume <identity>
to bring up the replacement from the recorded state.
to bring up the replacement with the new capabilities baked in.
"""
from __future__ import annotations
@@ -27,6 +28,7 @@ from .start import _launch_bottle
def cmd_resume(argv: list[str]) -> int:
parser = argparse.ArgumentParser(prog=f"{PROG} resume", add_help=True)
parser.add_argument("--dry-run", action="store_true")
parser.add_argument("--remote-control", action="store_true")
parser.add_argument(
"identity",
help="bottle identity from a prior `start` (see its session-end output)",
@@ -49,11 +51,11 @@ def cmd_resume(argv: list[str]) -> int:
copy_cwd=metadata.copy_cwd,
user_cwd=metadata.cwd or USER_CWD,
identity=metadata.identity,
bottle_names=tuple(metadata.bottle_names),
)
backend_name = metadata.backend or None
return _launch_bottle(
spec,
dry_run=args.dry_run,
remote_control=args.remote_control,
backend_name=backend_name,
)
+27 -312
View File
@@ -2,11 +2,6 @@
interactive claude-code session. The container is torn down when the
session ends.
`--headless` selects a non-interactive launch (agent/bottles/label from
flags, no TUI selectors, no y/N prompt) for orchestrators,
CI, and webhook dispatch. The agent still execs on the inherited
stdio/PTY, so an orchestrator that allocates the PTY drives the session.
The launch core is shared with `cli.py resume <identity>` through
the private orchestrator `_launch_bottle`.
"""
@@ -21,7 +16,7 @@ import tempfile
from pathlib import Path
from typing import Callable
from ..agent_provider import get_provider, runtime_for
from ..agent_provider import runtime_for
from ..backend import (
Bottle,
BottleSpec,
@@ -36,8 +31,9 @@ from ..bottle_state import (
is_preserved,
mark_preserved,
)
from ..log import info, die
from ..manifest import Manifest, ManifestIndex
# from ..backend.docker.capability_apply import snapshot_transcript
from ..log import info
from ..manifest import ManifestIndex
from ._common import PROG, USER_CWD, read_tty_line
from . import tui
@@ -46,6 +42,7 @@ def cmd_start(argv: list[str]) -> int:
parser = argparse.ArgumentParser(prog=f"{PROG} start", add_help=True)
parser.add_argument("--dry-run", action="store_true")
parser.add_argument("--cwd", action="store_true", help="copy host cwd into the running bottle")
parser.add_argument("--remote-control", action="store_true")
parser.add_argument(
"--backend",
choices=known_backend_names(),
@@ -55,39 +52,6 @@ def cmd_start(argv: list[str]) -> int:
"or host auto-selection). Overrides the env var when set."
),
)
parser.add_argument(
"--headless",
action="store_true",
help=(
"non-interactive launch: take agent/bottles/label from flags, "
"skip all prompts. For orchestrators, CI, and webhooks."
),
)
parser.add_argument(
"--bottle",
action="append",
default=None,
metavar="NAME",
help=(
"bottle to compose, repeatable (order = merge order). In "
"--headless, defaults to the agent's own bottle when omitted."
),
)
parser.add_argument(
"--label",
default=None,
help="bottle label / terminal title (--headless default: agent name)",
)
parser.add_argument(
"--color",
default=None,
help="bottle color, one of the 16 ANSI color names (--headless default: none)",
)
parser.add_argument(
"--prompt",
default=None,
help="initial task prompt delivered to the agent (required with --headless)",
)
parser.add_argument(
"name",
nargs="?",
@@ -99,22 +63,9 @@ def cmd_start(argv: list[str]) -> int:
dry_run = args.dry_run or os.environ.get("BOT_BOTTLE_DRY_RUN") == "1"
manifest = ManifestIndex.resolve(USER_CWD)
backend_name: str | None = args.backend
if args.headless:
return _start_headless(
manifest, args, dry_run=dry_run, backend_name=backend_name
)
agent_name: str | None = args.name
if agent_name is None:
if not manifest.all_agent_names:
print(
"bot-bottle: no agents defined. "
"Add an agent to ~/.bot-bottle/agents/ or ./bot-bottle/agents/ to get started.",
file=sys.stderr,
)
return 1
agent_name = tui.filter_select(
manifest.all_agent_names,
title="Select agent",
@@ -122,22 +73,7 @@ def cmd_start(argv: list[str]) -> int:
if agent_name is None:
return 0
# Bottle multiselect: always show after agent selection so operators
# can compose bottles at launch time without editing agent manifests.
available_bottles = manifest.all_bottle_names
lineage_map = _bottle_lineage(manifest)
display_labels = [lineage_map.get(n, n) for n in available_bottles]
label_to_name = {lineage_map.get(n, n): n for n in available_bottles}
initial_bottle = _peek_agent_bottle(manifest, agent_name)
initial_labels = [lineage_map.get(initial_bottle, initial_bottle)] if initial_bottle else []
selected_labels = tui.filter_multiselect(
display_labels,
title="Select bottles",
initial=initial_labels,
)
if selected_labels is None:
return 0
bottle_names = tuple(label_to_name.get(lbl, lbl) for lbl in selected_labels)
backend_name: str | None = args.backend
label, color = tui.name_color_modal(default_label=agent_name)
label, color = _resolve_unique_label(label, color)
@@ -149,93 +85,15 @@ def cmd_start(argv: list[str]) -> int:
user_cwd=USER_CWD,
label=label,
color=color,
bottle_names=bottle_names,
)
return _launch_bottle(
spec,
dry_run=dry_run,
remote_control=args.remote_control,
backend_name=backend_name,
)
# --- Headless launch -----------------------------------------------------
def _start_headless(
manifest: ManifestIndex,
args: argparse.Namespace,
*,
dry_run: bool,
backend_name: str | None,
) -> int:
"""Non-interactive launch path for orchestrators / CI / webhooks.
Resolves agent, bottles, label, and color from flags + manifest
defaults instead of the TUI selectors, and auto-confirms the
preflight. Otherwise runs the same launch core as the interactive
path, so the agent still execs on the inherited stdio/PTY an
orchestrator allocates that PTY and relays it to its
desktop/mobile clients."""
agent_name = args.name
if not agent_name:
die("--headless requires an agent name: ./cli.py start <agent> --headless")
manifest.require_agent(agent_name) # raises ManifestError if unknown
prompt = args.prompt
if not prompt:
die(
"--headless requires --prompt: "
"./cli.py start <agent> --headless --prompt 'Do the thing'"
)
if args.bottle:
bottle_names: tuple[str, ...] = tuple(args.bottle)
else:
default_bottle = _peek_agent_bottle(manifest, agent_name)
if not default_bottle:
die(
f"--headless: agent '{agent_name}' has no default bottle; "
f"pass one or more --bottle NAME"
)
bottle_names = (default_bottle,)
label = _uniquify_label_headless(args.label or agent_name)
spec = BottleSpec(
manifest=manifest,
agent_name=agent_name,
copy_cwd=args.cwd,
user_cwd=USER_CWD,
label=label,
color=args.color or "",
bottle_names=bottle_names,
headless=True,
)
return _launch_bottle(
spec,
dry_run=dry_run,
backend_name=backend_name,
assume_yes=True,
headless_prompt_text=prompt,
)
def _uniquify_label_headless(label: str) -> str:
"""Non-interactive analog of `_resolve_unique_label`: if the label's
slug collides with a running bottle, append -2, -3, until free,
logging the chosen label. Orchestrators fire-and-forget many bottles,
so silently picking a free name beats erroring on every collision."""
active_slugs = {a.slug for a in enumerate_active_agents()}
if docker_mod.slugify(label) not in active_slugs:
return label
n = 2
while docker_mod.slugify(f"{label}-{n}") in active_slugs:
n += 1
chosen = f"{label}-{n}"
info(f"label '{label}' already in use; using '{chosen}'")
return chosen
# --- Launch helpers ------------------------------------------------------
@@ -243,7 +101,7 @@ def prepare_with_preflight(
spec: BottleSpec,
*,
stage_dir: Path,
render_preflight: Callable[[DockerBottlePlan, str], None],
render_preflight: Callable[[DockerBottlePlan], None],
prompt_yes: Callable[[], bool],
dry_run: bool = False,
backend_name: str | None = None,
@@ -264,7 +122,7 @@ def prepare_with_preflight(
plan = backend.prepare(spec, stage_dir=stage_dir)
identity = _identity_from_plan(plan)
render_preflight(plan, backend.name)
render_preflight(plan)
if dry_run:
info("dry-run requested; not starting container.")
@@ -276,7 +134,7 @@ def prepare_with_preflight(
def attach_agent(
bottle: Bottle, *, resume: bool = False,
bottle: Bottle, *, remote_control: bool = False, resume: bool = False,
agent_provider_template: str = "claude",
startup_args: tuple[str, ...] = (),
) -> int:
@@ -295,6 +153,8 @@ def attach_agent(
"(Ctrl-D or 'exit' to leave; container will be removed)"
)
agent_args = list(runtime.bypass_args)
if remote_control:
agent_args.extend(runtime.remote_control_args)
agent_args.extend(startup_args)
if resume:
agent_args.extend(runtime.resume_args)
@@ -334,38 +194,6 @@ def _identity_from_plan(plan: object) -> str:
return getattr(plan, "slug", "")
def _peek_agent_bottle(manifest: ManifestIndex, agent_name: str) -> str:
"""Return the `bottle:` value from the named agent's frontmatter without
fully parsing the agent file, or "" when absent or unreadable.
Used to pre-populate the bottle multiselect with the agent's default
bottle so operators who haven't removed `bottle:` from their manifests
don't need to re-select it every time."""
if manifest.home_md is None:
# Eager mode (from_json_obj): agent is pre-parsed.
if agent_name in manifest.agents:
return manifest.agents[agent_name].bottle
return ""
from ..manifest_loader import scan_agent_names
from ..yaml_subset import YamlSubsetError, parse_frontmatter
home_agents = scan_agent_names(manifest.home_md / "agents")
cwd_agents: dict[str, Path] = {}
if manifest.cwd_md is not None:
cwd_agents = scan_agent_names(manifest.cwd_md / "agents")
merged = {**home_agents, **cwd_agents}
path = merged.get(agent_name)
if path is None:
return ""
try:
fm, _ = parse_frontmatter(path.read_text())
bottle = fm.get("bottle", "")
return str(bottle) if isinstance(bottle, str) else ""
except (OSError, YamlSubsetError):
return ""
def _resolve_unique_label(label: str, color: str) -> tuple[str, str]:
"""Re-prompt with a disclaimer until the label's slug is not already
in use among running bottles. Passes through unchanged when no
@@ -390,141 +218,30 @@ def _text_prompt_yes() -> bool:
return reply in ("y", "Y", "yes", "YES")
def _text_render_preflight():
def _render(plan: DockerBottlePlan, backend_name: str) -> None:
print(file=sys.stderr)
print(f"backend: {backend_name}", file=sys.stderr)
print(_manifest_to_yaml(plan.manifest), file=sys.stderr)
def _text_render_preflight(*, remote_control: bool):
def _render(plan: DockerBottlePlan) -> None:
plan.print(remote_control=remote_control)
return _render
def _bottle_lineage(manifest: ManifestIndex) -> dict[str, str]:
"""Return {bottle_name: lineage_label} for bottles that have an extends chain.
Bottles without a parent are omitted (the caller falls back to the bare name).
Labels show the chain root-first: e.g. 'dev -> bot-bottle-dev -> claude-dev'."""
if manifest.home_md is None:
return {}
bottles_dir = manifest.home_md / "bottles"
if not bottles_dir.is_dir():
return {}
from ..yaml_subset import YamlSubsetError, parse_frontmatter
extends_of: dict[str, str] = {}
for path in bottles_dir.glob("*.md"):
try:
fm, _ = parse_frontmatter(path.read_text())
parent = fm.get("extends", "")
if isinstance(parent, str) and parent:
extends_of[path.stem] = parent
except (OSError, YamlSubsetError):
pass
labels: dict[str, str] = {}
for name in extends_of:
chain = [name]
seen = {name}
cur = name
while cur in extends_of:
par = extends_of[cur]
if par in seen:
break
chain.append(par)
seen.add(par)
cur = par
labels[name] = " -> ".join(reversed(chain))
return labels
def _manifest_to_yaml(manifest: Manifest) -> str:
"""Serialize the resolved Manifest to a YAML string for preflight display."""
lines: list[str] = []
agent = manifest.agent
lines.append("agent:")
if agent.skills:
lines.append(" skills:")
for s in agent.skills:
lines.append(f" - {s}")
if not agent.git_user.is_empty():
lines.append(" git-gate:")
lines.append(" user:")
if agent.git_user.name:
lines.append(f" name: {agent.git_user.name}")
if agent.git_user.email:
lines.append(f" email: {agent.git_user.email}")
bottle = manifest.bottle
lines.append("bottle:")
if bottle.agent_provider.template != "claude" or bottle.agent_provider.dockerfile:
lines.append(" agent_provider:")
lines.append(f" template: {bottle.agent_provider.template}")
if bottle.agent_provider.dockerfile:
lines.append(f" dockerfile: {bottle.agent_provider.dockerfile}")
if bottle.env:
lines.append(" env:")
for k, v in sorted(bottle.env.items()):
lines.append(f" {k}: {v}")
has_git_gate = not bottle.git_user.is_empty() or bottle.git
if has_git_gate:
lines.append(" git-gate:")
if not bottle.git_user.is_empty():
lines.append(" user:")
if bottle.git_user.name:
lines.append(f" name: {bottle.git_user.name}")
if bottle.git_user.email:
lines.append(f" email: {bottle.git_user.email}")
if bottle.git:
lines.append(" repos:")
for entry in bottle.git:
lines.append(f" {entry.Name}:")
lines.append(f" url: {entry.Upstream}")
if bottle.egress.routes:
lines.append(" egress:")
lines.append(" routes:")
for r in bottle.egress.routes:
lines.append(f" - host: {r.Host}")
if r.AuthScheme:
lines.append(f" auth:")
lines.append(f" scheme: {r.AuthScheme}")
lines.append(f" supervise: {'true' if bottle.supervise else 'false'}")
return "\n".join(lines)
def _launch_bottle(
spec: BottleSpec,
*,
dry_run: bool,
remote_control: bool,
backend_name: str | None = None,
assume_yes: bool = False,
headless_prompt_text: str = "",
) -> int:
"""Shared launch core for `start` and `resume`. Builds the plan,
prints / dry-runs / prompts as appropriate, brings the bottle up,
attaches claude, and prints the resume hint on session end.
`assume_yes` skips the interactive y/N confirmation (headless /
orchestrator launches), where there is no human at the prompt.
`headless_prompt_text` is passed to the provider's `headless_prompt`
method and the resulting args are appended to startup_args so the
agent receives the initial task without interactive input."""
attaches claude, and prints the resume hint on session end."""
stage_dir = Path(tempfile.mkdtemp(prefix="bot-bottle-stage."))
identity = ""
try:
plan, identity = prepare_with_preflight(
spec,
stage_dir=stage_dir,
render_preflight=_text_render_preflight(),
prompt_yes=(lambda: True) if assume_yes else _text_prompt_yes,
render_preflight=_text_render_preflight(remote_control=remote_control),
prompt_yes=_text_prompt_yes,
dry_run=dry_run,
backend_name=backend_name,
)
@@ -534,17 +251,11 @@ def _launch_bottle(
backend = get_bottle_backend(backend_name)
with backend.launch(plan) as bottle:
agent_provider_template = getattr(plan, "agent_provider_template", "claude")
extra_args: tuple[str, ...] = ()
if headless_prompt_text:
extra_args = tuple(
get_provider(agent_provider_template).headless_prompt(
headless_prompt_text
)
)
exit_code = attach_agent(
bottle,
remote_control=remote_control,
agent_provider_template=agent_provider_template,
startup_args=plan.agent_provision.startup_args + extra_args,
startup_args=plan.agent_provision.startup_args,
)
info(
f"session ended (exit {exit_code}); "
@@ -552,8 +263,12 @@ def _launch_bottle(
)
# While the container is still alive: always snapshot the
# transcript and — if the agent exited non-zero — mark
# the state for preservation. This picks up crashes /
# Ctrl-Cs / OOM kills before cleanup removes the state dir.
# the state for preservation. Capability-block already
# did both before triggering teardown from the dashboard;
# this picks up crashes / Ctrl-Cs / OOM kills the same
# way. snapshot_transcript is best-effort so the
# capability-block path's prior snapshot isn't clobbered
# when the container is already gone.
if agent_provider_template == "claude":
capture_claude_session_state(identity, exit_code)
return 0
+57 -18
View File
@@ -2,8 +2,9 @@
act on them (approve / modify / reject).
Curses-based TUI; modify-then-approve shells out to $EDITOR. The
Egress proposals are queued for operator review as full routes.yaml
updates.
approval handler wires to PRD 0016 (capability-block), which rebuilds
the bottle Dockerfile. Egress proposals are queued for operator review
as full routes.yaml updates.
"""
from __future__ import annotations
@@ -21,6 +22,10 @@ from pathlib import Path
from .. import supervise as _supervise
from ..bottle_state import read_metadata
# from ..backend.docker.capability_apply import (
# CapabilityApplyError,
# apply_capability_change,
# )
from ..backend.docker.egress_apply import (
EgressApplyError,
applicator as _docker_applicator,
@@ -28,8 +33,15 @@ from ..backend.docker.egress_apply import (
from ..backend.macos_container.egress_apply import (
applicator as _macos_applicator,
)
from ..backend.smolmachines.egress_apply import (
applicator as _smolmachines_applicator,
)
from ..log import Die, error, info
class CapabilityApplyError(RuntimeError):
"""Placeholder while capability_apply is disabled."""
from ..supervise import (
COMPONENT_FOR_TOOL,
AuditEntry,
@@ -38,11 +50,13 @@ from ..supervise import (
STATUS_APPROVED,
STATUS_MODIFIED,
STATUS_REJECTED,
TOOL_CAPABILITY_BLOCK,
TOOL_EGRESS_ALLOW,
TOOL_EGRESS_BLOCK,
TOOL_GITLEAKS_ALLOW,
TOOL_EGRESS_TOKEN_ALLOW,
list_all_pending_proposals,
archive_proposal,
list_pending_proposals,
render_diff,
write_audit_entry,
write_response,
@@ -60,15 +74,16 @@ _REPORT_ONLY_TOOLS: tuple[str, ...] = (TOOL_GITLEAKS_ALLOW, TOOL_EGRESS_TOKEN_AL
@dataclass(frozen=True)
class QueuedProposal:
"""A pending proposal from the supervise queue."""
"""A pending proposal plus the queue dir it was found in."""
proposal: Proposal
queue_dir: Path
# Errors any remediation engine may raise. Caught by the TUI key
# handlers and surfaced in the status line so a failed apply keeps
# the proposal pending rather than crashing curses.
ApplyError = (EgressApplyError,)
ApplyError = (CapabilityApplyError, EgressApplyError)
def apply_routes_change(slug: str, content: str) -> tuple[str, str]:
@@ -76,15 +91,22 @@ def apply_routes_change(slug: str, content: str) -> tuple[str, str]:
backend = meta.backend if meta is not None else ""
if backend == "macos-container":
return _macos_applicator.apply_routes_change(slug, content)
if backend == "smolmachines":
return _smolmachines_applicator.apply_routes_change(slug, content)
return _docker_applicator.apply_routes_change(slug, content)
def discover_pending() -> list[QueuedProposal]:
"""Collect pending proposals across bottles."""
out = [
QueuedProposal(proposal=proposal)
for proposal in list_all_pending_proposals()
]
"""Walk ~/.bot-bottle/queue/* and collect pending proposals."""
queue_root = _supervise.bot_bottle_root() / "queue"
if not queue_root.is_dir():
return []
out: list[QueuedProposal] = []
for slug_dir in sorted(queue_root.iterdir()):
if not slug_dir.is_dir():
continue
for proposal in list_pending_proposals(slug_dir):
out.append(QueuedProposal(proposal=proposal, queue_dir=slug_dir))
out.sort(key=lambda q: q.proposal.arrival_timestamp)
return out
@@ -107,6 +129,7 @@ def _detail_lines(
(f"tool: {p.tool}", 0),
(f"id: {p.id}", 0),
(f"arrived: {p.arrival_timestamp}", 0),
(f"queue: {qp.queue_dir}", 0),
("", 0),
("justification:", 0),
]
@@ -120,6 +143,8 @@ def _detail_lines(
def _suffix_for_tool(tool: str) -> str:
if tool == TOOL_CAPABILITY_BLOCK:
return ".dockerfile"
if tool in (TOOL_EGRESS_ALLOW, TOOL_EGRESS_BLOCK):
return ".yaml"
if tool in (TOOL_GITLEAKS_ALLOW, TOOL_EGRESS_TOKEN_ALLOW):
@@ -141,6 +166,17 @@ def approve(
file_to_apply = final_file if final_file is not None else qp.proposal.proposed_file
diff_before, diff_after = "", ""
# if qp.proposal.tool == TOOL_CAPABILITY_BLOCK:
# _meta = read_metadata(qp.proposal.bottle_slug)
# if _meta is not None and not _meta.compose_project:
# raise CapabilityApplyError(
# "capability-block remediation is not supported for smolmachines "
# "bottles. Reject this proposal or handle the capability change "
# "manually, then restart the bottle."
# )
# diff_before, diff_after = apply_capability_change(
# qp.proposal.bottle_slug, file_to_apply,
# )
if qp.proposal.tool in (TOOL_EGRESS_ALLOW, TOOL_EGRESS_BLOCK):
diff_before, diff_after = apply_routes_change(
qp.proposal.bottle_slug,
@@ -153,11 +189,14 @@ def approve(
notes=notes,
final_file=final_file,
)
write_response(qp.proposal.bottle_slug, response)
write_response(qp.queue_dir, response)
_write_audit(
qp, action=status, notes=notes,
diff_before=diff_before, diff_after=diff_after,
)
if qp.proposal.tool == TOOL_CAPABILITY_BLOCK:
archive_proposal(qp.queue_dir, qp.proposal.id)
def reject(qp: QueuedProposal, *, reason: str) -> None:
"""Write a rejection response and an audit entry."""
@@ -167,7 +206,7 @@ def reject(qp: QueuedProposal, *, reason: str) -> None:
notes=reason,
final_file=None,
)
write_response(qp.proposal.bottle_slug, response)
write_response(qp.queue_dir, response)
_write_audit(qp, action=STATUS_REJECTED, notes=reason, diff_before="", diff_after="")
@@ -307,7 +346,7 @@ def _list_once() -> int:
return 0
def _try_init_green() -> int: # pragma: no cover
def _try_init_green() -> int:
"""Initialise a green color pair and return its attr, or 0."""
try:
curses.start_color()
@@ -318,7 +357,7 @@ def _try_init_green() -> int: # pragma: no cover
return 0
def _main_loop(stdscr: "curses._CursesWindow") -> None: # type: ignore # pragma: no cover
def _main_loop(stdscr: "curses._CursesWindow") -> None: # type: ignore
curses.curs_set(0)
stdscr.timeout(_REFRESH_INTERVAL_MS)
green_attr = _try_init_green()
@@ -408,7 +447,7 @@ def _render(
status_line: str,
*,
green_attr: int = 0, # noqa: F841 — unused, but required by interface
) -> None: # pragma: no cover
) -> None:
stdscr.erase()
h, w = stdscr.getmaxyx()
header = f"bot-bottle supervise ({len(pending)} pending)"
@@ -459,7 +498,7 @@ def _detail_view(
qp: QueuedProposal,
*,
green_attr: int = 0,
) -> None: # pragma: no cover
) -> None:
"""Render the full proposal. Scrollable. Press q to return."""
lines = _detail_lines(qp, green_attr=green_attr)
offset = 0
@@ -511,7 +550,7 @@ def _detail_view(
return
def _modify(stdscr: "curses._CursesWindow", qp: QueuedProposal) -> str | None: # type: ignore # pragma: no cover
def _modify(stdscr: "curses._CursesWindow", qp: QueuedProposal) -> str | None: # type: ignore
"""Suspend curses, open $EDITOR on the proposed file, return edited content."""
suffix = _suffix_for_tool(qp.proposal.tool)
curses.endwin()
@@ -522,7 +561,7 @@ def _modify(stdscr: "curses._CursesWindow", qp: QueuedProposal) -> str | None:
return edited
def _prompt(stdscr: "curses._CursesWindow", label: str) -> str: # type: ignore # pragma: no cover
def _prompt(stdscr: "curses._CursesWindow", label: str) -> str: # type: ignore
"""One-line input at the bottom of the screen."""
curses.curs_set(1)
h, _ = stdscr.getmaxyx()
-300
View File
@@ -17,43 +17,6 @@ import sys
from typing import Any, Optional
def filter_multiselect(
items: list[str],
*,
title: str = "",
initial: Optional[list[str]] = None,
tty_path: str = "/dev/tty",
) -> Optional[list[str]]:
"""Render a multi-select picker over *items*.
Returns the ordered list of selected items, or ``None`` if the user
cancelled (Esc / ``q`` / Ctrl-C / Ctrl-D with no items).
Press Space to toggle the item under the cursor.
Press Enter to confirm the current selection.
Press Ctrl-D to confirm the current selection (returns even if empty).
Press Esc/q to cancel (returns None).
*initial* pre-populates the selection in insertion order. Items
added are appended; removed items leave the remaining order unchanged.
"""
if not items:
return []
try:
tty_fd = open(tty_path, "r+b", buffering=0)
except OSError:
return None
try:
fd_dup = os.dup(tty_fd.fileno())
return _run_multiselect(
items, title=title, initial=list(initial or []), tty_fd=fd_dup
)
finally:
tty_fd.close()
def filter_select(
items: list[str],
*,
@@ -258,269 +221,6 @@ def _addstr_safe(screen: Any, row: int, col: int, text: str, attr: int = curses.
pass
# ---------------------------------------------------------------------------
# filter_multiselect internals
# ---------------------------------------------------------------------------
_KEY_SPACE = 32
def _run_multiselect(
items: list[str], *, title: str, initial: list[str], tty_fd: int
) -> Optional[list[str]]:
"""Drive a curses multi-select session on *tty_fd*."""
os.environ.setdefault("TERM", "xterm-256color")
orig_stdin = sys.__stdin__
orig_stdout = sys.__stdout__
try:
import io
tty_text = io.TextIOWrapper(io.FileIO(tty_fd, mode='r+'), write_through=True)
sys.__stdin__ = tty_text # type: ignore[assignment]
sys.__stdout__ = tty_text # type: ignore[assignment]
screen = curses.initscr()
curses.noecho()
curses.cbreak()
screen.keypad(True)
try:
result = _multiselect_loop(screen, items, title=title, initial=initial)
finally:
screen.keypad(False)
curses.nocbreak()
curses.echo()
curses.endwin()
except Exception: # noqa: W0718
return None
finally:
sys.__stdin__ = orig_stdin # type: ignore[assignment]
sys.__stdout__ = orig_stdout # type: ignore[assignment]
return result
def _toggle_membership(items: list[str], item: str) -> None:
"""Add `item` if absent, remove it if present (in place)."""
if item in items:
items.remove(item)
else:
items.append(item)
def _handle_order_key(key: int, selected: list[str], order_cursor: int) -> int:
"""Apply a keypress in 'order' focus: navigate, reorder, or remove the
item at `order_cursor`. Mutates `selected` in place and returns the new
order cursor."""
if key in (curses.KEY_UP, ord("k")):
if order_cursor > 0:
order_cursor -= 1
elif key in (curses.KEY_DOWN, ord("j")):
if order_cursor < len(selected) - 1:
order_cursor += 1
elif key == ord("K"):
# Move selected item up (earlier in order).
if order_cursor > 0:
i = order_cursor
selected[i - 1], selected[i] = selected[i], selected[i - 1]
order_cursor -= 1
elif key == ord("J"):
# Move selected item down (later in order).
if order_cursor < len(selected) - 1:
i = order_cursor
selected[i], selected[i + 1] = selected[i + 1], selected[i]
order_cursor += 1
elif key in (curses.KEY_ENTER, _KEY_ENTER_ALT, ord("\r"), _KEY_SPACE):
# Remove item from selection while in order mode.
del selected[order_cursor]
if order_cursor >= len(selected) and order_cursor > 0:
order_cursor -= 1
return order_cursor
def _multiselect_loop(
screen: Any, items: list[str], *, title: str, initial: list[str]
) -> Optional[list[str]]:
query = ""
cursor = 0
selected: list[str] = [s for s in initial if s in items]
# focus = "filter": navigate + toggle items in the filterable list
# focus = "order": navigate + reorder items in the selected list
focus = "filter"
order_cursor = 0
while True:
filtered = _filter_items(items, query)
if not filtered:
cursor = 0
elif cursor >= len(filtered):
cursor = len(filtered) - 1
if not selected:
order_cursor = 0
if focus == "order":
focus = "filter"
elif order_cursor >= len(selected):
order_cursor = len(selected) - 1
try:
_render_multiselect(
screen, filtered, cursor,
query=query, title=title, selected=selected,
focus=focus, order_cursor=order_cursor,
)
except curses.error:
return None
try:
key = screen.getch()
except KeyboardInterrupt:
return None
if key in (_KEY_ESC, _KEY_CTRL_C, ord("q")):
return None
if key == _KEY_CTRL_D:
return list(selected)
# Tab toggles between filter and order focus.
if key == ord("\t"):
if focus == "filter" and selected:
focus = "order"
order_cursor = 0
else:
focus = "filter"
continue
if focus == "filter":
if key in (curses.KEY_ENTER, _KEY_ENTER_ALT, ord("\r")):
return list(selected)
elif key == _KEY_SPACE:
if filtered:
_toggle_membership(selected, filtered[cursor])
elif key in (curses.KEY_UP, ord("k")):
if cursor > 0:
cursor -= 1
elif key in (curses.KEY_DOWN, ord("j")):
if cursor < len(filtered) - 1:
cursor += 1
elif key in (curses.KEY_BACKSPACE, _KEY_BACKSPACE_WIN, 127):
query = query[:-1]
new_filtered = _filter_items(items, query)
if cursor >= len(new_filtered):
cursor = max(0, len(new_filtered) - 1)
elif 32 <= key <= 126 and key != _KEY_SPACE:
query += chr(key)
cursor = 0
else: # focus == "order"
order_cursor = _handle_order_key(key, selected, order_cursor)
def _render_multiselect(
screen: Any,
filtered: list[str],
cursor: int,
*,
query: str,
title: str,
selected: list[str],
focus: str = "filter",
order_cursor: int = 0,
) -> None:
screen.erase()
rows, cols = screen.getmaxyx()
min_rows = 7
if rows < min_rows:
raise curses.error("terminal too small")
sep = "" * min(cols - 1, 40)
row = 0
if title and row < rows - 1:
_addstr_safe(screen, row, 0, title[:cols - 1], curses.A_BOLD)
row += 1
# Filter line — dim when focus is on the order panel.
filter_label = f"Filter: {query}"
filter_hint = " [Tab: reorder]" if focus == "filter" and selected else ""
filter_attr = curses.A_DIM if focus == "order" else curses.A_NORMAL
if row < rows - 1:
_addstr_safe(screen, row, 0, (filter_label + filter_hint)[:cols - 1], filter_attr)
row += 1
if row < rows - 1:
_addstr_safe(screen, row, 0, sep)
row += 1
# Compute how many rows the bottom order panel needs.
# Cap the visible selected list to keep the filter list legible.
order_rows = min(len(selected), max(1, (rows - row) // 3)) if selected else 0
# Bottom reserved: sep + order_rows + sep + help = order_rows + 3
bottom_reserved = order_rows + 3
list_start = row
list_rows = rows - list_start - bottom_reserved
if list_rows < 1:
list_rows = 1
selected_set = set(selected)
filter_dim = focus == "order"
scroll = max(0, cursor - list_rows + 1)
visible = filtered[scroll: scroll + list_rows]
for idx, item in enumerate(visible):
abs_idx = scroll + idx
mark = "[*]" if item in selected_set else "[ ]"
prefix = "> " if (abs_idx == cursor and focus == "filter") else " "
line = (prefix + mark + " " + item)[:cols - 1]
item_attr = curses.A_DIM if filter_dim else (
curses.A_REVERSE if abs_idx == cursor else curses.A_NORMAL
)
if row < rows - bottom_reserved:
_addstr_safe(screen, row, 0, line, item_attr)
row += 1
# Separator before the order panel.
if row < rows - (order_rows + 2):
_addstr_safe(screen, row, 0, sep)
row += 1
# Order panel.
order_scroll = max(0, order_cursor - order_rows + 1)
order_visible = selected[order_scroll: order_scroll + order_rows]
for idx, item in enumerate(order_visible):
abs_idx = order_scroll + idx
is_active = focus == "order" and abs_idx == order_cursor
prefix = "> " if is_active else " "
line = (prefix + item)[:cols - 1]
attr = curses.A_REVERSE if is_active else curses.A_NORMAL
if row < rows - 2:
_addstr_safe(screen, row, 0, line, attr)
row += 1
if row < rows - 1:
_addstr_safe(screen, row, 0, sep)
row += 1
if focus == "filter":
help_line = "[↑↓/jk] move [Space] toggle [Enter] confirm [Tab] reorder [Esc/q] cancel"
else:
help_line = "[↑↓/jk] cursor [K/J] reorder [Space/Enter] remove [Tab] back [Ctrl-D] done"
if row < rows:
_addstr_safe(screen, min(rows - 1, row), 0, help_line[:cols - 1])
screen.refresh()
# ---------------------------------------------------------------------------
# name_color_modal — two-step label + color picker
# ---------------------------------------------------------------------------
+1 -1
View File
@@ -21,7 +21,7 @@ FROM node:22-slim
# to it) works against egress's bumped TLS without the agent needing
# local DNS.
RUN apt-get update \
&& apt-get install -y --no-install-recommends git ca-certificates curl ripgrep iproute2 dnsutils \
&& apt-get install -y --no-install-recommends git ca-certificates curl \
&& rm -rf /var/lib/apt/lists/*
# App-specific deps. Python isn't required by claude-code itself
+4 -10
View File
@@ -91,6 +91,7 @@ _RUNTIME = AgentProviderRuntime(
prompt_mode="append_file",
bypass_args=("--dangerously-skip-permissions",),
resume_args=("--continue",),
remote_control_args=("--remote-control",),
)
@@ -217,7 +218,7 @@ class ClaudeAgentProvider(AgentProvider):
if not agent.skills:
return
skills_dir = _skills_dir(plan.guest_home)
bottle.exec(f"mkdir -p {shlex.quote(skills_dir)}", user="root")
bottle.exec(f"mkdir -p {skills_dir}", user="root")
for name in agent.skills:
src = host_skill_dir(name)
if not os.path.isdir(src):
@@ -227,13 +228,9 @@ class ClaudeAgentProvider(AgentProvider):
)
dst = f"{skills_dir}/{name}"
info(f"copying skill {name} into {bottle.name}:{dst}")
# Defense in depth: skill names are validated kebab-case at
# manifest load, but quote the path so a future unvalidated
# field can't inject shell metacharacters here either.
dst_q = shlex.quote(dst)
bottle.exec(f"rm -rf {dst_q} && mkdir -p {dst_q}", user="root")
bottle.exec(f"rm -rf {dst} && mkdir -p {dst}", user="root")
bottle.cp_in(f"{src}/.", f"{dst}/")
bottle.exec(f"chown -R node:node {dst_q}", user="root")
bottle.exec(f"chown -R node:node {dst}", user="root")
def provision_prompt(self, plan: "BottlePlan", bottle: "Bottle") -> str | None:
"""Copy the prompt file into the guest, fix ownership/mode.
@@ -313,9 +310,6 @@ class ClaudeAgentProvider(AgentProvider):
f"claude mcp add --scope user --transport http supervise {supervise_url}"
)
def headless_prompt(self, prompt: str) -> list[str]:
return ["-p", prompt]
def _exec(bottle: "Bottle", script: str, error: str) -> None:
result = bottle.exec(script, user="root")
+6 -9
View File
@@ -1,12 +1,12 @@
# bot-bottle Codex provider image.
#
# Mirrors the default Claude image shape: Node LTS, git/network tooling,
# non-root node user, and the provider CLI installed for that user.
# non-root node user, and the provider CLI installed globally.
FROM node:22-slim
RUN apt-get update \
&& apt-get install -y --no-install-recommends git ca-certificates curl procps ripgrep \
&& apt-get install -y --no-install-recommends git ca-certificates curl \
&& rm -rf /var/lib/apt/lists/*
# App-specific deps. Python isn't required by codex itself
@@ -17,15 +17,12 @@ RUN apt-get update \
&& apt-get install -y --no-install-recommends python3 python3-pip python3-venv \
&& rm -rf /var/lib/apt/lists/*
RUN npm install -g --no-fund --no-audit @openai/codex@0.136.0 \
&& npm cache clean --force
USER node
WORKDIR /home/node
ENV PATH="/home/node/.local/bin:${PATH}"
# Remote-control support requires the standalone Codex install layout
# under ~/.codex/packages/standalone/current. The npm package can run
# the TUI, but remote-control commands expect this installer-owned path.
RUN mkdir -p /home/node/.codex \
&& curl -fsSL https://chatgpt.com/codex/install.sh | sh
RUN mkdir -p /home/node/.codex
CMD ["codex"]
+7 -20
View File
@@ -34,12 +34,6 @@ if TYPE_CHECKING:
_SUPERVISE_MCP_NAME = "supervise"
_CODEX_CLI = "/home/node/.codex/packages/standalone/current/bin/codex"
_CODEX_CLI_PATH = (
"/home/node/.local/bin:"
"/home/node/.codex/packages/standalone/current/bin:"
"/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"
)
def _skills_dir(guest_home: str) -> str:
@@ -56,11 +50,12 @@ def _prompt_path(guest_home: str) -> str:
_RUNTIME = AgentProviderRuntime(
template="codex",
command=_CODEX_CLI,
command="codex",
image="bot-bottle-codex:latest",
prompt_mode="read_prompt_file",
bypass_args=("--dangerously-bypass-approvals-and-sandbox",),
resume_args=("resume", "--last"),
remote_control_args=(),
)
@@ -151,8 +146,7 @@ class CodexAgentProvider(AgentProvider):
"env",
f"HOME={guest_home}",
f"CODEX_HOME={auth_dir}",
f"PATH={_CODEX_CLI_PATH}",
_CODEX_CLI, "login", "status",
"codex", "login", "status",
), (
"codex host credentials: dummy auth was copied into the "
"guest, but Codex did not accept it"
@@ -190,7 +184,7 @@ class CodexAgentProvider(AgentProvider):
if not agent.skills:
return
skills_dir = _skills_dir(plan.guest_home)
bottle.exec(f"mkdir -p {shlex.quote(skills_dir)}", user="root")
bottle.exec(f"mkdir -p {skills_dir}", user="root")
for name in agent.skills:
src = host_skill_dir(name)
if not os.path.isdir(src):
@@ -200,13 +194,9 @@ class CodexAgentProvider(AgentProvider):
)
dst = f"{skills_dir}/{name}"
info(f"copying skill {name} into {bottle.name}:{dst}")
# Defense in depth: skill names are validated kebab-case at
# manifest load, but quote the path so a future unvalidated
# field can't inject shell metacharacters here either.
dst_q = shlex.quote(dst)
bottle.exec(f"rm -rf {dst_q} && mkdir -p {dst_q}", user="root")
bottle.exec(f"rm -rf {dst} && mkdir -p {dst}", user="root")
bottle.cp_in(f"{src}/.", f"{dst}/")
bottle.exec(f"chown -R node:node {dst_q}", user="root")
bottle.exec(f"chown -R node:node {dst}", user="root")
def provision_prompt(self, plan: "BottlePlan", bottle: "Bottle") -> str | None:
"""Copy the prompt file into the guest, fix ownership/mode.
@@ -274,7 +264,7 @@ class CodexAgentProvider(AgentProvider):
return
info(f"registering supervise MCP server in agent codex config → {supervise_url}")
r = bottle.exec(
f"{shlex.quote(_CODEX_CLI)} mcp add {_SUPERVISE_MCP_NAME} --url "
f"codex mcp add {_SUPERVISE_MCP_NAME} --url "
f"{shlex.quote(supervise_url)}",
user="node",
)
@@ -286,9 +276,6 @@ class CodexAgentProvider(AgentProvider):
f"codex mcp add supervise --url {shlex.quote(supervise_url)}"
)
def headless_prompt(self, prompt: str) -> list[str]:
return [prompt]
def _exec(bottle: "Bottle", script: str, error: str) -> None:
result = bottle.exec(script, user="root")
@@ -21,11 +21,6 @@ from pathlib import Path
from ...deploy_key_provisioner import DeployKeyCollisionError, DeployKeyProvisioner
# Timeout for ssh-keygen and Gitea API HTTP calls. A hung Gitea instance at
# prepare time would stall bottle launch indefinitely without this bound.
_API_TIMEOUT_SECS = 30
_KEYGEN_TIMEOUT_SECS = 10
class GiteaDeployKeyProvisioner(DeployKeyProvisioner):
"""Manages deploy keys on a Gitea instance."""
@@ -51,7 +46,6 @@ class GiteaDeployKeyProvisioner(DeployKeyProvisioner):
check=True,
stdout=subprocess.DEVNULL,
stderr=subprocess.DEVNULL,
timeout=_KEYGEN_TIMEOUT_SECS,
)
private_key = key_path.read_bytes()
public_key = key_path.with_suffix(".pub").read_text().strip()
@@ -73,7 +67,7 @@ class GiteaDeployKeyProvisioner(DeployKeyProvisioner):
method="POST",
)
try:
with urllib.request.urlopen(req, timeout=_API_TIMEOUT_SECS) as resp:
with urllib.request.urlopen(req) as resp:
body = json.loads(resp.read())
except urllib.error.HTTPError as exc:
_body = _read_error_body(exc)
@@ -104,7 +98,7 @@ class GiteaDeployKeyProvisioner(DeployKeyProvisioner):
method="DELETE",
)
try:
with urllib.request.urlopen(req, timeout=_API_TIMEOUT_SECS):
with urllib.request.urlopen(req):
pass
except urllib.error.HTTPError as exc:
if exc.code == 404:
+4 -10
View File
@@ -166,6 +166,7 @@ _RUNTIME = AgentProviderRuntime(
prompt_mode="append_system_prompt",
bypass_args=(),
resume_args=(),
remote_control_args=(),
)
@@ -238,7 +239,7 @@ class PiAgentProvider(AgentProvider):
if not agent.skills:
return
skills_dir = _skills_dir(plan.guest_home)
bottle.exec(f"mkdir -p {shlex.quote(skills_dir)}", user="root")
bottle.exec(f"mkdir -p {skills_dir}", user="root")
for name in agent.skills:
src = host_skill_dir(name)
if not os.path.isdir(src):
@@ -248,13 +249,9 @@ class PiAgentProvider(AgentProvider):
)
dst = f"{skills_dir}/{name}"
info(f"copying skill {name} into {bottle.name}:{dst}")
# Defense in depth: skill names are validated kebab-case at
# manifest load, but quote the path so a future unvalidated
# field can't inject shell metacharacters here either.
dst_q = shlex.quote(dst)
bottle.exec(f"rm -rf {dst_q} && mkdir -p {dst_q}", user="root")
bottle.exec(f"rm -rf {dst} && mkdir -p {dst}", user="root")
bottle.cp_in(f"{src}/.", f"{dst}/")
bottle.exec(f"chown -R node:node {dst_q}", user="root")
bottle.exec(f"chown -R node:node {dst}", user="root")
def provision_prompt(self, plan: "BottlePlan", bottle: "Bottle") -> str | None:
prompt_path = _prompt_path(plan.guest_home)
@@ -315,9 +312,6 @@ class PiAgentProvider(AgentProvider):
) -> None:
del plan, bottle, supervise_url
def headless_prompt(self, prompt: str) -> list[str]:
return ["-p", prompt]
def _exec(bottle: "Bottle", script: str, error: str) -> None:
result = bottle.exec(script, user="root")
-59
View File
@@ -1,59 +0,0 @@
"""Shared SQLite-backed store base class for bot-bottle (PRD 0013)."""
from __future__ import annotations
import sqlite3
from pathlib import Path
try:
from .migrations import TableMigrations
except ImportError:
from migrations import TableMigrations # type: ignore[import-not-found] # pylint: disable=import-error,no-name-in-module
class DbVersionError(Exception):
"""Raised when the on-disk schema is behind the current migration list."""
class DbStore:
"""Base for SQLite-backed stores. Subclasses resolve db_path then call super().__init__."""
def __init__(self, db_path: Path, migrations: TableMigrations) -> None:
self.db_path = db_path
self._migrations = migrations
self.db_path.parent.mkdir(parents=True, exist_ok=True)
def _connect(self) -> sqlite3.Connection:
conn = sqlite3.connect(self.db_path)
conn.row_factory = sqlite3.Row
return conn
def is_migrated(self) -> bool:
"""Return True if the DB is fully up-to-date, False if migration is needed."""
if not self.db_path.exists():
return False
try:
with self._connect() as conn:
row = conn.execute(
"SELECT version FROM schema_versions WHERE module = ?",
(self._migrations.schema_key,),
).fetchone()
except sqlite3.OperationalError:
return False
version = row[0] if row else 0
return version == len(self._migrations.migrations)
def migrate(self) -> None:
"""Apply any pending migrations and set permissions on the DB file."""
with self._connect() as conn:
self._migrations.apply(conn)
self._chmod()
def _chmod(self) -> None:
try:
self.db_path.chmod(0o600)
except OSError:
pass
__all__ = ["DbStore", "DbVersionError"]
+19 -80
View File
@@ -11,7 +11,6 @@ the same try/except import shim pattern.
from __future__ import annotations
import base64
import functools
import gzip
import re
import typing
@@ -127,29 +126,8 @@ def redact_tokens(
# Known secrets detector
# ---------------------------------------------------------------------------
# Encoded-variant cache. Provisioned secrets are stable for the life of the
# proxy, but `_encoded_variants` is on the per-request hot path — it runs for
# every secret on every redaction and known-secret scan (host, path, each
# header, body). Deriving the variant set is relatively expensive (gzip +
# nine encodings), so memoize it per distinct secret. The proxy process
# already holds these values in `os.environ`, so caching them here adds no
# new exposure. The cache is bounded (lru_cache maxsize) so a long-lived
# proxy that sees rotating secrets evicts the oldest rather than growing
# without limit; 256 comfortably covers the EGRESS_TOKEN_* set in practice.
_VARIANT_CACHE_MAXSIZE = 256
def _encoded_variants(secret: str) -> list[str]:
"""Return the secret plus common encoded variants for exfil detection.
The variant set is computed once per distinct secret and cached; callers
get a fresh list so they can't mutate the shared cached tuple."""
return list(_compute_encoded_variants(secret))
@functools.lru_cache(maxsize=_VARIANT_CACHE_MAXSIZE)
def _compute_encoded_variants(secret: str) -> tuple[str, ...]:
"""Derive the secret plus its encoded variants (memoized, bounded)."""
"""Return the secret plus common encoded variants for exfil detection."""
seen: set[str] = {secret}
variants: list[str] = [secret]
@@ -183,7 +161,7 @@ def _compute_encoded_variants(secret: str) -> tuple[str, ...]:
# gzip + base64 (deterministic: mtime=0); recognisable by H4sI prefix
_add(base64.b64encode(gzip.compress(secret_bytes, mtime=0)).decode("ascii"))
return tuple(variants)
return variants
# ---------------------------------------------------------------------------
@@ -209,24 +187,18 @@ def _alnum_projection(text: str) -> str:
def _find_partial_window(secret_alnum: str, text_alnum: str, min_len: int) -> int | None:
"""Return the earliest position in text_alnum holding a min_len-char window
that also appears in secret_alnum, or None.
"""Return the position in text_alnum where any min_len-char window of
secret_alnum first appears, or None.
The secret's set of min_len-grams is small (bounded by the secret length),
so building it once and sweeping the text a single time is O(len(text))
rather than the O(len(secret) * len(text)) of repeated substring searches
which matters because this runs per provisioned secret on every request
body. Coverage is unchanged: a hit still means at least min_len consecutive
alphanumeric characters of the secret leaked into the text.
Slides a window of width min_len across secret_alnum and searches for
each window in text_alnum. The first hit position is returned.
"""
if len(secret_alnum) < min_len or len(text_alnum) < min_len:
return None
secret_grams = {
secret_alnum[i:i + min_len]
for i in range(len(secret_alnum) - min_len + 1)
}
for pos in range(len(text_alnum) - min_len + 1):
if text_alnum[pos:pos + min_len] in secret_grams:
for i in range(len(secret_alnum) - min_len + 1):
window = secret_alnum[i:i + min_len]
pos = text_alnum.find(window)
if pos >= 0:
return pos
return None
@@ -392,52 +364,19 @@ JAILBREAK_PHRASES: tuple[re.Pattern[str], ...] = (
PROXIMITY_CHARS = 500
def _match_gap(a: re.Match[str], b: re.Match[str]) -> int:
"""Character gap between two match spans; 0 when they overlap or touch."""
return max(0, max(a.start(), b.start()) - min(a.end(), b.end()))
def _closest_pair(
a_matches: list[re.Match[str]],
b_matches: list[re.Match[str]],
*,
within: int | None = None,
) -> tuple[re.Match[str], re.Match[str]] | None:
"""Return the (a, b) pair with the smallest character gap, or None when
either list is empty.
Runs in O(n log n) sort + O(n) merge rather than the O(n*m) cross product:
both lists are sorted by start offset and swept with a two-pointer merge,
advancing whichever span ends first (it can only get farther from any
later span in the other list). This matters because the inputs are
attacker-controlled response-body matches that have already passed the
body-size cap, so the quadratic form is a latent DoS.
When `within` is set, returns as soon as a pair with gap <= within is
found: the only caller blocks on any pair inside the proximity threshold,
so the exact global minimum past that point doesn't change the decision.
"""
if not a_matches or not b_matches:
return None
a_sorted = sorted(a_matches, key=lambda m: m.start())
b_sorted = sorted(b_matches, key=lambda m: m.start())
i = j = 0
"""Return the pair (a, b) with the smallest character gap, or None."""
best: tuple[re.Match[str], re.Match[str]] | None = None
best_gap: int | None = None
while i < len(a_sorted) and j < len(b_sorted):
a, b = a_sorted[i], b_sorted[j]
gap = _match_gap(a, b)
if best_gap is None or gap < best_gap:
best_gap = gap
best = (a, b)
if within is not None and gap <= within:
return best
# Advance the span that ends first; it cannot form a closer pair with
# any later (further-right) span from the other list.
if a.end() <= b.end():
i += 1
else:
j += 1
for a in a_matches:
for b in b_matches:
gap = max(0, max(a.start(), b.start()) - min(a.end(), b.end()))
if best_gap is None or gap < best_gap:
best_gap = gap
best = (a, b)
return best
@@ -447,9 +386,9 @@ def scan_naive_injection(text: str) -> ScanResult | None:
jailbreak_hits = [m for p in JAILBREAK_PHRASES for m in p.finditer(text)]
if disclosure_hits and jailbreak_hits:
pair = _closest_pair(disclosure_hits, jailbreak_hits, within=PROXIMITY_CHARS)
pair = _closest_pair(disclosure_hits, jailbreak_hits)
if pair is not None:
dist = _match_gap(pair[0], pair[1])
dist = max(0, max(pair[0].start(), pair[1].start()) - min(pair[0].end(), pair[1].end()))
if dist <= PROXIMITY_CHARS:
first = pair[0] if pair[0].start() <= pair[1].start() else pair[1]
return ScanResult(
+14 -28
View File
@@ -23,7 +23,6 @@ from .egress_addon_core import (
PathMatch as CorePathMatch,
Route,
)
from .errors import MissingEnvVarError
from .log import die
if TYPE_CHECKING:
@@ -211,17 +210,6 @@ def egress_token_env_map(
return out
def _yaml_str_escape(s: str) -> str:
"""Escape a string for use inside a YAML double-quoted scalar."""
return (
s.replace("\\", "\\\\")
.replace('"', '\\"')
.replace("\n", "\\n")
.replace("\r", "\\r")
.replace("\t", "\\t")
)
def _route_to_yaml_fields(r: Route) -> dict[str, object]:
fields: dict[str, object] = {"host": r.host}
if r.auth_scheme and r.token_env:
@@ -284,12 +272,12 @@ def _render_match_entry(entry: dict[str, object]) -> list[str]:
for pd in entry["paths"]: # type: ignore[union-attr]
pd_dict: dict[str, str] = pd # type: ignore[assignment]
if "type" in pd_dict:
lines.append(f' - type: "{_yaml_str_escape(pd_dict["type"])}"')
lines.append(f' value: "{_yaml_str_escape(pd_dict["value"])}"')
lines.append(f' - type: "{pd_dict["type"]}"')
lines.append(f' value: "{pd_dict["value"]}"')
else:
lines.append(f' - value: "{_yaml_str_escape(pd_dict["value"])}"')
lines.append(f' - value: "{pd_dict["value"]}"')
if "methods" in entry:
methods_str = ", ".join(f'"{_yaml_str_escape(m)}"' for m in entry["methods"]) # type: ignore[union-attr]
methods_str = ", ".join(f'"{m}"' for m in entry["methods"]) # type: ignore[union-attr]
prefix = " - " if first_key else " "
lines.append(f'{prefix}methods: [{methods_str}]')
first_key = False
@@ -299,8 +287,8 @@ def _render_match_entry(entry: dict[str, object]) -> list[str]:
first_key = False
for hd in entry["headers"]: # type: ignore[union-attr]
hd_dict: dict[str, str] = hd # type: ignore[assignment]
lines.append(f' - name: "{_yaml_str_escape(hd_dict["name"])}"')
lines.append(f' value: "{_yaml_str_escape(hd_dict["value"])}"')
lines.append(f' - name: "{hd_dict["name"]}"')
lines.append(f' value: "{hd_dict["value"]}"')
if first_key:
lines.append(" - {}")
return lines
@@ -320,10 +308,10 @@ def egress_render_routes(
return "\n".join(lines) + "\n"
for r in routes:
f = _route_to_yaml_fields(r)
lines.append(f' - host: "{_yaml_str_escape(str(f["host"]))}"')
lines.append(f' - host: "{f["host"]}"')
if "auth_scheme" in f:
lines.append(f' auth_scheme: "{_yaml_str_escape(str(f["auth_scheme"]))}"')
lines.append(f' token_env: "{_yaml_str_escape(str(f["token_env"]))}"')
lines.append(f' auth_scheme: "{f["auth_scheme"]}"')
lines.append(f' token_env: "{f["token_env"]}"')
if "matches" in f:
lines.append(" matches:")
for entry in f["matches"]: # type: ignore[union-attr]
@@ -343,7 +331,7 @@ def egress_render_routes(
items_str = ", ".join(f'"{x}"' for x in dv)
lines.append(f" {dk}: [{items_str}]")
elif isinstance(dv, str):
lines.append(f' {dk}: "{_yaml_str_escape(dv)}"')
lines.append(f' {dk}: "{dv}"')
return "\n".join(lines) + "\n"
@@ -355,18 +343,16 @@ def egress_resolve_token_values(
for token_env, token_ref in token_env_map.items():
value = host_env.get(token_ref)
if value is None:
raise MissingEnvVarError(
token_ref,
die(
f"egress: host env var '{token_ref}' is unset. Set it "
f"before launching, or remove the corresponding auth block "
f"from bottle.egress.routes.",
f"from bottle.egress.routes."
)
if not value:
raise MissingEnvVarError(
token_ref,
die(
f"egress: host env var '{token_ref}' is empty. The "
f"egress will not inject an empty token; set it to "
f"the real value or remove the route's auth block.",
f"the real value or remove the route's auth block."
)
out[token_env] = value
return out
+9 -6
View File
@@ -79,13 +79,14 @@ class EgressAddon:
# only — a restart re-prompts. Mutated only from the asyncio loop that
# runs the addon hooks, so no lock is needed.
self.safe_tokens: set[str] = set()
self._supervise_queue_dir = os.environ.get("SUPERVISE_QUEUE_DIR", "").strip()
self._supervise_slug = os.environ.get("SUPERVISE_BOTTLE_SLUG", "").strip()
self._token_allow_timeout = _token_allow_timeout_from_env(os.environ)
self._reload(initial=True)
self._install_sighup()
def _supervise_available(self) -> bool:
return bool(self._supervise_slug)
return bool(self._supervise_queue_dir and self._supervise_slug)
def _reload(self, *, initial: bool = False) -> None:
try:
@@ -392,8 +393,9 @@ class EgressAddon:
justification=_TOKEN_ALLOW_JUSTIFICATION,
current_file_hash=_sv.sha256_hex(payload),
)
queue_dir = Path(self._supervise_queue_dir)
try:
_sv.write_proposal(proposal)
_sv.write_proposal(queue_dir, proposal)
except OSError as e:
sys.stderr.write(
f"egress: could not queue token-allow proposal: {e}; "
@@ -409,8 +411,8 @@ class EgressAddon:
**self._req_ctx(flow),
}) + "\n")
response = await self._await_token_response(proposal.id)
_sv.archive_proposal(self._supervise_slug, proposal.id)
response = await self._await_token_response(queue_dir, proposal.id)
_sv.archive_proposal(queue_dir, proposal.id)
if response is not None and response.status in (
_sv.STATUS_APPROVED, _sv.STATUS_MODIFIED,
@@ -437,15 +439,16 @@ class EgressAddon:
async def _await_token_response(
self,
queue_dir: Path,
proposal_id: str,
) -> "_sv.Response | None":
"""Poll the DB for the operator's response without blocking the
"""Poll the queue dir for the operator's response without blocking the
proxy event loop. Returns the Response, or None on timeout."""
loop = asyncio.get_running_loop()
deadline = loop.time() + self._token_allow_timeout
while True:
try:
return _sv.read_response(self._supervise_slug, proposal_id)
return _sv.read_response(queue_dir, proposal_id)
except (OSError, ValueError, KeyError):
# Not written yet, or a partial/malformed write — retry until
# the deadline, then fail closed.
+89 -30
View File
@@ -21,32 +21,6 @@ try:
except ImportError: # pragma: no cover - host-side path
from .yaml_subset import YamlSubsetError, parse_yaml_subset
# DLP detector-config parsing lives in a sibling module (also flat-bundled
# into the sidecar — see Dockerfile.sidecars). Re-exported below so existing
# `from egress_addon_core import ON_MATCH_*` callers keep working.
try:
from egress_dlp_config import ( # type: ignore[import-not-found]
DEFAULT_OUTBOUND_ON_MATCH,
INBOUND_DETECTOR_NAMES,
ON_MATCH_BLOCK,
ON_MATCH_REDACT,
ON_MATCH_SUPERVISE,
OUTBOUND_DETECTOR_NAMES,
OUTBOUND_ON_MATCH_VALUES,
parse_dlp_block,
)
except ImportError: # pragma: no cover - host-side path
from .egress_dlp_config import (
DEFAULT_OUTBOUND_ON_MATCH,
INBOUND_DETECTOR_NAMES,
ON_MATCH_BLOCK,
ON_MATCH_REDACT,
ON_MATCH_SUPERVISE,
OUTBOUND_DETECTOR_NAMES,
OUTBOUND_ON_MATCH_VALUES,
parse_dlp_block,
)
# ---------------------------------------------------------------------------
# Match types (Gateway API HTTPRoute vocabulary, PRD 0053)
@@ -60,6 +34,18 @@ VALID_METHODS = frozenset({
"CONNECT",
})
OUTBOUND_DETECTOR_NAMES = frozenset({"token_patterns", "known_secrets", "entropy"})
INBOUND_DETECTOR_NAMES = frozenset({"naive_injection_detection"})
# Per-route policy for what the proxy does when an outbound DLP detector
# matches a token (PRD 0062).
ON_MATCH_BLOCK = "block" # hard 403, never overridable
ON_MATCH_REDACT = "redact" # scrub the matched value, forward the request
ON_MATCH_SUPERVISE = "supervise" # queue for operator approval, hold the request
OUTBOUND_ON_MATCH_VALUES = (ON_MATCH_BLOCK, ON_MATCH_REDACT, ON_MATCH_SUPERVISE)
# Unset resolves to supervise (fall back to block when supervise is not wired).
DEFAULT_OUTBOUND_ON_MATCH = ON_MATCH_SUPERVISE
@dataclass(frozen=True)
class PathMatch:
@@ -244,6 +230,72 @@ def _parse_match_entry(idx: int, k: int, raw: object) -> MatchEntry:
return MatchEntry(paths=paths, methods=methods, headers=headers)
def _parse_detectors(
idx: int,
host: str,
raw_dict: dict[str, object],
) -> tuple[tuple[str, ...] | None, tuple[str, ...] | None, str]:
"""Parse the optional `dlp` block on a route, returning
(outbound_detectors, inbound_detectors, outbound_on_match)."""
dlp_raw = raw_dict.get("dlp")
if dlp_raw is None:
return None, None, ""
label = f"route[{idx}] ({host})"
if not isinstance(dlp_raw, dict):
raise ValueError(f"{label}: 'dlp' must be an object")
dlp = typing.cast(dict[str, object], dlp_raw)
def _parse_detector_field(
field: str,
valid_names: frozenset[str],
) -> tuple[str, ...] | None:
val = dlp.get(field)
if val is None:
return None
if val is False:
return ()
if not isinstance(val, list):
raise ValueError(
f"{label}: dlp.{field} must be false, a list, or omitted"
)
items = typing.cast(list[object], val)
names: list[str] = []
for j, item in enumerate(items):
if not isinstance(item, str):
raise ValueError(
f"{label}: dlp.{field}[{j}] must be a string"
)
if item not in valid_names:
raise ValueError(
f"{label}: dlp.{field}[{j}] {item!r} is not a valid "
f"detector name; valid names: {', '.join(sorted(valid_names))}"
)
names.append(item)
return tuple(names)
outbound = _parse_detector_field("outbound_detectors", OUTBOUND_DETECTOR_NAMES)
inbound = _parse_detector_field("inbound_detectors", INBOUND_DETECTOR_NAMES)
on_match = ""
on_match_raw = dlp.get("outbound_on_match")
if on_match_raw is not None:
if not isinstance(on_match_raw, str) or on_match_raw not in OUTBOUND_ON_MATCH_VALUES:
raise ValueError(
f"{label}: dlp.outbound_on_match must be one of "
f"{', '.join(OUTBOUND_ON_MATCH_VALUES)} (got {on_match_raw!r})"
)
on_match = on_match_raw
for k in dlp:
if k not in ("outbound_detectors", "inbound_detectors", "outbound_on_match"):
raise ValueError(
f"{label}: dlp has unknown key {k!r}; accepted keys "
f"are 'outbound_detectors', 'inbound_detectors', "
f"'outbound_on_match'"
)
return outbound, inbound, on_match
def parse_routes(payload: object) -> tuple[Route, ...]:
if not isinstance(payload, dict):
raise ValueError("routes payload: top-level must be an object")
@@ -312,7 +364,7 @@ def _parse_one(idx: int, raw: object) -> Route:
)
# dlp detectors
outbound_detectors, inbound_detectors, outbound_on_match = parse_dlp_block(
outbound_detectors, inbound_detectors, outbound_on_match = _parse_detectors(
idx, host, raw_dict,
)
@@ -387,6 +439,15 @@ def route_to_yaml_dict(r: Route) -> dict[str, object]:
return d
def load_routes(text: str) -> tuple[Route, ...]:
"""Parse YAML text → routes."""
try:
payload = parse_yaml_subset(text)
except YamlSubsetError as e:
raise ValueError(f"routes payload: invalid YAML: {e}") from e
return parse_routes(payload)
def parse_config(payload: object) -> "Config":
"""Parse a full egress config payload (top-level log level + routes)."""
if not isinstance(payload, dict):
@@ -785,9 +846,6 @@ __all__ = [
"ON_MATCH_SUPERVISE",
"OUTBOUND_ON_MATCH_VALUES",
"DEFAULT_OUTBOUND_ON_MATCH",
"OUTBOUND_DETECTOR_NAMES",
"INBOUND_DETECTOR_NAMES",
"parse_dlp_block",
"Config",
"Decision",
"HeaderMatch",
@@ -804,6 +862,7 @@ __all__ = [
"is_git_push_request",
"is_git_fetch_request",
"load_config",
"load_routes",
"match_route",
"outbound_scan_headers",
"parse_config",
-92
View File
@@ -1,92 +0,0 @@
"""DLP detector-config parsing for egress routes (PRD 0053, PRD 0062).
A route's optional `dlp:` block names which outbound/inbound detectors run
and what the proxy does when an outbound detector matches a token
(`outbound_on_match`). This module owns parsing and validating that block,
kept apart from the request-time scan/decision flow in `egress_addon_core`
so each half reads top-to-bottom without scrolling past the other.
Stdlib-only; ships flat into the sidecar bundle image alongside
`egress_addon_core.py` see `Dockerfile.sidecars`."""
from __future__ import annotations
import typing
OUTBOUND_DETECTOR_NAMES = frozenset({"token_patterns", "known_secrets", "entropy"})
INBOUND_DETECTOR_NAMES = frozenset({"naive_injection_detection"})
# Per-route policy for what the proxy does when an outbound DLP detector
# matches a token (PRD 0062).
ON_MATCH_BLOCK = "block" # hard 403, never overridable
ON_MATCH_REDACT = "redact" # scrub the matched value, forward the request
ON_MATCH_SUPERVISE = "supervise" # queue for operator approval, hold the request
OUTBOUND_ON_MATCH_VALUES = (ON_MATCH_BLOCK, ON_MATCH_REDACT, ON_MATCH_SUPERVISE)
# Unset resolves to supervise (fall back to block when supervise is not wired).
DEFAULT_OUTBOUND_ON_MATCH = ON_MATCH_SUPERVISE
def parse_dlp_block(
idx: int,
host: str,
raw_dict: dict[str, object],
) -> tuple[tuple[str, ...] | None, tuple[str, ...] | None, str]:
"""Parse the optional `dlp` block on a route, returning
(outbound_detectors, inbound_detectors, outbound_on_match)."""
dlp_raw = raw_dict.get("dlp")
if dlp_raw is None:
return None, None, ""
label = f"route[{idx}] ({host})"
if not isinstance(dlp_raw, dict):
raise ValueError(f"{label}: 'dlp' must be an object")
dlp = typing.cast(dict[str, object], dlp_raw)
def _parse_detector_field(
field: str,
valid_names: frozenset[str],
) -> tuple[str, ...] | None:
val = dlp.get(field)
if val is None:
return None
if val is False:
return ()
if not isinstance(val, list):
raise ValueError(
f"{label}: dlp.{field} must be false, a list, or omitted"
)
items = typing.cast(list[object], val)
names: list[str] = []
for j, item in enumerate(items):
if not isinstance(item, str):
raise ValueError(
f"{label}: dlp.{field}[{j}] must be a string"
)
if item not in valid_names:
raise ValueError(
f"{label}: dlp.{field}[{j}] {item!r} is not a valid "
f"detector name; valid names: {', '.join(sorted(valid_names))}"
)
names.append(item)
return tuple(names)
outbound = _parse_detector_field("outbound_detectors", OUTBOUND_DETECTOR_NAMES)
inbound = _parse_detector_field("inbound_detectors", INBOUND_DETECTOR_NAMES)
on_match = ""
on_match_raw = dlp.get("outbound_on_match")
if on_match_raw is not None:
if not isinstance(on_match_raw, str) or on_match_raw not in OUTBOUND_ON_MATCH_VALUES:
raise ValueError(
f"{label}: dlp.outbound_on_match must be one of "
f"{', '.join(OUTBOUND_ON_MATCH_VALUES)} (got {on_match_raw!r})"
)
on_match = on_match_raw
for k in dlp:
if k not in ("outbound_detectors", "inbound_detectors", "outbound_on_match"):
raise ValueError(
f"{label}: dlp has unknown key {k!r}; accepted keys "
f"are 'outbound_detectors', 'inbound_detectors', "
f"'outbound_on_match'"
)
return outbound, inbound, on_match
+2 -2
View File
@@ -37,8 +37,8 @@ if [ -n "$EGRESS_UPSTREAM_PROXY" ]; then
fi
# Bind address. Docker backend wants `0.0.0.0` (agent dials egress
# directly via the docker network alias). A VM backend uses
# EGRESS_LISTEN_HOST when a non-default binding is needed.
# directly via the docker network alias). Smolmachines backend
# uses EGRESS_LISTEN_HOST when a non-default binding is needed.
LISTEN_HOST_FLAG=""
if [ -n "$EGRESS_LISTEN_HOST" ]; then
LISTEN_HOST_FLAG="--listen-host $EGRESS_LISTEN_HOST"
+2 -4
View File
@@ -35,7 +35,6 @@ import re
import sys
from dataclasses import dataclass, field
from .errors import MissingEnvVarError
from .log import die
from .manifest import Manifest
@@ -137,10 +136,9 @@ def resolve_env(manifest: Manifest) -> ResolvedEnv:
host_var = env_entry_interpolated_from(raw)
host_value = os.environ.get(host_var, "")
if not host_value:
raise MissingEnvVarError(
host_var,
die(
f"env entry {name} is interpolated from ${host_var}, "
f"but ${host_var} is unset or empty in the host environment.",
f"but ${host_var} is unset or empty in the host environment."
)
forwarded[name] = host_value
else: # literal
-18
View File
@@ -1,18 +0,0 @@
"""bot-bottle application-level exception hierarchy.
Exceptions here are caught by the CLI dispatcher (``cli/__init__.py``)
and rendered as ``bot-bottle: error: `` lines same UX as ``die()``,
but without coupling library code to stderr output."""
from __future__ import annotations
class MissingEnvVarError(Exception):
"""Raised when a required host environment variable is unset or empty.
``var_name`` is the exact name of the missing variable so callers
can display it without re-parsing the message string."""
def __init__(self, var_name: str, message: str) -> None:
self.var_name = var_name
super().__init__(message)
+570 -47
View File
@@ -27,36 +27,51 @@ dataclass (`GitGatePlan`). The sidecar's start/stop lifecycle is
backend-specific and lives on concrete subclasses (see
`bot_bottle/backend/docker/git_gate.py`)."""
from __future__ import annotations
import dataclasses
import os
import shlex
from abc import ABC
from dataclasses import dataclass
from pathlib import Path
from .manifest import ManifestBottle
from .log import info
from .manifest import ManifestBottle, ManifestGitEntry
# Short network alias for git-gate inside the sidecar bundle. The
# agent's `.gitconfig` insteadOf rewrites resolve through this name.
GIT_GATE_HOSTNAME = "git-gate"
# Bound half-open git client sessions. If an agent/tool runner is
# interrupted during push, git daemon should reap the receive-pack
# child instead of keeping the gate wedged indefinitely.
GIT_GATE_DAEMON_TIMEOUT_SECS = 15
@dataclass(frozen=True)
class GitGateUpstream:
"""One bare repo on the gate. `name` drives the bare-repo path
(`/git/<name>.git`), the agent's URL after insteadOf rewrite
(`git://<gate>/<name>.git`), and the per-upstream credential
paths inside the gate (`/git-gate/creds/<name>-key` and
`/git-gate/creds/<name>-known_hosts`).
`identity_file` is the host-side absolute path the gate's start
step will docker-cp into the container. `known_host_key` is the
KnownHostKey string from the manifest; the gate's start step
materialises it into a known_hosts file if non-empty.
the gate credential paths inside the running sidecar."""
name: str
upstream_url: str
upstream_host: str
upstream_port: str
identity_file: str
known_host_key: str
known_hosts_file: Path = Path()
# Rendering and the deploy-key lifecycle live in sibling modules; the
# names are re-exported here (see __all__) so existing
# `from bot_bottle.git_gate import …` callers are unchanged.
from .git_gate_render import (
GIT_GATE_HOSTNAME,
GIT_GATE_TIMEOUT_SECS,
GitGateUpstream,
git_gate_known_hosts_line,
git_gate_render_access_hook,
git_gate_render_entrypoint,
git_gate_render_gitconfig,
git_gate_render_hook,
git_gate_upstreams_for_bottle,
_gitconfig_validate_value,
)
from .git_gate_provision import (
provision_git_gate_dynamic_keys,
revoke_git_gate_provisioned_keys,
_provision_dynamic_key,
_resolve_identity_file,
)
@dataclass(frozen=True)
class GitGatePlan:
@@ -81,6 +96,528 @@ class GitGatePlan:
egress_network: str = ""
def git_gate_upstreams_for_bottle(bottle: ManifestBottle) -> tuple[GitGateUpstream, ...]:
"""Lift each `bottle.git` entry into a GitGateUpstream. Unique-Name
validation already ran in `manifest.ManifestBottle.from_dict`."""
return tuple(
GitGateUpstream(
name=e.Name,
upstream_url=e.Upstream,
upstream_host=e.UpstreamHost,
upstream_port=e.UpstreamPort,
identity_file=e.IdentityFile,
known_host_key=e.KnownHostKey,
)
for e in bottle.git
)
def git_gate_render_gitconfig(
entries: tuple[ManifestGitEntry, ...], gate_host: str, *, scheme: str = "git",
) -> str:
"""Render the agent's ~/.gitconfig content for git-gate
`insteadOf` rewrites. Pure host-side, no docker / smolvm;
exposed for tests + reuse across backends.
`gate_host` is the part of the URL between `<scheme>://` and the
repo path backends differ here:
- docker: `git-gate` (the short network alias)
- smolmachines: `<bundle_ip>:<port>` (no DNS in the
TSI-allowlisted guest)
Empty `entries` returns an empty string so callers can no-op
cleanly without conditional formatting at the call site."""
if not entries:
return ""
out = [
"# bot-bottle git-gate (PRD 0008): every git operation against\n",
"# a declared upstream routes through the gate, which mirrors\n",
"# the upstream bidirectionally (gitleaks-scanned push;\n",
"# fetch-from-upstream-before-every-upload-pack via access-hook).\n",
]
for entry in entries:
out.append(f'[url "{scheme}://{gate_host}/{entry.Name}.git"]\n')
out.append(f"\tinsteadOf = {entry.Upstream}\n")
if entry.RemoteKey and entry.RemoteKey != entry.UpstreamHost:
port = (
f":{entry.UpstreamPort}"
if entry.UpstreamPort and entry.UpstreamPort != "22"
else ""
)
alias = (
f"ssh://{entry.UpstreamUser}@{entry.RemoteKey}{port}/"
f"{entry.UpstreamPath}"
)
out.append(f"\tinsteadOf = {alias}\n")
return "".join(out)
def git_gate_known_hosts_line(host: str, port: str, key: str) -> str:
"""Format `host[:port] key` for OpenSSH's known_hosts. Non-default
ports use the bracketed `[host]:port` form (the form OpenSSH writes
on disk for hosts reached via a non-22 port)."""
if port and port != "22":
target = f"[{host}]:{port}"
else:
target = host
return f"{target} {key}\n"
def git_gate_render_entrypoint(upstreams: tuple[GitGateUpstream, ...]) -> str:
"""Posix-sh entrypoint. One `init_repo` call per upstream, then
`exec git daemon`. The function reads
`/git-gate/creds/<name>-{key,known_hosts}` (bind-mounted into
the bundle by the renderer) and wires them into each bare repo's
config; the access-hook + pre-receive hook pick those paths up
at fetch / push time."""
lines = [
"#!/bin/sh",
"set -eu",
"",
"init_repo() {",
" name=$1",
" upstream_url=$2",
" keyfile=/git-gate/creds/${name}-key",
" hostsfile=/git-gate/creds/${name}-known_hosts",
"",
# `|| true`: PRD 0018 chunk 3+ bind-mounts these RO from the
# host, so chmod-syscalls fail with EROFS. The files already
# have the right perms on the host (SSH requires 0600 to load
# the key in the first place), so the chmod is best-effort
# cleanup for the legacy docker-cp path where the file
# landed at the host's umask perms.
" chmod 600 \"$keyfile\" 2>/dev/null || true",
" if [ -f \"$hostsfile\" ]; then",
" chmod 600 \"$hostsfile\" 2>/dev/null || true",
" fi",
"",
" repo=/git/${name}.git",
" if [ ! -d \"$repo\" ]; then",
" git init --bare \"$repo\" >/dev/null",
# --mirror=fetch sets remote.origin.fetch = +refs/*:refs/* so",
# a later `git fetch origin` mirrors the upstream's full ref",
# graph (heads, tags, notes) into the bare repo at canonical",
# paths. It does NOT set remote.origin.mirror=true, so an",
# explicit `git push origin <ref>:<ref>` still pushes one ref.",
" git -C \"$repo\" remote add --mirror=fetch origin \"$upstream_url\"",
" fi",
" git -C \"$repo\" config git-gate.identityFile \"$keyfile\"",
" git -C \"$repo\" config git-gate.knownHosts \"$hostsfile\"",
" git -C \"$repo\" config receive.denyCurrentBranch ignore",
" git -C \"$repo\" config receive.advertisePushOptions true",
" git -C \"$repo\" config http.receivepack true",
" install -m 755 /etc/git-gate/pre-receive \"$repo/hooks/pre-receive\"",
"}",
"",
"mkdir -p /git",
]
for u in upstreams:
lines.append(f"init_repo {shlex.quote(u.name)} {shlex.quote(u.upstream_url)}")
lines.extend([
"",
"exec git daemon \\",
" --reuseaddr \\",
f" --timeout={GIT_GATE_DAEMON_TIMEOUT_SECS} \\",
f" --init-timeout={GIT_GATE_DAEMON_TIMEOUT_SECS} \\",
" --base-path=/git \\",
" --export-all \\",
" --enable=receive-pack \\",
" --access-hook=/etc/git-gate/access-hook \\",
" --verbose",
])
return "\n".join(lines) + "\n"
def git_gate_render_hook() -> str:
"""The shared pre-receive hook: gitleaks-scan all incoming refs,
then forward each accepted ref to the real upstream (`origin`)
using the per-repo credential. Failure in either phase aborts
the push so the agent sees a real rejection. POSIX sh.
Two phases (scan all, then push all) keeps a hit on ref N from
half-pushing refs 1..N-1; both phases re-read stdin from a temp
file because pre-receive's stdin is a one-shot stream."""
return r"""#!/bin/sh
# git-gate pre-receive (PRD 0008). Stdin: <old> <new> <ref> per line.
set -u
refs_file=$(mktemp)
trap 'rm -f "$refs_file"' EXIT
cat > "$refs_file"
zero=0000000000000000000000000000000000000000
supervise_gitleaks_allow() {
log_opts=$1
ref=$2
report_file=$(mktemp)
if ! gitleaks git \
--log-opts="$log_opts" \
--no-banner \
--redact \
--ignore-gitleaks-allow \
--report-format=json \
--report-path="$report_file" \
--exit-code 0 \
1>&2; then
rm -f "$report_file"
echo "git-gate: gitleaks inline-suppression scan failed for $ref" >&2
return 1
fi
proposal_id=$(
GITLEAKS_ALLOW_REF="$ref" python3 - "$report_file" <<'PY'
import datetime
import hashlib
import json
import os
import sys
import uuid
from pathlib import Path
report_path = Path(sys.argv[1])
queue_dir = os.environ.get("SUPERVISE_QUEUE_DIR", "")
slug = os.environ.get("SUPERVISE_BOTTLE_SLUG", "")
if not queue_dir or not slug:
sys.exit(2)
try:
raw = json.loads(report_path.read_text() or "[]")
except json.JSONDecodeError:
sys.exit(3)
if not isinstance(raw, list):
sys.exit(3)
if not raw:
sys.exit(0)
ref = os.environ.get("GITLEAKS_ALLOW_REF", "")
lines = [
"gitleaks inline suppression requires supervisor approval",
f"ref: {ref}",
"",
]
for i, finding in enumerate(raw, 1):
if not isinstance(finding, dict):
continue
file_path = finding.get("File", "")
line_no = finding.get("StartLine", finding.get("Line", ""))
rule_id = finding.get("RuleID", "")
commit = finding.get("Commit", "")
line = finding.get("Line", "")
lines.extend([
f"finding {i}:",
f" file: {file_path}",
f" line: {line_no}",
f" rule: {rule_id}",
f" commit: {commit}",
f" code: {line}",
"",
])
payload = "\n".join(lines).rstrip() + "\n"
proposal_id = str(uuid.uuid4())
proposal = {
"id": proposal_id,
"bottle_slug": slug,
"tool": "gitleaks-allow",
"proposed_file": payload,
"justification": (
"git-gate found gitleaks findings hidden by # gitleaks:allow; "
"approve only for dummy test fixtures or confirmed false positives"
),
"arrival_timestamp": datetime.datetime.now(
datetime.timezone.utc
).isoformat(),
"current_file_hash": hashlib.sha256(payload.encode("utf-8")).hexdigest(),
}
queue = Path(queue_dir)
queue.mkdir(parents=True, exist_ok=True)
path = queue / f"{proposal_id}.proposal.json"
tmp = path.with_suffix(path.suffix + ".tmp")
with tmp.open("w", encoding="utf-8") as f:
json.dump(proposal, f, indent=2)
f.write("\n")
os.chmod(tmp, 0o600)
os.replace(tmp, path)
print(proposal_id)
PY
)
rc=$?
rm -f "$report_file"
if [ "$rc" -eq 0 ] && [ -z "$proposal_id" ]; then
return 0
fi
if [ "$rc" -ne 0 ]; then
echo "git-gate: cannot route # gitleaks:allow finding to supervisor; refusing push" >&2
return 1
fi
queue_dir=${SUPERVISE_QUEUE_DIR:-}
response_file="$queue_dir/${proposal_id}.response.json"
timeout=${SUPERVISE_GITLEAKS_ALLOW_TIMEOUT_SECONDS:-300}
case "$timeout" in
''|*[!0-9]*)
echo "git-gate: invalid SUPERVISE_GITLEAKS_ALLOW_TIMEOUT_SECONDS=$timeout" >&2
return 1
;;
esac
echo "git-gate: queued # gitleaks:allow supervisor approval $proposal_id" >&2
echo "git-gate: approve with './cli.py supervise' to continue this push" >&2
waited=0
while [ "$waited" -lt "$timeout" ]; do
if [ -f "$response_file" ]; then
status=$(python3 - "$response_file" <<'PY'
import json
import sys
try:
with open(sys.argv[1], encoding="utf-8") as f:
raw = json.load(f)
except (OSError, json.JSONDecodeError):
sys.exit(1)
status = raw.get("status")
if not isinstance(status, str):
sys.exit(1)
print(status)
PY
) || status=""
case "$status" in
approved|modified)
mkdir -p "$queue_dir/processed"
mv -f "$queue_dir/${proposal_id}.proposal.json" "$queue_dir/processed/" 2>/dev/null || true
mv -f "$queue_dir/${proposal_id}.response.json" "$queue_dir/processed/" 2>/dev/null || true
echo "git-gate: supervisor approved # gitleaks:allow for $ref" >&2
return 0
;;
rejected)
echo "git-gate: supervisor rejected # gitleaks:allow for $ref" >&2
return 1
;;
*)
echo "git-gate: invalid supervisor response for # gitleaks:allow" >&2
return 1
;;
esac
fi
sleep 1
waited=$((waited + 1))
done
echo "git-gate: supervisor approval timed out for # gitleaks:allow; refusing push" >&2
return 1
}
# Phase 1: gitleaks scan each ref's incoming commits.
while IFS=' ' read -r old new ref; do
[ -z "$ref" ] && continue
[ "$new" = "$zero" ] && continue
if [ "$old" = "$zero" ]; then
# New ref: scan only the commits this push introduces — those
# reachable from $new but not from any ref the gate already has.
# Everything already on the gate arrived via upstream mirror-fetch
# or a previously gitleaks-scanned push, so it's already-upstream
# or already-scanned; re-scanning it (the old `$new` full-ancestry
# range) only resurfaces historical findings and blocks every new
# branch. See PRD 0028 / issue #106.
log_opts="$new --not --all"
else
log_opts="$old..$new"
fi
echo "git-gate: gitleaks scanning $ref ($log_opts)" >&2
if ! gitleaks git --log-opts="$log_opts" --no-banner --redact 1>&2; then
echo "git-gate: gitleaks rejected push to $ref" >&2
exit 1
fi
if ! supervise_gitleaks_allow "$log_opts" "$ref"; then
exit 1
fi
done < "$refs_file"
# Phase 2: forward each ref to the upstream (`origin`, configured
# in the entrypoint via `git remote add --mirror=fetch`).
keyfile=$(git config --get git-gate.identityFile)
hostsfile=$(git config --get git-gate.knownHosts)
if [ ! -f "$hostsfile" ]; then
echo "git-gate: no KnownHostKey configured for this upstream; refusing to push" >&2
echo "git-gate: add KnownHostKey to the bottle.git entry and restart the bottle" >&2
exit 1
fi
ssh_cmd="ssh -i $keyfile -o UserKnownHostsFile=$hostsfile -o StrictHostKeyChecking=yes -o IdentitiesOnly=yes -o BatchMode=yes -o ConnectTimeout=10"
push_option_count=${GIT_PUSH_OPTION_COUNT:-0}
case "$push_option_count" in
''|*[!0-9]*)
echo "git-gate: invalid GIT_PUSH_OPTION_COUNT=$push_option_count" >&2
exit 1
;;
esac
set --
i=0
while [ "$i" -lt "$push_option_count" ]; do
opt=$(printenv "GIT_PUSH_OPTION_$i" || :)
set -- "$@" --push-option="$opt"
i=$((i + 1))
done
while IFS=' ' read -r old new ref; do
[ -z "$ref" ] && continue
if [ "$new" = "$zero" ]; then
refspec=":$ref"
elif [ "$old" != "$zero" ] && ! git merge-base --is-ancestor "$old" "$new" 2>/dev/null; then
refspec="+$new:$ref"
else
refspec="$new:$ref"
fi
echo "git-gate: forwarding $ref to origin" >&2
if ! GIT_SSH_COMMAND="$ssh_cmd" git push "$@" origin "$refspec" 1>&2; then
echo "git-gate: upstream push failed for $ref" >&2
exit 1
fi
done < "$refs_file"
exit 0
"""
def git_gate_render_access_hook() -> str:
"""`git daemon --access-hook` script. Runs before each protocol
service; for `upload-pack` (fetch / clone / ls-remote / pull) it
refreshes the bare repo from upstream first, so the response
reflects upstream's current state. For other services (notably
`receive-pack`) it returns 0 immediately and lets the existing
pre-receive hook gate the operation. POSIX sh.
The hook receives:
$1 service name (`upload-pack`, `receive-pack`, ...)
$2 absolute path to the resolved repo
$3 client hostname (unused)
$4 client tcp address (unused)
Fail-closed on upstream errors: the agent's fetch fails too,
so it never silently sees stale data matches the PRD's
'equivalent to operations against the upstream' contract."""
return r"""#!/bin/sh
# git-gate access-hook (PRD 0008). $1=service $2=repo $3=host $4=peer
set -u
service=$1
repo_dir=$2
# Push path keeps its own gating in pre-receive (gitleaks +
# forward). Only refresh-from-upstream on fetch operations.
if [ "$service" != "upload-pack" ]; then
exit 0
fi
keyfile=$(git -C "$repo_dir" config --get git-gate.identityFile 2>/dev/null || true)
hostsfile=$(git -C "$repo_dir" config --get git-gate.knownHosts 2>/dev/null || true)
if [ -z "$keyfile" ] || [ ! -f "$hostsfile" ]; then
echo "git-gate: missing credentials for $repo_dir; refusing fetch" >&2
exit 1
fi
ssh_cmd="ssh -i $keyfile -o UserKnownHostsFile=$hostsfile -o StrictHostKeyChecking=yes -o IdentitiesOnly=yes -o BatchMode=yes -o ConnectTimeout=10"
echo "git-gate: refreshing $repo_dir from upstream" >&2
if ! GIT_SSH_COMMAND="$ssh_cmd" git -C "$repo_dir" fetch origin --prune >&2; then
echo "git-gate: upstream fetch failed for $repo_dir; refusing to serve stale data" >&2
exit 1
fi
# Sync the bare repo's HEAD to upstream's HEAD on the first fetch
# (when it still points at the `git init --bare` default of
# refs/heads/master and upstream uses something else, the cloned
# checkout would fail with "remote HEAD refers to nonexistent ref").
# Costs one extra ls-remote on first fetch only; subsequent fetches
# skip the branch. If upstream's default branch changes after the
# gate has cached it, restart the bottle to resync.
if ! git -C "$repo_dir" rev-parse --verify HEAD >/dev/null 2>&1; then
upstream_head=$(GIT_SSH_COMMAND="$ssh_cmd" git -C "$repo_dir" \
ls-remote --symref origin HEAD 2>/dev/null \
| awk '/^ref:/ {print $2; exit}')
if [ -n "$upstream_head" ]; then
git -C "$repo_dir" symbolic-ref HEAD "$upstream_head" || true
fi
fi
exit 0
"""
def _provision_dynamic_key(
entry: ManifestGitEntry,
slug: str,
stage_dir: Path,
) -> str:
"""Generate a fresh ed25519 keypair, register the public half with
the forge, and persist the private key + key ID under `stage_dir`.
Returns the host-side path to the private key file so the caller
can inject it into the GitGateUpstream as `identity_file`."""
from .deploy_key_provisioner import get_provisioner
pk = entry.Key
token = os.environ.get(pk.forge_token_env)
if token is None:
raise RuntimeError(
f"git-gate.repos[{entry.Name!r}] key.forge_token_env"
f" = {pk.forge_token_env!r}: env var is not set"
)
api_url = pk.api_url or f"https://{entry.UpstreamHost}"
provisioner = get_provisioner(pk.provider, token, api_url)
owner_repo = entry.UpstreamPath
if owner_repo.endswith(".git"):
owner_repo = owner_repo[:-4]
title = f"bot-bottle:{slug}:{entry.Name}"
info(f"provisioning deploy key for git-gate.repos[{entry.Name!r}]")
key_id, private_key_bytes = provisioner.create(owner_repo, title)
key_file = stage_dir / f"{entry.Name}-key"
key_file.write_bytes(private_key_bytes)
key_file.chmod(0o600)
id_file = stage_dir / f"{entry.Name}-deploy-key-id"
id_file.write_text(key_id)
id_file.chmod(0o600)
info(f"provisioned deploy key {key_id} for git-gate.repos[{entry.Name!r}]")
return str(key_file)
def revoke_git_gate_provisioned_keys(bottle: ManifestBottle, stage_dir: Path) -> None:
"""Revoke all deploy keys provisioned for `bottle` during prepare.
Called at teardown after containers stop. Raises if any revocation
fails a stranded key is a security concern that the operator must
address manually."""
from .deploy_key_provisioner import get_provisioner
for entry in bottle.git:
if entry.Key.provider != "gitea":
continue
pk = entry.Key
id_file = stage_dir / f"{entry.Name}-deploy-key-id"
if not id_file.exists():
continue
key_id = id_file.read_text().strip()
token = os.environ.get(pk.forge_token_env)
if token is None:
raise RuntimeError(
f"git-gate.repos[{entry.Name!r}] key.forge_token_env"
f" = {pk.forge_token_env!r}: env var is not set;"
f" cannot revoke deploy key {key_id}"
)
api_url = pk.api_url or f"https://{entry.UpstreamHost}"
provisioner = get_provisioner(pk.provider, token, api_url)
owner_repo = entry.UpstreamPath
if owner_repo.endswith(".git"):
owner_repo = owner_repo[:-4]
info(f"revoking deploy key {key_id} for git-gate.repos[{entry.Name!r}]")
provisioner.delete(owner_repo, key_id)
info(f"revoked deploy key {key_id} for git-gate.repos[{entry.Name!r}]")
def _resolve_identity_file(entry: ManifestGitEntry, slug: str, stage_dir: Path) -> str:
"""Return the host-side SSH identity file path for this entry.
For gitea entries, provisions a fresh deploy key first."""
if entry.Key.provider == "gitea":
return _provision_dynamic_key(entry, slug, stage_dir)
return entry.IdentityFile
class GitGate(ABC):
@@ -94,14 +631,20 @@ class GitGate(ABC):
entrypoint, pre-receive hook, and access-hook scripts (mode
600) under `stage_dir`. Pure host-side, no docker subprocess.
For `gitea` key entries, the returned upstream intentionally
has an empty identity file. Backend launch fills that in after
the operator confirms the preflight.
For `gitea` key entries, also generates and registers
a fresh deploy key via the forge API and writes the private key
+ key ID to `stage_dir`.
Returned plan is incomplete: the launch step must fill
`internal_network` / `egress_network` via `dataclasses.replace`
before passing the plan to `.start`."""
upstreams = git_gate_upstreams_for_bottle(bottle)
upstreams_list = list(git_gate_upstreams_for_bottle(bottle))
for i, entry in enumerate(bottle.git):
upstreams_list[i] = dataclasses.replace(
upstreams_list[i],
identity_file=_resolve_identity_file(entry, slug, stage_dir),
)
upstreams = tuple(upstreams_list)
entrypoint = stage_dir / "git_gate_entrypoint.sh"
entrypoint.write_text(git_gate_render_entrypoint(upstreams))
entrypoint.chmod(0o600)
@@ -143,23 +686,3 @@ class GitGate(ABC):
access_hook_script=access_hook,
upstreams=tuple(upstreams_with_files),
)
__all__ = [
"GIT_GATE_HOSTNAME",
"GIT_GATE_TIMEOUT_SECS",
"GitGateUpstream",
"GitGatePlan",
"GitGate",
"git_gate_upstreams_for_bottle",
"git_gate_render_gitconfig",
"git_gate_known_hosts_line",
"git_gate_render_entrypoint",
"git_gate_render_hook",
"git_gate_render_access_hook",
"provision_git_gate_dynamic_keys",
"revoke_git_gate_provisioned_keys",
"_gitconfig_validate_value",
"_provision_dynamic_key",
"_resolve_identity_file",
]
-268
View File
@@ -1,268 +0,0 @@
"""Preflight host-key population for git-gate upstreams (issue #333).
When a git-gate repo entry lacks a `host_key`, this module either:
- headless: dies with a clear config error.
- interactive: fetches the key via ssh-keyscan, prompts the operator to
confirm, and optionally persists it to the bottle config file on disk.
Public entry point: `preflight_host_keys(manifest, headless=..., home_md=...)`.
"""
from __future__ import annotations
import dataclasses
import subprocess
import sys
from pathlib import Path
from typing import cast
from .log import die, info
from .manifest import Manifest
from .yaml_subset import YamlSubsetError, parse_frontmatter, serialize_yaml_subset
# Preferred key types, most secure first.
_KEY_TYPE_PREFERENCE = (
"ssh-ed25519",
"ecdsa-sha2-nistp256",
"ecdsa-sha2-nistp384",
"ecdsa-sha2-nistp521",
"ssh-rsa",
)
def fetch_host_key(host: str, port: str) -> str:
"""Return an SSH public key for `host`:`port` via ssh-keyscan.
Returns the key in `<type> <base64-data>` format (the host prefix is
stripped so the result can be stored in `host_key` and later formatted
into a known_hosts line by `git_gate_known_hosts_line`).
Prefers ed25519 > ecdsa > rsa; falls back to the first key type
returned if none of the preferred types are present.
Raises `RuntimeError` on subprocess failure, timeout, or no result.
Uses only the Python stdlib (subprocess)."""
args = ["ssh-keyscan"]
if port and port != "22":
args += ["-p", port]
args.append(host)
try:
result = subprocess.run(
args, capture_output=True, text=True, timeout=15, check=False,
)
except OSError as e:
raise RuntimeError(
f"ssh-keyscan: could not launch for {host}:{port}: {e}"
) from e
except subprocess.TimeoutExpired as e:
raise RuntimeError(f"ssh-keyscan timed out for {host}:{port}") from e
# known_hosts format: "[host]:port type data" or "host type data"
# Strip the host/port prefix; collect "type -> type data" by type.
found: dict[str, str] = {}
for line in result.stdout.splitlines():
line = line.strip()
if not line or line.startswith("#"):
continue
parts = line.split(None, 2)
if len(parts) == 3 and parts[1] not in found:
found[parts[1]] = f"{parts[1]} {parts[2]}"
for preferred in _KEY_TYPE_PREFERENCE:
if preferred in found:
return found[preferred]
if found:
return next(iter(found.values()))
raise RuntimeError(
f"ssh-keyscan returned no host key for {host}:{port}."
+ (f" stderr: {result.stderr.strip()!r}" if result.stderr.strip() else "")
)
# ---------------------------------------------------------------------------
# Frontmatter editing
# ---------------------------------------------------------------------------
def add_host_key_to_frontmatter(file_text: str, repo_name: str, host_key: str) -> str:
"""Return an updated copy of `file_text` with `host_key` set on the
named repo entry in the YAML frontmatter.
Parses the frontmatter into a dict, sets the key, and re-serializes.
Returns the original text unchanged when: the file has no frontmatter,
the git-gate.repos.<repo_name> entry is absent or already has a
host_key, or the frontmatter cannot be parsed."""
try:
fm, body = parse_frontmatter(file_text)
except YamlSubsetError:
return file_text
git_gate = fm.get("git-gate")
if not isinstance(git_gate, dict):
return file_text
repos = git_gate.get("repos")
if not isinstance(repos, dict):
return file_text
repo = repos.get(repo_name)
if not isinstance(repo, dict):
return file_text
if repo.get("host_key"):
return file_text
cast(dict[str, object], repo)["host_key"] = host_key
return f"---\n{serialize_yaml_subset(fm)}---\n{body}"
def find_repo_bottle_file(bottles_dir: Path, repo_name: str) -> Path | None:
"""Return the first `bottles_dir/*.md` that declares `repo_name` without
a `host_key`, without modifying anything. Returns None if not found."""
if not bottles_dir.is_dir():
return None
for path in sorted(bottles_dir.glob("*.md")):
try:
text = path.read_text(encoding="utf-8")
fm, _ = parse_frontmatter(text)
except (OSError, UnicodeDecodeError, YamlSubsetError):
continue
git_gate = fm.get("git-gate")
if not isinstance(git_gate, dict):
continue
repos = git_gate.get("repos")
if not isinstance(repos, dict):
continue
repo = repos.get(repo_name)
if not isinstance(repo, dict) or repo.get("host_key"):
continue
return path
return None
def find_and_update_bottle_file(
bottles_dir: Path, repo_name: str, host_key: str,
) -> bool:
"""Write `host_key` into the bottle file returned by `find_repo_bottle_file`.
Returns True on success, False when no suitable file is found or the
write fails."""
path = find_repo_bottle_file(bottles_dir, repo_name)
if path is None:
return False
try:
text = path.read_text(encoding="utf-8")
except (OSError, UnicodeDecodeError):
return False
updated = add_host_key_to_frontmatter(text, repo_name, host_key)
if updated == text:
return False
path.write_text(updated, encoding="utf-8")
info(f"wrote host_key for {repo_name!r} to {path}")
return True
# ---------------------------------------------------------------------------
# Interactive prompt helper
# ---------------------------------------------------------------------------
def prompt_tty(message: str) -> str:
"""Write `message` to stderr and read a line from /dev/tty (or stdin)."""
sys.stderr.write(message)
sys.stderr.flush()
try:
with open("/dev/tty", "r", encoding="utf-8") as tty:
return tty.readline().rstrip("\n")
except OSError:
return sys.stdin.readline().rstrip("\n")
# ---------------------------------------------------------------------------
# Public API
# ---------------------------------------------------------------------------
def preflight_host_keys(
manifest: Manifest,
*,
headless: bool,
home_md: Path | None,
) -> Manifest:
"""Ensure every git-gate repo entry has a `host_key` configured.
For entries whose `KnownHostKey` is empty:
- headless: calls `die()` with a clear message naming the repos.
- interactive: fetches the key via ssh-keyscan, shows it to the
operator, and requests confirmation. If accepted, optionally
persists it to the bottle config file on disk; the key is always
applied in memory for this launch regardless of the persistence
choice. Aborted confirmation calls `die()`.
Returns a (possibly updated) Manifest. If all entries already have
host keys the original manifest is returned unchanged."""
bottle = manifest.bottle
missing = [e for e in bottle.git if not e.KnownHostKey]
if not missing:
return manifest
if headless:
names = ", ".join(repr(e.Name) for e in missing)
die(
f"git-gate: no host_key configured for repo(s) {names}. "
f"Add host_key to each bottle git-gate.repos entry, or run "
f"interactively once to have it fetched and saved automatically."
)
bottles_dir = (home_md / "bottles") if home_md is not None else None
updated_entries = list(bottle.git)
for entry in missing:
host = entry.UpstreamHost
port = entry.UpstreamPort
label = f"git-gate.repos[{entry.Name!r}]"
info(f"{label}: no host_key configured; fetching from {host}:{port}")
try:
key = fetch_host_key(host, port)
except RuntimeError as e:
die(f"git-gate: {label}: {e}")
sys.stderr.write(f"\ngit-gate: host key for {label}:\n {key}\n\n")
confirm = prompt_tty("Is this host key correct? [y/N] ")
if confirm.strip().lower() not in ("y", "yes"):
die(f"git-gate: {label}: host key not confirmed; aborting launch")
if bottles_dir is not None:
target_file = find_repo_bottle_file(bottles_dir, entry.Name)
if target_file is not None:
save = prompt_tty(
f"Save host_key for {entry.Name!r} to {target_file}? [y/N] "
)
if save.strip().lower() in ("y", "yes"):
ok = find_and_update_bottle_file(bottles_dir, entry.Name, key)
if not ok:
sys.stderr.write(
f"git-gate: {label}: could not write to {target_file}; "
f"host_key kept in memory for this session only\n"
)
else:
sys.stderr.write(
f"git-gate: {label}: no bottle config file found for "
f"{entry.Name!r}; host_key kept in memory for this session only\n"
)
idx = next(i for i, e in enumerate(updated_entries) if e.Name == entry.Name)
updated_entries[idx] = dataclasses.replace(entry, KnownHostKey=key)
updated_bottle = dataclasses.replace(bottle, git=tuple(updated_entries))
return dataclasses.replace(manifest, bottle=updated_bottle)
__all__ = [
"fetch_host_key",
"preflight_host_keys",
"add_host_key_to_frontmatter",
"find_repo_bottle_file",
"find_and_update_bottle_file",
"prompt_tty",
]
-148
View File
@@ -1,148 +0,0 @@
"""git-gate deploy-key lifecycle for `gitea` upstreams (PRD 0047/0048).
Provisions a fresh ed25519 deploy key via the forge API at prepare time
and revokes it at teardown, so the agent never holds an upstream
credential. Split out of `git_gate.py`; the forge HTTP client is lazily
imported (`deploy_key_provisioner`) to keep its cost off the host path.
`git_gate` re-exports these names for API stability."""
from __future__ import annotations
import os
import dataclasses
from pathlib import Path
from typing import TYPE_CHECKING
from .errors import MissingEnvVarError
from .log import info
from .manifest import ManifestBottle, ManifestGitEntry
from .git_gate_render import GitGateUpstream
if TYPE_CHECKING:
from .git_gate import GitGatePlan
def _provision_dynamic_key(
entry: ManifestGitEntry,
slug: str,
stage_dir: Path,
) -> str:
"""Generate a fresh ed25519 keypair, register the public half with
the forge, and persist the private key + key ID under `stage_dir`.
Returns the host-side path to the private key file so the caller
can inject it into the GitGateUpstream as `identity_file`."""
from .deploy_key_provisioner import get_provisioner
pk = entry.Key
token = os.environ.get(pk.forge_token_env)
if token is None:
raise MissingEnvVarError(
pk.forge_token_env,
f"git-gate.repos[{entry.Name!r}] key.forge_token_env"
f" = {pk.forge_token_env!r}: env var is not set",
)
api_url = pk.api_url or f"https://{entry.UpstreamHost}"
provisioner = get_provisioner(pk.provider, token, api_url)
owner_repo = entry.UpstreamPath
if owner_repo.endswith(".git"):
owner_repo = owner_repo[:-4]
title = f"bot-bottle:{slug}:{entry.Name}"
info(f"provisioning deploy key for git-gate.repos[{entry.Name!r}]")
key_id, private_key_bytes = provisioner.create(owner_repo, title)
key_file = stage_dir / f"{entry.Name}-key"
key_file.write_bytes(private_key_bytes)
key_file.chmod(0o600)
id_file = stage_dir / f"{entry.Name}-deploy-key-id"
id_file.write_text(key_id)
id_file.chmod(0o600)
info(f"provisioned deploy key {key_id} for git-gate.repos[{entry.Name!r}]")
return str(key_file)
def revoke_git_gate_provisioned_keys(bottle: ManifestBottle, stage_dir: Path) -> None:
"""Revoke all deploy keys provisioned for `bottle` during prepare.
Called at teardown after containers stop. Raises if any revocation
fails a stranded key is a security concern that the operator must
address manually."""
from .deploy_key_provisioner import get_provisioner
for entry in bottle.git:
if entry.Key.provider != "gitea":
continue
pk = entry.Key
id_file = stage_dir / f"{entry.Name}-deploy-key-id"
if not id_file.exists():
continue
key_id = id_file.read_text().strip()
token = os.environ.get(pk.forge_token_env)
if token is None:
raise MissingEnvVarError(
pk.forge_token_env,
f"git-gate.repos[{entry.Name!r}] key.forge_token_env"
f" = {pk.forge_token_env!r}: env var is not set;"
f" cannot revoke deploy key {key_id}",
)
api_url = pk.api_url or f"https://{entry.UpstreamHost}"
provisioner = get_provisioner(pk.provider, token, api_url)
owner_repo = entry.UpstreamPath
if owner_repo.endswith(".git"):
owner_repo = owner_repo[:-4]
info(f"revoking deploy key {key_id} for git-gate.repos[{entry.Name!r}]")
provisioner.delete(owner_repo, key_id)
info(f"revoked deploy key {key_id} for git-gate.repos[{entry.Name!r}]")
def _resolve_identity_file(entry: ManifestGitEntry, slug: str, stage_dir: Path) -> str:
"""Return the host-side SSH identity file path for this entry.
For gitea entries, provisions a fresh deploy key first."""
if entry.Key.provider == "gitea":
return _provision_dynamic_key(entry, slug, stage_dir)
return entry.IdentityFile
def provision_git_gate_dynamic_keys(
bottle: ManifestBottle,
plan: "GitGatePlan",
stage_dir: Path,
) -> "GitGatePlan":
"""Provision dynamic git-gate keys and return an updated plan.
This runs during backend launch, after the operator confirms the
preflight. Plan preparation intentionally stays side-effect-light:
dry-runs and aborted launches must not create remote deploy keys.
"""
if not plan.upstreams:
return plan
upstreams_by_name: dict[str, GitGateUpstream] = {
upstream.name: upstream for upstream in plan.upstreams
}
updated: list[GitGateUpstream] = []
for entry in bottle.git:
upstream = upstreams_by_name.get(entry.Name)
if upstream is None:
continue
if entry.Key.provider == "gitea":
identity_file = _provision_dynamic_key(entry, plan.slug, stage_dir)
upstream = dataclasses.replace(upstream, identity_file=identity_file)
updated.append(upstream)
if len(updated) != len(plan.upstreams):
updated_names = {u.name for u in updated}
for upstream in plan.upstreams:
if upstream.name not in updated_names:
updated.append(upstream)
return dataclasses.replace(plan, upstreams=tuple(updated))
__all__ = [
"revoke_git_gate_provisioned_keys",
"provision_git_gate_dynamic_keys",
"_provision_dynamic_key",
"_resolve_identity_file",
]
-506
View File
@@ -1,506 +0,0 @@
"""Pure host-side rendering for the per-agent git-gate (PRD 0008).
Builds the agent's `.gitconfig` insteadOf rewrites, the known_hosts
line, and the entrypoint / pre-receive / access-hook scripts the sidecar
runs. No docker or forge calls exposed for tests and reuse across
backends. Split out of `git_gate.py` so the control surface (`GitGate`)
and the deploy-key lifecycle (`git_gate_provision`) each read on their
own; `git_gate` re-exports these names for API stability."""
from __future__ import annotations
import shlex
from dataclasses import dataclass
from pathlib import Path
from .manifest import ManifestBottle, ManifestGitEntry
# Short network alias for git-gate inside the sidecar bundle. The
# agent's `.gitconfig` insteadOf rewrites resolve through this name.
GIT_GATE_HOSTNAME = "git-gate"
# Shared timeout (seconds) for all git-gate subprocess and CGI calls:
# git daemon (--timeout/--init-timeout), the access-hook subprocess in
# git_http_backend, and the git http-backend CGI subprocess.
GIT_GATE_TIMEOUT_SECS = 15
@dataclass(frozen=True)
class GitGateUpstream:
"""One bare repo on the gate. `name` drives the bare-repo path
(`/git/<name>.git`), the agent's URL after insteadOf rewrite
(`git://<gate>/<name>.git`), and the per-upstream credential
paths inside the gate (`/git-gate/creds/<name>-key` and
`/git-gate/creds/<name>-known_hosts`).
`identity_file` is the host-side absolute path the gate's start
step will docker-cp into the container. `known_host_key` is the
KnownHostKey string from the manifest; the gate's start step
materialises it into a known_hosts file if non-empty.
the gate credential paths inside the running sidecar."""
name: str
upstream_url: str
upstream_host: str
upstream_port: str
identity_file: str
known_host_key: str
known_hosts_file: Path = Path()
def git_gate_upstreams_for_bottle(bottle: ManifestBottle) -> tuple[GitGateUpstream, ...]:
"""Lift each `bottle.git` entry into a GitGateUpstream. Unique-Name
validation already ran in `manifest.ManifestBottle.from_dict`."""
return tuple(
GitGateUpstream(
name=e.Name,
upstream_url=e.Upstream,
upstream_host=e.UpstreamHost,
upstream_port=e.UpstreamPort,
identity_file=e.IdentityFile,
known_host_key=e.KnownHostKey,
)
for e in bottle.git
)
def _gitconfig_validate_value(field: str, value: str) -> None:
"""Raise ValueError if value contains characters that break gitconfig line syntax."""
if "\n" in value or "\r" in value:
raise ValueError(
f"git-gate: {field} contains a newline, which would inject "
f"arbitrary gitconfig keys; rejecting manifest entry"
)
def git_gate_render_gitconfig(
entries: tuple[ManifestGitEntry, ...], gate_host: str, *, scheme: str = "git",
) -> str:
"""Render the agent's ~/.gitconfig content for git-gate
`insteadOf` rewrites. Pure host-side, no docker / VM;
exposed for tests + reuse across backends.
`gate_host` is the part of the URL between `<scheme>://` and the
repo path backends differ here:
- docker: `git-gate` (the short network alias)
- firecracker: `<bundle_ip>:<port>` (no DNS on the
point-to-point TAP link)
Empty `entries` returns an empty string so callers can no-op
cleanly without conditional formatting at the call site."""
if not entries:
return ""
out = [
"# bot-bottle git-gate (PRD 0008): every git operation against\n",
"# a declared upstream routes through the gate, which mirrors\n",
"# the upstream bidirectionally (gitleaks-scanned push;\n",
"# fetch-from-upstream-before-every-upload-pack via access-hook).\n",
]
for entry in entries:
_gitconfig_validate_value(f"repos[{entry.Name!r}].url", entry.Upstream)
out.append(f'[url "{scheme}://{gate_host}/{entry.Name}.git"]\n')
out.append(f"\tinsteadOf = {entry.Upstream}\n")
if entry.RemoteKey and entry.RemoteKey != entry.UpstreamHost:
port = (
f":{entry.UpstreamPort}"
if entry.UpstreamPort and entry.UpstreamPort != "22"
else ""
)
alias = (
f"ssh://{entry.UpstreamUser}@{entry.RemoteKey}{port}/"
f"{entry.UpstreamPath}"
)
_gitconfig_validate_value(f"repos[{entry.Name!r}].url (resolved alias)", alias)
out.append(f"\tinsteadOf = {alias}\n")
return "".join(out)
def git_gate_known_hosts_line(host: str, port: str, key: str) -> str:
"""Format `host[:port] key` for OpenSSH's known_hosts. Non-default
ports use the bracketed `[host]:port` form (the form OpenSSH writes
on disk for hosts reached via a non-22 port)."""
if port and port != "22":
target = f"[{host}]:{port}"
else:
target = host
return f"{target} {key}\n"
def git_gate_render_entrypoint(upstreams: tuple[GitGateUpstream, ...]) -> str:
"""Posix-sh entrypoint. One `init_repo` call per upstream, then
`exec git daemon`. The function reads
`/git-gate/creds/<name>-{key,known_hosts}` (bind-mounted into
the bundle by the renderer) and wires them into each bare repo's
config; the access-hook + pre-receive hook pick those paths up
at fetch / push time."""
lines = [
"#!/bin/sh",
"set -eu",
"",
"init_repo() {",
" name=$1",
" upstream_url=$2",
" keyfile=/git-gate/creds/${name}-key",
" hostsfile=/git-gate/creds/${name}-known_hosts",
"",
# `|| true`: PRD 0018 chunk 3+ bind-mounts these RO from the
# host, so chmod-syscalls fail with EROFS. The files already
# have the right perms on the host (SSH requires 0600 to load
# the key in the first place), so the chmod is best-effort
# cleanup for the legacy docker-cp path where the file
# landed at the host's umask perms.
" chmod 600 \"$keyfile\" 2>/dev/null || true",
" if [ -f \"$hostsfile\" ]; then",
" chmod 600 \"$hostsfile\" 2>/dev/null || true",
" fi",
"",
" repo=/git/${name}.git",
" if [ ! -d \"$repo\" ]; then",
" git init --bare \"$repo\" >/dev/null",
# --mirror=fetch sets remote.origin.fetch = +refs/*:refs/* so",
# a later `git fetch origin` mirrors the upstream's full ref",
# graph (heads, tags, notes) into the bare repo at canonical",
# paths. It does NOT set remote.origin.mirror=true, so an",
# explicit `git push origin <ref>:<ref>` still pushes one ref.",
" git -C \"$repo\" remote add --mirror=fetch origin \"$upstream_url\"",
" fi",
" git -C \"$repo\" config git-gate.identityFile \"$keyfile\"",
" git -C \"$repo\" config git-gate.knownHosts \"$hostsfile\"",
" git -C \"$repo\" config receive.denyCurrentBranch ignore",
" git -C \"$repo\" config receive.advertisePushOptions true",
" git -C \"$repo\" config http.receivepack true",
" install -m 755 /etc/git-gate/pre-receive \"$repo/hooks/pre-receive\"",
"}",
"",
"mkdir -p /git",
]
for u in upstreams:
lines.append(f"init_repo {shlex.quote(u.name)} {shlex.quote(u.upstream_url)}")
lines.extend([
"",
"exec git daemon \\",
" --reuseaddr \\",
f" --timeout={GIT_GATE_TIMEOUT_SECS} \\",
f" --init-timeout={GIT_GATE_TIMEOUT_SECS} \\",
" --base-path=/git \\",
" --export-all \\",
" --enable=receive-pack \\",
" --access-hook=/etc/git-gate/access-hook \\",
" --verbose",
])
return "\n".join(lines) + "\n"
def git_gate_render_hook() -> str:
"""The shared pre-receive hook: gitleaks-scan all incoming refs,
then forward each accepted ref to the real upstream (`origin`)
using the per-repo credential. Failure in either phase aborts
the push so the agent sees a real rejection. POSIX sh.
Two phases (scan all, then push all) keeps a hit on ref N from
half-pushing refs 1..N-1; both phases re-read stdin from a temp
file because pre-receive's stdin is a one-shot stream."""
return r"""#!/bin/sh
# git-gate pre-receive (PRD 0008). Stdin: <old> <new> <ref> per line.
set -u
refs_file=$(mktemp)
trap 'rm -f "$refs_file"' EXIT
cat > "$refs_file"
zero=0000000000000000000000000000000000000000
supervise_gitleaks_allow() {
log_opts=$1
ref=$2
report_file=$(mktemp)
if ! gitleaks git \
--log-opts="$log_opts" \
--no-banner \
--redact \
--ignore-gitleaks-allow \
--report-format=json \
--report-path="$report_file" \
--exit-code 0 \
1>&2; then
rm -f "$report_file"
echo "git-gate: gitleaks inline-suppression scan failed for $ref" >&2
return 1
fi
proposal_id=$(
PYTHONPATH="/app${PYTHONPATH:+:$PYTHONPATH}" GITLEAKS_ALLOW_REF="$ref" python3 - "$report_file" <<'PY'
import datetime
import hashlib
import json
import os
import sys
from pathlib import Path
try:
import supervise as _sv
except ImportError:
from bot_bottle import supervise as _sv
report_path = Path(sys.argv[1])
slug = os.environ.get("SUPERVISE_BOTTLE_SLUG", "")
if not slug:
sys.exit(2)
try:
raw = json.loads(report_path.read_text() or "[]")
except json.JSONDecodeError:
sys.exit(3)
if not isinstance(raw, list):
sys.exit(3)
if not raw:
sys.exit(0)
ref = os.environ.get("GITLEAKS_ALLOW_REF", "")
lines = [
"gitleaks inline suppression requires supervisor approval",
f"ref: {ref}",
"",
]
for i, finding in enumerate(raw, 1):
if not isinstance(finding, dict):
continue
file_path = finding.get("File", "")
line_no = finding.get("StartLine", finding.get("Line", ""))
rule_id = finding.get("RuleID", "")
commit = finding.get("Commit", "")
line = finding.get("Line", "")
lines.extend([
f"finding {i}:",
f" file: {file_path}",
f" line: {line_no}",
f" rule: {rule_id}",
f" commit: {commit}",
f" code: {line}",
"",
])
payload = "\n".join(lines).rstrip() + "\n"
proposal = _sv.Proposal.new(
bottle_slug=slug,
tool=_sv.TOOL_GITLEAKS_ALLOW,
proposed_file=payload,
justification=(
"git-gate found gitleaks findings hidden by # gitleaks:allow; "
"approve only for dummy test fixtures or confirmed false positives"
),
current_file_hash=hashlib.sha256(payload.encode("utf-8")).hexdigest(),
now=datetime.datetime.now(datetime.timezone.utc),
)
_sv.write_proposal(proposal)
print(proposal.id)
PY
)
rc=$?
rm -f "$report_file"
if [ "$rc" -eq 0 ] && [ -z "$proposal_id" ]; then
return 0
fi
if [ "$rc" -ne 0 ]; then
echo "git-gate: cannot route # gitleaks:allow finding to supervisor; refusing push" >&2
return 1
fi
slug=${SUPERVISE_BOTTLE_SLUG:-}
timeout=${SUPERVISE_GITLEAKS_ALLOW_TIMEOUT_SECONDS:-300}
case "$timeout" in
''|*[!0-9]*)
echo "git-gate: invalid SUPERVISE_GITLEAKS_ALLOW_TIMEOUT_SECONDS=$timeout" >&2
return 1
;;
esac
echo "git-gate: queued # gitleaks:allow supervisor approval $proposal_id" >&2
echo "git-gate: approve with './cli.py supervise' to continue this push" >&2
waited=0
while [ "$waited" -lt "$timeout" ]; do
status=$(PYTHONPATH="/app${PYTHONPATH:+:$PYTHONPATH}" python3 - "$slug" "$proposal_id" <<'PY'
import sys
try:
import supervise as _sv
except ImportError:
from bot_bottle import supervise as _sv
slug = sys.argv[1]
try:
response = _sv.read_response(slug, sys.argv[2])
except FileNotFoundError:
sys.exit(2)
print(response.status)
PY
)
rc=$?
if [ "$rc" -eq 2 ]; then
status=""
elif [ "$rc" -ne 0 ]; then
status="invalid"
fi
if [ -n "$status" ]; then
case "$status" in
approved|modified)
PYTHONPATH="/app${PYTHONPATH:+:$PYTHONPATH}" python3 - "$slug" "$proposal_id" <<'PY' || true
import sys
try:
import supervise as _sv
except ImportError:
from bot_bottle import supervise as _sv
_sv.archive_proposal(sys.argv[1], sys.argv[2])
PY
echo "git-gate: supervisor approved # gitleaks:allow for $ref" >&2
return 0
;;
rejected)
echo "git-gate: supervisor rejected # gitleaks:allow for $ref" >&2
return 1
;;
*)
echo "git-gate: invalid supervisor response for # gitleaks:allow" >&2
return 1
;;
esac
fi
sleep 1
waited=$((waited + 1))
done
echo "git-gate: supervisor approval timed out for # gitleaks:allow; refusing push" >&2
return 1
}
# Phase 1: gitleaks scan each ref's incoming commits.
while IFS=' ' read -r old new ref; do
[ -z "$ref" ] && continue
[ "$new" = "$zero" ] && continue
if [ "$old" = "$zero" ]; then
# New ref: scan only the commits this push introduces — those
# reachable from $new but not from any ref the gate already has.
# Everything already on the gate arrived via upstream mirror-fetch
# or a previously gitleaks-scanned push, so it's already-upstream
# or already-scanned; re-scanning it (the old `$new` full-ancestry
# range) only resurfaces historical findings and blocks every new
# branch. See PRD 0028 / issue #106.
log_opts="$new --not --all"
else
log_opts="$old..$new"
fi
echo "git-gate: gitleaks scanning $ref ($log_opts)" >&2
if ! gitleaks git --log-opts="$log_opts" --no-banner --redact 1>&2; then
echo "git-gate: gitleaks rejected push to $ref" >&2
exit 1
fi
if ! supervise_gitleaks_allow "$log_opts" "$ref"; then
exit 1
fi
done < "$refs_file"
# Phase 2: forward each ref to the upstream (`origin`, configured
# in the entrypoint via `git remote add --mirror=fetch`).
keyfile=$(git config --get git-gate.identityFile)
hostsfile=$(git config --get git-gate.knownHosts)
if [ ! -f "$hostsfile" ]; then
echo "git-gate: no KnownHostKey configured for this upstream; refusing to push" >&2
echo "git-gate: add KnownHostKey to the bottle.git entry and restart the bottle" >&2
exit 1
fi
ssh_cmd="ssh -i $keyfile -o UserKnownHostsFile=$hostsfile -o StrictHostKeyChecking=yes -o IdentitiesOnly=yes -o BatchMode=yes -o ConnectTimeout=10"
push_option_count=${GIT_PUSH_OPTION_COUNT:-0}
case "$push_option_count" in
''|*[!0-9]*)
echo "git-gate: invalid GIT_PUSH_OPTION_COUNT=$push_option_count" >&2
exit 1
;;
esac
set --
i=0
while [ "$i" -lt "$push_option_count" ]; do
opt=$(printenv "GIT_PUSH_OPTION_$i" || :)
set -- "$@" --push-option="$opt"
i=$((i + 1))
done
while IFS=' ' read -r old new ref; do
[ -z "$ref" ] && continue
if [ "$new" = "$zero" ]; then
refspec=":$ref"
elif [ "$old" != "$zero" ] && ! git merge-base --is-ancestor "$old" "$new" 2>/dev/null; then
refspec="+$new:$ref"
else
refspec="$new:$ref"
fi
echo "git-gate: forwarding $ref to origin" >&2
if ! GIT_SSH_COMMAND="$ssh_cmd" git push "$@" origin "$refspec" 1>&2; then
echo "git-gate: upstream push failed for $ref" >&2
exit 1
fi
done < "$refs_file"
exit 0
"""
def git_gate_render_access_hook() -> str:
"""`git daemon --access-hook` script. Runs before each protocol
service; for `upload-pack` (fetch / clone / ls-remote / pull) it
refreshes the bare repo from upstream first, so the response
reflects upstream's current state. For other services (notably
`receive-pack`) it returns 0 immediately and lets the existing
pre-receive hook gate the operation. POSIX sh.
The hook receives:
$1 service name (`upload-pack`, `receive-pack`, ...)
$2 absolute path to the resolved repo
$3 client hostname (unused)
$4 client tcp address (unused)
Fail-closed on upstream errors: the agent's fetch fails too,
so it never silently sees stale data matches the PRD's
'equivalent to operations against the upstream' contract."""
return r"""#!/bin/sh
# git-gate access-hook (PRD 0008). $1=service $2=repo $3=host $4=peer
set -u
service=$1
repo_dir=$2
# Push path keeps its own gating in pre-receive (gitleaks +
# forward). Only refresh-from-upstream on fetch operations.
if [ "$service" != "upload-pack" ]; then
exit 0
fi
keyfile=$(git -C "$repo_dir" config --get git-gate.identityFile 2>/dev/null || true)
hostsfile=$(git -C "$repo_dir" config --get git-gate.knownHosts 2>/dev/null || true)
if [ -z "$keyfile" ] || [ ! -f "$hostsfile" ]; then
echo "git-gate: missing credentials for $repo_dir; refusing fetch" >&2
exit 1
fi
ssh_cmd="ssh -i $keyfile -o UserKnownHostsFile=$hostsfile -o StrictHostKeyChecking=yes -o IdentitiesOnly=yes -o BatchMode=yes -o ConnectTimeout=10"
echo "git-gate: refreshing $repo_dir from upstream" >&2
if ! GIT_SSH_COMMAND="$ssh_cmd" git -C "$repo_dir" fetch origin --prune >&2; then
echo "git-gate: upstream fetch failed for $repo_dir; refusing to serve stale data" >&2
exit 1
fi
# Sync the bare repo's HEAD to upstream's HEAD on the first fetch
# (when it still points at the `git init --bare` default of
# refs/heads/master and upstream uses something else, the cloned
# checkout would fail with "remote HEAD refers to nonexistent ref").
# Costs one extra ls-remote on first fetch only; subsequent fetches
# skip the branch. If upstream's default branch changes after the
# gate has cached it, restart the bottle to resync.
if ! git -C "$repo_dir" rev-parse --verify HEAD >/dev/null 2>&1; then
upstream_head=$(GIT_SSH_COMMAND="$ssh_cmd" git -C "$repo_dir" \
ls-remote --symref origin HEAD 2>/dev/null \
| awk '/^ref:/ {print $2; exit}')
if [ -n "$upstream_head" ]; then
git -C "$repo_dir" symbolic-ref HEAD "$upstream_head" || true
fi
fi
exit 0
"""
+4 -20
View File
@@ -1,9 +1,8 @@
"""Tiny smart-HTTP wrapper for git-gate repos.
Used where `git://` push traffic over a host-published Docker port can
hang before receive-pack reaches hooks (e.g. the firecracker backend,
where the guest reaches the sidecar over the point-to-point TAP). The
wrapper serves the same `/git/*.git` bare repos through
Used by the smolmachines backend where `git://` push traffic over the
host-published Docker port can hang before receive-pack reaches hooks.
The wrapper serves the same `/git/*.git` bare repos through
`git http-backend`, so pre-receive and upstream forwarding remain the
git-gate enforcement point.
"""
@@ -20,13 +19,6 @@ from urllib.parse import urlsplit
DEFAULT_PORT = 9420
# Mirrors git_gate_render.GIT_GATE_TIMEOUT_SECS. Duplicated rather than
# imported: this module ships as a flat top-level sibling in the sidecar
# bundle image (see Dockerfile.sidecars), not as part of the bot_bottle
# package, so `bot_bottle.git_gate` and its dependency chain aren't
# available at runtime.
GIT_GATE_TIMEOUT_SECS = 15
# Bound memory use while still allowing ordinary git push packfiles.
MAX_BODY_BYTES = 100 * 1024 * 1024
@@ -55,7 +47,6 @@ class GitHttpHandler(BaseHTTPRequestHandler):
[hook_path, "upload-pack", str(repo_dir), peer, peer],
capture_output=True,
check=False,
timeout=GIT_GATE_TIMEOUT_SECS,
)
if hook.returncode != 0:
detail = (hook.stderr or hook.stdout).decode(
@@ -119,7 +110,6 @@ class GitHttpHandler(BaseHTTPRequestHandler):
env=env,
capture_output=True,
check=False,
timeout=GIT_GATE_TIMEOUT_SECS,
)
self._write_cgi_response(proc.stdout)
@@ -158,13 +148,7 @@ class GitHttpHandler(BaseHTTPRequestHandler):
key, _, value = line.decode("latin1").partition(":")
value = value.strip()
if key.lower() == "status":
try:
status = int(value.split()[0])
except (ValueError, IndexError):
self.log_message(
"malformed CGI Status header %r; using 500", value,
)
status = 500
status = int(value.split()[0])
else:
headers.append((key, value))
self.send_response(status)
+145 -142
View File
@@ -62,25 +62,15 @@ from dataclasses import dataclass, field, replace
from pathlib import Path
from typing import Mapping
from .log import warn
from .manifest_util import ManifestError, as_json_object
from .manifest_agent import ManifestAgent, ManifestAgentProvider
from .manifest_bottle import ManifestBottle
from .manifest_egress import (
EGRESS_AUTH_SCHEMES,
ManifestEgressConfig,
ManifestEgressRoute,
)
from .manifest_extends import merge_bottles_runtime, resolve_bottles
from .manifest_git import ManifestGitEntry, ManifestGitUser, ManifestKeyConfig
from .manifest_loader import (
check_stale_json,
load_bottle_chain_from_dir,
scan_agent_names,
scan_bottle_names,
)
from .manifest_schema import validate_agent_frontmatter_keys
from .yaml_subset import YamlSubsetError, parse_frontmatter
from .manifest_git import ManifestGitEntry, ManifestGitUser, ManifestKeyConfig, parse_git_gate_config
from .manifest_schema import BOTTLE_KEYS
# Re-export everything that callers currently import from this module.
__all__ = [
@@ -99,6 +89,10 @@ __all__ = [
]
def _empty_str_dict() -> dict[str, str]:
return {}
def _section_dict(value: object, label: str) -> dict[str, object]:
"""Like as_json_object but treats absent/null as an empty section."""
if value is None:
@@ -106,6 +100,109 @@ def _section_dict(value: object, label: str) -> dict[str, object]:
return as_json_object(value, label)
@dataclass(frozen=True)
class ManifestBottle:
env: Mapping[str, str] = field(default_factory=_empty_str_dict)
agent_provider: ManifestAgentProvider = field(default_factory=ManifestAgentProvider)
git: tuple[ManifestGitEntry, ...] = ()
# Per-bottle git identity (issue #86). Empty default — bottles
# that don't set `git-gate.user:` in the manifest skip the
# `git config --global` step entirely. A bottle can declare a user
# identity without any git-gate.repos upstreams, and vice versa.
git_user: ManifestGitUser = field(default_factory=ManifestGitUser)
egress: ManifestEgressConfig = field(default_factory=ManifestEgressConfig)
# Per-bottle stuck-recovery sidecar (PRD 0013). When true (the
# default, issue #249), the launch step brings up a supervise
# sidecar that exposes MCP tools to the agent (egress-block,
# capability-block) plus mounts the current-config dir read-only
# into the agent at /etc/bot-bottle/current-config. Set
# `supervise: false` to skip the sidecar and mount.
supervise: bool = True
@classmethod
def from_dict(cls, name: str, raw: object) -> "ManifestBottle":
d = as_json_object(raw, f"bottle '{name}'")
if "runtime" in d:
raise ManifestError(
f"bottle '{name}' has a 'runtime' field, which is no longer "
f"supported. gVisor (runsc) is now auto-detected by the "
f"backend; remove the 'runtime' field from the bottle "
f"definition."
)
if "ssh" in d:
raise ManifestError(
f"bottle '{name}' has an 'ssh' field, which has been removed "
f"(PRD 0009). Declare upstreams under 'git-gate.repos' with "
f"url + identity + host_key; the git-gate sidecar (PRD 0008) "
f"holds the credential and gitleaks-scans pushes."
)
if "git" in d:
raise ManifestError(
f"bottle '{name}' uses 'git' which has been replaced by "
f"'git-gate' (PRD 0047). Move git.user → git-gate.user "
f"and git.remotes → git-gate.repos (fields: url, identity, host_key)."
)
if "git_user" in d:
raise ManifestError(
f"bottle '{name}' has a 'git_user' field, which has been "
f"removed. Move it under 'git-gate.user'."
)
unknown = set(d.keys()) - BOTTLE_KEYS
if unknown:
allowed = ", ".join(sorted(BOTTLE_KEYS))
raise ManifestError(
f"bottle '{name}' has unknown key(s) {sorted(unknown)}; "
f"allowed keys are {allowed}."
)
env: dict[str, str] = {}
env_raw = d.get("env")
if env_raw is not None:
env_dict = as_json_object(env_raw, f"bottle '{name}' env")
for var, value in env_dict.items():
if not isinstance(value, str):
raise ManifestError(
f"env entry {var} in bottle '{name}' must be a JSON string "
f"(was {type(value).__name__}). Use \"?<message>\" for prompt-at-runtime."
)
env[var] = value
git: tuple[ManifestGitEntry, ...] = ()
git_user = ManifestGitUser()
git_raw = d.get("git-gate")
if git_raw is not None:
git, git_user = parse_git_gate_config(name, git_raw)
agent_provider = (
ManifestAgentProvider.from_dict(name, d["agent_provider"])
if "agent_provider" in d
else ManifestAgentProvider()
)
egress = (
ManifestEgressConfig.from_dict(name, d["egress"])
if "egress" in d
else ManifestEgressConfig()
)
supervise_raw = d.get("supervise", True)
if not isinstance(supervise_raw, bool):
raise ManifestError(
f"bottle '{name}' supervise must be a boolean "
f"(was {type(supervise_raw).__name__})"
)
return cls(
env=env, agent_provider=agent_provider, git=git,
git_user=git_user, egress=egress, supervise=supervise_raw,
)
def _merge_git_user(
agent_user: ManifestGitUser, base_user: ManifestGitUser
) -> ManifestGitUser:
@@ -118,74 +215,6 @@ def _merge_git_user(
)
def _manifest_with_merged_git_user(
agent: "ManifestAgent", raw_bottle: "ManifestBottle"
) -> "Manifest":
"""Build the single-value Manifest, overlaying the agent's git-gate.user
onto the bottle (agent wins on non-empty, per-field). Shared by the eager
and lazy load_for_agent paths."""
merged = _merge_git_user(agent.git_user, raw_bottle.git_user)
bottle = (
raw_bottle if merged == raw_bottle.git_user
else replace(raw_bottle, git_user=merged)
)
return Manifest(agent=agent, bottle=bottle)
def _resolve_effective_bottle_eager(
agent_name: str,
agent: "ManifestAgent",
bottle_names: "tuple[str, ...]",
bottles: "Mapping[str, ManifestBottle]",
) -> "ManifestBottle":
"""Return the effective ManifestBottle for the eager (from_json_obj) path.
When bottle_names is non-empty they are merged in order. When empty, falls
back to agent.bottle. Raises ManifestError when neither is set."""
if bottle_names:
resolved: list[ManifestBottle] = []
for bn in bottle_names:
if bn not in bottles:
available = ", ".join(sorted(bottles.keys())) or "(none)"
raise ManifestError(
f"bottle '{bn}' not defined. Available: {available}"
)
resolved.append(bottles[bn])
return merge_bottles_runtime(resolved)
if not agent.bottle:
raise ManifestError(
f"agent '{agent_name}' has no 'bottle' field and no bottles were "
f"selected at launch. Select at least one bottle or add "
f"'bottle: <name>' to the agent manifest."
)
return bottles[agent.bottle]
def _resolve_effective_bottle_lazy(
agent_name: str,
agent_bottle: str,
bottle_names: "tuple[str, ...]",
bottles_dir: "Path",
) -> "ManifestBottle":
"""Return the effective ManifestBottle for the lazy (from_md_dirs) path.
When bottle_names is non-empty they are resolved from disk and merged in
order. When empty, falls back to agent_bottle. Raises ManifestError when
neither is set."""
if bottle_names:
resolved = [load_bottle_chain_from_dir(bn, bottles_dir) for bn in bottle_names]
return merge_bottles_runtime(resolved)
if not agent_bottle:
raise ManifestError(
f"agent '{agent_name}' has no 'bottle' field and no bottles were "
f"selected at launch. Select at least one bottle or add "
f"'bottle: <name>' to the agent manifest."
)
return load_bottle_chain_from_dir(agent_bottle, bottles_dir)
@dataclass(frozen=True)
class Manifest:
"""Single-agent/bottle value type. Returned by ManifestIndex.load_for_agent().
@@ -258,6 +287,8 @@ class ManifestIndex:
home_md = home_dir / ".bot-bottle"
cwd_md = cwd_dir / ".bot-bottle"
from .manifest_loader import check_stale_json
check_stale_json(home_dir, home_md, "$HOME")
if cwd_dir.resolve() != home_dir.resolve():
check_stale_json(cwd_dir, cwd_md, "$CWD")
@@ -297,6 +328,7 @@ class ManifestIndex:
files = sorted(stale_bottles.glob("*.md"))
if files:
names = ", ".join(p.name for p in files)
from .log import warn
warn(
f"ignoring bottle file(s) under "
f"{stale_bottles}: {names}. Bottles can only "
@@ -318,6 +350,7 @@ class ManifestIndex:
raw_bottles: dict[str, dict[str, object]] = {}
for n, b in raw_bottles_obj.items():
raw_bottles[n] = as_json_object(b, f"bottle '{n}'")
from .manifest_extends import resolve_bottles
bottles = resolve_bottles(raw_bottles)
@@ -327,17 +360,6 @@ class ManifestIndex:
}
return cls(bottles=bottles, agents=agents)
@property
def all_bottle_names(self) -> list[str]:
"""Sorted list of all discoverable bottle names.
In names-only mode (from resolve/from_md_dirs) this scans bottle
filenames without reading their content. In eager mode (from
from_json_obj) it returns the pre-parsed bottles' names."""
if self.home_md is not None:
return scan_bottle_names(self.home_md / "bottles")
return sorted(self.bottles.keys())
@property
def all_agent_names(self) -> list[str]:
"""Sorted list of all discoverable agent names.
@@ -346,6 +368,7 @@ class ManifestIndex:
filenames without reading their content. In eager mode (from
from_json_obj) it returns the pre-parsed agents' names."""
if self.home_md is not None:
from .manifest_loader import scan_agent_names
home_names = set(scan_agent_names(self.home_md / "agents").keys())
cwd_names: set[str] = set()
if self.cwd_md is not None:
@@ -353,18 +376,9 @@ class ManifestIndex:
return sorted(home_names | cwd_names)
return sorted(self.agents.keys())
def load_for_agent(
self,
agent_name: str,
bottle_names: "tuple[str, ...] | None" = None,
) -> "Manifest":
def load_for_agent(self, agent_name: str) -> "Manifest":
"""Parse the named agent and its bottle; return a single-value Manifest.
`bottle_names` is an ordered list of bottles selected at launch time.
When non-empty they are resolved and merged in order (index 0 = base;
later entries override). When empty or None, falls back to the agent's
own `bottle:` field. Raises ManifestError when neither is set.
In lazy mode (from resolve/from_md_dirs) the agent file and its
bottle chain are read from disk for the first time here. In eager
mode (from_json_obj) the data is already parsed; this just filters
@@ -375,34 +389,25 @@ class ManifestIndex:
Always raises ManifestError if the agent is unknown or invalid.
Backends call this at preflight inside _validate."""
effective_bottle_names: tuple[str, ...] = bottle_names or ()
if self.home_md is None:
return self._load_for_agent_eager(agent_name, effective_bottle_names)
return self._load_for_agent_lazy(agent_name, effective_bottle_names)
# Eager manifest (from_json_obj): data already parsed; filter to
# the one requested agent and its bottle so the returned Manifest
# always holds exactly one agent and one bottle regardless of path.
if agent_name not in self.agents:
available = ", ".join(sorted(self.agents.keys())) or "(none)"
raise ManifestError(
f"agent '{agent_name}' not defined. Available: {available}"
)
agent = self.agents[agent_name]
raw_bottle = self.bottles[agent.bottle]
merged = _merge_git_user(agent.git_user, raw_bottle.git_user)
bottle = raw_bottle if merged == raw_bottle.git_user else replace(raw_bottle, git_user=merged)
return Manifest(agent=agent, bottle=bottle)
def _load_for_agent_eager(
self, agent_name: str, bottle_names: tuple[str, ...]
) -> "Manifest":
"""Eager path (from_json_obj): data is already parsed; filter to the one
requested agent and its bottle so the returned Manifest always holds
exactly one agent and one bottle regardless of path."""
if agent_name not in self.agents:
available = ", ".join(sorted(self.agents.keys())) or "(none)"
raise ManifestError(
f"agent '{agent_name}' not defined. Available: {available}"
)
agent = self.agents[agent_name]
raw_bottle = _resolve_effective_bottle_eager(
agent_name, agent, bottle_names, self.bottles
)
return _manifest_with_merged_git_user(agent, raw_bottle)
from .manifest_loader import load_bottle_chain_from_dir, scan_agent_names
from .manifest_schema import validate_agent_frontmatter_keys
from .yaml_subset import YamlSubsetError, parse_frontmatter
def _load_for_agent_lazy(
self, agent_name: str, bottle_names: tuple[str, ...]
) -> "Manifest":
"""Lazy path (resolve/from_md_dirs): read and parse the agent file and
its bottle chain from disk for the first time here."""
assert self.home_md is not None # guaranteed by load_for_agent dispatch
# Locate the agent file; cwd wins over home on name collision.
home_agents = scan_agent_names(self.home_md / "agents")
cwd_agents: dict[str, Path] = {}
@@ -426,32 +431,30 @@ class ManifestIndex:
validate_agent_frontmatter_keys(agent_path, fm.keys())
# Determine the effective bottle name(s).
agent_bottle = fm.get("bottle") or ""
bottle_name = fm.get("bottle")
if not isinstance(bottle_name, str) or not bottle_name:
raise ManifestError(
f"agent '{agent_name}' must declare a 'bottle' field "
f"naming a defined bottle"
)
# Load the bottle chain (may raise ManifestError).
bottles_dir = self.home_md / "bottles"
raw_bottle = _resolve_effective_bottle_lazy(
agent_name, str(agent_bottle), bottle_names, bottles_dir
)
effective_bottle_name = (
bottle_names[-1] if bottle_names else str(agent_bottle)
)
raw_bottle = load_bottle_chain_from_dir(bottle_name, bottles_dir)
# Build and validate the full ManifestAgent.
agent_dict: dict[str, object] = {
"bottle": bottle_name,
"skills": fm.get("skills", []),
"prompt": body.strip(),
}
if agent_bottle:
agent_dict["bottle"] = agent_bottle
if "git-gate" in fm:
agent_dict["git-gate"] = fm["git-gate"]
# Pass the effective bottle name as the known-bottles set so agents
# that have bottle: set are validated; agents without bottle: pass {}
# since bottle_names were already resolved above.
known = {effective_bottle_name} if effective_bottle_name else set()
agent = ManifestAgent.from_dict(agent_name, agent_dict, known)
agent = ManifestAgent.from_dict(agent_name, agent_dict, {bottle_name})
return _manifest_with_merged_git_user(agent, raw_bottle)
merged_user = _merge_git_user(agent.git_user, raw_bottle.git_user)
bottle = raw_bottle if merged_user == raw_bottle.git_user else replace(raw_bottle, git_user=merged_user)
return Manifest(agent=agent, bottle=bottle)
def has_agent(self, name: str) -> bool:
return name in self.agents
+14 -27
View File
@@ -8,7 +8,7 @@ from typing import cast
from .agent_provider import PROVIDER_TEMPLATES
from .manifest_util import ManifestError, as_json_object
from .manifest_git import ManifestGitUser
from .manifest_schema import AGENT_MODEL_KEYS, is_valid_entity_name
from .manifest_schema import AGENT_MODEL_KEYS
@dataclass(frozen=True)
@@ -109,8 +109,7 @@ class ManifestAgentProvider:
@dataclass(frozen=True)
class ManifestAgent:
# Optional: when empty the operator selects bottles at launch time.
bottle: str = ""
bottle: str
skills: tuple[str, ...] = ()
prompt: str = ""
# Per-agent git identity (issue #94). Overlays the referenced
@@ -130,20 +129,18 @@ class ManifestAgent:
f"allowed keys are {allowed}."
)
bottle_raw = d.get("bottle")
bottle = ""
if bottle_raw is not None:
if not isinstance(bottle_raw, str) or not bottle_raw:
raise ManifestError(
f"agent '{name}' bottle must be a non-empty string when declared"
)
if bottle_raw not in bottle_names:
available = ", ".join(sorted(bottle_names)) or "(none defined)"
raise ManifestError(
f"agent '{name}' references bottle '{bottle_raw}', which is not defined. "
f"Available: {available}"
)
bottle = bottle_raw
bottle = d.get("bottle")
if not isinstance(bottle, str) or not bottle:
raise ManifestError(
f"agent '{name}' must declare a 'bottle' field naming a "
f"defined bottle"
)
if bottle not in bottle_names:
available = ", ".join(sorted(bottle_names)) or "(none defined)"
raise ManifestError(
f"agent '{name}' references bottle '{bottle}', which is not defined. "
f"Available: {available}"
)
skills: tuple[str, ...] = ()
skills_raw = d.get("skills")
@@ -161,16 +158,6 @@ class ManifestAgent:
f"agent '{name}' skills[{i}] must be a string "
f"(was {type(skill).__name__})"
)
# Skill names become host/guest path segments and are
# interpolated into provisioning shell commands, so they
# must fit the same kebab-case convention as bottle/agent
# filenames — rejecting anything that could break out of a
# path segment or inject shell metacharacters.
if not is_valid_entity_name(skill):
raise ManifestError(
f"agent '{name}' skills[{i}] {skill!r} is not a valid "
f"skill name; must match [a-z][a-z0-9-]*"
)
collected.append(skill)
skills = tuple(collected)
-129
View File
@@ -1,129 +0,0 @@
"""The `ManifestBottle` value type.
Split out of `manifest.py` so the `extends:`/loader resolvers can import it
without a circular dependency: `manifest.py` imports those resolvers, while
they only need this value type. Everything here depends on leaf modules
(`manifest_util`, `manifest_agent`, `manifest_egress`, `manifest_git`,
`manifest_schema`), so this module sits at the bottom of the manifest layer.
`manifest.py` re-exports `ManifestBottle`, so existing
`from .manifest import ManifestBottle` callers are unaffected.
"""
from __future__ import annotations
from dataclasses import dataclass, field
from typing import Mapping
from .manifest_util import ManifestError, as_json_object
from .manifest_agent import ManifestAgentProvider
from .manifest_egress import ManifestEgressConfig
from .manifest_git import ManifestGitEntry, ManifestGitUser, parse_git_gate_config
from .manifest_schema import BOTTLE_KEYS
__all__ = ["ManifestBottle"]
def _empty_str_dict() -> dict[str, str]:
return {}
@dataclass(frozen=True)
class ManifestBottle:
env: Mapping[str, str] = field(default_factory=_empty_str_dict)
agent_provider: ManifestAgentProvider = field(default_factory=ManifestAgentProvider)
git: tuple[ManifestGitEntry, ...] = ()
# Per-bottle git identity (issue #86). Empty default — bottles
# that don't set `git-gate.user:` in the manifest skip the
# `git config --global` step entirely. A bottle can declare a user
# identity without any git-gate.repos upstreams, and vice versa.
git_user: ManifestGitUser = field(default_factory=ManifestGitUser)
egress: ManifestEgressConfig = field(default_factory=ManifestEgressConfig)
# Per-bottle stuck-recovery sidecar (PRD 0013). When true (the
# default, issue #249), the launch step brings up a supervise
# sidecar that exposes egress MCP tools to the agent. Set
# `supervise: false` to skip the sidecar.
supervise: bool = True
@classmethod
def from_dict(cls, name: str, raw: object) -> "ManifestBottle":
d = as_json_object(raw, f"bottle '{name}'")
if "runtime" in d:
raise ManifestError(
f"bottle '{name}' has a 'runtime' field, which is no longer "
f"supported. gVisor (runsc) is now auto-detected by the "
f"backend; remove the 'runtime' field from the bottle "
f"definition."
)
if "ssh" in d:
raise ManifestError(
f"bottle '{name}' has an 'ssh' field, which has been removed "
f"(PRD 0009). Declare upstreams under 'git-gate.repos' with "
f"url + identity + host_key; the git-gate sidecar (PRD 0008) "
f"holds the credential and gitleaks-scans pushes."
)
if "git" in d:
raise ManifestError(
f"bottle '{name}' uses 'git' which has been replaced by "
f"'git-gate' (PRD 0047). Move git.user → git-gate.user "
f"and git.remotes → git-gate.repos (fields: url, identity, host_key)."
)
if "git_user" in d:
raise ManifestError(
f"bottle '{name}' has a 'git_user' field, which has been "
f"removed. Move it under 'git-gate.user'."
)
unknown = set(d.keys()) - BOTTLE_KEYS
if unknown:
allowed = ", ".join(sorted(BOTTLE_KEYS))
raise ManifestError(
f"bottle '{name}' has unknown key(s) {sorted(unknown)}; "
f"allowed keys are {allowed}."
)
env: dict[str, str] = {}
env_raw = d.get("env")
if env_raw is not None:
env_dict = as_json_object(env_raw, f"bottle '{name}' env")
for var, value in env_dict.items():
if not isinstance(value, str):
raise ManifestError(
f"env entry {var} in bottle '{name}' must be a JSON string "
f"(was {type(value).__name__}). Use \"?<message>\" for prompt-at-runtime."
)
env[var] = value
git: tuple[ManifestGitEntry, ...] = ()
git_user = ManifestGitUser()
git_raw = d.get("git-gate")
if git_raw is not None:
git, git_user = parse_git_gate_config(name, git_raw)
agent_provider = (
ManifestAgentProvider.from_dict(name, d["agent_provider"])
if "agent_provider" in d
else ManifestAgentProvider()
)
egress = (
ManifestEgressConfig.from_dict(name, d["egress"])
if "egress" in d
else ManifestEgressConfig()
)
supervise_raw = d.get("supervise", True)
if not isinstance(supervise_raw, bool):
raise ManifestError(
f"bottle '{name}' supervise must be a boolean "
f"(was {type(supervise_raw).__name__})"
)
return cls(
env=env, agent_provider=agent_provider, git=git,
git_user=git_user, egress=egress, supervise=supervise_raw,
)
+24 -52
View File
@@ -2,59 +2,11 @@
from __future__ import annotations
from .manifest_bottle import ManifestBottle
from .manifest_egress import ManifestEgressConfig, validate_egress_routes
from .manifest_git import ManifestGitUser, parse_git_gate_config
from .manifest_util import ManifestError, as_json_object
from typing import TYPE_CHECKING
def merge_bottles_runtime(bottles: "list[ManifestBottle]") -> "ManifestBottle":
"""Merge an ordered list of pre-resolved ManifestBottle objects.
Index 0 is the base; each subsequent entry is applied on top using
the same field-merge rules as the file-based extends machinery:
env: dict merge, later wins; git_user: per-field overlay, later
wins on non-empty; git (repos): union by name, later wins; egress
routes: concatenate; agent_provider, supervise: later replaces.
"""
if not bottles:
raise ValueError("merge_bottles_runtime requires at least one bottle")
result = bottles[0]
for override in bottles[1:]:
result = _merge_two_bottles_runtime(result, override)
return result
def _merge_two_bottles_runtime(base: "ManifestBottle", override: "ManifestBottle") -> "ManifestBottle":
merged_env = {**base.env, **override.env}
merged_git_user = ManifestGitUser(
name=override.git_user.name or base.git_user.name,
email=override.git_user.email or base.git_user.email,
)
# git repos: union keyed by Name, override wins per-name.
base_repos_by_name = {entry.Name: entry for entry in base.git}
override_repos_by_name = {entry.Name: entry for entry in override.git}
merged_repos_names = list(base_repos_by_name) + [
n for n in override_repos_by_name if n not in base_repos_by_name
]
merged_git = tuple(
override_repos_by_name.get(n, base_repos_by_name[n])
for n in merged_repos_names
)
merged_routes = base.egress.routes + override.egress.routes
merged_egress = ManifestEgressConfig(routes=merged_routes, Log=override.egress.Log)
return ManifestBottle(
env=merged_env,
agent_provider=override.agent_provider,
git=merged_git,
git_user=merged_git_user,
egress=merged_egress,
supervise=override.supervise,
)
if TYPE_CHECKING:
from .manifest import ManifestBottle
from .manifest_egress import ManifestEgressConfig
def resolve_bottles(raws: dict[str, dict[str, object]]) -> dict[str, ManifestBottle]:
@@ -77,6 +29,8 @@ def _resolve_one_bottle(
repos_cache: dict[str, dict[str, object]],
seen: tuple[str, ...],
) -> ManifestBottle:
from .manifest import ManifestBottle, ManifestError
if name in cache:
return cache[name]
if name in seen:
@@ -168,6 +122,11 @@ def _fold_two_bottles(
later_repos_raw: dict[str, object],
) -> tuple[ManifestBottle, dict[str, object]]:
"""Combine two resolved parent bottles; later wins over earlier."""
from .manifest import ManifestBottle, ManifestGitUser
from .manifest_egress import ManifestEgressConfig
from .manifest_git import parse_git_gate_config
from .manifest_util import as_json_object
merged_env = {**earlier.env, **later.env}
merged_git_user = ManifestGitUser(
@@ -216,6 +175,10 @@ def _merge_bottles(
name: str,
) -> ManifestBottle:
"""Apply PRD 0025 merge rules."""
from .manifest import ManifestBottle, ManifestGitUser
from .manifest_egress import validate_egress_routes
from .manifest_util import as_json_object
# git-gate.repos: when the child declares repos, inject the already
# name-merged repo set (computed by _resolve_repos_raw) so the child
# parses with the full inherited+overridden list (issue #237).
@@ -288,6 +251,8 @@ def _resolve_repos_raw(
inherits the parent's set verbatim; an explicit empty dict clears it.
Otherwise parent and child unite by name, with same-name entries
field-merged (parent fields are defaults, child fields win)."""
from .manifest_util import as_json_object
if not _child_declares_git_gate_repos(child_raw):
return parent_repos
child_repos = _declared_repos_raw(child_raw)
@@ -307,6 +272,8 @@ def _resolve_repos_raw(
def _declared_repos_raw(child_raw: dict[str, object]) -> dict[str, object]:
"""Return the child's explicitly declared git-gate.repos as raw dicts,
or an empty dict when none are declared."""
from .manifest_util import as_json_object
if not _child_declares_git_gate_repos(child_raw):
return {}
git_raw = as_json_object(child_raw.get("git-gate", {}), "child git-gate")
@@ -314,6 +281,8 @@ def _declared_repos_raw(child_raw: dict[str, object]) -> dict[str, object]:
def _child_declares_git_gate_repos(child_raw: dict[str, object]) -> bool:
from .manifest_util import as_json_object
git_raw = child_raw.get("git-gate")
if git_raw is None:
return False
@@ -326,6 +295,9 @@ def _merge_egress(
child: ManifestEgressConfig,
child_raw: dict[str, object],
) -> ManifestEgressConfig:
from .manifest_egress import ManifestEgressConfig
from .manifest_util import as_json_object
child_egress_raw = as_json_object(child_raw.get("egress"), "child egress")
routes = parent.routes + child.routes
log = child.Log if "log" in child_egress_raw else parent.Log
+6 -21
View File
@@ -3,10 +3,9 @@
from __future__ import annotations
from pathlib import Path
from typing import TYPE_CHECKING
from .log import warn
from .manifest_bottle import ManifestBottle
from .manifest_extends import resolve_bottles
from .manifest_schema import (
entity_name_from_path,
validate_bottle_frontmatter_keys,
@@ -14,6 +13,9 @@ from .manifest_schema import (
from .manifest_util import ManifestError
from .yaml_subset import YamlSubsetError, parse_frontmatter
if TYPE_CHECKING:
from .manifest import ManifestBottle
def check_stale_json(dir_path: Path, md_dir: Path, label: str) -> None:
"""Die if `<dir_path>/bot-bottle.json` exists but `md_dir` does
@@ -30,25 +32,6 @@ def check_stale_json(dir_path: Path, md_dir: Path, label: str) -> None:
)
def scan_bottle_names(bottles_dir: Path) -> list[str]:
"""Scan `<bottles_dir>/*.md` for valid filenames and return sorted bottle names.
No file content is read. Invalid filenames are skipped with a warning."""
result: list[str] = []
if not bottles_dir.is_dir():
return result
for path in sorted(bottles_dir.glob("*.md")):
name = entity_name_from_path(path)
if name is None:
warn(
f"skipping {path}: filename must match "
f"[a-z][a-z0-9-]*.md (got {path.name!r})"
)
continue
result.append(name)
return result
def scan_agent_names(agents_dir: Path) -> dict[str, Path]:
"""Scan `<agents_dir>/*.md` for valid filenames and return `{name: path}`.
@@ -76,6 +59,8 @@ def load_bottle_chain_from_dir(
Only the files in the extends chain are read unrelated bottle files
are never touched. Raises ManifestError on parse or validation failure."""
from .manifest_extends import resolve_bottles
raws: dict[str, dict[str, object]] = {}
to_load = [bottle_name]
while to_load:
+3 -10
View File
@@ -18,8 +18,8 @@ _FILENAME_RX = re.compile(r"^[a-z][a-z0-9-]*$")
BOTTLE_KEYS = frozenset(
{"env", "extends", "agent_provider", "git-gate", "egress", "supervise"}
)
AGENT_KEYS_REQUIRED: frozenset[str] = frozenset()
AGENT_KEYS_OPTIONAL = frozenset({"bottle", "skills", "git-gate"})
AGENT_KEYS_REQUIRED = frozenset({"bottle"})
AGENT_KEYS_OPTIONAL = frozenset({"skills", "git-gate"})
# Claude Code subagent fields bot-bottle ignores at launch but does
# not reject. This lets the same file double as
@@ -33,20 +33,13 @@ AGENT_KEYS = (
AGENT_MODEL_KEYS = AGENT_KEYS | frozenset({"prompt"})
def is_valid_entity_name(name: str) -> bool:
"""True if `name` fits the kebab-case `[a-z][a-z0-9-]*` convention
shared by bottle/agent filenames and skill names. Names that satisfy
this are also safe to interpolate into a host/guest path segment."""
return bool(_FILENAME_RX.match(name))
def entity_name_from_path(path: Path) -> str | None:
"""Return the entity name implied by the filename, or None if the
filename does not fit the [a-z][a-z0-9-]* convention."""
if path.suffix != ".md":
return None
stem = path.stem
if not is_valid_entity_name(stem):
if not _FILENAME_RX.match(stem):
return None
return stem
-37
View File
@@ -1,37 +0,0 @@
"""SQLite migration runner for bot-bottle stores."""
from __future__ import annotations
import sqlite3
class TableMigrations:
"""Runs a sequential list of DDL migrations tracked by schema_key in schema_versions."""
def __init__(self, schema_key: str, migrations: list[str]) -> None:
self.schema_key = schema_key
self.migrations = migrations
def apply(self, conn: sqlite3.Connection) -> None:
conn.execute(
"""
CREATE TABLE IF NOT EXISTS schema_versions (
module TEXT PRIMARY KEY,
version INTEGER NOT NULL DEFAULT 0
)
"""
)
row = conn.execute(
"SELECT version FROM schema_versions WHERE module = ?",
(self.schema_key,),
).fetchone()
version = row[0] if row else 0
for i, sql in enumerate(self.migrations[version:], start=version + 1):
conn.execute(sql)
conn.execute(
"INSERT OR REPLACE INTO schema_versions (module, version) VALUES (?, ?)",
(self.schema_key, i),
)
__all__ = ["TableMigrations"]
-215
View File
@@ -1,215 +0,0 @@
"""SQLite-backed queue store for supervise proposals and responses (PRD 0013)."""
from __future__ import annotations
import os
import sqlite3
from pathlib import Path
try:
from .supervise_types import Proposal, Response, host_db_path
from .db_store import DbStore
from .migrations import TableMigrations
except ImportError:
from supervise_types import Proposal, Response, host_db_path # type: ignore[import-not-found] # pylint: disable=import-error,no-name-in-module
from db_store import DbStore # type: ignore[import-not-found] # pylint: disable=import-error,no-name-in-module
from migrations import TableMigrations # type: ignore[import-not-found] # pylint: disable=import-error,no-name-in-module
class QueueStore(DbStore):
"""SQLite-backed persistent store for supervise proposals and responses."""
def __init__(self, queue_key: str, db_path: Path | None = None) -> None:
self.queue_key = queue_key
if db_path is not None:
resolved = db_path
else:
# In the sidecar container SUPERVISE_DB_PATH points at the
# bind-mounted host DB. On the host this env var is never set,
# so we always fall through to host_db_path().
env_path = os.environ.get("SUPERVISE_DB_PATH", "").strip()
resolved = Path(env_path) if env_path else host_db_path()
# One entry per schema version: migrations[0] brings a fresh DB to
# version 1, [1] to version 2, etc. Add new entries at the end; never
# edit existing ones.
migrations = TableMigrations("queue_store", [
# v1 — proposals table
"""
CREATE TABLE IF NOT EXISTS supervise_proposals (
queue_key TEXT NOT NULL,
id TEXT NOT NULL,
bottle_slug TEXT NOT NULL,
tool TEXT NOT NULL,
proposed_file TEXT NOT NULL,
justification TEXT NOT NULL,
arrival_timestamp TEXT NOT NULL,
current_file_hash TEXT NOT NULL,
archived INTEGER NOT NULL DEFAULT 0,
PRIMARY KEY (queue_key, id)
)
""",
# v2 — responses table
"""
CREATE TABLE IF NOT EXISTS supervise_responses (
queue_key TEXT NOT NULL,
proposal_id TEXT NOT NULL,
status TEXT NOT NULL,
notes TEXT NOT NULL,
final_file TEXT,
archived INTEGER NOT NULL DEFAULT 0,
PRIMARY KEY (queue_key, proposal_id)
)
""",
])
super().__init__(resolved, migrations)
def write_proposal(self, proposal: Proposal) -> Path:
with self._connect() as conn:
conn.execute(
"""
INSERT OR REPLACE INTO supervise_proposals (
queue_key, id, bottle_slug, tool, proposed_file, justification,
arrival_timestamp, current_file_hash, archived
) VALUES (?, ?, ?, ?, ?, ?, ?, ?, 0)
""",
(
self.queue_key,
proposal.id,
proposal.bottle_slug,
proposal.tool,
proposal.proposed_file,
proposal.justification,
proposal.arrival_timestamp,
proposal.current_file_hash,
),
)
self._chmod()
return self.db_path
def read_proposal(self, proposal_id: str) -> Proposal:
with self._connect() as conn:
row = conn.execute(
"""
SELECT * FROM supervise_proposals
WHERE queue_key = ? AND id = ? AND archived = 0
""",
(self.queue_key, proposal_id),
).fetchone()
if row is None:
raise FileNotFoundError(proposal_id)
return self._row_to_proposal(row)
def list_pending_proposals(self) -> list[Proposal]:
if not self.db_path.is_file():
return []
with self._connect() as conn:
rows = conn.execute(
"""
SELECT p.* FROM supervise_proposals p
WHERE p.archived = 0
AND p.queue_key = ?
AND NOT EXISTS (
SELECT 1 FROM supervise_responses r
WHERE r.queue_key = p.queue_key
AND r.proposal_id = p.id
AND r.archived = 0
)
ORDER BY p.arrival_timestamp, p.id
""",
(self.queue_key,),
).fetchall()
return [self._row_to_proposal(row) for row in rows]
def list_all_pending_proposals(self) -> list[Proposal]:
if not self.db_path.is_file():
return []
with self._connect() as conn:
rows = conn.execute(
"""
SELECT p.* FROM supervise_proposals p
WHERE p.archived = 0
AND NOT EXISTS (
SELECT 1 FROM supervise_responses r
WHERE r.queue_key = p.queue_key
AND r.proposal_id = p.id
AND r.archived = 0
)
ORDER BY p.arrival_timestamp, p.id
"""
).fetchall()
return [self._row_to_proposal(row) for row in rows]
def write_response(self, response: Response) -> Path:
with self._connect() as conn:
conn.execute(
"""
INSERT OR REPLACE INTO supervise_responses (
queue_key, proposal_id, status, notes, final_file, archived
) VALUES (?, ?, ?, ?, ?, 0)
""",
(
self.queue_key,
response.proposal_id,
response.status,
response.notes,
response.final_file,
),
)
self._chmod()
return self.db_path
def read_response(self, proposal_id: str) -> Response:
with self._connect() as conn:
row = conn.execute(
"""
SELECT * FROM supervise_responses
WHERE queue_key = ? AND proposal_id = ? AND archived = 0
""",
(self.queue_key, proposal_id),
).fetchone()
if row is None:
raise FileNotFoundError(proposal_id)
return self._row_to_response(row)
def archive_proposal(self, proposal_id: str) -> None:
if not self.db_path.is_file():
return
with self._connect() as conn:
conn.execute(
"""
UPDATE supervise_proposals SET archived = 1
WHERE queue_key = ? AND id = ?
""",
(self.queue_key, proposal_id),
)
conn.execute(
"""
UPDATE supervise_responses SET archived = 1
WHERE queue_key = ? AND proposal_id = ?
""",
(self.queue_key, proposal_id),
)
@staticmethod
def _row_to_proposal(row: sqlite3.Row) -> Proposal:
return Proposal(
id=row["id"],
bottle_slug=row["bottle_slug"],
tool=row["tool"],
proposed_file=row["proposed_file"],
justification=row["justification"],
arrival_timestamp=row["arrival_timestamp"],
current_file_hash=row["current_file_hash"],
)
@staticmethod
def _row_to_response(row: sqlite3.Row) -> Response:
return Response(
proposal_id=row["proposal_id"],
status=row["status"],
notes=row["notes"],
final_file=row["final_file"],
)
__all__ = ["QueueStore"]
-60
View File
@@ -1,60 +0,0 @@
"""Singleton manager for all bot-bottle SQLite stores (PRD 0013)."""
from __future__ import annotations
from pathlib import Path
try:
from .audit_store import AuditStore
from .queue_store import QueueStore
except ImportError:
from audit_store import AuditStore # type: ignore[import-not-found] # pylint: disable=import-error,no-name-in-module
from queue_store import QueueStore # type: ignore[import-not-found] # pylint: disable=import-error,no-name-in-module
_instance: StoreManager | None = None
class StoreManager:
"""Owns db_path and delegates migrate/is_migrated across all stores.
Use instance() for normal access. Call reset(db_path) in tests to swap
the singleton to a temp path, then reset() with no args to restore the
default."""
def __init__(self, db_path: Path | None = None) -> None:
if db_path is None:
# Lazy import to avoid a circular dependency: supervise imports
# StoreManager at module level; StoreManager must not import
# supervise at module level in return.
try:
from .supervise import host_db_path
except ImportError:
from supervise import host_db_path # type: ignore[import-not-found] # pylint: disable=import-error,no-name-in-module
db_path = host_db_path()
self.db_path = db_path
@classmethod
def instance(cls) -> StoreManager:
global _instance
if _instance is None:
_instance = cls()
return _instance
@classmethod
def reset(cls, db_path: Path | None = None) -> None:
"""Replace the singleton. Pass db_path for test isolation; omit to restore default."""
global _instance
_instance = cls(db_path)
def is_migrated(self) -> bool:
return (
QueueStore("", self.db_path).is_migrated()
and AuditStore(self.db_path).is_migrated()
)
def migrate(self) -> None:
QueueStore("", self.db_path).migrate()
AuditStore(self.db_path).migrate()
__all__ = ["StoreManager"]
+386 -117
View File
@@ -2,21 +2,23 @@
The supervise plane is the per-bottle MCP sidecar plus its host-side
queue/audit support. The sidecar (bot_bottle.supervise_server)
sits on the bottle's internal network and exposes MCP tools the agent
calls when it needs an operator-reviewed egress change:
sits on the bottle's internal network and exposes three MCP tools the
agent calls when it hits a stuck-recovery category:
* egress-block / allow agent proposes a new routes.yaml
* capability-block agent proposes a new agent Dockerfile
Each tool call: the agent passes the full proposed file plus a
justification text. The sidecar validates the proposal syntactically,
writes it to the host SQLite queue table, and holds the tool-call
writes it to the host's per-bottle queue dir, and holds the tool-call
connection open. The operator's supervise TUI
(bot_bottle.cli.supervise) sees the proposal, accepts
approve / modify / reject, and writes a response row. The sidecar sees
the response and returns `{status, notes}` to the agent.
approve / modify / reject, and writes a response file alongside the
proposal. The sidecar sees the response and returns `{status, notes}`
to the agent.
This module defines the host-side library: dataclasses for the queue
record shapes, queue read/write helpers, the audit log writer, and the
file shapes, queue read/write helpers, the audit log writer, and the
diff renderer. The in-container sidecar lives in
bot_bottle/supervise_server.py; the supervise daemon's container
lifecycle is owned by the sidecar bundle (PRD 0024).
@@ -30,142 +32,290 @@ remediation engines that wire real config changes land in PRDs 0014,
from __future__ import annotations
import dataclasses
import difflib
import hashlib
import json
import os
import time
import uuid
from abc import ABC
from dataclasses import dataclass
from datetime import datetime, timezone
from pathlib import Path
try:
from .supervise_types import (
ACTION_OPERATOR_EDIT,
AuditEntry,
HOST_DB_FILENAME,
Proposal,
Response,
STATUSES,
STATUS_APPROVED,
STATUS_MODIFIED,
STATUS_REJECTED,
TOOLS,
TOOL_EGRESS_ALLOW,
TOOL_EGRESS_BLOCK,
TOOL_EGRESS_TOKEN_ALLOW,
TOOL_GITLEAKS_ALLOW,
TOOL_LIST_EGRESS_ROUTES,
bot_bottle_root,
)
except ImportError:
from supervise_types import ( # type: ignore[import-not-found,no-redef] # pylint: disable=import-error,no-name-in-module
ACTION_OPERATOR_EDIT,
AuditEntry,
HOST_DB_FILENAME,
Proposal,
Response,
STATUSES,
STATUS_APPROVED,
STATUS_MODIFIED,
STATUS_REJECTED,
TOOLS,
TOOL_EGRESS_ALLOW,
TOOL_EGRESS_BLOCK,
TOOL_EGRESS_TOKEN_ALLOW,
TOOL_GITLEAKS_ALLOW,
TOOL_LIST_EGRESS_ROUTES,
bot_bottle_root,
)
SUPERVISE_HOSTNAME = "supervise"
SUPERVISE_PORT = 9100
TOOL_CAPABILITY_BLOCK = "capability-block"
TOOL_EGRESS_BLOCK = "egress-block"
TOOL_EGRESS_ALLOW = "egress-allow"
TOOL_GITLEAKS_ALLOW = "gitleaks-allow"
# Written directly by the egress addon (not an agent-facing MCP tool) when an
# outbound DLP token block is routed to the operator for override (PRD 0062).
TOOL_EGRESS_TOKEN_ALLOW = "egress-token-allow"
TOOL_LIST_EGRESS_ROUTES = "list-egress-routes"
TOOLS: tuple[str, ...] = (
TOOL_EGRESS_ALLOW,
TOOL_CAPABILITY_BLOCK,
TOOL_EGRESS_BLOCK,
TOOL_GITLEAKS_ALLOW,
TOOL_EGRESS_TOKEN_ALLOW,
TOOL_LIST_EGRESS_ROUTES,
)
# The supervise sidecar uses these to query egress's
# introspection endpoint for the `list-egress-routes` MCP
# tool. The hostname + port match egress's docker network
# listen port (see backend.docker.egress.EGRESS_PORT). The supervise
# daemon runs inside the sidecar bundle alongside egress, so loopback
# is the stable address across docker, firecracker, and Apple
# is the stable address across docker, smolmachines, and Apple
# Container backends.
EGRESS_FORWARD_PROXY = "http://127.0.0.1:9099"
EGRESS_INTROSPECT_URL = "http://_egress.local/allowlist"
# capability-block has no on-disk config the operator edits in place
# (the Dockerfile is rebuilt, not patched), so it has no audit log
# here — those changes are captured by git history + the rebuild record
# laid down in PRD 0016.
COMPONENT_FOR_TOOL: dict[str, str] = {
TOOL_EGRESS_ALLOW: "egress",
TOOL_EGRESS_BLOCK: "egress",
}
DB_PATH_IN_CONTAINER = "/run/supervise/bot-bottle.db"
STATUS_APPROVED = "approved"
STATUS_MODIFIED = "modified"
STATUS_REJECTED = "rejected"
STATUSES: tuple[str, ...] = (STATUS_APPROVED, STATUS_MODIFIED, STATUS_REJECTED)
# Operator-initiated audit entries (no tool call). PRD 0014's
# `routes edit <bottle>` verb writes entries with this action.
ACTION_OPERATOR_EDIT = "operator-edit"
QUEUE_DIR_IN_CONTAINER = "/run/supervise/queue"
CURRENT_CONFIG_DIR_IN_AGENT = "/etc/bot-bottle/current-config"
DEFAULT_POLL_INTERVAL_SEC = 0.5
# --- Paths -----------------------------------------------------------------
def bot_bottle_root() -> Path:
return Path.home() / ".bot-bottle"
def queue_dir_for_slug(slug: str) -> Path:
return bot_bottle_root() / "queue" / slug
def audit_dir() -> Path:
return bot_bottle_root() / "audit"
def host_db_path() -> Path:
# Calls bot_bottle_root() through this module's globals so that patches
# on supervise.bot_bottle_root propagate to callers going through
# supervise.host_db_path().
#
# Kept in its own "db" subdirectory (see supervise_types.host_db_path
# for why) — must stay in sync with that copy.
return bot_bottle_root() / "db" / HOST_DB_FILENAME
def audit_log_path(component: str, slug: str) -> Path:
return audit_dir() / f"{component}-{slug}.log"
try:
from .queue_store import QueueStore
from .audit_store import AuditStore
from .store_manager import StoreManager
except ImportError:
# Sidecar bundle: files are flat-copied under /app, not a package.
from queue_store import QueueStore # type: ignore[import-not-found] # pylint: disable=import-error,no-name-in-module
from audit_store import AuditStore # type: ignore[import-not-found] # pylint: disable=import-error,no-name-in-module
from store_manager import StoreManager # type: ignore[import-not-found] # pylint: disable=import-error,no-name-in-module
# --- Dataclasses -----------------------------------------------------------
@dataclass(frozen=True)
class Proposal:
"""One pending tool-call from the agent. The sidecar writes one
of these to the queue dir on a tool call; the operator's TUI
reads them; the sidecar polls for a matching Response."""
id: str
bottle_slug: str
tool: str
proposed_file: str
justification: str
arrival_timestamp: str
current_file_hash: str
@classmethod
def new(
cls,
*,
bottle_slug: str,
tool: str,
proposed_file: str,
justification: str,
current_file_hash: str,
now: datetime | None = None,
) -> "Proposal":
ts = (now or datetime.now(timezone.utc)).isoformat()
return cls(
id=str(uuid.uuid4()),
bottle_slug=bottle_slug,
tool=tool,
proposed_file=proposed_file,
justification=justification,
arrival_timestamp=ts,
current_file_hash=current_file_hash,
)
def to_dict(self) -> dict[str, object]:
return dataclasses.asdict(self)
@classmethod
def from_dict(cls, raw: dict[str, object]) -> "Proposal":
tool = _require_str(raw, "tool")
if tool not in TOOLS:
raise ValueError(f"tool must be one of {TOOLS}; got {tool!r}")
return cls(
id=_require_str(raw, "id"),
bottle_slug=_require_str(raw, "bottle_slug"),
tool=tool,
proposed_file=_require_str(raw, "proposed_file"),
justification=_require_str(raw, "justification"),
arrival_timestamp=_require_str(raw, "arrival_timestamp"),
current_file_hash=_require_str(raw, "current_file_hash"),
)
@dataclass(frozen=True)
class Response:
"""The operator's decision on a proposal. The TUI writes one of
these to the queue dir; the sidecar reads it and returns the
`{status, notes}` pair to the agent's tool call.
`final_file` carries the file content the supervisor will
actually apply: for `approved`, equal to the proposal's
`proposed_file`; for `modified`, the operator's edited version
(the audit diff is current final_file, not current
proposed_file); for `rejected`, None."""
proposal_id: str
status: str
notes: str
final_file: str | None = None
def to_dict(self) -> dict[str, object]:
return dataclasses.asdict(self)
@classmethod
def from_dict(cls, raw: dict[str, object]) -> "Response":
status = _require_str(raw, "status")
if status not in STATUSES:
raise ValueError(
f"response status must be one of {STATUSES}; got {status!r}"
)
final = raw.get("final_file")
if final is not None and not isinstance(final, str):
raise ValueError(
f"final_file must be a string or null; got {type(final).__name__}"
)
return cls(
proposal_id=_require_str(raw, "proposal_id"),
status=status,
notes=_require_str(raw, "notes"),
final_file=final,
)
@dataclass(frozen=True)
class AuditEntry:
"""One row of the per-bottle audit log. JSON-Lines, append-only."""
timestamp: str
bottle_slug: str
component: str
operator_action: str
operator_notes: str
justification: str
diff: str
def to_dict(self) -> dict[str, object]:
return dataclasses.asdict(self)
# --- Queue I/O -------------------------------------------------------------
def write_proposal(proposal: Proposal) -> Path:
"""Persist `proposal` in the queue database, mode 0o600.
def _proposal_filename(proposal_id: str) -> str:
return f"{proposal_id}.proposal.json"
def _response_filename(proposal_id: str) -> str:
return f"{proposal_id}.response.json"
def _id_from_proposal_filename(path: Path) -> str | None:
name = path.name
if not name.endswith(".proposal.json"):
return None
return name[: -len(".proposal.json")]
def write_proposal(queue_dir: Path, proposal: Proposal) -> Path:
"""Persist `proposal` as JSON in the queue dir, mode 0o600.
Directory is created if missing."""
return QueueStore(proposal.bottle_slug).write_proposal(proposal)
queue_dir.mkdir(parents=True, exist_ok=True)
path = queue_dir / _proposal_filename(proposal.id)
payload = json.dumps(proposal.to_dict(), indent=2) + "\n"
_atomic_write(path, payload, mode=0o600)
return path
def read_proposal(bottle_slug: str, proposal_id: str) -> Proposal:
return QueueStore(bottle_slug).read_proposal(proposal_id)
def read_proposal(queue_dir: Path, proposal_id: str) -> Proposal:
path = queue_dir / _proposal_filename(proposal_id)
with path.open() as f:
raw = json.load(f)
if not isinstance(raw, dict):
raise ValueError(f"{path}: top-level must be an object")
return Proposal.from_dict(raw)
def list_pending_proposals(bottle_slug: str) -> list[Proposal]:
"""All proposals for `bottle_slug` that do not yet have a matching
response. Sorted by `arrival_timestamp` so the operator
def list_pending_proposals(queue_dir: Path) -> list[Proposal]:
"""All proposals in `queue_dir` that do not yet have a matching
response file. Sorted by `arrival_timestamp` so the operator
sees the queue FIFO."""
return QueueStore(bottle_slug).list_pending_proposals()
if not queue_dir.is_dir():
return []
out: list[Proposal] = []
for path in sorted(queue_dir.glob("*.proposal.json")):
proposal_id = _id_from_proposal_filename(path)
if proposal_id is None:
continue
if (queue_dir / _response_filename(proposal_id)).exists():
continue
try:
with path.open() as f:
raw = json.load(f)
except (OSError, json.JSONDecodeError):
continue
if not isinstance(raw, dict):
continue
try:
out.append(Proposal.from_dict(raw))
except (KeyError, ValueError):
continue
out.sort(key=lambda p: p.arrival_timestamp)
return out
def list_all_pending_proposals() -> list[Proposal]:
"""All pending proposals across bottles, sorted FIFO."""
return QueueStore("").list_all_pending_proposals()
def write_response(queue_dir: Path, response: Response) -> Path:
queue_dir.mkdir(parents=True, exist_ok=True)
path = queue_dir / _response_filename(response.proposal_id)
payload = json.dumps(response.to_dict(), indent=2) + "\n"
_atomic_write(path, payload, mode=0o600)
return path
def write_response(bottle_slug: str, response: Response) -> Path:
return QueueStore(bottle_slug).write_response(response)
def read_response(bottle_slug: str, proposal_id: str) -> Response:
return QueueStore(bottle_slug).read_response(proposal_id)
def read_response(queue_dir: Path, proposal_id: str) -> Response:
path = queue_dir / _response_filename(proposal_id)
with path.open() as f:
raw = json.load(f)
if not isinstance(raw, dict):
raise ValueError(f"{path}: top-level must be an object")
return Response.from_dict(raw)
def wait_for_response(
bottle_slug: str,
queue_dir: Path,
proposal_id: str,
*,
poll_interval: float = DEFAULT_POLL_INTERVAL_SEC,
@@ -176,35 +326,90 @@ def wait_for_response(
which the wait raises TimeoutError. None waits forever the
natural shape, since the operator's response time is unbounded.
Polls SQLite so the implementation stays portable and stdlib-only."""
store = QueueStore(bottle_slug)
Polls the filesystem so the implementation stays portable and
stdlib-only."""
path = queue_dir / _response_filename(proposal_id)
while True:
try:
return store.read_response(proposal_id)
except FileNotFoundError:
pass
if path.exists():
try:
with path.open() as f:
raw = json.load(f)
except (OSError, json.JSONDecodeError):
raw = None
if isinstance(raw, dict):
try:
return Response.from_dict(raw)
except (KeyError, ValueError):
pass
if deadline is not None and time.monotonic() >= deadline:
raise TimeoutError(f"no response for proposal {proposal_id!r}")
time.sleep(poll_interval)
def archive_proposal(bottle_slug: str, proposal_id: str) -> None:
"""Mark both proposal and response rows processed.
Idempotent missing rows are silently skipped."""
QueueStore(bottle_slug).archive_proposal(proposal_id)
def archive_proposal(queue_dir: Path, proposal_id: str) -> None:
"""Move both proposal and response files to `<queue_dir>/processed/`.
Idempotent missing files are silently skipped."""
processed = queue_dir / "processed"
processed.mkdir(parents=True, exist_ok=True)
for name in (_proposal_filename(proposal_id), _response_filename(proposal_id)):
src = queue_dir / name
if src.exists():
src.rename(processed / name)
# --- Audit log -------------------------------------------------------------
def write_audit_entry(entry: AuditEntry) -> Path:
"""Append `entry` to the host supervise audit table."""
return AuditStore().write_audit_entry(entry)
"""Append `entry` as one JSON-Lines record to the per-bottle
audit log. Acquires an advisory exclusive lock so concurrent
writers don't interleave bytes."""
path = audit_log_path(entry.component, entry.bottle_slug)
path.parent.mkdir(parents=True, exist_ok=True)
line = json.dumps(entry.to_dict(), sort_keys=False) + "\n"
fd = os.open(path, os.O_WRONLY | os.O_APPEND | os.O_CREAT, 0o600)
try:
_try_flock(fd)
try:
os.write(fd, line.encode("utf-8"))
finally:
_try_funlock(fd)
finally:
os.close(fd)
return path
def read_audit_entries(component: str, slug: str) -> list[AuditEntry]:
"""Load all audit entries for the given component+slug."""
return AuditStore().read_audit_entries(component, slug)
"""Load all audit entries for the given component+slug. Empty
list if the log doesn't exist."""
path = audit_log_path(component, slug)
if not path.is_file():
return []
out: list[AuditEntry] = []
with path.open() as f:
for raw_line in f:
raw_line = raw_line.strip()
if not raw_line:
continue
try:
raw = json.loads(raw_line)
except json.JSONDecodeError:
continue
if not isinstance(raw, dict):
continue
try:
out.append(AuditEntry(
timestamp=_require_str(raw, "timestamp"),
bottle_slug=_require_str(raw, "bottle_slug"),
component=_require_str(raw, "component"),
operator_action=_require_str(raw, "operator_action"),
operator_notes=_require_str(raw, "operator_notes"),
justification=_require_str(raw, "justification"),
diff=_require_str(raw, "diff"),
))
except ValueError:
continue
return out
# --- Diff rendering --------------------------------------------------------
@@ -233,51 +438,115 @@ def sha256_hex(content: str) -> str:
# --- Sidecar plan + abstract lifecycle -------------------------------------
# Filename of the staged Dockerfile inside the agent's read-only
# current-config mount. The capability-block tool's description
# points the agent at this exact path so it can read the current
# Dockerfile and propose modifications.
#
# routes.yaml + allowlist used to live here too; PRD 0017 chunk 3
# moved them behind the `list-egress-routes` MCP tool (live state
# from egress's introspection endpoint) so the agent always sees
# current data rather than a launch-time snapshot.
CURRENT_CONFIG_DOCKERFILE = "Dockerfile"
@dataclass(frozen=True)
class SupervisePlan:
"""Output of Supervise.prepare; consumed by .start.
`db_path` is the host database bind-mounted into the sidecar at
/run/supervise/bot-bottle.db. `internal_network` is empty at
prepare time; the backend's launch step fills it via
`queue_dir` is the host directory bind-mounted into the sidecar
at /run/supervise/queue. `current_config_dir` is the host
directory bind-mounted (read-only) into the *agent* container
at /etc/bot-bottle/current-config currently holds only the
Dockerfile snapshot (routes.yaml + allowlist moved to the
`list-egress-routes` MCP tool). `internal_network` is
empty at prepare time; the backend's launch step fills it via
dataclasses.replace before calling .start."""
slug: str
db_path: Path
queue_dir: Path
current_config_dir: Path
internal_network: str = ""
class Supervise(ABC):
"""Per-bottle supervise sidecar. Encapsulates host-side database
staging; the sidecar's start/stop lifecycle is backend-specific."""
"""Per-bottle supervise sidecar. Encapsulates the host-side
prepare (queue dir + current-config staging); the sidecar's
start/stop lifecycle is backend-specific."""
def prepare(
self,
slug: str,
stage_dir: Path,
) -> SupervisePlan:
"""Stage the host database. Returns the plan; `internal_network`
must be set by the launch step before .start runs."""
del stage_dir
mgr = StoreManager.instance()
mgr.migrate()
"""Stage the per-bottle queue dir on the host and the
current-config dir under `stage_dir`. Returns the plan;
`internal_network` must be set by the launch step before
.start runs."""
queue_dir = queue_dir_for_slug(slug)
queue_dir.mkdir(parents=True, exist_ok=True)
current_config_dir = stage_dir / "current-config"
current_config_dir.mkdir(parents=True, exist_ok=True)
return SupervisePlan(
slug=slug,
db_path=mgr.db_path,
queue_dir=queue_dir,
current_config_dir=current_config_dir,
)
# --- Helpers ---------------------------------------------------------------
def _require_str(raw: dict[str, object], key: str) -> str:
value = raw.get(key)
if not isinstance(value, str):
raise ValueError(f"missing or non-string field {key!r}")
return value
def _atomic_write(path: Path, content: str, *, mode: int) -> None:
"""Atomic: write to a sibling tmp file, fsync, rename."""
tmp = path.with_suffix(path.suffix + ".tmp")
fd = os.open(tmp, os.O_WRONLY | os.O_CREAT | os.O_TRUNC, mode)
try:
os.write(fd, content.encode("utf-8"))
os.fsync(fd)
finally:
os.close(fd)
os.replace(tmp, path)
try:
import fcntl as _fcntl
def _try_flock(fd: int) -> None: # type: ignore[reportRedeclaration]
try:
_fcntl.flock(fd, _fcntl.LOCK_EX)
except OSError:
pass
def _try_funlock(fd: int) -> None: # type: ignore[reportRedeclaration]
try:
_fcntl.flock(fd, _fcntl.LOCK_UN)
except OSError:
pass
except ImportError: # pragma: no cover — Windows path
def _try_flock(fd: int) -> None: # noqa: F841 — Windows fallback
return None
def _try_funlock(fd: int) -> None: # noqa: F841 — Windows fallback
return None
__all__ = [
"ACTION_OPERATOR_EDIT",
"AuditEntry",
"AuditStore",
"COMPONENT_FOR_TOOL",
"CURRENT_CONFIG_DIR_IN_AGENT",
"CURRENT_CONFIG_DOCKERFILE",
"DEFAULT_POLL_INTERVAL_SEC",
"DB_PATH_IN_CONTAINER",
"Proposal",
"QueueStore",
"QUEUE_DIR_IN_CONTAINER",
"Response",
"StoreManager",
"STATUSES",
"STATUS_APPROVED",
"STATUS_MODIFIED",
@@ -289,6 +558,7 @@ __all__ = [
"TOOLS",
"EGRESS_FORWARD_PROXY",
"EGRESS_INTROSPECT_URL",
"TOOL_CAPABILITY_BLOCK",
"TOOL_EGRESS_ALLOW",
"TOOL_EGRESS_BLOCK",
"TOOL_GITLEAKS_ALLOW",
@@ -298,9 +568,8 @@ __all__ = [
"audit_dir",
"audit_log_path",
"bot_bottle_root",
"host_db_path",
"list_pending_proposals",
"list_all_pending_proposals",
"queue_dir_for_slug",
"read_audit_entries",
"read_proposal",
"read_response",
+142 -111
View File
@@ -1,19 +1,20 @@
"""Supervise sidecar HTTP server (PRD 0013).
Per-bottle MCP server exposing tools the agent calls to propose egress
config changes when stuck. The tools are `egress-allow`,
`egress-block`, and `list-egress-routes`.
Per-bottle MCP server exposing tools the agent calls to propose config
changes when stuck. The tools are `allow`, `egress-block`,
`capability-block`, and `list-egress-routes`.
Each queued tool call:
1. Validates the proposed file syntactically.
2. Writes a Proposal to the host SQLite database.
3. Blocks polling for a matching Response row.
2. Writes a Proposal to /run/supervise/queue/ (bind-mounted from
the host's ~/.bot-bottle/queue/<slug>/).
3. Blocks polling for a matching Response file.
4. Returns the operator's `{status, notes}` to the agent.
The bottle slug arrives via SUPERVISE_BOTTLE_SLUG env (stamped at
container creation by the backend's start step). SUPERVISE_DB_PATH
points at the bind-mounted host database.
container creation by the backend's start step). The queue dir comes
from SUPERVISE_QUEUE_DIR (default `/run/supervise/queue`).
Speaks MCP over HTTP+JSON-RPC. Methods handled:
@@ -41,15 +42,16 @@ import typing
import urllib.error
import urllib.request
from dataclasses import dataclass
from pathlib import Path
try:
# Same-directory imports inside the bundle container; these files are
# COPYed flat under /app by Dockerfile.sidecars.
from egress_addon_core import LOG_OFF, load_config
from egress_addon_core import load_routes
import supervise as _sv
except ModuleNotFoundError:
# Package imports for host-side tests and tooling.
from .egress_addon_core import LOG_OFF, load_config
from .egress_addon_core import load_routes
from . import supervise as _sv
@@ -88,19 +90,19 @@ def parse_jsonrpc(body: bytes) -> JsonRpcRequest:
try:
raw = json.loads(body)
except json.JSONDecodeError as e:
raise _RpcClientError(ERR_PARSE, f"parse error: {e}") from e
raise _RpcError(ERR_PARSE, f"parse error: {e}") from e
if not isinstance(raw, dict):
raise _RpcClientError(ERR_INVALID_REQUEST, "request must be a JSON object")
raise _RpcError(ERR_INVALID_REQUEST, "request must be a JSON object")
if raw.get("jsonrpc") != JSONRPC_VERSION:
raise _RpcClientError(ERR_INVALID_REQUEST, "jsonrpc field must be '2.0'")
raise _RpcError(ERR_INVALID_REQUEST, "jsonrpc field must be '2.0'")
method = raw.get("method")
if not isinstance(method, str):
raise _RpcClientError(ERR_INVALID_REQUEST, "method must be a string")
raise _RpcError(ERR_INVALID_REQUEST, "method must be a string")
params = raw.get("params", {})
if params is None:
params = {}
if not isinstance(params, dict):
raise _RpcClientError(ERR_INVALID_PARAMS, "params must be an object")
raise _RpcError(ERR_INVALID_PARAMS, "params must be an object")
rpc_id = raw.get("id", _NO_ID)
is_notification = rpc_id is _NO_ID
return JsonRpcRequest(
@@ -115,23 +117,12 @@ _NO_ID = object()
class _RpcError(Exception):
"""Base class for all typed RPC errors that surface as JSON-RPC error responses."""
def __init__(self, code: int, message: str):
super().__init__(message)
self.code = code
self.message = message
class _RpcClientError(_RpcError):
"""Caller sent a bad request; returned verbatim, no server-side logging."""
class _RpcInternalError(_RpcError):
"""Server-side fault; logged at ERROR with cause, always returns ERR_INTERNAL."""
def __init__(self, message: str) -> None:
super().__init__(ERR_INTERNAL, message)
def jsonrpc_result(request_id: object, result: object) -> bytes:
payload = {"jsonrpc": JSONRPC_VERSION, "id": request_id, "result": result}
return (json.dumps(payload) + "\n").encode("utf-8")
@@ -149,49 +140,6 @@ def jsonrpc_error(request_id: object, code: int, message: str) -> bytes:
# --- Tool definitions ------------------------------------------------------
# Shared by both proposal tools (egress-allow / egress-block): they take the
# same arguments and differ only in their top-level tool description. Kept as a
# single source of truth so the schema can't drift between the two tools.
_ROUTES_YAML_DESCRIPTION = (
"Full proposed /etc/egress/routes.yaml content. "
"Each route entry accepts these keys:\n"
" host: <hostname> (required)\n"
" auth_scheme: Bearer|token (must pair with token_env)\n"
" token_env: <ENV_VAR_NAME> (must pair with auth_scheme)\n"
" matches: (optional list of match entries)\n"
" - paths: [{type: prefix|exact|regex, value: /...}]\n"
" methods: [GET, POST, ...]\n"
" headers: [{name: X-Hdr, value: val, type: exact|regex}]\n"
" git: (optional; omit to block git clone/fetch)\n"
" fetch: true\n"
" dlp: (optional DLP scanner overrides)\n"
" outbound_detectors: [token_patterns, known_secrets]\n"
" inbound_detectors: [naive_injection_detection]\n"
" outbound_on_match: block|redact|supervise (default supervise)\n"
"Omit any key that should use its default. "
"`list-egress-routes` returns routes in this same format."
)
def _proposal_input_schema() -> dict[str, object]:
"""Build a fresh input schema for a routes.yaml proposal tool. Returns a
new dict per call so the two tool definitions don't alias one object."""
return {
"type": "object",
"properties": {
"routes_yaml": {
"type": "string",
"description": _ROUTES_YAML_DESCRIPTION,
},
"justification": {
"type": "string",
"description": "Why this egress route is needed.",
},
},
"required": ["routes_yaml", "justification"],
}
TOOL_DEFINITIONS: list[dict[str, object]] = [
{
"name": _sv.TOOL_LIST_EGRESS_ROUTES,
@@ -219,7 +167,38 @@ TOOL_DEFINITIONS: list[dict[str, object]] = [
"`list-egress-routes` first so the proposal preserves existing "
"routes."
),
"inputSchema": _proposal_input_schema(),
"inputSchema": {
"type": "object",
"properties": {
"routes_yaml": {
"type": "string",
"description": (
"Full proposed /etc/egress/routes.yaml content. "
"Each route entry accepts these keys:\n"
" host: <hostname> (required)\n"
" auth_scheme: Bearer|token (must pair with token_env)\n"
" token_env: <ENV_VAR_NAME> (must pair with auth_scheme)\n"
" matches: (optional list of match entries)\n"
" - paths: [{type: prefix|exact|regex, value: /...}]\n"
" methods: [GET, POST, ...]\n"
" headers: [{name: X-Hdr, value: val, type: exact|regex}]\n"
" git: (optional; omit to block git clone/fetch)\n"
" fetch: true\n"
" dlp: (optional DLP scanner overrides)\n"
" outbound_detectors: [token_patterns, known_secrets]\n"
" inbound_detectors: [naive_injection_detection]\n"
" outbound_on_match: block|redact|supervise (default supervise)\n"
"Omit any key that should use its default. "
"`list-egress-routes` returns routes in this same format."
),
},
"justification": {
"type": "string",
"description": "Why this egress route is needed.",
},
},
"required": ["routes_yaml", "justification"],
},
},
{
"name": _sv.TOOL_EGRESS_BLOCK,
@@ -230,7 +209,66 @@ TOOL_DEFINITIONS: list[dict[str, object]] = [
"`list-egress-routes` first so the proposal preserves existing "
"routes."
),
"inputSchema": _proposal_input_schema(),
"inputSchema": {
"type": "object",
"properties": {
"routes_yaml": {
"type": "string",
"description": (
"Full proposed /etc/egress/routes.yaml content. "
"Each route entry accepts these keys:\n"
" host: <hostname> (required)\n"
" auth_scheme: Bearer|token (must pair with token_env)\n"
" token_env: <ENV_VAR_NAME> (must pair with auth_scheme)\n"
" matches: (optional list of match entries)\n"
" - paths: [{type: prefix|exact|regex, value: /...}]\n"
" methods: [GET, POST, ...]\n"
" headers: [{name: X-Hdr, value: val, type: exact|regex}]\n"
" git: (optional; omit to block git clone/fetch)\n"
" fetch: true\n"
" dlp: (optional DLP scanner overrides)\n"
" outbound_detectors: [token_patterns, known_secrets]\n"
" inbound_detectors: [naive_injection_detection]\n"
" outbound_on_match: block|redact|supervise (default supervise)\n"
"Omit any key that should use its default. "
"`list-egress-routes` returns routes in this same format."
),
},
"justification": {
"type": "string",
"description": "Why this egress route is needed.",
},
},
"required": ["routes_yaml", "justification"],
},
},
{
"name": _sv.TOOL_CAPABILITY_BLOCK,
"description": (
"Call when the bottle is missing a tool, skill, permission, "
"or env var you need — something that lives in the agent "
"Dockerfile rather than in the egress routes. "
"Read the current Dockerfile from "
"/etc/bot-bottle/current-config/Dockerfile, compose a "
"modified version, and pass the full new file plus a "
"justification. On approval the supervisor rebuilds the "
"bottle from the new Dockerfile and starts a replacement on "
"the same branch (wired in PRD 0016; v1 acknowledges only)."
),
"inputSchema": {
"type": "object",
"properties": {
"dockerfile": {
"type": "string",
"description": "Full proposed Dockerfile content.",
},
"justification": {
"type": "string",
"description": "Why this capability is needed.",
},
},
"required": ["dockerfile", "justification"],
},
},
]
@@ -239,6 +277,7 @@ TOOL_DEFINITIONS: list[dict[str, object]] = [
# payload (stored in Proposal.proposed_file).
PROPOSED_FILE_FIELD: dict[str, str] = {
_sv.TOOL_EGRESS_ALLOW: "routes_yaml",
_sv.TOOL_CAPABILITY_BLOCK: "dockerfile",
_sv.TOOL_EGRESS_BLOCK: "routes_yaml",
}
@@ -251,22 +290,21 @@ def validate_proposed_file(tool: str, content: str) -> None:
catches obvious paste-errors / wrong-tool selections before they
enter the queue."""
if not content.strip():
raise _RpcClientError(ERR_INVALID_PARAMS, f"{tool}: proposed file is empty")
if tool in (_sv.TOOL_EGRESS_ALLOW, _sv.TOOL_EGRESS_BLOCK):
raise _RpcError(ERR_INVALID_PARAMS, f"{tool}: proposed file is empty")
if tool == _sv.TOOL_CAPABILITY_BLOCK:
# Dockerfiles are too varied to validate syntactically beyond
# non-empty. The operator reads the diff in the TUI.
pass
elif tool in (_sv.TOOL_EGRESS_ALLOW, _sv.TOOL_EGRESS_BLOCK):
try:
config = load_config(content)
load_routes(content)
except ValueError as e:
raise _RpcClientError(
raise _RpcError(
ERR_INVALID_PARAMS,
f"{tool}: proposed routes.yaml is not valid: {e}",
) from e
if config.log != LOG_OFF:
raise _RpcClientError(
ERR_INVALID_PARAMS,
f"{tool}: proposed routes.yaml must not change egress logging",
)
else:
raise _RpcClientError(ERR_INVALID_PARAMS, f"unknown tool {tool!r}")
raise _RpcError(ERR_INVALID_PARAMS, f"unknown tool {tool!r}")
# --- MCP handlers ----------------------------------------------------------
@@ -275,6 +313,7 @@ def validate_proposed_file(tool: str, content: str) -> None:
@dataclass(frozen=True)
class ServerConfig:
bottle_slug: str
queue_dir: Path
response_timeout_seconds: float = DEFAULT_RESPONSE_TIMEOUT_SECONDS
@@ -338,17 +377,17 @@ def handle_tools_call(
doesn't need operator approval."""
name = params.get("name")
if not isinstance(name, str):
raise _RpcClientError(ERR_INVALID_PARAMS, "tools/call missing 'name'")
raise _RpcError(ERR_INVALID_PARAMS, "tools/call missing 'name'")
if name == _sv.TOOL_LIST_EGRESS_ROUTES:
return handle_list_egress_routes(typing.cast(dict[str, object], params.get("arguments", {})), config)
args_raw = params.get("arguments", {})
if not isinstance(args_raw, dict):
raise _RpcClientError(ERR_INVALID_PARAMS, "tools/call 'arguments' must be an object")
raise _RpcError(ERR_INVALID_PARAMS, "tools/call 'arguments' must be an object")
justification = args_raw.get("justification")
if not isinstance(justification, str) or not justification.strip():
raise _RpcClientError(
raise _RpcError(
ERR_INVALID_PARAMS,
f"{name}: 'justification' is required and must be a non-empty string",
)
@@ -357,13 +396,13 @@ def handle_tools_call(
file_field = PROPOSED_FILE_FIELD[name]
proposed_file = args_raw.get(file_field)
if not isinstance(proposed_file, str):
raise _RpcClientError(
raise _RpcError(
ERR_INVALID_PARAMS,
f"{name}: '{file_field}' is required and must be a string",
)
validate_proposed_file(name, proposed_file)
else:
raise _RpcClientError(ERR_INVALID_PARAMS, f"unknown tool {name!r}")
raise _RpcError(ERR_INVALID_PARAMS, f"unknown tool {name!r}")
proposal = _sv.Proposal.new(
bottle_slug=config.bottle_slug,
@@ -372,10 +411,7 @@ def handle_tools_call(
justification=justification,
current_file_hash=_sv.sha256_hex(proposed_file),
)
try:
_sv.write_proposal(proposal)
except OSError as e:
raise _RpcInternalError(f"failed to write proposal to queue: {e}") from e
_sv.write_proposal(config.queue_dir, proposal)
sys.stderr.write(
f"supervise: queued proposal {proposal.id} ({name}) "
f"for bottle {config.bottle_slug}; waiting for operator...\n"
@@ -384,7 +420,7 @@ def handle_tools_call(
deadline = time.monotonic() + config.response_timeout_seconds
try:
response = _sv.wait_for_response(
config.bottle_slug,
config.queue_dir,
proposal.id,
poll_interval=MIN_RESPONSE_POLL_INTERVAL_SECONDS,
deadline=deadline,
@@ -395,10 +431,7 @@ def handle_tools_call(
"content": [{"type": "text", "text": text}],
"isError": False,
}
try:
_sv.archive_proposal(config.bottle_slug, proposal.id)
except OSError as e:
raise _RpcInternalError(f"failed to archive proposal: {e}") from e
_sv.archive_proposal(config.queue_dir, proposal.id)
text = format_response_text(response)
return {
@@ -432,8 +465,9 @@ def format_pending_response_text(timeout_seconds: float) -> str:
# --- HTTP transport --------------------------------------------------------
# Max request body the server accepts. 1 MB is well above any realistic
# routes.yaml proposal.
# Max request body the server accepts. Generous because Dockerfile
# proposals can be a few KB; routes.json is small. 1 MB is well above
# any realistic config file.
MAX_BODY_BYTES = 1 * 1024 * 1024
@@ -473,7 +507,7 @@ class MCPHandler(http.server.BaseHTTPRequestHandler):
try:
req = parse_jsonrpc(body)
except _RpcClientError as e:
except _RpcError as e:
self._write_jsonrpc(jsonrpc_error(None, e.code, e.message))
return
@@ -481,19 +515,11 @@ class MCPHandler(http.server.BaseHTTPRequestHandler):
try:
result = self._dispatch(req, config)
except _RpcClientError as e:
except _RpcError as e:
self._write_jsonrpc(jsonrpc_error(req.id, e.code, e.message))
return
except _RpcInternalError as e:
cause = e.__cause__
detail = f": {cause}" if cause else ""
sys.stderr.write(f"supervise: internal error: {e.message}{detail}\n")
sys.stderr.flush()
self._write_jsonrpc(jsonrpc_error(req.id, ERR_INTERNAL, "internal error"))
return
except Exception as e: # noqa: W0718 — unexpected errors
sys.stderr.write(f"supervise: unexpected error: {type(e).__name__}: {e}\n")
sys.stderr.flush()
except Exception as e: # noqa: W0718 — catch-all for RPC dispatch errors
sys.stderr.write(f"supervise: internal error: {e}\n")
self._write_jsonrpc(jsonrpc_error(req.id, ERR_INTERNAL, "internal error"))
return
@@ -512,7 +538,7 @@ class MCPHandler(http.server.BaseHTTPRequestHandler):
return handle_tools_list(req.params)
if method == "tools/call":
return handle_tools_call(req.params, config)
raise _RpcClientError(ERR_METHOD_NOT_FOUND, f"method not found: {method}")
raise _RpcError(ERR_METHOD_NOT_FOUND, f"method not found: {method}")
def _write_jsonrpc(self, body: bytes) -> None:
self.send_response(200)
@@ -536,7 +562,7 @@ class MCPHandler(http.server.BaseHTTPRequestHandler):
class MCPServer(socketserver.ThreadingMixIn, http.server.HTTPServer):
allow_reuse_address = True
daemon_threads = True
config: ServerConfig = ServerConfig(bottle_slug="")
config: ServerConfig = ServerConfig(bottle_slug="", queue_dir=Path())
# --- Entry point -----------------------------------------------------------
@@ -545,18 +571,21 @@ class MCPServer(socketserver.ThreadingMixIn, http.server.HTTPServer):
def serve(
*,
bottle_slug: str,
queue_dir: Path,
port: int = _sv.SUPERVISE_PORT,
bind: str = "0.0.0.0",
response_timeout_seconds: float = DEFAULT_RESPONSE_TIMEOUT_SECONDS,
) -> typing.NoReturn:
queue_dir.mkdir(parents=True, exist_ok=True)
server = MCPServer((bind, port), MCPHandler)
server.config = ServerConfig(
bottle_slug=bottle_slug,
queue_dir=queue_dir,
response_timeout_seconds=response_timeout_seconds,
)
sys.stderr.write(
f"supervise listening on {bind}:{port}; "
f"slug={bottle_slug!r}; "
f"slug={bottle_slug!r}; queue={queue_dir}; "
f"tools: {', '.join(t['name'] for t in TOOL_DEFINITIONS)}\n" # type: ignore[arg-type]
)
sys.stderr.flush()
@@ -575,6 +604,7 @@ def main(argv: list[str]) -> int:
if not bottle_slug:
sys.stderr.write("supervise: SUPERVISE_BOTTLE_SLUG env is unset\n")
return 2
queue_dir = Path(os.environ.get("SUPERVISE_QUEUE_DIR", _sv.QUEUE_DIR_IN_CONTAINER))
port = int(os.environ.get("SUPERVISE_PORT", str(_sv.SUPERVISE_PORT)))
bind = os.environ.get("SUPERVISE_BIND", "0.0.0.0")
try:
@@ -584,6 +614,7 @@ def main(argv: list[str]) -> int:
return 2
serve(
bottle_slug=bottle_slug,
queue_dir=queue_dir,
port=port,
bind=bind,
response_timeout_seconds=response_timeout_seconds,
-189
View File
@@ -1,189 +0,0 @@
"""Shared types and path helpers for supervise stores (PRD 0013).
Extracted from supervise.py so queue_store and audit_store can import
Proposal, Response, AuditEntry, and host_db_path without creating a
circular import (supervise imports from queue_store/audit_store and
vice-versa).
Patching bot_bottle_root on this module propagates through host_db_path
because host_db_path looks up bot_bottle_root via sys.modules[__name__]
at call time rather than capturing it at import time.
"""
from __future__ import annotations
import dataclasses
import sys
import uuid
from dataclasses import dataclass
from datetime import datetime, timezone
from pathlib import Path
HOST_DB_FILENAME = "bot-bottle.db"
TOOL_EGRESS_BLOCK = "egress-block"
TOOL_EGRESS_ALLOW = "egress-allow"
TOOL_GITLEAKS_ALLOW = "gitleaks-allow"
TOOL_EGRESS_TOKEN_ALLOW = "egress-token-allow"
TOOL_LIST_EGRESS_ROUTES = "list-egress-routes"
TOOLS: tuple[str, ...] = (
TOOL_EGRESS_ALLOW,
TOOL_EGRESS_BLOCK,
TOOL_GITLEAKS_ALLOW,
TOOL_EGRESS_TOKEN_ALLOW,
TOOL_LIST_EGRESS_ROUTES,
)
STATUS_APPROVED = "approved"
STATUS_MODIFIED = "modified"
STATUS_REJECTED = "rejected"
STATUSES: tuple[str, ...] = (STATUS_APPROVED, STATUS_MODIFIED, STATUS_REJECTED)
ACTION_OPERATOR_EDIT = "operator-edit"
def bot_bottle_root() -> Path:
return Path.home() / ".bot-bottle"
def host_db_path() -> Path:
# Look up bot_bottle_root through this module's live namespace so that
# monkey-patches on supervise_types.bot_bottle_root take effect.
#
# Lives in its own "db" subdirectory, not directly under
# bot_bottle_root(), so backends that can only bind-mount
# directories (macos_container's `container run --mount`, which
# rejects file sources with "is not a directory") can share this
# one file with a sidecar without also exposing bot_bottle_root()'s
# other contents (git-gate keys, per-bottle state, etc.).
return sys.modules[__name__].bot_bottle_root() / "db" / HOST_DB_FILENAME
def _require_str(raw: dict[str, object], key: str) -> str:
value = raw.get(key)
if not isinstance(value, str):
raise ValueError(f"missing or non-string field {key!r}")
return value
@dataclass(frozen=True)
class Proposal:
"""One pending tool-call from the agent."""
id: str
bottle_slug: str
tool: str
proposed_file: str
justification: str
arrival_timestamp: str
current_file_hash: str
@classmethod
def new(
cls,
*,
bottle_slug: str,
tool: str,
proposed_file: str,
justification: str,
current_file_hash: str,
now: datetime | None = None,
) -> "Proposal":
ts = (now or datetime.now(timezone.utc)).isoformat()
return cls(
id=str(uuid.uuid4()),
bottle_slug=bottle_slug,
tool=tool,
proposed_file=proposed_file,
justification=justification,
arrival_timestamp=ts,
current_file_hash=current_file_hash,
)
def to_dict(self) -> dict[str, object]:
return dataclasses.asdict(self)
@classmethod
def from_dict(cls, raw: dict[str, object]) -> "Proposal":
tool = _require_str(raw, "tool")
if tool not in TOOLS:
raise ValueError(f"tool must be one of {TOOLS}; got {tool!r}")
return cls(
id=_require_str(raw, "id"),
bottle_slug=_require_str(raw, "bottle_slug"),
tool=tool,
proposed_file=_require_str(raw, "proposed_file"),
justification=_require_str(raw, "justification"),
arrival_timestamp=_require_str(raw, "arrival_timestamp"),
current_file_hash=_require_str(raw, "current_file_hash"),
)
@dataclass(frozen=True)
class Response:
"""The operator's decision on a proposal."""
proposal_id: str
status: str
notes: str
final_file: str | None = None
def to_dict(self) -> dict[str, object]:
return dataclasses.asdict(self)
@classmethod
def from_dict(cls, raw: dict[str, object]) -> "Response":
status = _require_str(raw, "status")
if status not in STATUSES:
raise ValueError(
f"response status must be one of {STATUSES}; got {status!r}"
)
final = raw.get("final_file")
if final is not None and not isinstance(final, str):
raise ValueError(
f"final_file must be a string or null; got {type(final).__name__}"
)
return cls(
proposal_id=_require_str(raw, "proposal_id"),
status=status,
notes=_require_str(raw, "notes"),
final_file=final,
)
@dataclass(frozen=True)
class AuditEntry:
"""One row of the per-bottle audit log."""
timestamp: str
bottle_slug: str
component: str
operator_action: str
operator_notes: str
justification: str
diff: str
def to_dict(self) -> dict[str, object]:
return dataclasses.asdict(self)
__all__ = [
"ACTION_OPERATOR_EDIT",
"AuditEntry",
"HOST_DB_FILENAME",
"Proposal",
"Response",
"STATUSES",
"STATUS_APPROVED",
"STATUS_MODIFIED",
"STATUS_REJECTED",
"TOOLS",
"TOOL_EGRESS_ALLOW",
"TOOL_EGRESS_BLOCK",
"TOOL_EGRESS_TOKEN_ALLOW",
"TOOL_GITLEAKS_ALLOW",
"TOOL_LIST_EGRESS_ROUTES",
"bot_bottle_root",
"host_db_path",
]

Some files were not shown because too many files have changed in this diff Show More