Compare commits

..

5 Commits

Author SHA1 Message Date
didericis-codex fe82dbc7b8 refactor(smolmachines): drop obsolete docker-sidecar helpers
lint / lint (push) Successful in 2m5s
test / unit (pull_request) Successful in 1m2s
test / integration (pull_request) Successful in 20s
test / coverage (pull_request) Successful in 1m11s
2026-07-09 18:14:56 +00:00
didericis-codex bfd3e659e8 refactor(smolmachines): inline sidecar proxy host
lint / lint (push) Successful in 2m1s
test / unit (pull_request) Successful in 57s
test / integration (pull_request) Successful in 18s
test / coverage (pull_request) Successful in 1m10s
2026-07-09 13:53:44 -04:00
didericis-codex 9f0d56b75d test(smolmachines): cover sidecar vm launch paths 2026-07-09 13:53:44 -04:00
didericis-codex ce8e2b3874 feat: run smolmachines sidecar as vm 2026-07-09 13:53:44 -04:00
didericis-codex 46a77aec31 docs: draft smolmachines sidecar vm prd 2026-07-09 13:53:44 -04:00
223 changed files with 9521 additions and 12167 deletions
+11 -10
View File
@@ -71,15 +71,11 @@ jobs:
- name: Run integration tests
run: python3 -m unittest discover -t . -s tests/integration -v
# Combined unit+integration coverage report (informational). See
# docs/decisions/0004-coverage-policy.md.
#
# The hard diff-coverage gate (changed lines >= 90%) is DEFERRED: the
# Firecracker backend's VM/SSH orchestration is covered by the integration
# suite, which needs /dev/kvm + the provisioned TAP/nft pool — a
# container-based runner skips it and those lines read uncovered, so the
# gate can't pass here. Re-enabling it on a self-hosted KVM runner is
# tracked separately (see PRD 0069 / #348 and the ci-runner branch).
# Combined unit+integration coverage + the diff-coverage gate.
# See docs/decisions/0004-coverage-policy.md. The hard gate is diff
# coverage (new/changed lines >= 90%); the combined + critical reports
# are informational and degrade gracefully when the runner has no
# Docker (integration tests skip, those modules just read lower).
coverage:
runs-on: ubuntu-latest
steps:
@@ -96,5 +92,10 @@ jobs:
- name: Install dev requirements
run: python3 -m pip install -r requirements-dev.txt
- name: Combined coverage report (unit + integration)
- name: Combined coverage (unit + integration)
run: PYTHON=python3 bash scripts/coverage.sh critical
- name: Diff-coverage gate (changed lines >= 90%)
run: |
git fetch --no-tags origin main:refs/remotes/origin/main
python3 scripts/diff_coverage.py --base origin/main --min 90
+7 -7
View File
@@ -8,12 +8,12 @@ broad permissions inside a sandbox, so a misbehaving agent cannot reach the
host. A Python CLI (entry point `cli.py`, package `bot_bottle/`) orchestrates
the runtime lifecycle and the copying of skills and env vars into it.
The default backend on compatible macOS hosts is macos-container:
agents and gateways run through Apple's `container` CLI without
requiring Docker. On KVM-capable Linux hosts the default is firecracker:
agents run in a Firecracker microVM reached over SSH on a point-to-point
TAP, while the gateway still uses Docker. The legacy Docker
backend remains available with `BOT_BOTTLE_BACKEND=docker` or
`--backend=docker`.
agents and sidecar bundles run through Apple's `container` CLI without
requiring Docker. The smolmachines backend remains available with
`BOT_BOTTLE_BACKEND=smolmachines` or `--backend=smolmachines`; agents
run in a libkrun micro-VM, while the sidecar bundle still uses Docker.
The legacy Docker backend remains available with `BOT_BOTTLE_BACKEND=docker`
or `--backend=docker`.
## Goals
@@ -59,7 +59,7 @@ backend remains available with `BOT_BOTTLE_BACKEND=docker` or
in a PRD, research note, or decision record.
- Low dependencies by default. The project is Python, stdlib-first (no
runtime pip dependencies in the package itself; the only language
runtime is the Python 3.13 used by the CLI + companion containers). Ask before
runtime is the Python 3.13 used by the CLI + sidecars). Ask before
adding new tools, runtimes, or package managers.
- Commit messages follow [Conventional Commits](https://www.conventionalcommits.org/en/v1.0.0/):
`<type>[(scope)][!]: <description>`, where `<type>` is one of `feat`, `fix`,
-21
View File
@@ -1,21 +0,0 @@
# Firecracker single infra-VM image (PRD 0070 Stage B).
#
# The per-host infra VM runs BOTH the orchestrator control plane and the
# gateway data plane in one microVM (see backend/firecracker/infra_vm.py).
# It layers the stdlib-only control-plane source onto the gateway
# data-plane image, so the single VM has mitmproxy / git / gitleaks /
# supervise (gateway) AND `bot_bottle.orchestrator` (control plane) —
# reusing the gateway payload rather than copying it into a third image.
#
# The docker backend keeps the two images separate (a lean orchestrator
# container + a gateway container); this combined image exists only for
# the Firecracker single-VM cut. If the egress blast radius ever warrants
# it, splitting egress back into its own VM is a routing change, not a
# repackaging (see PRD 0070's "secret concentration").
FROM bot-bottle-gateway:latest
# The gateway image copies only its daemon modules flat under /app; the
# control plane needs the whole (stdlib-only) package. Both coexist:
# `/app/gateway_init.py` keeps its flat-sibling imports, and
# `python3 -m bot_bottle.orchestrator` resolves /app/bot_bottle.
COPY bot_bottle /app/bot_bottle
-62
View File
@@ -1,62 +0,0 @@
# Orchestrator control-plane image (PRD 0070, #384) + in-VM agent-image
# builder (PRD 0069 Stage 3).
#
# The per-host orchestrator runs `python3 -m bot_bottle.orchestrator`.
# The `bot_bottle` package is **stdlib-only** by design, so the control
# plane itself needs nothing but a Python runtime — none of the
# gateway's mitmproxy / git / gitleaks payload (that is the separate
# `bot-bottle-gateway` image, Dockerfile.gateway). Splitting them keeps
# the secret-dense control plane (it concentrates every bottle's egress
# tokens — see PRD 0070's "secret concentration") on a minimal
# dependency surface.
#
# The repo is bind-mounted read-only into the container at run time (see
# `orchestrator/lifecycle.py`), so the source is NOT copied in here: the
# image is just the runtime. `ensure_running` recreates the container
# only when the bind-mounted source hash changes (#381), which is why
# the code stays a mount rather than a baked layer.
FROM python:3.12-slim
# --- in-VM agent-image builder (PRD 0069 Stage 3) -------------------
# The Firecracker backend removes the host Docker daemon by building
# users' agent Dockerfiles *inside the orchestrator VM* with buildah
# (rootless, daemonless) instead of on the host. The host then needs no
# Docker daemon and no root-equivalent `docker` group; an untrusted
# Dockerfile builds inside the confined orchestrator VM, never on the
# host — strictly more isolated than host `docker build`.
#
# `git` lets buildah resolve git-context builds and lets the backend
# clone/pull; `ca-certificates` is needed for `FROM` pulls over TLS.
# buildah's runtime helpers (stripped by --no-install-recommends, so
# listed explicitly): `crun` is the OCI runtime that executes build
# steps; `netavark` + `aardvark-dns` are the network backend buildah
# uses to give `FROM` pulls and `RUN` steps network access.
RUN apt-get update \
&& apt-get install -y --no-install-recommends \
buildah crun netavark aardvark-dns git ca-certificates \
&& rm -rf /var/lib/apt/lists/*
# vfs storage + chroot isolation: buildah then needs neither
# fuse-overlayfs / an overlay kernel module nor configured subuid maps
# (newuidmap/newgidmap), so it works unconditionally as root in a
# minimal microVM rootfs. vfs copies layers rather than stacking them —
# slower and heavier than overlay; a build-cache optimization (overlay
# where available, a persistent cache disk) is tracked for later.
ENV STORAGE_DRIVER=vfs \
BUILDAH_ISOLATION=chroot
# No third-party *Python* deps — the control plane stays stdlib only.
WORKDIR /app
# Bake the (stdlib-only) control-plane source so the image is
# self-contained — it runs from a built image, no runtime bind-mount.
# This is what the Firecracker infra VM boots (a guest can't bind-mount
# host source); the docker backend may still bind-mount /app for dev
# live-reload, which simply overlays this baked copy. `.dockerignore`
# keeps .git/docs/*.md out of the context.
COPY bot_bottle /app/bot_bottle
# Documentation only; lifecycle.py overrides the entrypoint to
# `python3 -m bot_bottle.orchestrator` with the runtime flags.
ENTRYPOINT ["python3", "-m", "bot_bottle.orchestrator"]
+7 -16
View File
@@ -1,15 +1,8 @@
# Gateway data-plane image (PRD 0024 bundle shape; PRD 0070 gateway).
# Per-bottle sidecar bundle image (PRD 0024).
#
# The egress / git-gate / supervise *data plane* — one image, run by
# the consolidated per-host gateway (PRD 0070). It is NOT the
# orchestrator control plane: that is the separate, lean
# `bot-bottle-orchestrator` image (Dockerfile.orchestrator, #384), which
# ships only python + the stdlib-only `bot_bottle` package and none of
# this image's mitmproxy / git / gitleaks payload.
#
# Collapses the prior per-daemon images (egress, git-gate,
# Collapses the prior per-sidecar images (egress, git-gate,
# supervise) into one. A small stdlib-Python init supervisor at
# /app/gateway_init.py spawns all daemons, forwards SIGTERM, and
# /app/sidecar_init.py spawns all daemons, forwards SIGTERM, and
# propagates per-daemon stdout/stderr to the container log with a
# `[name]` prefix. See PRD 0024 for the rationale.
#
@@ -19,7 +12,7 @@
# /app/egress_addon.py + siblings mitmproxy addon (egress)
# /app/egress-entrypoint.sh mitmdump launcher
# /app/supervise_server.py + .py supervise MCP server
# /app/gateway_init.py PID 1 supervisor
# /app/sidecar_init.py PID 1 supervisor
# /etc/egress/routes.yaml bind-mounted at run time
# /etc/git-gate/pre-receive docker-cp'd at start time
# /git-gate-entrypoint.sh docker-cp'd at start time
@@ -31,7 +24,7 @@
# Exposed ports inside the container:
# 9099 egress (mitmproxy, agent-facing HTTPS proxy)
# 9418 git-gate (git-daemon)
# 9420 git-gate smart HTTP (VM-backend agent-facing transport)
# 9420 git-gate smart HTTP (smolmachines agent-facing transport)
# 9100 supervise (MCP HTTP)
# Stage 1: gitleaks binary. The upstream gitleaks image is alpine
@@ -71,10 +64,8 @@ COPY --from=gitleaks-src /usr/bin/gitleaks /usr/bin/gitleaks
COPY bot_bottle/egress_addon_core.py /app/egress_addon_core.py
COPY bot_bottle/egress_dlp_config.py /app/egress_dlp_config.py
COPY bot_bottle/egress_addon.py /app/egress_addon.py
COPY bot_bottle/policy_resolver.py /app/policy_resolver.py
COPY bot_bottle/dlp_detectors.py /app/dlp_detectors.py
COPY bot_bottle/yaml_subset.py /app/yaml_subset.py
COPY bot_bottle/paths.py /app/paths.py
COPY bot_bottle/migrations.py /app/migrations.py
COPY bot_bottle/db_store.py /app/db_store.py
COPY bot_bottle/supervise_types.py /app/supervise_types.py
@@ -83,7 +74,7 @@ COPY bot_bottle/audit_store.py /app/audit_store.py
COPY bot_bottle/store_manager.py /app/store_manager.py
COPY bot_bottle/supervise.py /app/supervise.py
COPY bot_bottle/supervise_server.py /app/supervise_server.py
COPY bot_bottle/gateway_init.py /app/gateway_init.py
COPY bot_bottle/sidecar_init.py /app/sidecar_init.py
COPY bot_bottle/git_http_backend.py /app/git_http_backend.py
COPY bot_bottle/egress_entrypoint.sh /app/egress-entrypoint.sh
RUN chmod +x /app/egress-entrypoint.sh
@@ -109,4 +100,4 @@ WORKDIR /app
# PID 1 is the supervisor. It owns signal handling and exit-code
# propagation; no `exec` chain in the entrypoint itself.
ENTRYPOINT ["python3", "/app/gateway_init.py"]
ENTRYPOINT ["python3", "/app/sidecar_init.py"]
+19 -18
View File
@@ -5,7 +5,7 @@
# bot-bottle
[![test](https://gitea.dideric.is/didericis/bot-bottle/actions/workflows/test.yml/badge.svg?branch=main)](https://gitea.dideric.is/didericis/bot-bottle/actions?workflow=test.yml)
[![coverage](https://img.shields.io/badge/coverage-83%25-brightgreen)](https://coverage.readthedocs.io/)
[![coverage](https://img.shields.io/badge/coverage-84%25-brightgreen)](https://coverage.readthedocs.io/)
[![core coverage](https://img.shields.io/badge/core%20coverage-95%25-brightgreen)](https://gitea.dideric.is/didericis/bot-bottle/src/branch/main/docs/decisions/0004-coverage-policy.md)
**Problem:** Developer wants to run a coding agent without supervision, but they don't want a prompt injected or misbehaving agent wrecking their environment or exfiltrating sensitive data.
@@ -16,7 +16,7 @@
- **Per-bottle egress allowlist** — TLS-bumped HTTP/HTTPS chokepoint with a per-manifest host allowlist; per-route path/method/header `matches` filtering; outbound DLP scanning for known tokens and secrets, inbound DLP scanning for prompt-injection attempts; DoH and arbitrary hosts blocked by default.
- **Per-route token-match policy** — each egress route picks what happens when the outbound DLP catches a token via `dlp.outbound_on_match`: `supervise` (default) holds the request and surfaces it in `./cli.py supervise` for approval (an approved value is remembered for the life of the proxy); `redact` scrubs the value and forwards; `block` is a hard `403`. Cuts false-positive friction without weakening default-deny.
- **Tokens the agent never sees** — host secrets live in a gateway; the agent dials `http://gateway:9099/<path>` and the proxy strips inbound `Authorization` and injects the real token before forwarding. `printenv` in the agent shows proxy URLs only.
- **Tokens the agent never sees** — host secrets live in a sidecar; the agent dials `http://sidecar:9099/<path>` and the proxy strips inbound `Authorization` and injects the real token before forwarding. `printenv` in the agent shows proxy URLs only.
- **Gitleaks-scanned push (git-gate)** — `bottle.git` remotes route through a per-bottle `git daemon` that gitleaks-scans incoming refs pre-receive and forwards clean refs upstream over SSH. The agent never holds the upstream credential.
- **Manifest-scoped skills + secrets** — each bottle declares its skills, env, git identity, remotes, and egress routes; unknown keys die at load.
- **Trust boundary at `$HOME`** — bottles (credentials, egress, remotes) live only under `~/.bot-bottle/bottles/`. Repos may ship agents but not bottles, so a cloned repo can't redirect an env var to an attacker host.
@@ -24,17 +24,17 @@
- **Parallel, isolated bottles** — each bottle runs in its own backend-owned isolation boundary; bottles don't share state or talk to each other.
- **Provider templates (Claude, Codex)** — `Dockerfile.claude` / `Dockerfile.codex`, or a bottle-supplied Dockerfile. Claude auth via long-lived OAuth token; Codex via opt-in host device-auth forwarding.
- **gVisor auto-detect** — on Linux hosts where `runsc` is registered with Docker, every bottle launches under it for a userspace syscall barrier; no manifest config required.
- **Apple Container backend (macOS default when available)** — runs the agent and gateway with Apple's `container` CLI, using a host-only agent network plus a separate gateway egress network.
- **Firecracker backend (Linux default when available)** — runs the agent in a KVM Firecracker microVM reached over SSH on a point-to-point TAP, with the gateway in Docker. A dedicated, fail-closed `nftables` table isolates the guest, closing the raw DNS/IP exfiltration gap that exists in the legacy Docker backend. Requires KVM (`/dev/kvm`) and a one-time privileged network-pool setup.
- **Legacy Docker backend** — still available for examples, CI, and hosts without Apple Container or KVM via `BOT_BOTTLE_BACKEND=docker` or `--backend=docker`.
- **Apple Container backend (macOS default when available)** — runs the agent and sidecar bundle with Apple's `container` CLI, using a host-only agent network plus a separate sidecar egress network.
- **Smolmachines backend** — runs the agent in a libkrun micro-VM while the sidecar bundle stays in Docker. TSI and smolmachines DNS filtering close the raw DNS exfiltration gap that exists in the legacy Docker backend. Runs on macOS (Hypervisor.framework) and Linux (KVM, `/dev/kvm`).
- **Legacy Docker backend** — still available for examples, CI, and hosts without Apple Container via `BOT_BOTTLE_BACKEND=docker` or `--backend=docker`.
## Architecture
On the default macOS Apple Container backend, a bottle is an agent container on a host-only internal network plus a gateway attached to both that internal network and a NAT egress network. The agent gets HTTP(S)_PROXY and CA bundle env vars pointing at the gateway's internal-network IP, so HTTP/HTTPS traffic flows through the gateway instead of direct egress. `bottle.git` / git-gate is intentionally deferred on this backend until a safe Apple Container key-delivery path exists.
On the default macOS Apple Container backend, a bottle is an agent container on a host-only internal network plus a sidecar bundle attached to both that internal network and a NAT egress network. The agent gets HTTP(S)_PROXY and CA bundle env vars pointing at the sidecar's internal-network IP, so HTTP/HTTPS traffic flows through the sidecar instead of direct egress. `bottle.git` / git-gate is intentionally deferred on this backend until a safe Apple Container key-delivery path exists.
On the Firecracker backend, a bottle is an agent microVM plus a Docker gateway for egress, git-gate, and supervise. The VM reaches the gateway over a per-bottle point-to-point TAP link; a dedicated fail-closed `nftables` table (`inet bot_bottle_fc`) confines the guest to that link, so nothing leaves the box except through the gateway. The TAP pool and nft table are provisioned once (root); per-launch needs no privilege.
On the smolmachines backend, a bottle is an agent micro-VM plus a Docker sidecar bundle for egress, git-gate, and supervise. The VM reaches the sidecars through a per-bottle loopback alias allowed by TSI; smolmachines handles DNS filtering below the guest OS.
On the legacy Docker backend, the same logical bottle is two containers per agent: an `agent` container and a `companion containers` container. They share a per-agent Docker `--internal` network; the agent has no default route off-box.
On the legacy Docker backend, the same logical bottle is two containers per agent: an `agent` container and a `sidecars` container. They share a per-agent Docker `--internal` network; the agent has no default route off-box.
The Docker topology looks like this:
@@ -67,28 +67,29 @@ The Docker topology looks like this:
└─────────────────────────────────────────────────────────────────────┘
```
When the agent exits, `cli.py` tears down every gateway and both networks; nothing about a bottle persists between runs.
When the agent exits, `cli.py` tears down every sidecar and both networks; nothing about a bottle persists between runs.
## Quickstart
On compatible macOS hosts, the default backend requires Apple's `container` CLI and does not require Docker. The Firecracker backend (Linux) requires Docker on the host for the gateway plus the `firecracker` binary and KVM. The legacy Docker backend requires Docker. Claude bottles also need a long-lived Claude Code OAuth token (`claude setup-token`) exported as `BOT_BOTTLE_CLAUDE_OAUTH_TOKEN`.
On compatible macOS hosts, the default backend requires Apple's `container` CLI and does not require Docker. The smolmachines backend requires Docker on the host for the sidecar bundle plus `smolvm` (macOS or Linux). The legacy Docker backend requires Docker. Claude bottles also need a long-lived Claude Code OAuth token (`claude setup-token`) exported as `BOT_BOTTLE_CLAUDE_OAUTH_TOKEN`.
Use `BOT_BOTTLE_BACKEND=docker ./cli.py start <agent>` on hosts where neither Apple Container nor KVM is available and Docker is the desired backend.
Use `BOT_BOTTLE_BACKEND=docker ./cli.py start <agent>` on hosts where Apple Container is not installed and Docker is the desired backend.
### Firecracker on Linux
### smolmachines on Linux
On Linux, a KVM-capable host defaults to the Firecracker backend. It needs:
The smolmachines backend runs on Linux as well as macOS. On Linux, `smolvm`/libkrun use KVM, so the host needs:
- **`/dev/kvm`** present and accessible. Load `kvm-intel` or `kvm-amd` (and enable virtualization in BIOS/firmware). The invoking user must be in the `kvm` group: `sudo usermod -aG kvm "$USER"` then re-login. bot-bottle preflights this and reports exactly what's missing.
- **`firecracker`** on `PATH`: grab a release from <https://github.com/firecracker-microvm/firecracker/releases>. Start flows print this pointer when the binary is missing.
- **Docker** for the gateway and image build.
- **A one-time privileged network setup** — the per-bottle TAP pool plus the fail-closed `nftables` isolation table. Run `./cli.py backend setup --backend=firecracker` for the host-appropriate config (a NixOS module, a `sudo` script elsewhere); `./cli.py backend status --backend=firecracker` reports what's present, including whether the pool range collides with an existing route. The pool defaults to `10.243.0.0/16` (an obscure RFC-1918 block that dodges docker/libvirt/LAN and, deliberately, Tailscale's `100.64.0.0/10` CGNAT range); override with `BOT_BOTTLE_FC_IP_BASE` if it clashes on your host.
- **`smolvm`** on `PATH`: `curl -sSL https://smolmachines.com/install.sh | sh`.
- **Docker** for the sidecar bundle and image build, same as macOS.
Per-bottle isolation works the same as macOS without any `ifconfig`/sudo step — all of `127.0.0.0/8` is already loopback on Linux, so each bottle's sidecar bundle is published on its own `127.0.0.<N>` and TSI's allowlist is scoped to that `/32`.
```sh
BOT_BOTTLE_BACKEND=firecracker ./cli.py start <agent>
BOT_BOTTLE_BACKEND=smolmachines ./cli.py start <agent>
```
> **NixOS:** enable `virtualisation.docker`, ensure the KVM module is loaded (`boot.kernelModules = [ "kvm-intel" ];` or `kvm-amd`), and add your user to the `kvm` and `docker` groups. For the network pool, consume the flake module — `imports = [ inputs.bot-bottle.nixosModules.firecracker-netpool ]; services.bot-bottle-firecracker = { enable = true; owner = "you"; };` — then `nixos-rebuild switch` (imperative nft/TAP rules don't survive a rebuild; channel users can `imports = [ <bot-bottle>/nix/firecracker-netpool.nix ]`). `firecracker` isn't in nixpkgs by default as a user binary — install the release binary (pin the version) and put it on `PATH`.
> **NixOS:** enable `virtualisation.docker`, ensure the KVM module is loaded (`boot.kernelModules = [ "kvm-intel" ];` or `kvm-amd`), and add your user to the `kvm` and `docker` groups. If you run bottles from a Gitea Actions runner, use a `host`-label runner so Docker, `smolvm`, and `/dev/kvm` are all reachable from the job. `smolvm` isn't in nixpkgs — install the release binary (pin the version) and put it on the runner's `PATH`.
```sh
./cli.py start <agent> # builds the image on first run, drops you into claude
+5 -12
View File
@@ -61,13 +61,6 @@ class AgentProviderRuntime:
prompt_mode: PromptMode
bypass_args: tuple[str, ...]
resume_args: tuple[str, ...]
# argv run inside a throwaway container of a freshly built agent
# image, right after `build_image()`, to catch a build that
# exited 0 but produced a broken CLI (e.g. an npm
# optionalDependencies fetch for a platform-native binary that
# silently no-ops on a transient failure). Empty tuple skips the
# check — not every provider has opted in yet.
smoke_test: tuple[str, ...] = ()
@dataclass(frozen=True)
@@ -211,9 +204,9 @@ class AgentProvider(ABC):
bottle: "Bottle",
supervise_url: str,
) -> None:
"""Register the per-bottle supervise daemon as an MCP server
"""Register the per-bottle supervise sidecar as an MCP server
in the provider's in-guest config. Called by the backend after
the supervise daemon is reachable. No-op when
the supervise sidecar is reachable. No-op when
`plan.supervise_plan is None`."""
@abstractmethod
@@ -234,9 +227,9 @@ class AgentProvider(ABC):
from .backend.util import AGENT_CA_PATH, log_ca_fingerprint, select_ca_cert
from .log import die
cert_host_path, label = select_ca_cert(plan.egress_plan)
# Ensure the target directory exists. A backend's rootfs build
# may not preserve the empty /usr/local/share/ca-certificates/
# directory; mkdir -p is idempotent and safe for all backends.
# Ensure the target directory exists. smolvm's pack step may not
# preserve the empty /usr/local/share/ca-certificates/ directory
# on Linux; mkdir -p is idempotent and safe for all backends.
bottle.exec("mkdir -p /usr/local/share/ca-certificates", user="root")
bottle.cp_in(str(cert_host_path), AGENT_CA_PATH)
r = bottle.exec(
+2 -4
View File
@@ -6,13 +6,11 @@ import sqlite3
from pathlib import Path
try:
from .supervise_types import AuditEntry
from .paths import host_db_path
from .supervise_types import AuditEntry, host_db_path
from .db_store import DbStore
from .migrations import TableMigrations
except ImportError:
from supervise_types import AuditEntry # type: ignore[import-not-found] # pylint: disable=import-error,no-name-in-module
from paths import host_db_path # type: ignore[import-not-found] # pylint: disable=import-error,no-name-in-module
from supervise_types import AuditEntry, host_db_path # type: ignore[import-not-found] # pylint: disable=import-error,no-name-in-module
from db_store import DbStore # type: ignore[import-not-found] # pylint: disable=import-error,no-name-in-module
from migrations import TableMigrations # type: ignore[import-not-found] # pylint: disable=import-error,no-name-in-module
+23 -64
View File
@@ -26,9 +26,8 @@ backend exposes five methods:
Selection is driven by `--backend` on `start` or BOT_BOTTLE_BACKEND
(env var). When neither is set, compatible macOS hosts default to
`macos-container`; Linux hosts with KVM default to `firecracker`;
otherwise `docker`. Per PRD 0003 the manifest does not carry a
backend field; the host picks.
`macos-container`; other hosts default to `smolmachines`. Per PRD 0003
the manifest does not carry a backend field; the host picks.
"""
from __future__ import annotations
@@ -98,14 +97,14 @@ class BottlePlan(ABC):
@property
def git_gate_insteadof_host(self) -> str:
"""Host (and optional port) used in git-gate insteadOf URLs.
Docker uses the compose-network DNS alias; VM backends may
override with an IP:port when the guest has no DNS."""
Docker uses the compose-network DNS alias; smolmachines
overrides with a loopback IP:port since TSI has no DNS."""
return "git-gate"
@property
def git_gate_insteadof_scheme(self) -> str:
"""URL scheme for git-gate insteadOf rewrites. 'git' for
Docker (git daemon); VM backends may override (e.g. 'http'
Docker (git daemon); 'http' for smolmachines (HTTP proxy
over a published host port)."""
return "git"
egress_plan: EgressPlan
@@ -183,8 +182,8 @@ class BottleCleanupPlan(ABC):
class ExecResult:
"""Captured result of `Bottle.exec`. Backend-neutral: the Docker
impl populates it from a `subprocess.CompletedProcess`, but a
VM backend could populate it from any source that produces a
returncode + captured streams."""
future fly/smolmachines backend could populate it from any source
that produces a returncode + captured streams."""
returncode: int
stdout: str
@@ -199,10 +198,10 @@ class ActiveAgent:
bottle is the container, the agent is what runs in it.)
Fields are deliberately backend-neutral. `services` is the set
of gateway daemons currently up for this bottle (`egress`,
of sidecar daemons currently up for this bottle (`egress`,
`git-gate`, `supervise`); the dashboard uses it to
gate edit verbs. `backend_name` is the matching key in
`_BACKENDS` (`docker` / `firecracker` / `macos-container`) — used by the active-
`_BACKENDS` (`docker` / `smolmachines` / `macos-container`) — used by the active-
list rendering to disambiguate and by the dashboard's
re-attach path."""
@@ -251,7 +250,7 @@ class Bottle(ABC):
`user` (default `node`, matching the agent image's USER
directive) and return the captured stdout/stderr/returncode.
The bottle's environment (including HTTPS_PROXY pointing at
the egress daemon) is inherited by the child. Non-zero
the egress sidecar) is inherited by the child. Non-zero
exit does not raise — callers inspect `returncode`
themselves.
@@ -454,7 +453,7 @@ class BottleBackend(ABC, Generic[PlanT, CleanupT]):
declarative provision-plan apply, supervise MCP registration)
live on the `AgentProvider` plugin. The backend only owns the
steps that are about backend infrastructure (CA, workspace,
git) and surfaces the supervise daemon URL its launch step
git) and surfaces the supervise sidecar URL its launch step
knows about via `supervise_mcp_url`.
PRD 0017: cred-proxy's agent-side dotfile rewrites (~/.npmrc,
@@ -502,12 +501,12 @@ class BottleBackend(ABC, Generic[PlanT, CleanupT]):
def supervise_mcp_url(self, plan: PlanT) -> str:
"""Return the agent-side URL of the per-bottle supervise
gateway, or "" when this bottle has no gateway. The provider
sidecar, or "" when this bottle has no sidecar. The provider
plugin's `provision_supervise_mcp` uses it to register the
MCP entry inside the guest.
Default returns "" so backends without supervise support
don't have to implement it. Docker and firecracker override."""
don't have to implement it. Docker and smolmachines override."""
del plan
return ""
@@ -524,60 +523,27 @@ class BottleBackend(ABC, Generic[PlanT, CleanupT]):
def enumerate_active(self) -> Sequence[ActiveAgent]:
"""Return every currently-running agent on this backend.
Empty when none. Backend-specific: docker queries `docker
compose ls`; firecracker cross-references its running gateway
containers against per-bottle metadata."""
compose ls`; smolmachines queries `smolvm machine ls --json`
+ cross-references its bundle container."""
@classmethod
@abstractmethod
def is_available(cls) -> bool:
"""Whether this backend's runtime prerequisites are satisfied
on the current host. Docker → `docker` on PATH; firecracker →
Linux + KVM. Used by the cross-backend
on the current host. Docker → `docker` on PATH; smolmachines
→ `smolvm` on PATH. Used by the cross-backend
`enumerate_active_agents` / `cmd_cleanup` to skip backends
the operator hasn't installed, so a docker-only host
doesn't fail when `cli.py list active` walks past
firecracker."""
@classmethod
@abstractmethod
def setup(cls) -> int:
"""Emit this backend's one-time host setup — privileged network
pool, daemon bring-up, install pointers, etc. — as
host-appropriate config or commands. Prints to stdout/stderr and
returns a shell exit code (0 = nothing to report / success). A
backend that needs no host setup prints a short note and returns
0. Invoked generically by `./cli.py backend setup [--backend=…]`
so operators can provision any backend without a
backend-specific command. Classmethod (like `is_available`) —
it's a host query, not per-bottle state."""
@classmethod
@abstractmethod
def status(cls) -> int:
"""Report whether this backend's prerequisites are satisfied on
the host — binaries, daemon reachability, network pool, range
conflicts, etc. Prints a human-readable summary; returns 0 when
the backend is ready to launch and non-zero when something is
missing. Invoked by `./cli.py backend status [--backend=…]`."""
@classmethod
@abstractmethod
def teardown(cls) -> int:
"""Undo `setup()` — the inverse operation, surfaced as
`./cli.py backend teardown [--backend=…]` (uninstall). Symmetric
with setup: where setup is advisory (prints the privileged
commands / declarative config to apply), teardown prints the
commands / config change to remove the host prerequisites. A
backend with no host setup prints a short note and returns 0.
Not called by the launch path or the test suite."""
smolmachines."""
# Import concrete backend classes AFTER the base types are defined, so
# each backend module can pull BottleSpec / BottlePlan / BottleBackend
# via `from . import ...` without hitting a partially-initialized module.
from .docker import DockerBottleBackend # noqa: E402 # pylint: disable=wrong-import-position
from .firecracker import FirecrackerBottleBackend # noqa: E402 # pylint: disable=wrong-import-position
from .macos_container import MacosContainerBottleBackend # noqa: E402 # pylint: disable=wrong-import-position
from .smolmachines import SmolmachinesBottleBackend # noqa: E402 # pylint: disable=wrong-import-position
# Freezer is imported after the backend classes for the same reason:
# Freezer.commit_slug constructs ActiveAgent, which must be fully
@@ -591,8 +557,8 @@ from .freeze import CommitCancelled, Freezer, get_freezer # noqa: E402 # pylin
# unparameterized methods (prepare → plan → launch(plan), cleanup, etc.).
_BACKENDS: dict[str, BottleBackend[Any, Any]] = {
"docker": DockerBottleBackend(),
"firecracker": FirecrackerBottleBackend(),
"macos-container": MacosContainerBottleBackend(),
"smolmachines": SmolmachinesBottleBackend(),
}
@@ -605,8 +571,7 @@ def get_bottle_backend(
1. explicit arg (CLI `--backend=<name>` passes through here)
2. BOT_BOTTLE_BACKEND env var
3. `macos-container` on compatible macOS hosts
4. `firecracker` on KVM-capable Linux hosts
5. default `docker`
4. default `smolmachines`
Dies with a pointer at the known backends if the chosen name
isn't implemented."""
@@ -620,13 +585,7 @@ def get_bottle_backend(
def _default_backend_name() -> str:
if has_backend("macos-container"):
return "macos-container"
# A KVM-capable Linux host defaults to firecracker even when the
# `firecracker` binary isn't installed yet: selecting it here routes
# start through firecracker's preflight, which prints an install
# pointer, instead of silently falling back to docker.
if FirecrackerBottleBackend.is_host_capable():
return "firecracker"
return "docker"
return "smolmachines"
def known_backend_names() -> tuple[str, ...]:
@@ -640,7 +599,7 @@ def has_backend(name: str) -> bool:
"""Whether the named backend's runtime prerequisites are
available on the current host. Cross-backend callers (list,
cleanup) skip unavailable backends so a docker-only host
doesn't fail when the firecracker backend isn't usable,
doesn't fail when the smolmachines backend isn't installed,
and vice versa.
Returns False for unknown names so callers can pass
+1 -21
View File
@@ -54,21 +54,6 @@ class DockerBottleBackend(BottleBackend["DockerBottlePlan", "DockerBottleCleanup
launch."""
return shutil.which("docker") is not None
@classmethod
def setup(cls) -> int:
from . import setup as _setup
return _setup.setup()
@classmethod
def status(cls) -> int:
from . import setup as _setup
return _setup.status()
@classmethod
def teardown(cls) -> int:
from . import setup as _setup
return _setup.teardown()
def _preflight(self) -> None:
_resolve_plan.preflight()
@@ -106,16 +91,11 @@ class DockerBottleBackend(BottleBackend["DockerBottlePlan", "DockerBottleCleanup
yield bottle
def supervise_mcp_url(self, plan: DockerBottlePlan) -> str:
"""Docker bottles reach the supervise daemon via the
"""Docker bottles reach the supervise sidecar via the
compose-network alias `supervise:9100`. No per-bottle URL
plumbing needed; the alias resolves inside the bridge."""
if plan.supervise_plan is None:
return ""
# Consolidated: the supervise daemon lives on the shared gateway, so
# the agent registers the gateway address (NO_PROXY bypasses egress),
# not the per-bottle `supervise` alias.
if plan.agent_supervise_url:
return plan.agent_supervise_url
return f"http://{SUPERVISE_HOSTNAME}:{SUPERVISE_PORT}/"
def prepare_cleanup(self) -> DockerBottleCleanupPlan:
-19
View File
@@ -28,30 +28,11 @@ class DockerBottlePlan(BottlePlan):
# accidental log of the plan dataclass.
forwarded_env: dict[str, str] = field(repr=False)
use_runsc: bool
# Consolidated mode (PRD 0070): the agent reaches git-gate over HTTP at the
# shared gateway's address (`http://<gateway_ip>:9420`), set at launch.
# Empty → single-tenant defaults (the `git-gate` alias over git://).
agent_git_gate_url: str = ""
# Likewise the supervise MCP endpoint at the gateway (`http://<gw>:9100/`);
# empty → the single-tenant `supervise` alias.
agent_supervise_url: str = ""
@property
def container_name(self) -> str:
return self.agent_provision.instance_name
@property
def git_gate_insteadof_host(self) -> str:
if self.agent_git_gate_url.startswith("http://"):
return self.agent_git_gate_url.removeprefix("http://").rstrip("/")
return super().git_gate_insteadof_host
@property
def git_gate_insteadof_scheme(self) -> str:
if self.agent_git_gate_url.startswith("http://"):
return "http"
return super().git_gate_insteadof_scheme
@property
def image(self) -> str:
return self.agent_provision.image
+7 -6
View File
@@ -18,7 +18,8 @@ scan, just as a fallback bucket alongside the project list.
`cleanup` removes everything in the plan.
Active-agent enumeration lives in `backend/docker/enumerate.py`.
Active-agent enumeration lives in `backend/docker/enumerate.py`
(mirror of `backend/smolmachines/enumerate.py`).
"""
from __future__ import annotations
@@ -26,7 +27,7 @@ from __future__ import annotations
import shutil
import subprocess
from ...paths import bot_bottle_root
from ... import supervise as _supervise
from ...log import info, warn
from . import util as docker_mod
from .bottle_cleanup_plan import DockerBottleCleanupPlan
@@ -92,9 +93,9 @@ def _list_orphan_state_dirs(
`protected_identities` is the set of slugs that are live in
ANY backend — used so this docker-side check doesn't reap a
running non-docker bottle's state dir (the layout is shared
across backends)."""
state_root = bot_bottle_root() / "state"
running smolmachines bottle's state dir (the layout is shared
across both backends)."""
state_root = _supervise.bot_bottle_root() / "state"
if not state_root.is_dir():
return []
orphans: list[str] = []
@@ -118,7 +119,7 @@ def prepare_cleanup() -> DockerBottleCleanupPlan:
Pulls the union of live identities across backends via
`enumerate_active_agents()` so the orphan-state-dir bucket
doesn't include slugs whose non-docker bottle is still up."""
doesn't include slugs whose smolmachines VM is still up."""
docker_mod.require_docker()
projects = list_compose_projects()
project_set = set(projects)
+243 -6
View File
@@ -1,10 +1,20 @@
"""Docker compose lifecycle helpers (PRD 0018).
"""Compose-spec rendering for a Docker bottle (PRD 0018, chunk 1).
Serialize a compose spec to disk, drive `docker compose up/down`,
dump the merged log on teardown, and enumerate `bot-bottle-*`
projects. The spec itself is built by `consolidated_compose.py`
(the consolidated per-host gateway topology); this module owns the
I/O side that persists and runs it.
`bottle_plan_to_compose(plan)` returns a Compose v2 spec dict
describing the per-bottle container topology — one project per
bottle instance, services for the agent + every applicable sidecar,
two networks, no named volumes.
Pure function. No I/O, no subprocess. Expects every launch-time
field (network names, CA host paths, etc.) on the plan's inner
plans to be populated; chunks 2+3 own that ordering.
Conditional services follow the plan content:
- agent + sidecars bundle: always.
- git-gate: iff plan.git_gate_plan.upstreams.
- egress: iff plan.egress_plan.routes.
- supervise: iff plan.supervise_plan is not None.
"""
from __future__ import annotations
@@ -15,7 +25,233 @@ import sys
from pathlib import Path
from typing import Any
from ...egress import (
EGRESS_HOSTNAME,
EGRESS_ROUTES_IN_CONTAINER,
egress_agent_env_entries,
egress_sidecar_env_entries,
)
from ...git_gate import GIT_GATE_HOSTNAME
from ...log import die, warn
from ...supervise import (
DB_PATH_IN_CONTAINER,
SUPERVISE_HOSTNAME,
SUPERVISE_PORT,
)
from ...util import expand_tilde
from ..util import AGENT_CA_BUNDLE, AGENT_CA_PATH
from .bottle_plan import DockerBottlePlan
from .egress import (
EGRESS_CA_IN_CONTAINER,
EGRESS_PORT,
)
from .git_gate import (
GIT_GATE_ACCESS_HOOK_IN_CONTAINER,
GIT_GATE_CREDS_DIR_IN_CONTAINER,
GIT_GATE_ENTRYPOINT_IN_CONTAINER,
GIT_GATE_HOOK_IN_CONTAINER,
)
from . import network as network_mod
from .sidecar_bundle import (
SIDECAR_BUNDLE_DOCKERFILE,
SIDECAR_BUNDLE_IMAGE,
sidecar_bundle_container_name,
)
# Repo root, used as the build context for the bundle Dockerfile.
_REPO_DIR = str(Path(__file__).resolve().parent.parent.parent.parent)
def bottle_plan_to_compose(plan: DockerBottlePlan) -> dict[str, Any]:
"""Render a Compose v2 spec dict from a fully-resolved
DockerBottlePlan.
The plan must have its inner plans (`git_gate_plan`,
`egress_plan`, `supervise_plan`) populated with launch-time
fields — network names, CA host paths. The renderer doesn't
validate; callers feed it a fully-resolved plan or get an
incomplete compose spec back.
"""
project = f"bot-bottle-{plan.slug}"
services: dict[str, Any] = {
"sidecars": _sidecar_bundle_service(plan),
"agent": _agent_service(plan),
}
return {
"name": project,
"services": services,
"networks": _networks(plan),
}
def _networks(plan: DockerBottlePlan) -> dict[str, Any]:
"""Compose-managed networks with explicit `name:` matching the
existing slug-suffixed convention. Compose creates them on `up`
and destroys them on `down`. The internal one is `--internal`
(no default gateway); the egress one is a normal user-defined
bridge."""
return {
"internal": {
"name": network_mod.network_name_for_slug(plan.slug),
"internal": True,
},
"egress": {
"name": network_mod.network_egress_name_for_slug(plan.slug),
},
}
def _bind(host: str | Path, target: str, *, read_only: bool = True) -> dict[str, Any]:
"""One bind-mount entry in the long-form `volumes:` shape.
Long form is preferred over `host:target:ro` strings because
it's easier to inspect in tests and survives whitespace in
host paths."""
return {
"type": "bind",
"source": str(host),
"target": target,
"read_only": read_only,
}
def _sidecar_bundle_service(plan: DockerBottlePlan) -> dict[str, Any]:
"""The `sidecars` service: one container per bottle, bundle
image, all daemons under a Python init supervisor.
Daemon subset narrows via `BOT_BOTTLE_SIDECAR_DAEMONS` env.
egress is always present; git-gate / supervise are conditional.
"""
daemons: list[str] = ["egress"]
if plan.git_gate_plan.upstreams:
daemons.append("git-gate")
if plan.supervise_plan is not None:
daemons.append("supervise")
env: list[str] = [f"BOT_BOTTLE_SIDECAR_DAEMONS={','.join(daemons)}"]
volumes: list[dict[str, Any]] = []
# --- egress -------------------------------------------------------
ep = plan.egress_plan
volumes.append(_bind(ep.mitmproxy_ca_host_path, EGRESS_CA_IN_CONTAINER))
if ep.routes:
volumes.append(_bind(ep.routes_path.parent, str(Path(EGRESS_ROUTES_IN_CONTAINER).parent)))
env.extend(egress_sidecar_env_entries(ep))
# --- git-gate -----------------------------------------------------
gp = plan.git_gate_plan
if gp.upstreams:
volumes += [
_bind(gp.entrypoint_script, GIT_GATE_ENTRYPOINT_IN_CONTAINER),
_bind(gp.hook_script, GIT_GATE_HOOK_IN_CONTAINER),
_bind(gp.access_hook_script, GIT_GATE_ACCESS_HOOK_IN_CONTAINER),
]
for u in gp.upstreams:
keypath = expand_tilde(u.identity_file)
volumes.append(_bind(
keypath,
f"{GIT_GATE_CREDS_DIR_IN_CONTAINER}/{u.name}-key",
))
if u.known_hosts_file:
volumes.append(_bind(
u.known_hosts_file,
f"{GIT_GATE_CREDS_DIR_IN_CONTAINER}/{u.name}-known_hosts",
))
# --- supervise ----------------------------------------------------
sp = plan.supervise_plan
if sp is not None:
env += [
f"SUPERVISE_BOTTLE_SLUG={plan.slug}",
f"SUPERVISE_DB_PATH={DB_PATH_IN_CONTAINER}",
f"SUPERVISE_PORT={SUPERVISE_PORT}",
]
volumes.append({
"type": "bind",
"source": str(sp.db_path),
"target": DB_PATH_IN_CONTAINER,
"read_only": False,
})
internal_aliases = [EGRESS_HOSTNAME]
if gp.upstreams:
internal_aliases.append(GIT_GATE_HOSTNAME)
if sp is not None:
internal_aliases.append(SUPERVISE_HOSTNAME)
service: dict[str, Any] = {
"image": SIDECAR_BUNDLE_IMAGE,
"build": {
"context": _REPO_DIR,
"dockerfile": SIDECAR_BUNDLE_DOCKERFILE,
},
"container_name": sidecar_bundle_container_name(plan.slug),
"networks": {
"internal": {"aliases": internal_aliases},
"egress": None,
},
"environment": env,
"volumes": volumes,
}
return service
def _agent_service(plan: DockerBottlePlan) -> dict[str, Any]:
"""Agent container. Runs `sleep infinity`; claude is `docker
exec -it`'d into it later. HTTP_PROXY/HTTPS_PROXY point at the
egress sidecar."""
proxy_url = _agent_proxy_url(plan)
no_proxy = _agent_no_proxy(plan)
env: list[str] = [
f"HTTPS_PROXY={proxy_url}",
f"HTTP_PROXY={proxy_url}",
f"https_proxy={proxy_url}",
f"http_proxy={proxy_url}",
f"NO_PROXY={no_proxy}",
f"no_proxy={no_proxy}",
f"NODE_EXTRA_CA_CERTS={AGENT_CA_PATH}",
f"SSL_CERT_FILE={AGENT_CA_BUNDLE}",
f"REQUESTS_CA_BUNDLE={AGENT_CA_BUNDLE}",
]
for name, value in sorted(plan.agent_provision.guest_env.items()):
env.append(f"{name}={value}")
# Forwarded vars (OAuth token, manifest host-interpolations):
# bare name → inherits from compose-up process env, value
# never lands on argv or in the compose file.
for name in sorted(plan.forwarded_env.keys()):
env.append(name)
env.extend(egress_agent_env_entries(plan.egress_plan))
service: dict[str, Any] = {
"image": plan.image,
"container_name": plan.container_name,
"command": ["sleep", "infinity"],
"networks": {"internal": None},
"environment": env,
}
if plan.use_runsc:
service["runtime"] = "runsc"
# The init supervisor inside the bundle owns intra-bundle
# daemon ordering, so the agent only waits for the bundle
# container itself.
service["depends_on"] = ["sidecars"]
return service
def _agent_proxy_url(plan: DockerBottlePlan) -> str:
"""Agent's HTTP_PROXY — always points at egress."""
return f"http://{EGRESS_HOSTNAME}:{EGRESS_PORT}"
def _agent_no_proxy(plan: DockerBottlePlan) -> str:
"""NO_PROXY for the agent: loopback always; supervise hostname
when the supervise sidecar is up (MCP long-poll must bypass
the egress proxy)."""
hosts = ["localhost", "127.0.0.1"]
if plan.supervise_plan is not None:
hosts.append(SUPERVISE_HOSTNAME)
return ",".join(hosts)
# --- Lifecycle helpers (PRD 0018 chunk 3) ----------------------------------
@@ -206,6 +442,7 @@ __all__ = [
"COMPOSE_FILE_NAME",
"COMPOSE_LOG_NAME",
"COMPOSE_PROJECT_PREFIX",
"bottle_plan_to_compose",
"compose_down",
"compose_dump_logs",
"compose_file_path",
@@ -1,78 +0,0 @@
"""Agent-only compose for the consolidated docker backend (PRD 0070).
The per-bottle model rendered a compose project with the agent *and* a
gateway on two per-bottle networks. In the consolidated model the
per-bottle companion containers are gone — one shared gateway serves every bottle — so this renders
just the agent, attached to the **external shared gateway network** with the
pinned source IP the orchestrator allocated, and pointed at the gateway's
address for egress (and, around the proxy, for git-http / supervise).
Pure: it takes the launch-time `LaunchContext` values (gateway address,
source IP, network) and the prepared plan, and returns a compose dict — no
docker, so it's testable in isolation.
"""
from __future__ import annotations
from typing import Any
from ...egress import egress_agent_env_entries
from ..util import AGENT_CA_BUNDLE, AGENT_CA_PATH
from .bottle_plan import DockerBottlePlan
from .egress import EGRESS_PORT
def consolidated_agent_compose(
plan: DockerBottlePlan,
*,
gateway_ip: str,
source_ip: str,
network: str,
) -> dict[str, Any]:
"""A compose spec with only the agent service, on the external gateway
network at `source_ip`, proxying egress through `gateway_ip`."""
proxy_url = f"http://{gateway_ip}:{EGRESS_PORT}"
# git-http + supervise live on the gateway too and must NOT go through the
# egress proxy — the agent reaches them directly by the gateway address.
no_proxy = f"localhost,127.0.0.1,{gateway_ip}"
env: list[str] = [
f"HTTPS_PROXY={proxy_url}",
f"HTTP_PROXY={proxy_url}",
f"https_proxy={proxy_url}",
f"http_proxy={proxy_url}",
f"NO_PROXY={no_proxy}",
f"no_proxy={no_proxy}",
f"NODE_EXTRA_CA_CERTS={AGENT_CA_PATH}",
f"SSL_CERT_FILE={AGENT_CA_BUNDLE}",
f"REQUESTS_CA_BUNDLE={AGENT_CA_BUNDLE}",
]
for name, value in sorted(plan.agent_provision.guest_env.items()):
env.append(f"{name}={value}")
# Forwarded vars: bare name → inherits from the compose-up process env so
# the secret value never lands on argv or in the compose file.
for name in sorted(plan.forwarded_env.keys()):
env.append(name)
env.extend(egress_agent_env_entries(plan.egress_plan))
service: dict[str, Any] = {
"image": plan.image,
"container_name": plan.container_name,
"command": ["sleep", "infinity"],
# Pinned address on the shared gateway network — the orchestrator
# registered this IP, and the gateway attributes the bottle by it.
"networks": {network: {"ipv4_address": source_ip}},
"environment": env,
}
if plan.use_runsc:
service["runtime"] = "runsc"
return {
"name": f"bot-bottle-{plan.slug}",
"services": {"agent": service},
# The gateway network is created + owned by the orchestrator; compose
# attaches to it (external) and must not create or destroy it.
"networks": {network: {"external": True}},
}
__all__ = ["consolidated_agent_compose"]
@@ -1,158 +0,0 @@
"""Consolidated bottle launch sequence for the docker backend (PRD 0070).
Composes the orchestrator primitives into the register/teardown sequence that
replaces the per-bottle gateway:
1. ensure the orchestrator control plane + shared gateway are up;
2. allocate the bottle a pinned source IP on the gateway network (the
attribution key), skipping the gateway's own address + live bottles;
3. register it (egress policy blob + slug metadata) → bottle id + identity
token;
4. provision its git-gate repos/creds into the running gateway.
It returns a `LaunchContext` with everything the agent container needs to
attach — network, pinned IP, the gateway's address (its proxy target), the
orchestrator URL, and the identity token. The agent `docker run` itself is
the backend's job (it owns provider provisioning); this owns the
orchestrator-facing wiring so that sequence stays testable in isolation.
"""
from __future__ import annotations
from dataclasses import dataclass
from ...docker_cmd import run_docker
from ...egress import EgressPlan
from ...git_gate import GitGatePlan
from ...orchestrator.client import OrchestratorClient
from ...orchestrator.gateway import GATEWAY_NAME, GATEWAY_NETWORK
from ...orchestrator.lifecycle import OrchestratorService
from ...orchestrator.registration import registration_inputs
from .gateway_net import next_free_ip
from .gateway_provision import (
DockerGatewayTransport,
deprovision_git_gate,
provision_git_gate,
)
class ConsolidatedLaunchError(RuntimeError):
"""The consolidated register/provision sequence could not complete."""
@dataclass(frozen=True)
class LaunchContext:
"""What the agent container needs to join the shared gateway."""
bottle_id: str
identity_token: str
source_ip: str # the agent's pinned address (attribution key)
network: str # the shared gateway network to attach to
gateway_ip: str # the gateway's address — the agent's proxy target
orchestrator_url: str
def _network_cidr(network: str) -> str:
"""The gateway network's IPv4 subnet, or raise."""
proc = run_docker([
"docker", "network", "inspect",
"--format", "{{range .IPAM.Config}}{{.Subnet}}{{end}}", network,
])
cidr = proc.stdout.strip()
if proc.returncode != 0 or not cidr:
raise ConsolidatedLaunchError(
f"gateway network {network} has no subnet: {proc.stderr.strip()}"
)
return cidr
def _container_ip(name: str, network: str) -> str:
"""A container's IPv4 address on `network`, or raise."""
proc = run_docker([
"docker", "inspect", "--format",
f'{{{{(index .NetworkSettings.Networks "{network}").IPAddress}}}}', name,
])
ip = proc.stdout.strip()
if proc.returncode != 0 or not ip:
raise ConsolidatedLaunchError(
f"gateway {name} has no address on {network}: {proc.stderr.strip()}"
)
return ip
def _network_container_ips(network: str) -> list[str]:
"""Every address currently assigned on the gateway network — the ground
truth for "in use": the gateway + orchestrator infrastructure containers
and every live agent. Read from the network so a new bottle can't collide
with anything actually attached (a registry-only view would miss the
orchestrator/gateway containers)."""
proc = run_docker([
"docker", "network", "inspect", "--format",
"{{range .Containers}}{{.IPv4Address}} {{end}}", network,
])
ips: list[str] = []
for entry in proc.stdout.split():
# entries look like "172.20.0.2/16" — keep the address.
ips.append(entry.split("/", 1)[0])
return ips
def launch_consolidated(
egress_plan: EgressPlan,
git_gate_plan: GitGatePlan,
*,
image_ref: str = "",
tokens: dict[str, str] | None = None,
service: OrchestratorService | None = None,
gateway_name: str = GATEWAY_NAME,
network: str = GATEWAY_NETWORK,
) -> LaunchContext:
"""Ensure the orchestrator + gateway are up, allocate + register the
bottle, and provision its git-gate state. Returns the agent's attach
context. Raises `ConsolidatedLaunchError` (or the primitives' own errors)
if any step fails — the caller tears down on failure."""
service = service or OrchestratorService()
url = service.ensure_running()
client = OrchestratorClient(url)
cidr = _network_cidr(network)
gateway_ip = _container_ip(gateway_name, network)
source_ip = next_free_ip(cidr, _network_container_ips(network))
inputs = registration_inputs(egress_plan)
reg = client.register_bottle(
source_ip, image_ref=image_ref, policy=inputs.policy,
metadata=inputs.metadata, tokens=tokens,
)
try:
provision_git_gate(
DockerGatewayTransport(gateway_name), reg.bottle_id, git_gate_plan)
except Exception:
# Roll the registration back so a provisioning failure leaves no orphan.
client.teardown_bottle(reg.bottle_id)
raise
return LaunchContext(
bottle_id=reg.bottle_id,
identity_token=reg.identity_token,
source_ip=source_ip,
network=network,
gateway_ip=gateway_ip,
orchestrator_url=url,
)
def teardown_consolidated(
bottle_id: str, *, orchestrator_url: str, gateway_name: str = GATEWAY_NAME,
) -> None:
"""Deregister the bottle and remove its git-gate state from the gateway.
Both steps are idempotent so this is safe from a cleanup trap."""
OrchestratorClient(orchestrator_url).teardown_bottle(bottle_id)
deprovision_git_gate(DockerGatewayTransport(gateway_name), bottle_id)
__all__ = [
"LaunchContext",
"launch_consolidated",
"teardown_consolidated",
"ConsolidatedLaunchError",
]
+1 -1
View File
@@ -4,7 +4,7 @@ prepare-time routes-yaml rendering itself lives on the
platform-neutral `Egress` ABC — backends instantiate it directly.
The per-container `.start()` / `.stop()` lifecycle was removed in
PRD 0024 chunk 3; the gateway (PRD 0024) runs egress
PRD 0024 chunk 3; the sidecar bundle (PRD 0024) runs egress
under its python init supervisor."""
from __future__ import annotations
+44 -13
View File
@@ -1,29 +1,60 @@
"""Host-side egress route-apply for the docker backend.
"""Host-side helper for egress sidecar inspection and live updates.
The per-bottle companion container this used to signal (`docker kill
--signal HUP <container>`) was removed in the companion-container removal (#385).
In the consolidated model the shared gateway resolves egress policy
per-request against the orchestrator rather than reloading a per-bottle
routes file, so the live per-bottle reload is not supported here and
fails closed until the gateway-side apply lands.
The approve path uses this module to validate a proposed routes file,
write it to the bottle's live egress state dir, and signal the sidecar
bundle so the mitmproxy addon reloads it.
"""
from __future__ import annotations
import os
import subprocess
from ...egress import EGRESS_ROUTES_IN_CONTAINER
from ...log import warn
from ..egress_apply import EgressApplicator, EgressApplyError
from .sidecar_bundle import sidecar_bundle_container_name
def fetch_current_routes(slug: str) -> str:
container = sidecar_bundle_container_name(slug)
r = subprocess.run(
["docker", "exec", container, "cat", EGRESS_ROUTES_IN_CONTAINER],
capture_output=True, text=True, check=False,
)
if r.returncode != 0:
raise EgressApplyError(
f"could not read routes.yaml from {container}: "
f"{(r.stderr or '').strip() or 'container not running?'}"
)
return r.stdout
class DockerEgressApplicator(EgressApplicator):
def _signal_bundle_reload(self, slug: str) -> None:
del slug
raise EgressApplyError(
"live egress route-apply was removed with the per-bottle "
"companion container (#385); route changes will flow through "
"the consolidated gateway in a follow-up."
container = sidecar_bundle_container_name(slug)
result = subprocess.run(
["docker", "kill", "--signal", "HUP", container],
capture_output=True, text=True, check=False, env=os.environ,
)
if result.returncode != 0:
last_error = (result.stderr or "").strip() or (result.stdout or "").strip()
warn(
f"egress: routes updated on disk for {slug}, but bundle reload failed: "
f"{last_error or 'docker kill failed'}"
)
raise EgressApplyError(
f"could not reload egress bundle {container}: "
f"{last_error or 'docker kill failed'}"
)
applicator = DockerEgressApplicator()
__all__ = ["DockerEgressApplicator", "EgressApplyError", "applicator"]
__all__ = [
"DockerEgressApplicator",
"EgressApplyError",
"applicator",
"fetch_current_routes",
]
+2 -1
View File
@@ -1,6 +1,7 @@
"""Active-agent enumeration for the docker backend.
Returns `ActiveAgent` records the CLI `list active` command and the
Mirrors `backend/smolmachines/enumerate.py`: returns
`ActiveAgent` records the CLI `list active` command and the
dashboard agents pane consume. Empty when docker isn't reachable
— gated by `has_backend('docker')` at the cross-backend caller
so this module trusts that docker is available when called.
-47
View File
@@ -1,47 +0,0 @@
"""Shared-gateway source-IP allocation for the consolidated docker backend
(PRD 0070).
In the consolidated model one gateway container serves every bottle over a
single shared docker network, and each agent bottle attaches with a pinned,
deterministic address that the gateway uses as its **attribution key**. This
allocates those addresses from the network's subnet, skipping the reserved
ones — the network address and broadcast (excluded by `hosts()`), docker's
router `.1`, and everything already in use (`taken`: the gateway container
plus every live bottle, which the caller reads from the registry).
Pure `ipaddress` logic — the docker-specific bits (the subnet CIDR, the
gateway container's own address) are gathered by the caller and passed in, so
this stays testable without docker.
"""
from __future__ import annotations
import ipaddress
from collections.abc import Iterable
class NoFreeAddressError(RuntimeError):
"""The shared gateway network's subnet is exhausted — every host address
is reserved or already assigned to a bottle."""
def next_free_ip(cidr: str, taken: Iterable[str]) -> str:
"""The lowest host address in `cidr` not in `taken` and not docker's
router (`.1`). `taken` must include the gateway container's own address
and every live bottle's. Raises `NoFreeAddressError` if the subnet is
full."""
net = ipaddress.ip_network(cidr, strict=False)
reserved = {str(a) for a in taken}
# Docker assigns the network's first host (.1) to the bridge router; a
# bottle must never be handed that address.
reserved.add(str(net.network_address + 1))
for host in net.hosts(): # hosts() already excludes network + broadcast
candidate = str(host)
if candidate not in reserved:
return candidate
raise NoFreeAddressError(
f"no free address in {cidr} ({len(reserved)} reserved/assigned)"
)
__all__ = ["next_free_ip", "NoFreeAddressError"]
@@ -1,126 +0,0 @@
"""Provision one bottle's git-gate state into the running shared gateway
(PRD 0070, docker slice).
The consolidated gateway serves every bottle's repos under `/git/<bottle_id>/`
with per-repo credentials under `/git-gate/creds/<bottle_id>/`. When a bottle
is registered the launcher must place *its* deploy keys + known_hosts into
that per-bottle creds dir and init its bare repos there — so this copies the
credential files into the live gateway container and runs the (namespaced,
init-only) provisioning script produced by `git_gate_render_provision`.
Isolating each bottle's creds dir + repo root by id is what keeps one
bottle's push credentials out of another's repos on the shared gateway.
"""
from __future__ import annotations
import re
from typing import Protocol
from ...docker_cmd import run_docker
from ...git_gate import GitGatePlan, git_gate_render_provision
# bottle ids index the gateway's per-bottle repo + creds dirs; they land in
# exec/cp path arguments, so validate before any path is built (a traversal
# id like "../etc" must never reach the gateway). Registry ids are token_hex —
# this is defense in depth at the transport boundary.
_SAFE_BOTTLE_ID = re.compile(r"[A-Za-z0-9_-]+")
class GatewayProvisionError(RuntimeError):
"""A git-gate provisioning step against the running gateway failed."""
class GatewayTransport(Protocol):
"""How the launcher stages files + runs commands in the running gateway.
Backend-neutral so the same provisioning logic serves the docker gateway
(exec/cp over the docker socket) and the firecracker gateway VM (over
SSH)."""
def exec(self, argv: list[str]) -> None:
"""Run `argv` in the gateway, raising `GatewayProvisionError` on
failure."""
def cp_into(self, src: str, dest: str) -> None:
"""Copy host file `src` to `dest` in the gateway, raising on
failure."""
class DockerGatewayTransport:
"""`GatewayTransport` for the docker gateway container (exec/cp)."""
def __init__(self, gateway: str) -> None:
self.gateway = gateway
def exec(self, argv: list[str]) -> None:
proc = run_docker(["docker", "exec", self.gateway, *argv])
if proc.returncode != 0:
raise GatewayProvisionError(
f"gateway exec {argv!r} failed: {proc.stderr.strip()}"
)
def cp_into(self, src: str, dest: str) -> None:
proc = run_docker(["docker", "cp", src, f"{self.gateway}:{dest}"])
if proc.returncode != 0:
raise GatewayProvisionError(
f"gateway cp {src} -> {dest} failed: {proc.stderr.strip()}"
)
def _require_safe(bottle_id: str) -> None:
if not _SAFE_BOTTLE_ID.fullmatch(bottle_id):
raise GatewayProvisionError(f"unsafe bottle id {bottle_id!r}")
def _creds_dir(bottle_id: str) -> str:
return f"/git-gate/creds/{bottle_id}"
def provision_git_gate(
transport: GatewayTransport, bottle_id: str, plan: GitGatePlan,
) -> None:
"""Place `bottle_id`'s git-gate credentials into the running gateway and
init its bare repos under `/git/<bottle_id>/`.
Copies each upstream's identity key (and known_hosts, when present) into
`/git-gate/creds/<bottle_id>/`, then runs the namespaced provisioning
script. No-op for a bottle with no git upstreams."""
_require_safe(bottle_id)
if not plan.upstreams:
return
# The pre-receive + access hooks are bottle-agnostic and shared by every
# bottle's repos; install them into the gateway (idempotent — same content
# each time). The per-bottle model cp'd these into each bundle at start.
transport.exec(["mkdir", "-p", "/etc/git-gate"])
transport.cp_into(str(plan.hook_script), "/etc/git-gate/pre-receive")
transport.cp_into(str(plan.access_hook_script), "/etc/git-gate/access-hook")
creds = _creds_dir(bottle_id)
transport.exec(["mkdir", "-p", creds])
for u in plan.upstreams:
if u.identity_file:
transport.cp_into(u.identity_file, f"{creds}/{u.name}-key")
known_hosts = str(u.known_hosts_file)
if known_hosts and known_hosts != ".":
transport.cp_into(known_hosts, f"{creds}/{u.name}-known_hosts")
# Init the bare repos + per-repo credential config for this namespace.
script = git_gate_render_provision(bottle_id, plan.upstreams)
transport.exec(["sh", "-c", script])
def deprovision_git_gate(transport: GatewayTransport, bottle_id: str) -> None:
"""Remove a bottle's repos + creds from the gateway on teardown. Idempotent
— an already-absent namespace is a clean no-op (best effort; a stray dir
can't leak, since attribution is by source IP and the bottle is gone)."""
_require_safe(bottle_id)
try:
transport.exec([
"rm", "-rf", f"/git/{bottle_id}", _creds_dir(bottle_id),
])
except GatewayProvisionError:
pass # best-effort teardown; absent namespace is success
__all__ = [
"provision_git_gate", "deprovision_git_gate",
"GatewayProvisionError", "GatewayTransport", "DockerGatewayTransport",
]
+1 -1
View File
@@ -2,7 +2,7 @@
bind-mounts target + the listening port. The prepare-time entrypoint
/ hook render lives on the platform-neutral `GitGate` ABC — backends
instantiate it directly. The git-gate daemon's container lifecycle
is owned by the gateway (PRD 0024)."""
is owned by the sidecar bundle (PRD 0024)."""
from __future__ import annotations
+62 -61
View File
@@ -5,7 +5,7 @@ PRD 0018 chunk 3: each instance is one `docker compose` project.
The flow is:
1. Build the agent image from the provider Dockerfile (compose
builds the gateway image on first up).
builds the sidecar images via the `build:` directive on first up).
2. Mint the per-bottle egress CA (chunk 2 writes it under
state/<slug>/egress/).
3. Populate the inner plans with launch-time fields so the
@@ -36,13 +36,13 @@ from contextlib import ExitStack, contextmanager
from pathlib import Path
from typing import Callable, Generator
from ...agent_provider import runtime_for
from ...egress import egress_resolve_token_values
from ...git_gate import (
provision_git_gate_dynamic_keys,
revoke_git_gate_provisioned_keys,
)
from ...log import info, warn
from . import network as network_mod
from . import util as docker_mod
from .bottle import DockerBottle
from .bottle_plan import DockerBottlePlan
@@ -53,6 +53,7 @@ from ...bottle_state import (
read_committed_image,
)
from .compose import (
bottle_plan_to_compose,
compose_down,
compose_dump_logs,
compose_file_path,
@@ -61,9 +62,7 @@ from .compose import (
compose_up,
write_compose_file,
)
from .consolidated_compose import consolidated_agent_compose
from .consolidated_launch import launch_consolidated, teardown_consolidated
from ...orchestrator.gateway import DockerGateway
from .egress import egress_tls_init
# Where the repo root lives, for `docker build` context. Computed once.
@@ -98,7 +97,8 @@ def launch(
try:
# Step 1: agent image. Use a committed snapshot when one exists
# and is present in the local daemon; otherwise build from the
# Dockerfile. (The gateway image is built by the orchestrator.)
# Dockerfile. Sidecar images get built lazily by `docker compose
# up` via the renderer's `build:` directives.
committed = read_committed_image(plan.slug)
if committed and docker_mod.image_exists(committed):
info(f"using committed image {committed!r}")
@@ -111,86 +111,83 @@ def launch(
plan.image, _REPO_DIR,
dockerfile=plan.dockerfile_path,
)
docker_mod.verify_agent_image(
plan.image, runtime_for(plan.agent_provider_template).smoke_test,
)
# Step 2: mint the git-gate dynamic (gitea) deploy keys, if any, before
# provisioning the bottle's repos into the shared gateway.
internal_network = network_mod.network_name_for_slug(plan.slug)
egress_network = network_mod.network_egress_name_for_slug(plan.slug)
egress_ca_host, egress_ca_cert_only = egress_tls_init(
egress_state_dir(plan.slug),
)
git_gate_plan = plan.git_gate_plan
if git_gate_plan.upstreams:
git_gate_plan = provision_git_gate_dynamic_keys(
plan.manifest.bottle, git_gate_plan, git_gate_state_dir(plan.slug),
plan.manifest.bottle,
git_gate_plan,
git_gate_state_dir(plan.slug),
)
git_gate_plan = dataclasses.replace(
git_gate_plan,
internal_network=internal_network,
egress_network=egress_network,
)
# Step 3: register on the orchestrator + provision this bottle's
# git-gate state into the shared gateway; get the agent's attach
# context (pinned source IP, gateway address, shared network). The
# per-bottle egress auth tokens are resolved from the host env now and
# handed to the orchestrator (in memory) for the gateway to inject —
# the agent never sees them.
effective_env = {**os.environ, **plan.agent_provision.provisioned_env}
token_values = egress_resolve_token_values(
plan.egress_plan.token_env_map, effective_env,
)
ctx = launch_consolidated(
plan.egress_plan, git_gate_plan, image_ref=plan.image, tokens=token_values,
)
stack.callback(
teardown_consolidated, ctx.bottle_id, orchestrator_url=ctx.orchestrator_url,
)
# Step 4: install the SHARED gateway CA into the agent (replaces the
# per-bottle CA) — read it out of the running gateway.
ca_dir = egress_state_dir(plan.slug) / "gateway-ca"
ca_dir.mkdir(parents=True, exist_ok=True)
ca_file = ca_dir / "gateway-ca.pem"
ca_file.write_text(DockerGateway(network=ctx.network).ca_cert_pem())
egress_plan = dataclasses.replace(
plan.egress_plan,
mitmproxy_ca_host_path=ca_file,
mitmproxy_ca_cert_only_host_path=ca_file,
)
# Point the agent's git-gate insteadOf rewrites at the shared gateway's
# HTTP git endpoint (9420) instead of the dead per-bottle `git-gate`
# alias. git-http + supervise on the gateway bypass the egress proxy
# (NO_PROXY includes the gateway address).
git_gate_url = (
f"http://{ctx.gateway_ip}:9420" if git_gate_plan.upstreams else ""
)
supervise_url = (
f"http://{ctx.gateway_ip}:9100/" if plan.supervise_plan is not None else ""
internal_network=internal_network,
egress_network=egress_network,
mitmproxy_ca_host_path=egress_ca_host,
mitmproxy_ca_cert_only_host_path=egress_ca_cert_only,
)
supervise_plan = plan.supervise_plan
if supervise_plan is not None:
supervise_plan = dataclasses.replace(
supervise_plan,
internal_network=internal_network,
)
plan = dataclasses.replace(
plan,
git_gate_plan=git_gate_plan,
egress_plan=egress_plan,
agent_git_gate_url=git_gate_url,
agent_supervise_url=supervise_url,
supervise_plan=supervise_plan,
)
# Step 5: render + up the agent-only compose, pinned on the shared
# gateway network and proxied through the gateway's address.
# Step 6: render + write the compose file. metadata.json
# was written at prepare time and already carries
# compose_project; nothing to update here.
state_dir = bottle_state_dir(plan.slug)
spec = consolidated_agent_compose(
plan, gateway_ip=ctx.gateway_ip, source_ip=ctx.source_ip, network=ctx.network,
)
spec = bottle_plan_to_compose(plan)
compose_file = write_compose_file(spec, compose_file_path(state_dir))
project = compose_project_name(plan.slug)
# Forwarded vars (OAuth token, host interpolations) flow through the
# subprocess env as bare names so values never land in the file.
compose_env: dict[str, str] = {**os.environ, **plan.forwarded_env}
# Step 7: compose up. Token values + the OAuth placeholder
# flow through subprocess env; the compose file holds only
# bare names for the secret-carrying entries.
effective_env = {**dict(os.environ), **plan.agent_provision.provisioned_env}
token_values = egress_resolve_token_values(
plan.egress_plan.token_env_map, effective_env,
)
compose_env: dict[str, str] = {
**os.environ,
**plan.forwarded_env,
**token_values,
}
info(
f"docker compose up -d (project {project}, agent on shared "
f"gateway {ctx.gateway_ip}, ip {ctx.source_ip})"
f"docker compose up -d (project {project}, "
f"{len(spec['services'])} services)"
)
compose_up(project, compose_file, env=compose_env)
# Register teardown in reverse order: log dump first, then
# `compose down`. Networks come down last via callbacks
# registered in step 2.
stack.callback(compose_down, project, compose_file)
stack.callback(
compose_dump_logs, project, compose_file, compose_log_path(state_dir),
)
# Step 6: provision (CA install now uses the gateway CA) + yield.
# Step 8: provision. Create the bottle first so provisioners
# can use bottle.exec / bottle.cp_in; set the prompt path
# returned by provision_prompt after the fact.
bottle = DockerBottle(
plan.container_name,
teardown,
@@ -203,6 +200,10 @@ def launch(
agent_workdir=plan.workspace_plan.workdir,
)
bottle.prompt_path = provision(plan, bottle)
# Step 9: yield. exec_agent continues to use `docker exec -it`
# — the agent runs `sleep infinity` per the renderer's
# service spec.
yield bottle
finally:
teardown()
+1 -1
View File
@@ -76,7 +76,7 @@ def network_create_internal(slug: str) -> str:
def network_create_egress(slug: str) -> str:
"""Create a per-agent user-defined bridge (NOT the legacy `bridge`)
so the egress daemon has working DNS for upstream hostnames."""
so the egress sidecar has working DNS for upstream hostnames."""
return _network_create_with_prefix(network_egress_name_for_slug(slug), internal=False)
-89
View File
@@ -1,89 +0,0 @@
"""Host setup + status for the Docker backend.
Unlike Firecracker, the Docker backend needs no privileged one-time
host provisioning (no TAP pool / nft table) — networks and the gateway
bundle are created per-launch. So `setup()` is mostly an install/daemon
pointer, and `status()` reports whether docker is usable.
This is intentionally minimal; a richer version (daemon config checks,
gVisor/runsc install guidance, rootless-docker hints) is tracked
separately. Reached via `DockerBottleBackend.setup` / `.status`, which
the generic `./cli.py backend {setup,status}` dispatches to.
"""
from __future__ import annotations
import shutil
import subprocess
import sys
from . import util as _util
def _docker_on_path() -> bool:
return shutil.which("docker") is not None
def _daemon_reachable() -> bool:
if not _docker_on_path():
return False
return subprocess.run(
["docker", "info"],
stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL, check=False,
).returncode == 0
def _print_install_pointer() -> None:
sys.stderr.write("Docker is required but was not found on PATH.\n")
sys.stderr.write(" macOS: Docker Desktop https://docs.docker.com/desktop/install/mac-install/\n")
sys.stderr.write(" Linux: Docker Engine https://docs.docker.com/engine/install/\n")
def setup() -> int:
if not _docker_on_path():
_print_install_pointer()
return 1
sys.stderr.write(
"Docker backend: no privileged host setup required — networks and "
"the gateway are created per-launch.\n"
)
if not _daemon_reachable():
sys.stderr.write(
"The Docker daemon isn't reachable; start it (e.g. Docker "
"Desktop, or `systemctl start docker`).\n"
)
return 1
if not _util.runsc_available():
sys.stderr.write(
"Optional: register the gVisor (`runsc`) runtime with Docker for "
"a userspace syscall barrier; bottles auto-detect and use it.\n"
)
return 0
def teardown() -> int:
sys.stderr.write(
"Docker backend: nothing to undo — it provisions no privileged host "
"state (networks and the gateway are per-launch and are "
"removed by `./cli.py cleanup`). Docker itself is left installed.\n"
)
return 0
def status() -> int:
ok = True
if _docker_on_path():
sys.stderr.write("docker on PATH: yes\n")
else:
sys.stderr.write("docker on PATH: NO\n")
ok = False
if ok and _daemon_reachable():
sys.stderr.write("docker daemon: reachable\n")
elif ok:
sys.stderr.write("docker daemon: UNREACHABLE\n")
ok = False
runsc = _docker_on_path() and _util.runsc_available()
sys.stderr.write(f"gVisor runsc runtime: {'registered' if runsc else 'not registered (optional)'}\n")
if not ok:
sys.stderr.write("\nRun: ./cli.py backend setup --backend=docker\n")
return 0 if ok else 1
@@ -0,0 +1,30 @@
"""Sidecar bundle constants + helpers for the Docker backend
(PRD 0024).
The bundle image (built by Dockerfile.sidecars, PRD 0024 chunk 1)
runs egress + git-gate + supervise as one container per bottle
under a small Python init supervisor. As of chunk 5 the bundle
is the only shape — the legacy four-sidecar topology and its
`BOT_BOTTLE_SIDECAR_BUNDLE` feature flag are gone."""
from __future__ import annotations
import os
# Bundle image. Defaults to a built-locally tag (built from the
# repo's Dockerfile.sidecars via compose `build:`). Operators
# pinning to a published digest can override via env.
SIDECAR_BUNDLE_IMAGE = os.environ.get(
"BOT_BOTTLE_SIDECAR_IMAGE",
"bot-bottle-sidecars:latest",
)
SIDECAR_BUNDLE_DOCKERFILE = "Dockerfile.sidecars"
def sidecar_bundle_container_name(slug: str) -> str:
"""`bot-bottle-sidecars-<slug>`. Same prefix scheme as the
per-sidecar containers it replaces, so the dashboard's
discovery-by-prefix logic keeps working."""
return f"bot-bottle-sidecars-{slug}"
+31 -51
View File
@@ -4,13 +4,11 @@ existence, and building images."""
from __future__ import annotations
import os
import re
import shutil
import subprocess
from typing import Iterable, Iterator
from ...docker_cmd import run_docker
from ...log import die, info
# from ...workspace import WorkspacePlan
@@ -90,29 +88,6 @@ def docker_exec_root(container: str, argv: list[str]) -> None:
)
def docker_exec(container: str, argv: list[str], *, user: str = "") -> None:
"""Run `docker exec` in the named container, dying with the
command's own stderr on failure. Pass `user=\"0\"` to run as root."""
cmd = ["docker", "exec"]
if user:
cmd += ["-u", user]
cmd += [container, *argv]
result = run_docker(cmd)
if result.returncode != 0:
die(
f"docker exec in {container} failed: "
f"{(result.stderr or '').strip() or '<no stderr>'}"
)
def docker_cp(src: str, dest: str) -> None:
"""Run `docker cp`, dying with the command's own stderr on failure."""
result = run_docker(["docker", "cp", src, dest])
if result.returncode != 0:
die(f"docker cp {src} -> {dest} failed: "
f"{(result.stderr or '').strip() or '<no stderr>'}")
_SLUG_RE = re.compile(r"[^a-z0-9]+")
@@ -133,40 +108,15 @@ def build_image(ref: str, context: str, *, dockerfile: str = "") -> None:
`dockerfile` is an optional path (relative to `context`, or
absolute) for callers that need to build from a non-default
Dockerfile in the same context — e.g. `Dockerfile.git-gate`.
Set `BOT_BOTTLE_NO_CACHE=1` (the `start --no-cache` flag) to force
`--no-cache`. The npm/curl installers some provider Dockerfiles
shell out to can silently no-op on a transient network failure —
e.g. an `optionalDependencies` fetch for a platform-native binary —
and Docker will then cache that broken layer indefinitely."""
Dockerfile in the same context — e.g. `Dockerfile.git-gate`."""
info(f"building image {ref} from {context} (layer cache keeps repeat builds fast)")
args = ["docker", "build", "-t", ref]
if os.environ.get("BOT_BOTTLE_NO_CACHE") == "1":
args.append("--no-cache")
if dockerfile:
args.extend(["-f", dockerfile])
args.append(context)
subprocess.run(args, check=True)
def verify_agent_image(image: str, argv: tuple[str, ...]) -> None:
"""Run `argv` inside a throwaway container of a freshly built agent
image and die loudly if it fails, instead of shipping an image
whose CLI only breaks at first real use. No-op when the provider
hasn't declared a smoke test (`AgentProviderRuntime.smoke_test`)."""
if not argv:
return
result = run_docker(["docker", "run", "--rm", "--entrypoint", argv[0], image, *argv[1:]])
if result.returncode != 0:
detail = (result.stderr or result.stdout or "").strip()
die(
f"agent image {image!r} failed its post-build smoke test "
f"({' '.join(argv)}): {detail}\n"
f"Try rebuilding from scratch: bot-bottle start --no-cache"
)
# def build_image_with_cwd(
# derived: str,
# base: str,
@@ -217,6 +167,36 @@ def commit_container(container_name: str, image_tag: str) -> None:
info(f"committed {container_name!r}{image_tag!r}")
def image_id(ref: str) -> str:
"""Return the content-addressed image ID (e.g.
`sha256:abcd...`) for `ref`. The smolmachines backend keys its
`.smolmachine` artifact cache on this, so a Dockerfile change
that produces a new image automatically invalidates the cache."""
r = subprocess.run(
["docker", "image", "inspect", "--format", "{{.Id}}", ref],
capture_output=True,
text=True,
check=False,
)
if r.returncode != 0:
die(
f"docker image inspect for {ref!r} failed: "
f"{(r.stderr or '').strip() or '<no stderr>'}"
)
return r.stdout.strip()
def save(ref: str, output: str) -> None:
"""`docker save REF -o OUTPUT`. Writes a tarball of the image
layers + manifest to the host path. Used by smolmachines
prepare to hand the agent image to a containerized crane that
pushes it to the ephemeral registry — bypassing the docker
daemon's `docker push` (which on Docker Desktop can't reach a
host-loopback registry and refuses plain-HTTP pushes to
non-loopback hosts)."""
subprocess.run(["docker", "save", ref, "-o", output], check=True)
def _silent_run(cmd: Iterable[str]) -> int:
return subprocess.run(
list(cmd),
@@ -1,7 +0,0 @@
"""Firecracker backend: Linux KVM microVM isolation (issue #342)."""
from __future__ import annotations
from .backend import FirecrackerBottleBackend
__all__ = ["FirecrackerBottleBackend"]
-189
View File
@@ -1,189 +0,0 @@
"""Bottle handle for a Firecracker microVM, over SSH.
Every operation routes through SSH to the guest (dropbear, listening on
the TAP link). `exec` pipes a script over stdin and captures output;
`cp_in` uses scp; `exec_agent` uses `ssh -t` for an interactive PTY
session. `ssh -t` forwards the host terminal's SIGWINCH to the remote
PTY natively, so no separate resize bridge is needed.
Commands run as the image's `node` user via `runuser`, with HOME/USER/
PATH and the bottle env (HTTPS_PROXY at the gateway, CA paths, …) set
per-invocation through `env` (the VM itself just runs the init; it has
no baked-in process env like a `docker run` container would).
"""
from __future__ import annotations
import os
import shlex
import subprocess
import sys
from pathlib import Path
from typing import Mapping, cast
from ...agent_provider import PromptMode, prompt_args
from .. import Bottle, ExecResult
from ..terminal import exec_shell_script
from . import util
_HOME_FOR = {"node": "/home/node", "root": "/root"}
_DEFAULT_PATH_FOR = {
"node": (
"/home/node/.local/bin:"
"/home/node/.codex/packages/standalone/current/bin:"
"/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"
),
"root": "/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
}
def _env_assignments_for(user: str, env: Mapping[str, str]) -> list[str]:
home = _HOME_FOR.get(user, f"/home/{user}")
out = [f"HOME={home}", f"USER={user}"]
if "PATH" not in env:
out.append(f"PATH={_DEFAULT_PATH_FOR.get(user, _DEFAULT_PATH_FOR['root'])}")
for k, v in env.items():
out.append(f"{k}={v}")
return out
class FirecrackerBottle(Bottle):
def __init__(
self,
name: str,
*,
private_key: Path,
guest_ip: str,
guest_env: Mapping[str, str] | None = None,
prompt_path_in_guest: str | None = None,
agent_command: str = "claude",
agent_prompt_mode: PromptMode = "append_file",
agent_provider_template: str = "claude",
terminal_title: str = "",
terminal_color: str = "",
agent_workdir: str = "/home/node",
) -> None:
self.name = name
self._private_key = private_key
self._guest_ip = guest_ip
self._guest_env = dict(guest_env or {})
self.prompt_path = prompt_path_in_guest
self._agent_prompt_mode = agent_prompt_mode
self.agent_command = agent_command
self.terminal_title = terminal_title
self.terminal_color = terminal_color
self.agent_provider_template = agent_provider_template
self.agent_workdir = agent_workdir
def _ssh(self, *, tty: bool) -> list[str]:
argv = util.ssh_base_argv(self._private_key, self._guest_ip)
if tty:
# -t: allocate a remote PTY and forward window-size changes.
argv.insert(1, "-t")
return argv
def _agent_remote_argv(self, argv: list[str]) -> list[str]:
"""The `runuser … <agent> …` command run inside the guest."""
full_argv = list(argv)
full_argv.extend(
prompt_args(
cast(PromptMode, self._agent_prompt_mode),
self.prompt_path,
argv=full_argv,
)
)
# ALWAYS cd into the workdir before exec'ing the agent. The
# control SSH logs in as root, so the inherited cwd is /root —
# which the agent (running as node) can't even read now that
# /root is root-owned. Run the agent from the node workdir
# (default /home/node) so its cwd is node-accessible; otherwise
# Node's process.cwd(), the shell-snapshot machinery, and
# `/doctor` all fail on the unreadable /root.
# Run the agent from the node workdir (default /home/node), NOT
# the /root cwd inherited from the root SSH login — /root is now
# root-owned and unreadable by node, which breaks Node's
# process.cwd(), the shell-snapshot machinery, and `/doctor`.
# Use `env --chdir` rather than a `sh -c 'cd … && exec "$@"'`
# wrapper: ssh space-joins everything after the host into one
# string for the guest shell, so a quoted script + $@ would be
# re-split and mangled (exec'ing the $0 placeholder). All-simple
# words survive that join.
workdir = self.agent_workdir or _HOME_FOR["node"]
remote = ["runuser", "-u", "node", "--",
"env", f"--chdir={workdir}",
*_env_assignments_for("node", self._guest_env),
self.agent_command, *full_argv]
return remote
def agent_argv(self, argv: list[str], *, tty: bool = True) -> list[str]:
return [*self._ssh(tty=tty), "--", *self._agent_remote_argv(argv)]
def exec_agent(self, argv: list[str], *, tty: bool = True) -> int:
agent_argv = self.agent_argv(argv, tty=tty)
script = (
exec_shell_script(agent_argv, self.terminal_title, self.terminal_color)
if tty else None
)
if script is None:
return subprocess.run(agent_argv, check=False).returncode
return subprocess.run(["sh", "-lc", script], check=False).returncode
def exec(self, script: str, *, user: str = "node") -> ExecResult:
# Pipe the script over stdin (`sh -s`) so nothing needs shell
# quoting through the SSH command line.
remote = ["runuser", "-u", user, "--",
"env", *_env_assignments_for(user, self._guest_env),
"/bin/sh", "-s"]
result = subprocess.run(
[*self._ssh(tty=False), "--", *remote],
input=script, capture_output=True, text=True, check=False,
)
return ExecResult(
returncode=result.returncode,
stdout=result.stdout,
stderr=result.stderr,
)
def cp_in(self, host_path: str, container_path: str) -> None:
# Stream a tar over the SSH exec channel rather than scp: the
# guest runs dropbear, which ships no sftp-server/scp, but every
# agent image has `tar`. Copy so `container_path` becomes a copy
# of `host_path` (docker cp semantics — callers rm -rf the
# destination first).
host_parent = os.path.dirname(host_path.rstrip("/")) or "/"
host_base = os.path.basename(host_path.rstrip("/"))
guest_parent = os.path.dirname(container_path.rstrip("/")) or "/"
guest_base = os.path.basename(container_path.rstrip("/"))
remote = (
f"mkdir -p {shlex.quote(guest_parent)} && "
f"cd {shlex.quote(guest_parent)} && "
f"rm -rf {shlex.quote(guest_base)} && tar -xf - && "
f"{{ [ {shlex.quote(host_base)} = {shlex.quote(guest_base)} ] || "
f"mv {shlex.quote(host_base)} {shlex.quote(guest_base)}; }}"
)
tar = subprocess.Popen(
["tar", "-C", host_parent, "-cf", "-", host_base],
stdout=subprocess.PIPE,
)
# Pass the command as one arg: ssh space-joins everything after
# the host into a single string for the guest's login shell, so a
# `sh -c <remote>` split would drop everything past the first word
# (the guest shell runs `<remote>` directly; stdin carries the
# tar).
ssh = subprocess.run(
[*self._ssh(tty=False), "--", remote],
stdin=tar.stdout, capture_output=True, text=True, check=False,
)
tar.wait()
if tar.returncode != 0 or ssh.returncode != 0:
sys.stderr.write(ssh.stderr)
raise subprocess.CalledProcessError(
ssh.returncode or tar.returncode,
["cp_in", host_path, container_path],
)
def close(self) -> None:
# Real teardown (VM terminate, TAP release) lives on the launch
# ExitStack; this is the idempotent alias the ABC expects.
pass
@@ -1,29 +0,0 @@
"""Cleanup plan for the Firecracker backend."""
from __future__ import annotations
from dataclasses import dataclass
from ...log import info
from .. import BottleCleanupPlan
@dataclass(frozen=True)
class FirecrackerBottleCleanupPlan(BottleCleanupPlan):
# PIDs of orphaned firecracker VMM processes and the per-bottle run
# dirs left behind by previous bottles.
vm_pids: tuple[int, ...] = ()
run_dirs: tuple[str, ...] = ()
def print(self) -> None:
if self.empty:
info("firecracker cleanup: nothing to remove")
return
for pid in self.vm_pids:
info(f"firecracker VM process: pid {pid}")
for path in self.run_dirs:
info(f"firecracker run dir: {path}")
@property
def empty(self) -> bool:
return not (self.vm_pids or self.run_dirs)
@@ -1,63 +0,0 @@
"""Plan type for the Firecracker backend."""
from __future__ import annotations
from dataclasses import dataclass, field
from pathlib import Path
from ...agent_provider import PromptMode
from .. import BottlePlan
@dataclass(frozen=True)
class FirecrackerBottlePlan(BottlePlan):
slug: str
forwarded_env: dict[str, str] = field(repr=False)
# Stamped by launch once the gateway is up and its ports are
# published on the host-side TAP IP (empty at prepare time).
agent_proxy_url: str = ""
agent_git_gate_url: str = ""
agent_supervise_url: str = ""
@property
def container_name(self) -> str:
"""Instance name, reused for the gateway container + VM run dir.
Matches the `bot-bottle-<slug>` convention the other backends
use so cleanup/enumerate discovery-by-prefix keeps working."""
return self.agent_provision.instance_name
@property
def image(self) -> str:
return self.agent_provision.image
@property
def dockerfile_path(self) -> str:
return self.agent_provision.dockerfile
@property
def prompt_file(self) -> Path:
return self.agent_provision.prompt_file
@property
def agent_command(self) -> str:
return self.agent_provision.command
@property
def agent_prompt_mode(self) -> PromptMode:
return self.agent_provision.prompt_mode
@property
def agent_provider_template(self) -> str:
return self.agent_provision.template
@property
def git_gate_insteadof_host(self) -> str:
if self.agent_git_gate_url.startswith("http://"):
return self.agent_git_gate_url.removeprefix("http://").rstrip("/")
return super().git_gate_insteadof_host
@property
def git_gate_insteadof_scheme(self) -> str:
if self.agent_git_gate_url.startswith("http://"):
return "http"
return super().git_gate_insteadof_scheme
-69
View File
@@ -1,69 +0,0 @@
"""Cleanup for the Firecracker backend.
Orphans are: firecracker VMM processes whose config lives under our run
dir, and the per-bottle run dirs. TAP slots free themselves (the flock
drops when the launcher exits), so there is nothing to reclaim there.
"""
from __future__ import annotations
import os
import shutil
import signal
import subprocess
from pathlib import Path
from ...log import info
from . import util
from .bottle_cleanup_plan import FirecrackerBottleCleanupPlan
def _run_root() -> Path:
return util.cache_dir() / "run"
def _orphan_vm_pids() -> list[int]:
"""firecracker processes whose --config-file is under our run dir."""
run_root = str(_run_root())
result = subprocess.run(
["pgrep", "-a", "firecracker"],
capture_output=True, text=True, check=False,
)
if result.returncode != 0:
return []
pids: list[int] = []
for line in result.stdout.splitlines():
parts = line.split(None, 1)
if len(parts) != 2 or run_root not in parts[1]:
continue
try:
pids.append(int(parts[0]))
except ValueError:
continue
return pids
def _run_dirs() -> list[str]:
run_root = _run_root()
if not run_root.is_dir():
return []
return sorted(str(p) for p in run_root.iterdir() if p.is_dir())
def prepare_cleanup() -> FirecrackerBottleCleanupPlan:
return FirecrackerBottleCleanupPlan(
vm_pids=tuple(_orphan_vm_pids()),
run_dirs=tuple(_run_dirs()),
)
def cleanup(plan: FirecrackerBottleCleanupPlan) -> None:
for pid in plan.vm_pids:
info(f"kill firecracker VM pid {pid}")
try:
os.kill(pid, signal.SIGTERM)
except ProcessLookupError:
pass
for path in plan.run_dirs:
info(f"rm -rf {path}")
shutil.rmtree(path, ignore_errors=True)
@@ -1,109 +0,0 @@
"""Consolidated bottle launch sequence for the Firecracker backend
(PRD 0070, Stage B).
The shared gateway + orchestrator control plane run in a single persistent
per-host **infra VM** (`infra_vm.py`), not Docker containers. Agent VMs reach
the gateway's egress / supervise / git-http ports at the infra VM via a
PREROUTING DNAT on their own host-side TAP IP (see
`scripts/firecracker-netpool.sh`), and the host CLI reaches the control plane
over HTTP at the infra VM's guest IP.
Attribution is by the agent VM's guest IP, unspoofable by construction: the
/31 point-to-point TAP + the `bot_bottle_fc` nft table ensure only the
expected VM can source-IP that address.
Sequence:
1. ensure the infra VM (control plane + gateway) is up (a singleton — a
prior launcher may already have booted it);
2. register the bottle by its guest IP (attribution key) → bottle id +
identity token;
3. provision its git-gate repos/creds into the gateway VM (over SSH);
4. fetch the shared gateway CA for the provisioner to install in the rootfs.
The TAP slot allocation, rootfs build, and VM boot are the caller's job.
"""
from __future__ import annotations
from dataclasses import dataclass
from ...egress import EgressPlan
from ...git_gate import GitGatePlan
from ...orchestrator.client import OrchestratorClient
from ...orchestrator.lifecycle import (
OrchestratorStartError, # re-exported so callers can catch it
)
from ...orchestrator.registration import registration_inputs
from ..docker.gateway_provision import deprovision_git_gate, provision_git_gate
from . import infra_vm
class ConsolidatedLaunchError(RuntimeError):
"""The consolidated register/provision sequence could not complete."""
@dataclass(frozen=True)
class LaunchContext:
"""What the Firecracker launch needs from the consolidated sequence."""
bottle_id: str
identity_token: str
source_ip: str # the VM's guest IP — the attribution key
gateway_ca_pem: str # the shared gateway CA the provisioner installs
orchestrator_url: str
def launch_consolidated(
egress_plan: EgressPlan,
git_gate_plan: GitGatePlan,
*,
guest_ip: str,
image_ref: str = "",
tokens: dict[str, str] | None = None,
) -> LaunchContext:
"""Ensure the infra VM is up, register the bottle by its guest IP, and
provision its git-gate state into the gateway VM. Returns the context the
agent-VM launch needs. Raises on failure — the caller tears down."""
infra = infra_vm.ensure_running()
url = infra.control_plane_url
client = OrchestratorClient(url)
inputs = registration_inputs(egress_plan)
reg = client.register_bottle(
guest_ip, image_ref=image_ref, policy=inputs.policy,
metadata=inputs.metadata, tokens=tokens,
)
try:
provision_git_gate(
infra_vm.gateway_transport(), reg.bottle_id, git_gate_plan)
except Exception:
client.teardown_bottle(reg.bottle_id)
raise
# The shared gateway CA every agent on this host trusts for TLS
# interception — fetched from the infra VM over SSH.
return LaunchContext(
bottle_id=reg.bottle_id,
identity_token=reg.identity_token,
source_ip=guest_ip,
gateway_ca_pem=infra.gateway_ca_pem(),
orchestrator_url=url,
)
def teardown_consolidated(bottle_id: str, *, orchestrator_url: str) -> None:
"""Deregister the bottle and remove its git-gate state from the gateway
VM. Both steps are idempotent so this is safe from a cleanup trap. Does
NOT stop the infra VM — it's a persistent per-host singleton shared by
every bottle."""
OrchestratorClient(orchestrator_url).teardown_bottle(bottle_id)
deprovision_git_gate(infra_vm.gateway_transport(), bottle_id)
__all__ = [
"LaunchContext",
"launch_consolidated",
"teardown_consolidated",
"ConsolidatedLaunchError",
"OrchestratorStartError",
]
@@ -1,14 +0,0 @@
"""Active-agent enumeration for the Firecracker backend.
The backend is disabled during the companion-container removal (#385) — it can't
launch bottles, so there are none to enumerate. Real enumeration returns
with the backend's consolidated relaunch (#354).
"""
from __future__ import annotations
from .. import ActiveAgent
def enumerate_active() -> list[ActiveAgent]:
return []
@@ -1,194 +0,0 @@
"""Firecracker microVM process lifecycle.
A bottle VM is one `firecracker` process booted from a JSON config
(kernel + writable rootfs ext4 + one TAP network interface). We run it
`--no-api`: all configuration is in the config file, and teardown is a
process signal — the microVM dies with its VMM, which is exactly the
ephemeral-bottle semantics we want. No API socket, no runtime
reconfiguration.
The guest gets its IP from the kernel `ip=` cmdline (kernel-level
autoconfig, no in-guest iproute2) and its SSH pubkey from a `bb_pubkey=`
cmdline arg the init decodes.
"""
from __future__ import annotations
import base64
import json
import signal
import subprocess
import time
from dataclasses import dataclass
from pathlib import Path
from ...log import die, info
from . import util
# Serial console off the critical path but captured to a log for
# debugging boot failures. `reboot=k panic=1 pci=off` are the standard
# Firecracker guest args; `i8042.*` skip the (absent) PS/2 probe to
# shave boot time.
_BASE_BOOT_ARGS = (
"console=ttyS0 reboot=k panic=1 pci=off "
"i8042.noaux i8042.nomux i8042.nopnp i8042.dumbkbd "
"root=/dev/vda rw init=/bb-init"
)
@dataclass
class VmHandle:
"""A running microVM: its VMM process, guest IP, and console log."""
process: subprocess.Popen[bytes]
guest_ip: str
console_log: Path
def is_alive(self) -> bool:
return self.process.poll() is None
def terminate(self) -> None:
"""Stop the VMM (and thus the guest). SIGTERM, then SIGKILL."""
if self.process.poll() is not None:
return
self.process.send_signal(signal.SIGTERM)
try:
self.process.wait(timeout=5)
except subprocess.TimeoutExpired:
self.process.kill()
self.process.wait(timeout=5)
def _boot_args(guest_ip: str, host_ip: str, pubkey: str) -> str:
# ip=<client>::<gw>:<netmask>::<dev>:off — /31 point-to-point link,
# so netmask is 255.255.255.254 and the gateway is the host TAP IP.
ip_arg = f"ip={guest_ip}::{host_ip}:255.255.255.254::eth0:off"
pub_b64 = base64.b64encode(pubkey.encode()).decode()
return f"{_BASE_BOOT_ARGS} {ip_arg} bb_pubkey={pub_b64}"
def _config(
*,
rootfs: Path,
tap: str,
guest_ip: str,
host_ip: str,
pubkey: str,
vcpus: int,
mem_mib: int,
guest_mac: str,
data_drive: Path | None = None,
) -> dict[str, object]:
drives: list[dict[str, object]] = [
{
"drive_id": "rootfs",
"path_on_host": str(rootfs),
"is_root_device": True,
"is_read_only": False,
}
]
# A second virtio-block device (guest /dev/vdb) — the infra VM's
# persistent registry "volume", a host-side ext4 file that outlives the
# ephemeral rootfs across VM restarts.
if data_drive is not None:
drives.append({
"drive_id": "data",
"path_on_host": str(data_drive),
"is_root_device": False,
"is_read_only": False,
})
return {
"boot-source": {
"kernel_image_path": str(util.kernel_path()),
"boot_args": _boot_args(guest_ip, host_ip, pubkey),
},
"drives": drives,
"network-interfaces": [
{
"iface_id": "eth0",
"host_dev_name": tap,
"guest_mac": guest_mac,
}
],
"machine-config": {
"vcpu_count": vcpus,
"mem_size_mib": mem_mib,
},
}
def boot(
*,
name: str,
rootfs: Path,
tap: str,
guest_ip: str,
host_ip: str,
pubkey: str,
run_dir: Path,
vcpus: int = 2,
mem_mib: int = 2048,
guest_mac: str = "06:00:AC:10:00:02",
detached: bool = False,
data_drive: Path | None = None,
) -> VmHandle:
"""Write the config and launch the VMM. Returns once the process is
spawned; callers wait for SSH readiness separately.
`detached` starts the VMM in its own session (`start_new_session`) so it
survives the launcher exiting — used for the persistent per-host infra
VM, which must outlive the short-lived `start` process (agent VMs stay
attached and are torn down with the launcher)."""
run_dir.mkdir(parents=True, exist_ok=True)
config_path = run_dir / "config.json"
console_log = run_dir / "console.log"
config_path.write_text(json.dumps(
_config(
rootfs=rootfs, tap=tap, guest_ip=guest_ip, host_ip=host_ip,
pubkey=pubkey, vcpus=vcpus, mem_mib=mem_mib, guest_mac=guest_mac,
data_drive=data_drive,
),
indent=2,
))
info(f"booting microVM {name} on {tap} (guest {guest_ip})")
log_fh = console_log.open("wb")
process = subprocess.Popen(
["firecracker", "--no-api", "--config-file", str(config_path)],
stdout=log_fh, stderr=subprocess.STDOUT, stdin=subprocess.DEVNULL,
start_new_session=detached,
)
return VmHandle(process=process, guest_ip=guest_ip, console_log=console_log)
def wait_for_ssh(
vm: VmHandle, private_key: Path, *, timeout: float = 30.0,
) -> None:
"""Poll SSH until the guest accepts a command or the deadline
passes. Dies (with the console tail) if the VMM exits early or the
guest never comes up — a boot failure must be loud, not a hang."""
deadline = time.monotonic() + timeout
probe = util.ssh_base_argv(private_key, vm.guest_ip) + ["true"]
while time.monotonic() < deadline:
if not vm.is_alive():
die(f"microVM exited during boot (rc={vm.process.returncode}).\n"
f"{_console_tail(vm.console_log)}")
result = subprocess.run(
probe, stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL,
check=False,
)
if result.returncode == 0:
return
time.sleep(0.5)
die(f"microVM {vm.guest_ip} did not accept SSH within {timeout:.0f}s.\n"
f"{_console_tail(vm.console_log)}")
def _console_tail(console_log: Path, lines: int = 25) -> str:
try:
text = console_log.read_text(errors="replace").splitlines()
except OSError:
return "(no console log)"
tail = "\n".join(text[-lines:])
return f"--- guest console (last {lines} lines) ---\n{tail}"
-76
View File
@@ -1,76 +0,0 @@
"""FirecrackerFreezer — snapshot a running microVM to a Docker image.
The VM is live and can't be block-copied safely, so — like the macOS
backend — we stream the guest root filesystem out over the control
channel (SSH here) and rebuild an image from it. The bottle keeps
running after the snapshot.
"""
from __future__ import annotations
import json
import os
import subprocess
import tempfile
from pathlib import Path
from ...log import die, info
from .. import ActiveAgent
from ..freeze import Freezer
from . import util
class FirecrackerFreezer(Freezer):
backend_name = "firecracker"
def _freeze(self, agent: ActiveAgent) -> str:
run_dir = util.cache_dir() / "run" / agent.slug
private_key = run_dir / "bottle_id_ed25519"
guest_ip = _guest_ip_from_config(run_dir / "config.json")
if not private_key.is_file() or not guest_ip:
die(f"cannot freeze {agent.slug}: run dir {run_dir} is missing the "
f"SSH key or VM config (is the bottle still running?)")
image_tag = f"bot-bottle-committed-{agent.slug}:latest"
_commit_via_ssh(private_key, guest_ip, image_tag)
info(f"committed {agent.slug} -> {image_tag!r}")
return image_tag
def _export_hint(self, slug: str, image_ref: str) -> None:
info(f"to export for migration: docker image save {image_ref} "
f"-o {slug}.tar")
def _guest_ip_from_config(config_path: Path) -> str:
try:
cfg = json.loads(config_path.read_text())
except (OSError, json.JSONDecodeError):
return ""
boot_args = cfg.get("boot-source", {}).get("boot_args", "")
for token in boot_args.split():
if token.startswith("ip="):
# ip=<client>::<gw>:<mask>::<dev>:off
return token[len("ip="):].split(":", 1)[0]
return ""
def _commit_via_ssh(private_key: Path, guest_ip: str, image_tag: str) -> None:
with tempfile.TemporaryDirectory(prefix="bot-bottle-fc-commit.") as tmp:
rootfs_tar = os.path.join(tmp, "rootfs.tar")
ssh = util.ssh_base_argv(private_key, guest_ip)
with open(rootfs_tar, "wb") as tar_out:
result = subprocess.run(
[*ssh, "--", "tar", "--create", "--one-file-system",
"--exclude=./proc", "--exclude=./sys", "--exclude=./dev",
"--exclude=./run", "--file=-", "--directory=/", "."],
stdout=tar_out, stderr=subprocess.PIPE, check=False,
)
if result.returncode != 0:
die(f"ssh tar for {guest_ip} failed: "
f"{(result.stderr or b'').decode().strip() or '<no stderr>'}")
with open(os.path.join(tmp, "Dockerfile"), "w", encoding="utf-8") as f:
f.write("FROM scratch\nADD rootfs.tar /\nUSER node\nWORKDIR /home/node\n")
build = subprocess.run(
["docker", "build", "-t", image_tag, tmp], check=False,
)
if build.returncode != 0:
die(f"docker build for {image_tag!r} failed")
@@ -1,200 +0,0 @@
"""Docker-free agent-image builds for the Firecracker backend (PRD 0069 Stage 3).
Instead of `docker build` on the host, an agent's Dockerfile is built inside a
throwaway Firecracker *builder VM* running buildah (rootless, daemonless): no
host Docker daemon, no root-equivalent `docker` group, and the untrusted
Dockerfile executes in a confined microVM rather than on the host.
Flow (on a cache miss, keyed by Dockerfile content):
1. boot the builder VM (the orchestrator image, which carries buildah) on
the NAT'd orchestrator link (`netpool.orch_slot()`), so the Dockerfile's
`FROM` pulls + apt/npm reach the internet;
2. send the Dockerfile over SSH and `buildah build` it;
3. `buildah mount` the built image and stream its rootfs tar back to the
host, extracting it into the cache dir;
4. inject the guest boot bits + mark the cache ready — the same base-dir
shape `firecracker.util` turns into a bootable ext4 with `mke2fs -d`.
The builder shares the orchestrator's TAP; once the orchestrator is a
persistent VM (Stage B) these builds move onto its control plane instead of a
throwaway boot.
"""
from __future__ import annotations
import hashlib
import shutil
import subprocess
from pathlib import Path
from ...log import die, info
from . import firecracker_vm, netpool, util
# The builder runs the orchestrator image — it is the component that carries
# buildah (see Dockerfile.orchestrator). Built from the host docker image via
# the bootstrap export path until we pull a pre-built image instead.
_ORCHESTRATOR_IMAGE = "bot-bottle-orchestrator:latest"
# The builder VM makes DIRECT registry/apt connections (not through the
# gateway proxy the way agents do), and the kernel `ip=` cmdline sets no
# resolver — so give it one. Public for now; build-time egress through the
# gateway will remove the need for it.
_BUILDER_RESOLVER = "1.1.1.1"
# vfs + chroot: buildah works as root in the bare microVM (no fuse-overlayfs /
# overlay module / subuid maps). Must match Dockerfile.orchestrator's env.
# `--isolation` is a build/run-only flag; `from`/`mount` take just the store.
_BUILD_FLAGS = "--isolation chroot --storage-driver vfs"
_STORE_FLAG = "--storage-driver vfs"
_SSH_READY_TIMEOUT = 40.0
_BUILD_TIMEOUT_SECONDS = 900.0
_LOCAL_TAG = "bot-bottle-agent-build"
def _dockerfile_hash(dockerfile: Path) -> str:
"""Cache key: the Dockerfile's content. The shipped agent Dockerfiles
COPY nothing from the build context (see .dockerignore), so their content
fully determines the image; a Dockerfile that adds COPY will want the
context folded in here too."""
return hashlib.sha256(dockerfile.read_bytes()).hexdigest()[:16]
def build_agent_rootfs_dir(
dockerfile: Path, *, image_tag: str, smoke_test: tuple[str, ...] = (),
) -> Path:
"""Build `dockerfile` in a Firecracker builder VM (buildah, no host
docker), export its rootfs, inject the guest boot bits, and return the
cached base dir — the same shape `util.build_rootfs_ext4` consumes.
Cached by Dockerfile content, so a repeat launch skips the rebuild.
`smoke_test` (the provider's declared argv, e.g. `("claude","--version")`)
is run in the freshly built image before export, catching an npm
silent-failure image at build time rather than at first agent use."""
digest = _dockerfile_hash(dockerfile)
base = util.cache_dir() / "rootfs" / f"agent-{digest}"
ready = base / ".bb-ready"
if ready.is_file():
info(f"using cached agent rootfs {base.name}")
return base
if base.exists():
shutil.rmtree(base, ignore_errors=True)
base.mkdir(parents=True)
info(f"building agent image {image_tag!r} in a firecracker builder VM")
_build_in_vm(dockerfile, base, smoke_test)
util.inject_guest_boot(base)
ready.write_text("ok\n")
return base
def _build_in_vm(dockerfile: Path, base: Path, smoke_test: tuple[str, ...]) -> None:
"""Boot the builder VM, build the Dockerfile with buildah, and stream the
resulting rootfs into `base`. Tears the VM down on every exit path."""
slot = netpool.orch_slot()
if not netpool.tap_present(slot.iface):
die(f"orchestrator/builder link {slot.iface} not present.\n"
f" ./cli.py backend setup --backend=firecracker")
# The builder runs the orchestrator rootfs (bootstrap: exported from the
# host docker image; a pulled image replaces this later).
orch_base = util.build_base_rootfs_dir(_ORCHESTRATOR_IMAGE)
run_dir = util.cache_dir() / "run" / "builder"
run_dir.mkdir(parents=True, exist_ok=True)
rootfs = run_dir / "rootfs.ext4"
# vfs copies every layer, so the builder disk needs generous headroom.
util.build_rootfs_ext4(orch_base, rootfs, slack_mib=8192)
private_key, pubkey = util.generate_keypair(run_dir)
vm = firecracker_vm.boot(
name="bot-bottle-builder", rootfs=rootfs, tap=slot.iface,
guest_ip=slot.guest_ip, host_ip=slot.host_ip, pubkey=pubkey,
run_dir=run_dir, mem_mib=4096,
)
try:
firecracker_vm.wait_for_ssh(vm, private_key, timeout=_SSH_READY_TIMEOUT)
_ssh(private_key, slot.guest_ip,
f"printf 'nameserver {_BUILDER_RESOLVER}\\n' > /etc/resolv.conf "
f"&& mkdir -p /build/ctx")
_send_dockerfile(private_key, slot.guest_ip, dockerfile)
_buildah_build(private_key, slot.guest_ip)
_smoke_test(private_key, slot.guest_ip, smoke_test)
_stream_rootfs(private_key, slot.guest_ip, base)
finally:
vm.terminate()
def _ssh(private_key: Path, guest_ip: str, script: str,
*, timeout: float = 60.0) -> subprocess.CompletedProcess[str]:
return subprocess.run(
util.ssh_base_argv(private_key, guest_ip) + [script],
capture_output=True, text=True, timeout=timeout, check=False,
)
def _send_dockerfile(private_key: Path, guest_ip: str, dockerfile: Path) -> None:
proc = subprocess.run(
util.ssh_base_argv(private_key, guest_ip) + ["cat > /build/Dockerfile"],
input=dockerfile.read_bytes(), capture_output=True, timeout=30, check=False,
)
if proc.returncode != 0:
die(f"sending Dockerfile to the builder VM failed: "
f"{proc.stderr.decode(errors='replace').strip()}")
def _buildah_build(private_key: Path, guest_ip: str) -> None:
build = _ssh(
private_key, guest_ip,
f"buildah build {_BUILD_FLAGS} -t {_LOCAL_TAG} "
f"-f /build/Dockerfile /build/ctx",
timeout=_BUILD_TIMEOUT_SECONDS,
)
if build.returncode != 0:
tail = "\n".join((build.stdout + build.stderr).strip().splitlines()[-20:])
die(f"buildah build in the builder VM failed:\n{tail}")
def _smoke_test(private_key: Path, guest_ip: str, argv: tuple[str, ...]) -> None:
"""Run the provider's smoke argv inside the freshly built image
(`buildah run`, which uses the image's own PATH), failing the build
loudly if the CLI is a broken stub. No-op without a declared test."""
if not argv:
return
cmd = (
f"set -e; ctr=$(buildah from {_STORE_FLAG} {_LOCAL_TAG}); "
f"buildah run {_BUILD_FLAGS} \"$ctr\" -- {' '.join(argv)}; rc=$?; "
f"buildah rm \"$ctr\" >/dev/null 2>&1 || true; exit $rc"
)
result = _ssh(private_key, guest_ip, cmd, timeout=120)
if result.returncode != 0:
detail = (result.stdout + result.stderr).strip().splitlines()[-10:]
die(f"agent image failed its post-build smoke test "
f"({' '.join(argv)}):\n" + "\n".join(detail))
def _stream_rootfs(private_key: Path, guest_ip: str, base: Path) -> None:
"""`buildah mount` the built image in the VM and pipe its rootfs tar
straight into `base` on the host (extracted as the non-root host user, so
uid 0 isn't preserved — the guest init restores /root ownership, same as
the docker-export path)."""
export = (
f"set -e; ctr=$(buildah from {_STORE_FLAG} {_LOCAL_TAG}); "
f"mnt=$(buildah mount {_STORE_FLAG} \"$ctr\"); "
f"tar -C \"$mnt\" -cf - ."
)
ssh_proc = subprocess.Popen(
util.ssh_base_argv(private_key, guest_ip) + [export],
stdout=subprocess.PIPE, stderr=subprocess.PIPE,
)
assert ssh_proc.stdout is not None
untar = subprocess.run(
["tar", "-x", "-C", str(base)], stdin=ssh_proc.stdout, check=False,
)
ssh_proc.stdout.close()
ssh_err = (ssh_proc.stderr.read().decode(errors="replace")
if ssh_proc.stderr else "")
rc = ssh_proc.wait()
if rc != 0 or untar.returncode != 0:
die(f"exporting the built rootfs from the builder VM failed: "
f"{ssh_err.strip() or '<no stderr>'}")
-388
View File
@@ -1,388 +0,0 @@
"""The per-host infra VM for the Firecracker backend (PRD 0070 Stage B).
A single persistent microVM that runs the orchestrator **control plane** (and,
in a following step, the gateway **data plane**) — the trusted per-host service
the docker backend runs as containers. It boots on the NAT'd orchestrator link
(`netpool.orch_slot()`): the host CLI reaches its control plane over HTTP at the
guest IP, and agent VMs reach its gateway ports over VM-to-VM routing.
Build-from-source (the default while the design churns): the rootfs is exported
from the locally built orchestrator image, which bakes the stdlib-only
control-plane source. A pull-from-registry mode (Gitea's OCI registry) becomes
the default later.
SSH is left enabled for debugging; the control plane is the load-bearing
surface.
"""
from __future__ import annotations
import hashlib
import os
import shlex
import signal
import stat
import subprocess
import time
import urllib.error
import urllib.request
from dataclasses import dataclass
from pathlib import Path
from ...log import die, info
from ..docker import util as docker_mod
from ..docker.gateway_provision import GatewayProvisionError
from . import firecracker_vm, netpool, util
# The single infra-VM image: gateway data plane + baked control-plane source
# (Dockerfile.infra FROM the gateway image). Built from source by default;
# a pull-from-registry mode lands later.
_INFRA_IMAGE = "bot-bottle-infra:latest"
_GATEWAY_IMAGE = "bot-bottle-gateway:latest"
_REPO_ROOT = Path(__file__).resolve().parents[3]
CONTROL_PLANE_PORT = 8099
# Gateway data-plane ports (agent-facing): egress proxy, supervise MCP,
# git-http. Reached by agent VMs over VM-to-VM routing (added next).
EGRESS_PORT = 9099
SUPERVISE_PORT = 9100
GIT_HTTP_PORT = 9420
# mitmproxy writes its CA here a beat after start; agents install it to trust
# the gateway's TLS interception.
_GATEWAY_CA_PATH = "/home/mitmproxy/.mitmproxy/mitmproxy-ca-cert.pem"
# The infra VM makes direct upstream connections (gateway egress, and buildah
# during builds), and the kernel `ip=` cmdline sets no resolver. Public for
# now; routing DNS through a filtered path is a later refinement.
_INFRA_RESOLVER = "1.1.1.1"
_HEALTH_TIMEOUT_SECONDS = 45.0
_HEALTH_POLL_SECONDS = 0.5
_CA_TIMEOUT_SECONDS = 30.0
@dataclass
class InfraVm:
"""A handle to the per-host infra VM: its guest IP and the stable SSH key
used to fetch the gateway CA / provision git-gate. `vm` is the live VMM
handle when this process booted it, and None when adopting a singleton a
prior launcher started (teardown then goes through the PID file)."""
guest_ip: str
private_key: Path
vm: firecracker_vm.VmHandle | None = None
@property
def control_plane_url(self) -> str:
return f"http://{self.guest_ip}:{CONTROL_PLANE_PORT}"
def terminate(self) -> None:
"""Stop the infra VM — via the live handle if we booted it, else the
PID file (adopting-process case)."""
if self.vm is not None:
self.vm.terminate()
else:
_kill_pidfile()
_pid_file().unlink(missing_ok=True)
def gateway_ca_pem(self, *, timeout: float = _CA_TIMEOUT_SECONDS) -> str:
"""The gateway's mitmproxy CA (PEM) that agents install to trust its
TLS interception. Generated a moment after boot, so this polls over
SSH until it appears (mirrors DockerGateway.ca_cert_pem)."""
deadline = time.monotonic() + timeout
while True:
proc = subprocess.run(
util.ssh_base_argv(self.private_key, self.guest_ip)
+ [f"cat {_GATEWAY_CA_PATH}"],
capture_output=True, text=True, timeout=15, check=False,
)
if proc.returncode == 0 and "BEGIN CERTIFICATE" in proc.stdout:
return proc.stdout
if time.monotonic() >= deadline:
die(f"gateway CA not available after {timeout:g}s: "
f"{proc.stderr.strip() or 'empty'}")
time.sleep(_HEALTH_POLL_SECONDS)
def ensure_built() -> None:
"""Build the infra image from source (bootstrap via host docker): the
gateway data-plane image, then the infra image that layers the
control-plane source onto it. A pull-from-registry mode replaces this
later."""
docker_mod.build_image(
_GATEWAY_IMAGE, str(_REPO_ROOT), dockerfile="Dockerfile.gateway")
docker_mod.build_image(
_INFRA_IMAGE, str(_REPO_ROOT), dockerfile="Dockerfile.infra")
def build_infra_rootfs_dir() -> Path:
"""The infra VM's base rootfs: the infra image prepared with the
control-plane + gateway init as PID 1. The init's content is folded into
the cache key so an init change rebuilds the rootfs (the base image digest
alone wouldn't catch it)."""
init = _infra_init()
tag = hashlib.sha256(init.encode()).hexdigest()[:8]
return util.build_base_rootfs_dir(
_INFRA_IMAGE, variant=f"-infra-{tag}", init_script=init,
)
def ensure_running() -> InfraVm:
"""Idempotent per-host singleton. Adopt the infra VM if its control plane
is already healthy (a prior launcher booted it — it outlives short-lived
`start` processes); otherwise clear any stale VM and boot a fresh one.
Returns a handle usable for CA fetch / git-gate provisioning."""
slot = netpool.orch_slot()
url = f"http://{slot.guest_ip}:{CONTROL_PLANE_PORT}"
key = _infra_dir() / "id_ed25519"
if key.exists() and _health_ok(url):
info(f"adopting running infra VM at {url}")
return InfraVm(guest_ip=slot.guest_ip, private_key=key)
stop() # clear a stale/hung VM holding the link before booting fresh
ensure_built()
infra = boot()
wait_for_health(infra)
return infra
def stop() -> None:
"""Stop the infra VM singleton (idempotent — absent is success)."""
_kill_pidfile()
_pid_file().unlink(missing_ok=True)
def boot() -> InfraVm:
"""Boot the infra VM (detached, so it outlives the launcher) on the
orchestrator link, recording its PID. Prefer `ensure_running`."""
slot = netpool.orch_slot()
if not netpool.tap_present(slot.iface):
die(f"orchestrator link {slot.iface} not present.\n"
f" ./cli.py backend setup --backend=firecracker")
base = build_infra_rootfs_dir()
run_dir = _infra_dir()
rootfs = run_dir / "rootfs.ext4"
util.build_rootfs_ext4(base, rootfs, slack_mib=8192)
private_key, pubkey = _stable_keypair()
info(f"booting infra VM on {slot.iface} (guest {slot.guest_ip})")
vm = firecracker_vm.boot(
name="bot-bottle-infra", rootfs=rootfs, tap=slot.iface,
guest_ip=slot.guest_ip, host_ip=slot.host_ip, pubkey=pubkey,
run_dir=run_dir, mem_mib=4096, detached=True,
data_drive=_ensure_registry_volume(),
)
_pid_file().write_text(str(vm.process.pid))
return InfraVm(guest_ip=slot.guest_ip, private_key=private_key, vm=vm)
def _infra_dir() -> Path:
d = util.cache_dir() / "infra"
d.mkdir(parents=True, exist_ok=True)
return d
def _pid_file() -> Path:
return _infra_dir() / "vm.pid"
# The registry "volume": a host-side ext4 file attached to the infra VM as a
# second virtio-block device (guest /dev/vdb), mounted at the control plane's
# DB dir. It outlives the ephemeral rootfs, so the bottle registry survives an
# infra-VM restart — the firecracker analogue of a docker volume. It is a
# plain ext4 file: `sudo mount -o loop <path>` on the host (with the VM
# stopped) to inspect bot-bottle.db directly.
_REGISTRY_SIZE = "512M"
def registry_volume_path() -> Path:
return _infra_dir() / "registry.ext4"
def _ensure_registry_volume() -> Path:
"""Create the empty ext4 registry volume on first use; reuse it after."""
vol = registry_volume_path()
if vol.exists():
return vol
info(f"creating infra registry volume {vol} ({_REGISTRY_SIZE})")
proc = subprocess.run(
["mke2fs", "-q", "-t", "ext4", "-F", str(vol), _REGISTRY_SIZE],
capture_output=True, text=True, check=False,
)
if proc.returncode != 0:
vol.unlink(missing_ok=True)
die(f"creating registry volume failed: {proc.stderr.strip()}")
return vol
def _stable_keypair() -> tuple[Path, str]:
"""The infra VM's SSH keypair — generated once and reused, so any later
launcher can SSH in (fetch CA / provision) even though a different process
booted the VM. The pubkey is re-injected on every boot via the cmdline."""
d = _infra_dir()
key, pub = d / "id_ed25519", d / "id_ed25519.pub"
if key.exists() and pub.exists():
return key, pub.read_text().strip()
key.unlink(missing_ok=True)
pub.unlink(missing_ok=True)
subprocess.run(
["ssh-keygen", "-t", "ed25519", "-N", "", "-q", "-f", str(key),
"-C", "bot-bottle-infra"],
check=True,
)
return key, pub.read_text().strip()
def _kill_pidfile() -> None:
"""SIGTERM (then SIGKILL) the recorded infra VMM, if it's still ours.
Guards against a recycled PID by checking the process is firecracker."""
try:
pid = int(_pid_file().read_text().strip())
except (OSError, ValueError):
return
try:
comm = Path(f"/proc/{pid}/comm").read_text().strip()
except OSError:
return # already gone
if comm != "firecracker":
return # PID recycled by an unrelated process
try:
os.kill(pid, signal.SIGTERM)
for _ in range(50):
if not Path(f"/proc/{pid}").exists():
return
time.sleep(0.1)
os.kill(pid, signal.SIGKILL)
except OSError:
pass
def _health_ok(url: str) -> bool:
try:
with urllib.request.urlopen(f"{url}/health", timeout=1.0) as resp:
return resp.status == 200
except (urllib.error.URLError, TimeoutError, OSError):
return False
class SshGatewayTransport:
"""`GatewayTransport` for the gateway running in the infra VM — the docker
exec/cp equivalents over SSH (dropbear + the stable infra key)."""
def __init__(self, private_key: Path, guest_ip: str) -> None:
self._key = private_key
self._ip = guest_ip
def exec(self, argv: list[str]) -> None:
proc = subprocess.run(
util.ssh_base_argv(self._key, self._ip) + [shlex.join(argv)],
capture_output=True, text=True, timeout=60, check=False,
)
if proc.returncode != 0:
raise GatewayProvisionError(
f"infra gateway exec {argv!r} failed: {proc.stderr.strip()}")
def cp_into(self, src: str, dest: str) -> None:
# Preserve the source mode (docker cp does): the access-hook is staged
# 0700 and git-http execs it directly — a plain `cat >` would land it
# 0644 and the exec fails with EACCES; keys stay 0600.
mode = stat.S_IMODE(os.stat(src).st_mode)
q = shlex.quote(dest)
proc = subprocess.run(
util.ssh_base_argv(self._key, self._ip)
+ [f"cat > {q} && chmod {mode:o} {q}"],
input=Path(src).read_bytes(), capture_output=True, timeout=30, check=False,
)
if proc.returncode != 0:
raise GatewayProvisionError(
f"infra gateway cp {src} -> {dest} failed: "
f"{proc.stderr.decode(errors='replace').strip()}")
def gateway_transport() -> SshGatewayTransport:
"""git-gate provisioning transport for the gateway in the infra VM, built
from the stable key + the orchestrator link's guest IP. Needs no live VM
handle, so teardown can use it too."""
return SshGatewayTransport(
_infra_dir() / "id_ed25519", netpool.orch_slot().guest_ip)
def wait_for_health(
infra: InfraVm, *, timeout: float = _HEALTH_TIMEOUT_SECONDS,
) -> None:
"""Poll the control plane's /health until it answers 200 or the deadline
passes. Dies (with the console tail) if the VMM exits early."""
url = f"{infra.control_plane_url}/health"
deadline = time.monotonic() + timeout
while time.monotonic() < deadline:
if infra.vm is not None and not infra.vm.is_alive():
die(f"infra VM exited during boot (rc={infra.vm.process.returncode}).\n"
f"{firecracker_vm._console_tail(infra.vm.console_log)}")
try:
with urllib.request.urlopen(url, timeout=1.0) as resp:
if resp.status == 200:
info(f"infra control plane healthy at {infra.control_plane_url}")
return
except (urllib.error.URLError, TimeoutError, OSError):
pass
time.sleep(_HEALTH_POLL_SECONDS)
tail = (firecracker_vm._console_tail(infra.vm.console_log)
if infra.vm is not None else "")
die(f"infra control plane at {url} did not become healthy within "
f"{timeout:.0f}s.\n{tail}")
def _infra_init() -> str:
"""PID-1 init for the infra VM: mount the pseudo-filesystems, wire a
resolver, start dropbear (debug SSH), then launch the control plane and
the gateway data plane (multi-tenant against the local control plane)."""
return f"""#!/bin/sh
# bot-bottle Firecracker infra VM init (PID 1).
mount -t proc proc /proc 2>/dev/null
mount -t sysfs sys /sys 2>/dev/null
mount -t devtmpfs dev /dev 2>/dev/null
mkdir -p /dev/pts && mount -t devpts devpts /dev/pts 2>/dev/null
mount -o remount,rw / 2>/dev/null
# Export a real PATH: a bare-init shell resolves its own execs via a
# built-in default path, but that isn't in the *environment*, so
# gateway_init's subprocess daemons (spawned as `python3 ...`) would
# inherit no PATH and fail to find python3. Export it for all children.
export PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
# Direct upstream resolver (control-plane / gateway egress + buildah).
printf 'nameserver {_INFRA_RESOLVER}\\n' > /etc/resolv.conf 2>/dev/null
# Debug SSH: install the per-boot pubkey from the kernel cmdline.
KEY=$(sed -n 's/.*bb_pubkey=\\([^ ]*\\).*/\\1/p' /proc/cmdline | base64 -d 2>/dev/null)
if [ -n "$KEY" ]; then
mkdir -p /root/.ssh
printf '%s\\n' "$KEY" > /root/.ssh/authorized_keys
chmod 700 /root/.ssh && chmod 600 /root/.ssh/authorized_keys
fi
chown -R 0:0 /root 2>/dev/null || true
mkdir -p /etc/dropbear /run /var/lib/bot-bottle
# Persistent registry volume (second virtio-block device, /dev/vdb) mounted
# at the control plane's DB dir, so bot-bottle.db survives infra-VM restarts.
mount -t ext4 /dev/vdb /var/lib/bot-bottle 2>/dev/null || true
/bb-dropbear -R -E -p 22 &
# Control plane. Source is baked at /app; the package is stdlib-only.
cd /app
BOT_BOTTLE_ROOT=/var/lib/bot-bottle python3 -m bot_bottle.orchestrator \\
--host 0.0.0.0 --port {CONTROL_PLANE_PORT} --broker stub &
# Gateway data plane, multi-tenant: each request resolves source-IP ->
# policy against the local control plane. The VM backend reaches git over
# git-http (9420), so the git:// daemon (git-gate, needs a per-bottle
# entrypoint the consolidated model doesn't use) is left out.
BOT_BOTTLE_GATEWAY_DAEMONS=egress,git-http,supervise \\
BOT_BOTTLE_ORCHESTRATOR_URL=http://127.0.0.1:{CONTROL_PLANE_PORT} \\
python3 /app/gateway_init.py &
# Reap as PID 1; children are backgrounded, so `wait` blocks.
while : ; do wait ; done
"""
@@ -1,116 +0,0 @@
"""Empirical, post-boot isolation probe — the authoritative fail-closed
egress-boundary check.
The pre-boot nft check can't be trusted on every host (listing an
nftables table typically needs root; the launcher runs unprivileged).
So before the agent ever runs, we prove the boundary directly: open a
canary TCP listener on a host IP the VM must NOT be able to reach (the
host's primary non-TAP address), then have the guest try to connect. If
the isolation rules are in force the connection is dropped and times
out; if it succeeds, the VM can reach host services it shouldn't — a
sandbox escape — and we refuse to continue.
The listener is load-bearing: without it a "blocked" result could just
be connection-refused. With it, a reachable canary means a real leak
and a timeout means a real drop.
"""
from __future__ import annotations
import socket
import subprocess
import threading
from pathlib import Path
from ...log import die, info, warn
from . import util
def _host_primary_ip() -> str:
"""The source IP the host uses for off-box traffic (its LAN
address) — a canary the VM must not reach. Empty if the host has no
such route (isolated box)."""
result = subprocess.run(
["ip", "-o", "route", "get", "1.1.1.1"],
capture_output=True, text=True, check=False,
)
if result.returncode != 0:
return ""
tokens = result.stdout.split()
# `... src <addr> ...` — the address the host would use as source.
if "src" in tokens:
idx = tokens.index("src")
if idx + 1 < len(tokens):
return tokens[idx + 1]
return ""
# Guest-side connect test: exit 0 = reached the canary (LEAK), 1 =
# blocked (good), 2 = no usable tool (inconclusive -> fail-closed).
_GUEST_PROBE = r"""
h="$1"; p="$2"
if command -v python3 >/dev/null 2>&1; then
python3 - "$h" "$p" <<'PY'
import socket, sys
s = socket.socket(); s.settimeout(3)
sys.exit(0 if s.connect_ex((sys.argv[1], int(sys.argv[2]))) == 0 else 1)
PY
elif command -v bash >/dev/null 2>&1; then
timeout 3 bash -c "exec 3<>/dev/tcp/$h/$p" >/dev/null 2>&1
elif command -v nc >/dev/null 2>&1; then
nc -w3 -z "$h" "$p" >/dev/null 2>&1
else
exit 2
fi
"""
def verify_isolation(private_key: Path, guest_ip: str) -> None:
"""Prove the VM cannot reach the host's primary IP. Dies
(fail-closed) on a confirmed leak or if the guest has no tool to
run the test. Warns (does not fail) when the host has no canary
address to test against."""
canary_ip = _host_primary_ip()
if not canary_ip:
warn("isolation probe skipped: host has no non-TAP route to test "
"against (isolated box).")
return
listener = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
listener.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
listener.bind((canary_ip, 0))
listener.listen(1)
listener.settimeout(6)
canary_port = listener.getsockname()[1]
accepted: list[bool] = []
def _accept() -> None:
try:
conn, _ = listener.accept()
accepted.append(True)
conn.close()
except OSError:
pass
thread = threading.Thread(target=_accept, daemon=True)
thread.start()
ssh = util.ssh_base_argv(private_key, guest_ip)
result = subprocess.run(
[*ssh, "--", "sh", "-s", "--", canary_ip, str(canary_port)],
input=_GUEST_PROBE, capture_output=True, text=True, check=False,
)
thread.join(timeout=7)
listener.close()
if result.returncode == 0 or accepted:
die(f"ISOLATION FAILURE: the VM reached the host canary "
f"{canary_ip}:{canary_port}. The egress boundary is not in "
f"force — refusing to run the agent (fail-closed). Verify the "
f"nft table with: ./cli.py backend setup --backend=firecracker")
if result.returncode == 2:
die("isolation probe inconclusive: the guest has no python3/bash/nc "
"to run the connectivity test. Refusing to continue "
"(fail-closed).")
info(f"isolation verified: VM cannot reach host {canary_ip}")
-253
View File
@@ -1,253 +0,0 @@
"""Launch flow for the Firecracker backend (PRD 0070, consolidated).
Per bottle:
1. build the agent image (docker), export it to a cached ext4 rootfs;
2. ensure the per-host orchestrator + shared gateway are up;
3. claim a free TAP pool slot (rootless flock);
4. register the bottle on the orchestrator by the VM's guest IP (the
attribution key) and provision its git-gate state into the gateway;
5. boot the microVM on that TAP; wait for SSH;
6. provision (shared gateway CA, prompt, skills, workspace, git, supervise)
over SSH.
The per-bottle Docker sidecar bundle is gone. The shared gateway handles
egress / git-gate / supervise for every VM; Docker's PREROUTING DNAT routes
the VMs' traffic to it, and the nft table's `ct status dnat accept` rule
in the forward chain lets it pass. The VM still sends to `host_tap_ip:PORT`
— the address its world is, by nft design, limited to.
Isolation is enforced by the operator-provisioned nft table (checked
fail-closed in preflight): a VM reaches only the sidecar (DNAT'd from
the host TAP IP) and nothing else.
"""
from __future__ import annotations
import dataclasses
import os
from contextlib import ExitStack, contextmanager
from pathlib import Path
from typing import Callable, Generator
from ...agent_provider import runtime_for
from ...bottle_state import (
egress_state_dir,
git_gate_state_dir,
read_committed_image,
)
from ...egress import (
egress_agent_env_entries,
egress_resolve_token_values,
)
from ...git_gate import (
provision_git_gate_dynamic_keys,
revoke_git_gate_provisioned_keys,
)
from ...log import info, warn
from ...supervise import SUPERVISE_PORT
from ..docker import util as docker_mod
from ..docker.egress import EGRESS_PORT
from ..util import AGENT_CA_BUNDLE, AGENT_CA_PATH
from . import firecracker_vm, image_builder, isolation_probe, netpool, util
from .bottle import FirecrackerBottle
from .bottle_plan import FirecrackerBottlePlan
from .consolidated_launch import (
launch_consolidated,
teardown_consolidated,
)
_GIT_HTTP_PORT = 9420
@contextmanager
def launch(
plan: FirecrackerBottlePlan,
*,
provision: Callable[[FirecrackerBottlePlan, "FirecrackerBottle"], str | None],
) -> Generator[FirecrackerBottle, None, None]:
"""Build, launch, and provision a Firecracker bottle via the consolidated
orchestrator. Teardown on exit."""
stack = ExitStack()
bottle_for_revoke = plan.manifest.bottle
git_gate_dir_for_revoke = git_gate_state_dir(plan.slug)
def teardown() -> None:
teardown_exc: BaseException | None = None
try:
stack.close()
except BaseException as exc: # noqa: W0718 - teardown must continue
teardown_exc = exc
warn(f"firecracker teardown failed: {exc!r}")
revoke_git_gate_provisioned_keys(bottle_for_revoke, git_gate_dir_for_revoke)
if teardown_exc is not None:
raise teardown_exc
try:
# Step 1: agent rootfs. Built from the Dockerfile inside a Firecracker
# builder VM (buildah, no host docker); a committed snapshot is reused
# when present. Returns the base dir the per-bottle ext4 is made from.
plan, agent_base = _build_agent_base(plan)
# Step 2: mint the git-gate dynamic (gitea) deploy keys, if any.
git_gate_plan = plan.git_gate_plan
if git_gate_plan.upstreams:
git_gate_plan = provision_git_gate_dynamic_keys(
plan.manifest.bottle, git_gate_plan, git_gate_state_dir(plan.slug),
)
# Step 3: claim a TAP slot; the flock is held until teardown.
slot, lock = netpool.allocate(plan.slug)
stack.callback(lock.close)
info(f"firecracker slot {slot.iface}: host={slot.host_ip} "
f"guest={slot.guest_ip}")
# Step 4: register on the orchestrator + provision this bottle's
# git-gate state into the shared gateway. The per-bottle egress tokens
# are resolved from the host env now and handed to the orchestrator
# (in memory) for the gateway to inject — the agent never sees them.
# Attribution is by the VM's guest IP (unspoofable via /31 TAP + nft).
effective_env = {**os.environ, **plan.agent_provision.provisioned_env}
token_values = egress_resolve_token_values(
plan.egress_plan.token_env_map, effective_env,
)
ctx = launch_consolidated(
plan.egress_plan, git_gate_plan,
guest_ip=slot.guest_ip,
image_ref=plan.image,
tokens=token_values,
)
stack.callback(
teardown_consolidated, ctx.bottle_id,
orchestrator_url=ctx.orchestrator_url,
)
# Step 5: install the SHARED gateway CA (replaces the per-bottle CA).
# Write it to a stable host path so the provisioner can copy it over SSH.
ca_dir = egress_state_dir(plan.slug) / "gateway-ca"
ca_dir.mkdir(parents=True, exist_ok=True)
ca_file = ca_dir / "gateway-ca.pem"
ca_file.write_text(ctx.gateway_ca_pem)
egress_plan = dataclasses.replace(
plan.egress_plan,
mitmproxy_ca_host_path=ca_file,
mitmproxy_ca_cert_only_host_path=ca_file,
)
# Point the agent's git-gate insteadOf rewrites and supervise MCP URL
# at the shared gateway (reached at the slot's host TAP IP — the VM
# sends there and Docker DNAT routes to the gateway container).
git_gate_url = (
f"http://{slot.host_ip}:{_GIT_HTTP_PORT}" if git_gate_plan.upstreams else ""
)
supervise_url = (
f"http://{slot.host_ip}:{SUPERVISE_PORT}/"
if plan.supervise_plan is not None else ""
)
plan = dataclasses.replace(
plan,
git_gate_plan=git_gate_plan,
egress_plan=egress_plan,
agent_proxy_url=f"http://{slot.host_ip}:{EGRESS_PORT}",
agent_git_gate_url=git_gate_url,
agent_supervise_url=supervise_url,
)
# Step 6: build the per-bottle rootfs + SSH key, then boot.
run_dir = util.cache_dir() / "run" / plan.slug
run_dir.mkdir(parents=True, exist_ok=True)
rootfs = run_dir / "rootfs.ext4"
util.build_rootfs_ext4(agent_base, rootfs)
private_key, pubkey = util.generate_keypair(run_dir)
vm = firecracker_vm.boot(
name=plan.container_name,
rootfs=rootfs,
tap=slot.iface,
guest_ip=slot.guest_ip,
host_ip=slot.host_ip,
pubkey=pubkey,
run_dir=run_dir,
)
stack.callback(vm.terminate)
firecracker_vm.wait_for_ssh(vm, private_key)
# Authoritative fail-closed egress-boundary check, before the agent
# runs: prove the VM cannot reach the host directly.
isolation_probe.verify_isolation(private_key, slot.guest_ip)
bottle = FirecrackerBottle(
plan.container_name,
private_key=private_key,
guest_ip=slot.guest_ip,
guest_env=_agent_guest_env(plan, slot.host_ip),
agent_command=plan.agent_command,
agent_prompt_mode=plan.agent_prompt_mode,
agent_provider_template=plan.agent_provider_template,
terminal_title=(
f"{plan.spec.label} ({plan.spec.agent_name})"
if plan.spec.label else plan.spec.agent_name
),
terminal_color=plan.spec.color,
agent_workdir=plan.workspace_plan.workdir,
)
bottle.prompt_path = provision(plan, bottle)
yield bottle
finally:
teardown()
def _build_agent_base(
plan: FirecrackerBottlePlan,
) -> tuple[FirecrackerBottlePlan, Path]:
"""Produce the agent's base rootfs dir. Primary path: build the Dockerfile
inside a Firecracker builder VM (buildah, no host docker), smoke-testing
the image before export. A committed snapshot (freeze/migrate) is still
exported via the host docker path until that is ported too."""
committed = read_committed_image(plan.slug)
if committed and docker_mod.image_exists(committed):
info(f"using committed image {committed!r}")
plan = dataclasses.replace(
plan,
agent_provision=dataclasses.replace(plan.agent_provision, image=committed),
)
return plan, util.build_base_rootfs_dir(committed)
base = image_builder.build_agent_rootfs_dir(
Path(plan.dockerfile_path),
image_tag=plan.image,
smoke_test=runtime_for(plan.agent_provider_template).smoke_test,
)
return plan, base
# --- agent guest env -------------------------------------------------
def _agent_guest_env(plan: FirecrackerBottlePlan, host_ip: str) -> dict[str, str]:
"""Env injected into every agent/exec call over SSH. The VM has no
baked process env (it just runs init), so the proxy/CA/git/supervise
wiring is applied per-invocation."""
proxy_url = f"http://{host_ip}:{EGRESS_PORT}"
no_proxy = f"localhost,127.0.0.1,{host_ip}"
env: dict[str, str] = {
"HTTPS_PROXY": proxy_url, "HTTP_PROXY": proxy_url,
"https_proxy": proxy_url, "http_proxy": proxy_url,
"NO_PROXY": no_proxy, "no_proxy": no_proxy,
"NODE_EXTRA_CA_CERTS": AGENT_CA_PATH,
"SSL_CERT_FILE": AGENT_CA_BUNDLE,
"REQUESTS_CA_BUNDLE": AGENT_CA_BUNDLE,
}
if plan.agent_git_gate_url:
env["GIT_GATE_URL"] = plan.agent_git_gate_url
if plan.agent_supervise_url:
env["MCP_SUPERVISE_URL"] = plan.agent_supervise_url
for entry in egress_agent_env_entries(plan.egress_plan):
key, _, value = entry.partition("=")
env[key] = value
env.update(plan.agent_provision.guest_env)
# Forwarded (bare-name) env: resolve host values now, since the VM
# can't inherit them from a `docker run --env NAME`.
for name in plan.forwarded_env:
value = os.environ.get(name)
if value is not None:
env[name] = value
return env
@@ -1,25 +0,0 @@
# Firecracker network-pool defaults — the SINGLE source of these values.
#
# Read by every consumer so they can't drift:
# * netpool.py — parses this for the Python defaults (below).
# * scripts/firecracker-netpool.sh — falls back to these when the
# matching BOT_BOTTLE_FC_* env var is unset.
# * nix/firecracker-netpool.nix — readFile-parses this for its option
# defaults, then passes the resolved values back as Environment=.
#
# Plain KEY=VALUE (no quoting, no inline comments, no spaces around `=`)
# so it is bash-sourceable, systemd EnvironmentFile-compatible, and
# trivially parseable from Python and Nix. A real BOT_BOTTLE_FC_* env
# var of the same name always overrides the value here.
BOT_BOTTLE_FC_POOL_SIZE=8
BOT_BOTTLE_FC_IP_BASE=10.243.0.0
BOT_BOTTLE_FC_IFACE_PREFIX=bbfc
BOT_BOTTLE_FC_NFT_TABLE=bot_bottle_fc
# The orchestrator/gateway VM's own TAP — a dedicated link OUTSIDE the
# bbfc* agent pool. Unlike agent VMs (which reach only their gateway),
# the orchestrator is trusted infra that needs real NAT'd internet
# egress: to FROM-pull + apt/npm during in-VM agent-image builds
# (buildah) and to forward agent egress upstream (Stage B gateway). Its
# /31 is the top of the IP_BASE /16 (host x.y.255.0, guest x.y.255.1),
# clear of the pool near the bottom of the block.
BOT_BOTTLE_FC_ORCH_IFACE=bborch0
-346
View File
@@ -1,346 +0,0 @@
"""Firecracker network pool: constants, IP math, allocation, and the
config renderers (shell command + NixOS module) shown to operators.
The Firecracker backend needs a privileged one-time network setup:
a pool of point-to-point TAP devices (owned by the invoking user, so
`./cli.py start` never needs root) and a dedicated nftables table that
isolates every VM. The pool parameters live in exactly one place —
`netpool.defaults.env`, a plain KEY=VALUE file next to this module —
and every consumer reads *that*: this module (below), the shell script
(`scripts/firecracker-netpool.sh`), and the NixOS module
(`nix/firecracker-netpool.nix`). A `BOT_BOTTLE_FC_*` env var of the
same name always overrides the file, and the backend's fail-closed
preflight derives from these accessors, so nothing can drift.
Topology (per slot i):
* TAP ``bbfc{i}`` — no shared bridge, so no docker0 / virbr0 / cni0
/ br-* collisions.
* a /31 host<->guest link: host = base + 2i (the gateway the VM
routes through), guest = base + 2i + 1 (the VM's address).
* isolation via ``table inet bot_bottle_fc``: a VM reaches only its
own gateway (DNAT'd from the host TAP IP) and nothing else.
The default IP block is ``10.243.0.0/16`` — an intentionally obscure
corner of RFC-1918 private space. RFC-1918 is the range *designated*
for private links; we pick a high, unusual /16 to steer clear of the
common occupants (Docker's 172.17-31, libvirt's 192.168.122, k8s
10.42/10.244, and typical home LANs on 192.168.0/1.x or 10.0.0.x).
We deliberately avoid ``100.64.0.0/10`` (RFC-6598 CGNAT) because
Tailscale hands out node addresses from exactly that range. No default
is collision-proof, so `overlapping_routes()` is the real guard —
`backend setup`/`status` and the launch preflight call it to catch
a base that clashes with something already on the host.
"""
from __future__ import annotations
import fcntl
import ipaddress
import os
import subprocess
from dataclasses import dataclass
from pathlib import Path
from typing import IO
from ...log import die
# The pool defaults live in one shared file (see module docstring); the
# shell script and NixOS module read the same file, so the values can't
# drift. This is a packaged data file — a missing/broken install is a
# hard error, surfaced here rather than as confusing empty defaults.
DEFAULTS_FILE = Path(__file__).with_name("netpool.defaults.env")
def _load_defaults() -> dict[str, str]:
out: dict[str, str] = {}
for raw in DEFAULTS_FILE.read_text().splitlines():
line = raw.strip()
if not line or line.startswith("#") or "=" not in line:
continue
key, _, value = line.partition("=")
out[key.strip()] = value.strip()
return out
_DEFAULTS = _load_defaults()
def _cfg(key: str) -> str:
"""A `BOT_BOTTLE_FC_*` env var overrides the shared-file default."""
try:
return os.environ.get(key) or _DEFAULTS[key]
except KeyError:
die(f"{key} is missing from {DEFAULTS_FILE.name} (broken install)")
# Interface names are capped at 15 chars (IFNAMSIZ-1); "bbfc" + a small
# index stays well under that and is distinctive enough to grep for.
IFACE_PREFIX = _cfg("BOT_BOTTLE_FC_IFACE_PREFIX")
NFT_TABLE = _cfg("BOT_BOTTLE_FC_NFT_TABLE")
# The orchestrator/gateway VM's dedicated TAP — outside the bbfc* agent
# pool and, unlike it, NAT'd to the internet (see `orch_slot`). The
# orchestrator is trusted infra: it builds agent images in-VM (buildah
# needs to FROM-pull + apt/npm) and forwards agent egress upstream.
ORCH_IFACE = _cfg("BOT_BOTTLE_FC_ORCH_IFACE")
def pool_size() -> int:
return int(_cfg("BOT_BOTTLE_FC_POOL_SIZE"))
def ip_base() -> str:
return _cfg("BOT_BOTTLE_FC_IP_BASE")
# Gateway ports the VM reaches at its host-side TAP IP. Kept in sync
# with the backend constants (egress 9099, supervise 9100, git-http
# 9420); rendered into the setup output for operator visibility.
GATEWAY_PORTS = (9099, 9100, 9420)
@dataclass(frozen=True)
class Slot:
"""One pool slot: a TAP device and its host/guest /31 addresses."""
index: int
iface: str
host_ip: str
guest_ip: str
@property
def guest_cidr(self) -> str:
"""Guest address with the /31 prefix, for the kernel `ip=` arg."""
return f"{self.guest_ip}/31"
def slot(index: int) -> Slot:
base = int(ipaddress.IPv4Address(ip_base()))
return Slot(
index=index,
iface=f"{IFACE_PREFIX}{index}",
host_ip=str(ipaddress.IPv4Address(base + 2 * index)),
guest_ip=str(ipaddress.IPv4Address(base + 2 * index + 1)),
)
def all_slots() -> list[Slot]:
return [slot(i) for i in range(pool_size())]
def orch_slot() -> Slot:
"""The orchestrator/gateway VM's dedicated link — its own TAP
(`ORCH_IFACE`) on a /31 at the TOP of the IP_BASE /16 (host
x.y.255.0, guest x.y.255.1), well clear of the agent pool near the
bottom of the block. Unlike a pool `Slot`, this link is NAT'd out to
the internet by the setup (the orchestrator is trusted infra), so it
is deliberately *not* one of the isolated `bbfc*` slots.
`index` is -1 (sentinel: not a pool index)."""
base16 = int(ipaddress.IPv4Address(ip_base())) & 0xFFFF0000
host = base16 + 0xFF00
return Slot(
index=-1,
iface=ORCH_IFACE,
host_ip=str(ipaddress.IPv4Address(host)),
guest_ip=str(ipaddress.IPv4Address(host + 1)),
)
# --- fail-closed verification ---------------------------------------
def _run_ok(argv: list[str]) -> bool:
"""Run a probe command, treating a missing binary as failure
(rather than crashing) so callers can stay fail-closed."""
try:
return subprocess.run(
argv, stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL,
check=False,
).returncode == 0
except FileNotFoundError:
return False
def nft_table_present() -> bool:
"""True iff the isolation table exists in the active nftables
backend. The backend's preflight treats absence as fatal — the VM
must not boot without its egress boundary in place.
Listing may require root; when it can't be confirmed here the
launch path falls back to an empirical post-boot isolation probe.
Returns False (fail-closed) if `nft` is unavailable."""
return _run_ok(["nft", "list", "table", "inet", NFT_TABLE])
def tap_present(iface: str) -> bool:
# `ip link show` is unprivileged, so the TAP-pool check is reliable
# for the non-root launcher.
return _run_ok(["ip", "link", "show", iface])
def missing_taps() -> list[str]:
"""Pool TAPs that the one-time setup has not created yet."""
return [s.iface for s in all_slots() if not tap_present(s.iface)]
@dataclass(frozen=True)
class RouteConflict:
"""An existing host route whose destination overlaps the pool range
but isn't one of our own bbfc TAPs — i.e. the chosen IP base clashes
with something already on the host (a Tailscale CGNAT peer, a
docker/libvirt bridge, the LAN…)."""
dst: str
dev: str
def _pool_span() -> tuple[int, int]:
"""Inclusive [first, last] integer bounds of every address the pool
occupies: base .. base + 2*pool_size - 1."""
base = int(ipaddress.IPv4Address(ip_base()))
return base, base + 2 * pool_size() - 1
def overlapping_routes() -> list[RouteConflict]:
"""Existing routes whose destination overlaps the pool's address
range, excluding our own `bbfc*` TAP routes and the default route.
A non-empty result means `BOT_BOTTLE_FC_IP_BASE` collides with
something already configured on this host, so the pool would shadow
(or be shadowed by) it. Empty on success — or when `ip` is missing
or unparseable, since this is an advisory guard, not the fail-closed
isolation check (that's the nft table + post-boot probe)."""
import json
try:
proc = subprocess.run(
["ip", "-json", "route", "show", "table", "all"],
capture_output=True, text=True, check=False,
)
except FileNotFoundError:
return []
if proc.returncode != 0 or not proc.stdout.strip():
return []
try:
routes = json.loads(proc.stdout)
except json.JSONDecodeError:
return []
lo, hi = _pool_span()
conflicts: list[RouteConflict] = []
for r in routes:
if not isinstance(r, dict):
continue
dst = r.get("dst")
dev = r.get("dev", "")
if not isinstance(dst, str) or dst in ("", "default"):
continue
if isinstance(dev, str) and dev.startswith(IFACE_PREFIX):
continue # our own pool link
try:
net = ipaddress.ip_network(dst, strict=False)
except ValueError:
continue
if net.version != 4:
continue
r_lo = int(net.network_address)
r_hi = int(net.broadcast_address)
if r_lo <= hi and lo <= r_hi: # ranges intersect
conflicts.append(RouteConflict(dst=dst, dev=str(dev)))
return conflicts
# --- allocation ------------------------------------------------------
def _lock_dir() -> Path:
d = Path.home() / ".cache" / "bot-bottle" / "firecracker" / "pool"
d.mkdir(parents=True, exist_ok=True)
return d
def allocate(slug: str) -> tuple[Slot, IO[str]]:
"""Claim a free pool slot for one bottle. Returns the slot and the
held lock file — the caller keeps it open for the VM's lifetime and
closes it on teardown; the flock auto-releases if the launcher
crashes, so a slot is never leaked. Dies when the pool is
exhausted.
A per-slot `flock` (rather than inspecting running processes) makes
allocation race-free across concurrent launches without a central
registry: the first launcher to grab the lock owns the slot."""
del slug # logged by the caller; allocation is purely lock-driven
for s in all_slots():
lock_path = _lock_dir() / f"{s.iface}.lock"
handle = open(lock_path, "w", encoding="utf-8")
try:
fcntl.flock(handle, fcntl.LOCK_EX | fcntl.LOCK_NB)
except OSError:
handle.close()
continue
return s, handle
die(f"Firecracker TAP pool exhausted ({pool_size()} slots, all in "
f"use). Stop a running bottle or raise BOT_BOTTLE_FC_POOL_SIZE "
f"and re-run `./cli.py backend setup --backend=firecracker`.")
raise AssertionError("unreachable")
# --- config renderers (shown by `./cli.py backend setup`) -----------
# The persistent unit is the portable install: the same systemd oneshot
# on every systemd distro (Debian/Ubuntu/Fedora/RHEL/Arch/…).
SYSTEMD_UNIT = "bot-bottle-firecracker-netpool.service"
def render_shell_setup() -> str:
"""The imperative one-shot command — non-persistent fallback for
hosts without systemd (OpenRC/runit/manual)."""
env = _nondefault_env()
prefix = f"{env} " if env else ""
return f"sudo {prefix}./scripts/firecracker-netpool.sh up"
def render_systemd_unit(owner: str, script_path: str) -> str:
"""A portable systemd oneshot unit for the pool — identical across
every systemd distro. ExecStart/ExecStop delegate to the bundled
shell script (the single source of bring-up logic); pool params are
pinned via Environment= so the unit matches the CLI's current
settings and doesn't depend on $SUDO_USER at boot (systemd runs it
as root with no SUDO_USER, which would otherwise own the TAPs as
root and break the rootless launch)."""
return f"""[Unit]
Description=bot-bottle Firecracker TAP pool + nft isolation table
After=network-pre.target
Wants=network-pre.target
[Service]
Type=oneshot
RemainAfterExit=yes
Environment=BOT_BOTTLE_FC_POOL_SIZE={pool_size()}
Environment=BOT_BOTTLE_FC_IP_BASE={ip_base()}
Environment=BOT_BOTTLE_FC_IFACE_PREFIX={IFACE_PREFIX}
Environment=BOT_BOTTLE_FC_OWNER={owner}
ExecStart={script_path} up
ExecStop={script_path} down
[Install]
WantedBy=multi-user.target
"""
# The NixOS setup is a real, importable module (nix/firecracker-netpool.nix,
# exposed as the flake output nixosModules.firecracker-netpool) rather than a
# generated paste — see `backend setup` output.
def _nondefault_env() -> str:
"""Render any non-default pool env overrides so the printed shell
command reproduces the operator's current settings."""
pairs = []
if os.environ.get("BOT_BOTTLE_FC_POOL_SIZE"):
pairs.append(f"BOT_BOTTLE_FC_POOL_SIZE={pool_size()}")
if os.environ.get("BOT_BOTTLE_FC_IP_BASE"):
pairs.append(f"BOT_BOTTLE_FC_IP_BASE={ip_base()}")
if os.environ.get("BOT_BOTTLE_FC_IFACE_PREFIX"):
pairs.append(f"BOT_BOTTLE_FC_IFACE_PREFIX={IFACE_PREFIX}")
return " ".join(pairs)
@@ -1,47 +0,0 @@
"""Prepare step for the Firecracker backend."""
from __future__ import annotations
from pathlib import Path
from ...agent_provider import AgentProvisionPlan
from ...egress import EgressPlan
from ...env import ResolvedEnv
from ...git_gate import GitGatePlan
from ...manifest import Manifest
from ...supervise import SupervisePlan
from .. import BottleSpec
from . import util
from .bottle_plan import FirecrackerBottlePlan
def preflight() -> None:
util.require_firecracker()
def build_guest_env(resolved_env: ResolvedEnv) -> dict[str, str]:
return dict(resolved_env.literals)
def resolve_plan(
spec: BottleSpec,
manifest: Manifest,
slug: str,
resolved_env: ResolvedEnv,
agent_provision_plan: AgentProvisionPlan,
egress_plan: EgressPlan,
supervise_plan: SupervisePlan | None,
git_gate_plan: GitGatePlan,
stage_dir: Path,
) -> FirecrackerBottlePlan:
return FirecrackerBottlePlan(
spec=spec,
manifest=manifest,
stage_dir=stage_dir,
slug=slug,
forwarded_env=dict(resolved_env.forwarded),
git_gate_plan=git_gate_plan,
egress_plan=egress_plan,
supervise_plan=supervise_plan,
agent_provision=agent_provision_plan,
)
-276
View File
@@ -1,276 +0,0 @@
"""Host setup + status for the Firecracker backend.
`setup()` prints the host-appropriate config for the privileged,
one-time network pool (TAP devices + isolation nftables table) the
backend needs. On NixOS it points at the flake module (and prints a
paste-able fallback); elsewhere it prints the sudo command for the
bundled setup script. `status()` reports what's present, including
whether the pool range collides with an existing route.
Called through `FirecrackerBottleBackend.setup` / `.status`, which the
generic `./cli.py backend {setup,status}` command dispatches to.
"""
from __future__ import annotations
import os
import shutil
import subprocess
import sys
from pathlib import Path
from . import netpool
from . import util
_FC_RELEASES = "https://github.com/firecracker-microvm/firecracker/releases"
_UNIT_PATH = Path("/etc/systemd/system") / netpool.SYSTEMD_UNIT
def _owner() -> str:
# Under `sudo`, USER is root but SUDO_USER is the real invoker — the
# TAPs must be owned by them so `./cli.py start` stays rootless.
return os.environ.get("SUDO_USER") or os.environ.get("USER") or "youruser"
def _has_systemd() -> bool:
return Path("/run/systemd/system").is_dir()
def _module_path() -> str:
"""Absolute path to the importable NixOS module in this checkout."""
return str(Path(__file__).resolve().parents[3] / "nix" / "firecracker-netpool.nix")
def _script_path() -> str:
"""Absolute path to the bundled bring-up script in this checkout."""
return str(Path(__file__).resolve().parents[3] / "scripts" / "firecracker-netpool.sh")
def _print_prereqs() -> None:
"""The firecracker binary + KVM + guest artifacts, shown before the
privileged network-pool step so operators see the full picture."""
fc = shutil.which("firecracker")
if fc:
sys.stderr.write(f"1) firecracker binary: found ({fc}).\n")
else:
sys.stderr.write(
"1) firecracker binary: NOT found on PATH. Install a release binary "
"and put it on PATH:\n"
f" {_FC_RELEASES}\n"
" e.g.: download firecracker-vX.Y.Z-$(uname -m).tgz, extract, and\n"
" install -m755 release-*/firecracker-* ~/.local/bin/firecracker\n"
" (NixOS: not packaged as a user binary — fetch the release, pin\n"
" the version, and add it to PATH.)\n"
)
if util.is_host_capable():
sys.stderr.write(" KVM: /dev/kvm present.\n")
else:
sys.stderr.write(
" KVM: /dev/kvm missing/unusable — load kvm-intel/kvm-amd, enable\n"
" virtualization in firmware, and add your user to the `kvm` group.\n"
)
sys.stderr.write(
" Guest artifacts: a kernel (BOT_BOTTLE_FC_KERNEL) and static dropbear\n"
" (BOT_BOTTLE_FC_DROPBEAR) must be cached, and `mke2fs` (e2fsprogs) is\n"
" needed to build the rootfs.\n\n"
)
def _is_nixos() -> bool:
if Path("/etc/NIXOS").exists():
return True
try:
return "ID=nixos" in Path("/etc/os-release").read_text()
except OSError:
return False
def _warn_overlaps() -> None:
"""Warn if the chosen pool range collides with an existing host route
(Tailscale CGNAT peer, a docker/libvirt bridge, the LAN…)."""
conflicts = netpool.overlapping_routes()
if not conflicts:
return
detail = "\n".join(f" {c.dst} dev {c.dev}" for c in conflicts)
sys.stderr.write(
f"WARNING: pool range (base {netpool.ip_base()}, "
f"{netpool.pool_size()} slots) overlaps existing routes:\n"
f"{detail}\n"
"Set BOT_BOTTLE_FC_IP_BASE to a free range before setup, or the "
"pool may shadow / be shadowed by the above.\n\n"
)
def setup() -> int:
sys.stderr.write("Firecracker backend — one-time host setup.\n\n")
_print_prereqs()
slots = netpool.all_slots()
sys.stderr.write(
f"2) network pool: {len(slots)} slots "
f"({slots[0].iface}..{slots[-1].iface}), base {netpool.ip_base()}"
f"TAP devices + nft isolation table, privileged (needs root once).\n\n"
)
_warn_overlaps()
if _is_nixos():
sys.stderr.write(
"Detected NixOS. Import the module — it is NON-INVASIVE: it does "
"not flip networking.nftables.enable or systemd.network.enable, so "
"your existing (iptables) firewall and Docker are untouched. A "
"systemd oneshot brings the pool up alongside them.\n\n"
" # flake users:\n"
" inputs.bot-bottle.url = \"git+ssh://<your-bot-bottle-remote>\";\n"
" imports = [ inputs.bot-bottle.nixosModules.firecracker-netpool ];\n"
" # channel (non-flake) users — import the file directly:\n"
f" imports = [ {_module_path()} ];\n\n"
f" services.bot-bottle-firecracker = {{ enable = true; owner = \"{_owner()}\"; }};\n\n"
"Then `nixos-rebuild switch`.\n"
)
elif _has_systemd():
_setup_systemd()
else:
sys.stderr.write(
"No systemd detected. Run the one-time bring-up as root (and add "
"your own boot persistence — e.g. an OpenRC/runit service):\n\n"
)
sys.stdout.write(netpool.render_shell_setup() + "\n")
return 0
def _setup_systemd() -> None:
"""Install the pool as a persistent systemd unit — the portable path,
identical on every systemd distro. Performs the install directly when
run as root; otherwise prints a self-contained copy-paste block."""
unit = netpool.render_systemd_unit(_owner(), _script_path())
sys.stderr.write(
f"Persistent install (systemd — same on every systemd distro). Needs "
f"`nft` (nftables) and `ip` (iproute2); install via your package "
f"manager if `backend status` reports them missing.\n\n"
)
if os.geteuid() == 0:
_UNIT_PATH.write_text(unit)
subprocess.run(["systemctl", "daemon-reload"], check=False)
rc = subprocess.run(
["systemctl", "enable", "--now", netpool.SYSTEMD_UNIT], check=False,
).returncode
if rc == 0:
sys.stderr.write(
f"Installed and started {netpool.SYSTEMD_UNIT}. Verify with "
f"`./cli.py backend status --backend=firecracker`.\n"
)
else:
sys.stderr.write(
f"Wrote {_UNIT_PATH} but `systemctl enable --now` failed — "
f"check `systemctl status {netpool.SYSTEMD_UNIT}`.\n"
)
return
sys.stderr.write(
"Install the unit (one copy-paste; enables it on boot too):\n\n"
)
sys.stdout.write(
f"sudo tee {_UNIT_PATH} >/dev/null <<'UNIT'\n"
f"{unit}"
f"UNIT\n"
f"sudo systemctl daemon-reload\n"
f"sudo systemctl enable --now {netpool.SYSTEMD_UNIT}\n"
)
sys.stderr.write(
f"\n(Or re-run this as root to install it directly: "
f"sudo ./cli.py backend setup --backend=firecracker)\n"
)
def teardown() -> int:
slots = netpool.all_slots()
sys.stderr.write(
f"Undo the Firecracker network pool ({len(slots)} slots, base "
f"{netpool.ip_base()}) — a privileged operation.\n\n"
)
if _is_nixos():
sys.stderr.write(
"On NixOS: set `services.bot-bottle-firecracker.enable = false;` "
"(or drop the module import) and `nixos-rebuild switch`. The TAP "
"netdevs and nft table are removed declaratively.\n\n"
"To tear down imperatively before a rebuild (does not persist):\n\n"
)
sys.stdout.write("sudo ./scripts/firecracker-netpool.sh down\n")
return 0
if _has_systemd():
if os.geteuid() == 0:
subprocess.run(
["systemctl", "disable", "--now", netpool.SYSTEMD_UNIT],
check=False,
)
_UNIT_PATH.unlink(missing_ok=True)
subprocess.run(["systemctl", "daemon-reload"], check=False)
sys.stderr.write(
f"Stopped, disabled, and removed {netpool.SYSTEMD_UNIT}.\n"
)
else:
sys.stderr.write("Remove the persistent unit (one copy-paste):\n\n")
sys.stdout.write(
f"sudo systemctl disable --now {netpool.SYSTEMD_UNIT}\n"
f"sudo rm -f {_UNIT_PATH}\n"
f"sudo systemctl daemon-reload\n"
)
return 0
sys.stderr.write("Run the teardown as root:\n\n")
sys.stdout.write("sudo ./scripts/firecracker-netpool.sh down\n")
return 0
def status() -> int:
# Readiness == what the launch preflight hard-requires: the TAP pool
# present (unprivileged, authoritative) and no range overlap. Listing
# the nft table usually needs root, so — like the preflight — an
# unconfirmable table is reported but NOT treated as not-ready; the
# post-boot isolation probe is the authoritative check. This keeps an
# unprivileged `backend status` usable as a launch gate.
ok = True
missing = netpool.missing_taps()
total = netpool.pool_size()
if missing:
sys.stderr.write(f"TAP pool: {total - len(missing)}/{total} present "
f"(missing: {', '.join(missing)})\n")
ok = False
else:
sys.stderr.write(f"TAP pool: {total}/{total} present\n")
if shutil.which("nft") is None:
sys.stderr.write(f"nft table inet {netpool.NFT_TABLE}: unverified "
f"(nft not on PATH; enforced + checked post-boot)\n")
elif netpool.nft_table_present():
sys.stderr.write(f"nft table inet {netpool.NFT_TABLE}: present\n")
else:
sys.stderr.write(f"nft table inet {netpool.NFT_TABLE}: not confirmable "
f"unprivileged (listing needs root; verified post-boot)\n")
conflicts = netpool.overlapping_routes()
if conflicts:
detail = ", ".join(f"{c.dst} dev {c.dev}" for c in conflicts)
sys.stderr.write(f"range overlap: base {netpool.ip_base()} CLASHES "
f"with {detail}\n")
ok = False
else:
sys.stderr.write(f"range overlap: none (base {netpool.ip_base()})\n")
_report_persistence()
if not ok:
sys.stderr.write("\nRun: ./cli.py backend setup --backend=firecracker\n")
return 0 if ok else 1
def _report_persistence() -> None:
"""Report whether the pool is installed as the persistent systemd
unit (so it survives reboot) vs brought up imperatively. Advisory —
doesn't affect launch readiness."""
if not _has_systemd():
return
state = subprocess.run(
["systemctl", "is-active", netpool.SYSTEMD_UNIT],
capture_output=True, text=True, check=False,
).stdout.strip() or "unknown"
if state == "active":
sys.stderr.write(f"persistence: {netpool.SYSTEMD_UNIT} active "
f"(survives reboot)\n")
else:
sys.stderr.write(f"persistence: {netpool.SYSTEMD_UNIT} {state} — pool "
f"is not installed as a persistent unit (install with "
f"`backend setup` so it survives reboot)\n")
-332
View File
@@ -1,332 +0,0 @@
"""Host-side primitives for the Firecracker backend.
Covers the pieces that don't need root at launch time: locating the
firecracker binary / guest kernel / injected dropbear, the fail-closed
preflight (KVM + kernel + isolation table + TAP pool must all be
present before a VM boots), the rootless rootfs pipeline
(`docker export` -> `mke2fs -d`, no mount), and per-bottle SSH key
generation.
The privileged network setup (TAP pool + nft table) is a one-time
operator step — see `netpool.py`, `scripts/firecracker-netpool.sh`,
and `./cli.py backend setup --backend=firecracker`.
"""
from __future__ import annotations
import os
import platform
import shutil
import subprocess
from pathlib import Path
from ...log import die, info, warn
from . import netpool
# Guest agent images are Debian-family with USER node; the VM is
# reached over SSH as root (init drops the pubkey into both root's and
# node's authorized_keys) and commands `runuser` down to node.
GUEST_SSH_USER = "root"
# `/dev/kvm` must exist and be openable by the invoking user.
_KVM_DEVICE = "/dev/kvm"
def cache_dir() -> Path:
d = Path(
os.environ.get(
"BOT_BOTTLE_FC_CACHE",
str(Path.home() / ".cache" / "bot-bottle" / "firecracker"),
)
)
d.mkdir(parents=True, exist_ok=True)
return d
def kernel_path() -> Path:
return Path(os.environ.get("BOT_BOTTLE_FC_KERNEL", str(cache_dir() / "vmlinux")))
def dropbear_path() -> Path:
"""The static dropbear injected into every guest rootfs as its SSH
server. Must be statically linked — the guest has none of the
host's shared libraries."""
return Path(
os.environ.get("BOT_BOTTLE_FC_DROPBEAR", str(cache_dir() / "dropbear"))
)
# --- availability + preflight ---------------------------------------
def is_linux() -> bool:
return platform.system() == "Linux"
def is_host_capable() -> bool:
"""Whether this host *could* run firecracker — Linux with KVM —
regardless of whether the `firecracker` binary is installed. Used
for default-backend selection so a KVM Linux host that hasn't
installed firecracker yet still selects it and gets an install
pointer at launch (see `require_firecracker`), rather than silently
falling back to docker."""
return is_linux() and os.path.exists(_KVM_DEVICE)
def is_available() -> bool:
"""Cheap capability probe used by cross-backend enumeration —
firecracker on PATH, on Linux, with KVM. Does not check the
(operator-provisioned) kernel / pool, so an available-but-unset
host still shows up and gets an actionable error at launch."""
return is_host_capable() and shutil.which("firecracker") is not None
def require_firecracker() -> None:
"""Fail-closed preflight. Every check that gates the security
boundary (the isolation table, the TAP pool) errors rather than
booting a VM without it."""
if not is_linux():
die("firecracker backend is only supported on Linux (KVM). "
"On macOS use --backend=macos-container.")
if shutil.which("firecracker") is None:
info("Firecracker is required but was not found on PATH.")
info("Install: https://github.com/firecracker-microvm/firecracker/releases")
die("firecracker not found on PATH")
_require_kvm()
if not kernel_path().is_file():
die(f"guest kernel not found at {kernel_path()}. Set "
"BOT_BOTTLE_FC_KERNEL or place a vmlinux there.")
if not dropbear_path().is_file():
die(f"static dropbear not found at {dropbear_path()}. Set "
"BOT_BOTTLE_FC_DROPBEAR or cache one there.")
if shutil.which("mke2fs") is None:
die("mke2fs (e2fsprogs) not found — required to build guest rootfs")
_require_network_pool()
def _require_kvm() -> None:
if not os.path.exists(_KVM_DEVICE):
die(f"{_KVM_DEVICE} is missing. Enable KVM (load kvm-intel/kvm-amd; "
"confirm virtualization is on in firmware).")
if not os.access(_KVM_DEVICE, os.R_OK | os.W_OK):
die(f"{_KVM_DEVICE} exists but is not accessible. Add your user to "
"the `kvm` group and re-login.")
def _require_network_pool() -> None:
"""Fail-closed on the parts we can check unprivileged; defer the
rest to the post-boot isolation probe.
The TAP pool is verified here (`ip link show` is unprivileged). The
nft table can only be *confirmed present* here when `nft` is
queryable — which usually needs root — so a missing/absent nft is
not treated as fatal at this stage: the authoritative check is the
empirical isolation probe run after boot, before the agent starts
(see isolation_probe.verify_isolation). What we must never do is
boot without the TAP pool."""
conflicts = netpool.overlapping_routes()
if conflicts:
detail = "; ".join(f"{c.dst} dev {c.dev}" for c in conflicts)
warn(f"Firecracker pool range ({netpool.ip_base()}, "
f"{netpool.pool_size()} slots) overlaps existing routes: "
f"{detail}. This can shadow or be shadowed by that route; "
f"set BOT_BOTTLE_FC_IP_BASE to a free range and re-run "
f"./cli.py backend setup --backend=firecracker.")
missing = netpool.missing_taps()
if missing:
die(f"network pool incomplete — missing TAP devices: "
f"{', '.join(missing)}.\n ./cli.py backend setup --backend=firecracker")
if shutil.which("nft") is not None and not netpool.nft_table_present():
# nft is queryable and says the table is absent — that's a
# definite, catchable misconfiguration; fail early.
warn(f"isolation table `inet {netpool.NFT_TABLE}` not found via nft. "
"If this is a permissions issue it will be re-checked "
"empirically after boot; otherwise run: "
"./cli.py backend setup --backend=firecracker")
# --- rootfs pipeline (rootless) -------------------------------------
def docker_image_id(ref: str) -> str:
"""The image's content digest, used as the rootfs cache key."""
result = subprocess.run(
["docker", "image", "inspect", "--format", "{{.Id}}", ref],
capture_output=True, text=True, check=False,
)
if result.returncode != 0 or not result.stdout.strip():
die(f"docker image inspect for {ref!r} failed: "
f"{(result.stderr or '').strip() or '<no output>'}")
return result.stdout.strip().replace("sha256:", "")[:16]
def build_base_rootfs_dir(
image_ref: str, *, variant: str = "", init_script: str | None = None,
) -> Path:
"""Export the image's filesystem and inject the guest init + static
dropbear. Cached by image digest — the per-bottle bits
(authorized_keys, IP) are passed at boot via the kernel cmdline, so
this tree carries nothing bottle-specific and is safely shared.
`variant` suffixes the cache key so the same image can be prepared
with a different `init_script` (e.g. the infra VM boots the same
orchestrator image as the builder but runs the control plane as
PID 1, not the SSH-only agent init) without a cache collision.
Returns the prepared directory (read as the `mke2fs -d` source)."""
digest = docker_image_id(image_ref)
base = cache_dir() / "rootfs" / f"{digest}{variant}"
ready = base / ".bb-ready"
if ready.is_file():
return base
if base.exists():
shutil.rmtree(base, ignore_errors=True)
base.mkdir(parents=True)
info(f"exporting {image_ref} rootfs -> {base}")
cid = subprocess.run(
["docker", "create", image_ref, "sleep", "infinity"],
capture_output=True, text=True, check=False,
)
if cid.returncode != 0 or not cid.stdout.strip():
die(f"docker create {image_ref!r} failed: {cid.stderr.strip()}")
container = cid.stdout.strip()
try:
export = subprocess.Popen(
["docker", "export", container], stdout=subprocess.PIPE,
)
untar = subprocess.run(
["tar", "-x", "-C", str(base)], stdin=export.stdout, check=False,
)
export.wait()
if export.returncode != 0 or untar.returncode != 0:
die(f"exporting rootfs for {image_ref!r} failed")
finally:
subprocess.run(
["docker", "rm", "-f", container],
stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL, check=False,
)
inject_guest_boot(base, init_script=init_script)
ready.write_text("ok\n")
return base
def inject_guest_boot(rootfs: Path, init_script: str | None = None) -> None:
"""Drop the static dropbear and the PID-1 init into the rootfs.
`init_script` defaults to the SSH-only agent init; the infra VM
passes its own (control plane + gateway) init."""
shutil.copy2(dropbear_path(), rootfs / "bb-dropbear")
os.chmod(rootfs / "bb-dropbear", 0o755)
init = rootfs / "bb-init"
init.write_text(init_script or _GUEST_INIT)
os.chmod(init, 0o755)
def build_rootfs_ext4(base_dir: Path, out_path: Path, *, slack_mib: int = 1024) -> None:
"""Build a fresh, writable ext4 for one bottle from the cached base
dir. Rootless: `mke2fs -d` populates the image from a directory
without mounting. Each call produces an independent disk, so the
shared base dir stays untouched and concurrent bottles don't race."""
used_mib = _dir_size_mib(base_dir)
size_mib = used_mib + slack_mib
out_path.parent.mkdir(parents=True, exist_ok=True)
result = subprocess.run(
["mke2fs", "-q", "-t", "ext4", "-d", str(base_dir), "-F",
str(out_path), f"{size_mib}M"],
capture_output=True, text=True, check=False,
)
if result.returncode != 0:
die(f"mke2fs for {out_path} failed: {result.stderr.strip()}")
def _dir_size_mib(path: Path) -> int:
result = subprocess.run(
["du", "-sm", str(path)], capture_output=True, text=True, check=False,
)
try:
return int(result.stdout.split()[0])
except (ValueError, IndexError):
return 2048
# --- per-bottle SSH keys --------------------------------------------
def generate_keypair(dest_dir: Path) -> tuple[Path, str]:
"""Mint a per-bottle ed25519 keypair. Returns (private_key_path,
public_key_line). The public line is injected into the guest via
the boot cmdline; the private key authenticates host->guest SSH."""
dest_dir.mkdir(parents=True, exist_ok=True)
key = dest_dir / "bottle_id_ed25519"
if key.exists():
key.unlink()
(dest_dir / "bottle_id_ed25519.pub").unlink(missing_ok=True)
subprocess.run(
["ssh-keygen", "-t", "ed25519", "-N", "", "-q", "-f", str(key),
"-C", "bot-bottle-firecracker"],
check=True,
)
pub = (dest_dir / "bottle_id_ed25519.pub").read_text().strip()
return key, pub
def ssh_base_argv(private_key: Path, guest_ip: str) -> list[str]:
"""Common SSH options for host->guest control. The VM is ephemeral
and per-bottle, so host-key TOFU is meaningless — pin no known_hosts
and skip the check rather than accumulate churn."""
return [
"ssh",
"-i", str(private_key),
# Only offer the per-bottle key — don't let an agent or the
# operator's ~/.ssh/config inject other identities (newer
# OpenSSH otherwise may not reliably present the -i key).
"-o", "IdentitiesOnly=yes",
"-o", "StrictHostKeyChecking=no",
"-o", "UserKnownHostsFile=/dev/null",
"-o", "LogLevel=ERROR",
"-o", "ConnectTimeout=5",
f"{GUEST_SSH_USER}@{guest_ip}",
]
# PID-1 init injected into every guest. Kept dependency-light: relies
# only on coreutils + a POSIX shell (present in the Debian-family agent
# images). The kernel `ip=` cmdline configures eth0 before init runs,
# so no iproute2 is needed. The per-bottle SSH pubkey arrives base64 on
# the cmdline; dropbear generates ephemeral host keys with -R.
_GUEST_INIT = r"""#!/bin/sh
# bot-bottle Firecracker guest init (PID 1).
mount -t proc proc /proc 2>/dev/null
mount -t sysfs sys /sys 2>/dev/null
mount -t devtmpfs dev /dev 2>/dev/null
mkdir -p /dev/pts && mount -t devpts devpts /dev/pts 2>/dev/null
mount -o remount,rw / 2>/dev/null
# Install the per-bottle SSH pubkey from the kernel cmdline.
KEY=$(sed -n 's/.*bb_pubkey=\([^ ]*\).*/\1/p' /proc/cmdline | base64 -d 2>/dev/null)
if [ -n "$KEY" ]; then
for home in /root /home/node; do
mkdir -p "$home/.ssh"
printf '%s\n' "$KEY" > "$home/.ssh/authorized_keys"
chmod 700 "$home/.ssh"
chmod 600 "$home/.ssh/authorized_keys"
done
chown -R node:node /home/node/.ssh 2>/dev/null || true
fi
# The rootless rootfs build (`docker export | tar` as a non-root user)
# can't preserve uid 0, so every path lands owned by the build uid,
# which maps to `node` (uid 1000) in-guest — including /root. dropbear
# (like OpenSSH) refuses root's authorized_keys unless the home dir is
# owned by root, so restore root's ownership of its own home.
chown -R 0:0 /root 2>/dev/null || true
mkdir -p /etc/dropbear /run
# -R: generate host keys on demand. -E: log auth failures to stderr,
# captured in the host-side console.log for debugging.
/bb-dropbear -R -E -p 22 &
# Reap zombies as PID 1. dropbear is always a child, so `wait` blocks
# rather than busy-looping.
while : ; do wait ; done
"""
+4 -4
View File
@@ -90,11 +90,11 @@ def get_freezer(backend_name: str) -> Freezer:
if resolved == "macos-container":
from .macos_container.freezer import MacosContainerFreezer
return MacosContainerFreezer()
if resolved == "firecracker":
from .firecracker.freezer import FirecrackerFreezer
return FirecrackerFreezer()
if resolved == "smolmachines":
from .smolmachines.freezer import SmolmachinesFreezer
return SmolmachinesFreezer()
die(
f"commit is only supported for docker, macos-container, and "
f"firecracker; backend {backend_name!r} has no freezer"
f"smolmachines; backend {backend_name!r} has no freezer"
)
raise AssertionError("unreachable")
@@ -2,7 +2,7 @@
Selectable via `BOT_BOTTLE_BACKEND=macos-container`. This package owns
the Apple `container` CLI integration; launch remains gated until the
gateway network enforcement shape is implemented.
sidecar network enforcement shape is implemented.
"""
from .backend import MacosContainerBottleBackend
@@ -36,21 +36,6 @@ class MacosContainerBottleBackend(
def is_available(cls) -> bool:
return _container.is_available()
@classmethod
def setup(cls) -> int:
from . import setup as _setup
return _setup.setup()
@classmethod
def status(cls) -> int:
from . import setup as _setup
return _setup.status()
@classmethod
def teardown(cls) -> int:
from . import setup as _setup
return _setup.teardown()
def _preflight(self) -> None:
_resolve_plan.preflight()
@@ -9,6 +9,7 @@ from . import util as container_mod
from .bottle_cleanup_plan import MacosContainerBottleCleanupPlan
_PREFIX = "bot-bottle-"
_BUNDLE_PREFIX = "bot-bottle-sidecars-"
def _list_prefixed_containers() -> list[str]:
@@ -23,7 +24,7 @@ def _list_prefixed_containers() -> list[str]:
return []
return sorted(
name for name in (line.strip() for line in result.stdout.splitlines())
if name.startswith(_PREFIX)
if name.startswith(_PREFIX) or name.startswith(_BUNDLE_PREFIX)
)
@@ -1,24 +1,36 @@
"""Host-side egress route-apply for the macos-container backend.
"""Host-side egress apply for the macos-container backend.
The per-bottle companion container this used to signal (`container kill
--signal HUP <container>`) was removed in the companion-container removal (#385),
along with the disabled macOS launch path. Fails closed until the macOS
backend grows the consolidated gateway.
Uses `container kill --signal HUP` (Apple Container framework) instead
of `docker kill` to signal the sidecar bundle.
"""
from __future__ import annotations
import os
import subprocess
from ...log import warn
from ..egress_apply import EgressApplicator, EgressApplyError
from .launch import sidecar_container_name
class MacOSContainerEgressApplicator(EgressApplicator):
def _signal_bundle_reload(self, slug: str) -> None:
del slug
raise EgressApplyError(
"live egress route-apply was removed with the per-bottle "
"companion container (#385); the macos-container backend is "
"disabled until it uses the consolidated gateway."
container = sidecar_container_name(slug)
result = subprocess.run(
["container", "kill", "--signal", "HUP", container],
capture_output=True, text=True, check=False, env=os.environ,
)
if result.returncode != 0:
last_error = (result.stderr or "").strip() or (result.stdout or "").strip()
warn(
f"egress: routes updated on disk for {slug}, but bundle reload failed: "
f"{last_error or 'container kill failed'}"
)
raise EgressApplyError(
f"could not reload egress bundle {container}: "
f"{last_error or 'container kill failed'}"
)
applicator = MacOSContainerEgressApplicator()
@@ -1,14 +1,40 @@
"""Active-agent enumeration for the macOS Apple Container backend.
The backend is disabled during the companion-container removal (#385) — it can't
launch bottles, so there are none to enumerate. Enumeration returns when
the backend grows the consolidated gateway.
"""
"""Active-agent enumeration for the macOS Apple Container backend."""
from __future__ import annotations
import subprocess
from ...bottle_state import read_metadata
from .. import ActiveAgent
_PREFIX = "bot-bottle-"
_SIDECAR_PREFIX = "bot-bottle-sidecars-"
def enumerate_active() -> list[ActiveAgent]:
return []
result = subprocess.run(
["container", "list", "--quiet"],
capture_output=True,
text=True,
check=False,
)
if result.returncode != 0:
return []
out: list[ActiveAgent] = []
for name in sorted(line.strip() for line in result.stdout.splitlines()):
if not name.startswith(_PREFIX):
continue
if name.startswith(_SIDECAR_PREFIX):
continue
slug = name[len(_PREFIX):]
metadata = read_metadata(slug)
out.append(ActiveAgent(
backend_name="macos-container",
slug=slug,
agent_name=metadata.agent_name if metadata else "?",
started_at=metadata.started_at if metadata else "",
services=(),
label=metadata.label if metadata else "",
color=metadata.color if metadata else "",
))
return out
+440 -21
View File
@@ -1,39 +1,458 @@
"""Launch flow for the macOS Apple Container backend — disabled (#385).
"""Launch flow for the macOS Apple Container backend.
This backend launched a per-bottle companion container (the egress /
git-gate / supervise data plane) alongside the agent container, with the
agent's proxy env pointed at the companion's host-only IP. That
per-bottle-companion architecture was removed in the companion-container removal;
the macOS backend will be re-enabled once it grows the consolidated
per-host gateway the docker backend already uses.
Until then, launching a macOS bottle fails closed. `prepare` / `status`
/ cleanup still work.
This backend keeps the explicit proxy-env enforcement model for v1:
the agent container is attached only to a host-only Apple Container
network, while the sidecar bundle is attached to a NAT network first
and the host-only network second. The sidecar's host-only IP is
discovered from `container inspect` and stamped into the agent's
HTTP_PROXY / HTTPS_PROXY env vars.
"""
from __future__ import annotations
from contextlib import contextmanager
import dataclasses
import os
import subprocess
from contextlib import ExitStack, contextmanager
from pathlib import Path
from typing import Callable, Generator
from ...log import die
from ...bottle_state import (
egress_state_dir,
git_gate_state_dir,
read_committed_image,
)
from ...egress import (
EGRESS_ROUTES_IN_CONTAINER,
egress_agent_env_entries,
egress_resolve_token_values,
egress_sidecar_env_entries,
)
from ...git_gate import (
provision_git_gate_dynamic_keys,
revoke_git_gate_provisioned_keys,
)
from ...log import die, info, warn
from ...supervise import DB_PATH_IN_CONTAINER, SUPERVISE_PORT
from ...util import expand_tilde
from ..docker.egress import EGRESS_CA_IN_CONTAINER, EGRESS_PORT
from ..docker.git_gate import (
GIT_GATE_ACCESS_HOOK_IN_CONTAINER,
GIT_GATE_CREDS_DIR_IN_CONTAINER,
GIT_GATE_ENTRYPOINT_IN_CONTAINER,
GIT_GATE_HOOK_IN_CONTAINER,
)
from ..docker.sidecar_bundle import (
SIDECAR_BUNDLE_DOCKERFILE,
SIDECAR_BUNDLE_IMAGE,
)
from ..docker.egress import egress_tls_init
from ..util import AGENT_CA_BUNDLE, AGENT_CA_PATH
from . import util as container_mod
from .bottle import MacosContainerBottle
from .bottle_plan import MacosContainerBottlePlan
_REPO_DIR = str(Path(__file__).resolve().parent.parent.parent.parent)
_AGENT_SLEEP_SECONDS = "2147483647"
_GIT_HTTP_PORT = 9420
_GIT_GATE_READY_FILE = "/run/git-gate/ready"
def internal_network_name(slug: str) -> str:
return f"bot-bottle-net-{slug}"
def egress_network_name(slug: str) -> str:
return f"bot-bottle-egress-{slug}"
def sidecar_container_name(slug: str) -> str:
return f"bot-bottle-sidecars-{slug}"
@contextmanager
def launch(
plan: MacosContainerBottlePlan,
*,
provision: Callable[[MacosContainerBottlePlan, "MacosContainerBottle"], str | None],
) -> Generator[MacosContainerBottle, None, None]:
"""Fail closed: the macOS backend is disabled until it grows the
consolidated per-host gateway (the companion-container path it used
was removed in #385)."""
del plan, provision
die(
"the macos-container backend is temporarily disabled during the "
"companion-container removal (#385); it will return once it uses "
"the consolidated gateway. Use --backend=docker for now."
"""Build, run, provision, and yield an Apple Container bottle."""
stack = ExitStack()
bottle_for_revoke = plan.manifest.bottle
git_gate_dir_for_revoke = git_gate_state_dir(plan.slug)
def teardown() -> None:
teardown_exc: BaseException | None = None
try:
stack.close()
except BaseException as exc: # noqa: W0718 - teardown must continue
teardown_exc = exc
warn(f"macos-container teardown failed: {exc!r}")
revoke_git_gate_provisioned_keys(bottle_for_revoke, git_gate_dir_for_revoke)
if teardown_exc is not None:
raise teardown_exc
try:
plan = _mint_certs(plan)
plan = _build_images(plan)
internal_network = internal_network_name(plan.slug)
egress_network = egress_network_name(plan.slug)
_create_networks(internal_network, egress_network, stack)
plan = _provision_git_gate_keys(plan)
sidecar_name = sidecar_container_name(plan.slug)
container_mod.force_remove_container(sidecar_name)
_start_sidecar_bundle(plan, sidecar_name, internal_network, egress_network)
stack.callback(container_mod.force_remove_container, sidecar_name)
_stage_git_gate(plan, sidecar_name)
sidecar_ip = container_mod.container_ipv4_on_network(
sidecar_name, internal_network,
)
plan = _stamp_agent_urls(plan, sidecar_ip)
container_mod.force_remove_container(plan.container_name)
_start_agent(plan, internal_network, sidecar_ip)
stack.callback(container_mod.force_remove_container, plan.container_name)
bottle = MacosContainerBottle(
plan.container_name,
teardown,
None,
agent_command=plan.agent_command,
agent_prompt_mode=plan.agent_prompt_mode,
agent_provider_template=plan.agent_provider_template,
terminal_title=f"{plan.spec.label} ({plan.spec.agent_name})" if plan.spec.label else plan.spec.agent_name,
terminal_color=plan.spec.color,
agent_workdir=plan.workspace_plan.workdir,
)
bottle.prompt_path = provision(plan, bottle)
yield bottle
finally:
teardown()
def _mint_certs(plan: MacosContainerBottlePlan) -> MacosContainerBottlePlan:
egress_ca_host, egress_ca_cert_only = egress_tls_init(
egress_state_dir(plan.slug),
)
yield # unreachable — `die` raises; keeps this a generator/contextmanager
egress_plan = dataclasses.replace(
plan.egress_plan,
mitmproxy_ca_host_path=egress_ca_host,
mitmproxy_ca_cert_only_host_path=egress_ca_cert_only,
)
return dataclasses.replace(plan, egress_plan=egress_plan)
def _build_images(plan: MacosContainerBottlePlan) -> MacosContainerBottlePlan:
container_mod.build_image(
SIDECAR_BUNDLE_IMAGE,
_REPO_DIR,
dockerfile=SIDECAR_BUNDLE_DOCKERFILE,
)
committed = read_committed_image(plan.slug)
if committed and container_mod.image_exists(committed):
info(f"using committed image {committed!r}")
return dataclasses.replace(
plan,
agent_provision=dataclasses.replace(
plan.agent_provision,
image=committed,
),
)
container_mod.build_image(
plan.image,
_REPO_DIR,
dockerfile=plan.dockerfile_path,
)
return plan
def _create_networks(
internal_network: str,
egress_network: str,
stack: ExitStack,
) -> None:
container_mod.create_network(internal_network, internal=True)
stack.callback(container_mod.remove_network, internal_network)
container_mod.create_network(egress_network)
stack.callback(container_mod.remove_network, egress_network)
def _start_sidecar_bundle(
plan: MacosContainerBottlePlan,
sidecar_name: str,
internal_network: str,
egress_network: str,
) -> None:
argv = _sidecar_run_argv(plan, sidecar_name, internal_network, egress_network)
effective_env = {**dict(os.environ), **plan.agent_provision.provisioned_env}
token_values = egress_resolve_token_values(
plan.egress_plan.token_env_map, effective_env,
)
env = {**os.environ, **token_values}
info(f"container run sidecar bundle {sidecar_name}")
result = subprocess.run(
argv, capture_output=True, text=True, env=env, check=False,
)
if result.returncode != 0:
die(
f"container run for sidecar bundle {sidecar_name} failed: "
f"{(result.stderr or '').strip() or '<no stderr>'}"
)
def _start_agent(
plan: MacosContainerBottlePlan,
internal_network: str,
sidecar_ip: str,
) -> None:
argv = _agent_run_argv(plan, internal_network, sidecar_ip)
env = {
**os.environ,
**plan.forwarded_env,
}
info(f"container run agent {plan.container_name}")
result = subprocess.run(
argv, capture_output=True, text=True, env=env, check=False,
)
if result.returncode != 0:
die(
f"container run for agent {plan.container_name} failed: "
f"{(result.stderr or '').strip() or '<no stderr>'}"
)
def _stamp_agent_urls(
plan: MacosContainerBottlePlan,
sidecar_ip: str,
) -> MacosContainerBottlePlan:
proxy_url = f"http://{sidecar_ip}:{EGRESS_PORT}"
supervise_url = ""
if plan.supervise_plan is not None:
supervise_url = f"http://{sidecar_ip}:{SUPERVISE_PORT}/"
git_gate_url = ""
if plan.git_gate_plan.upstreams:
git_gate_url = f"http://{sidecar_ip}:{_GIT_HTTP_PORT}"
return dataclasses.replace(
plan,
agent_proxy_url=proxy_url,
agent_git_gate_url=git_gate_url,
agent_supervise_url=supervise_url,
)
def _provision_git_gate_keys(
plan: MacosContainerBottlePlan,
) -> MacosContainerBottlePlan:
if not plan.git_gate_plan.upstreams:
return plan
git_gate_plan = provision_git_gate_dynamic_keys(
plan.manifest.bottle,
plan.git_gate_plan,
git_gate_state_dir(plan.slug),
)
return dataclasses.replace(plan, git_gate_plan=git_gate_plan)
def _stage_git_gate(plan: MacosContainerBottlePlan, sidecar_name: str) -> None:
gp = plan.git_gate_plan
if not gp.upstreams:
return
container_mod.exec_container(
sidecar_name,
[
"mkdir",
"-p",
str(Path(GIT_GATE_HOOK_IN_CONTAINER).parent),
GIT_GATE_CREDS_DIR_IN_CONTAINER,
"/git",
str(Path(_GIT_GATE_READY_FILE).parent),
],
)
for host_path, container_path in _git_gate_files(plan):
container_mod.copy_into_container(
sidecar_name, host_path, container_path,
)
container_mod.exec_container(
sidecar_name,
[
"sh",
"-c",
"chmod 755 "
f"{GIT_GATE_ENTRYPOINT_IN_CONTAINER} "
f"{GIT_GATE_HOOK_IN_CONTAINER} "
f"{GIT_GATE_ACCESS_HOOK_IN_CONTAINER} && "
f"chmod 600 {GIT_GATE_CREDS_DIR_IN_CONTAINER}/* && "
f"touch {_GIT_GATE_READY_FILE}",
],
)
def _git_gate_files(
plan: MacosContainerBottlePlan,
) -> tuple[tuple[str, str], ...]:
gp = plan.git_gate_plan
files: list[tuple[str, str]] = [
(str(gp.entrypoint_script), GIT_GATE_ENTRYPOINT_IN_CONTAINER),
(str(gp.hook_script), GIT_GATE_HOOK_IN_CONTAINER),
(str(gp.access_hook_script), GIT_GATE_ACCESS_HOOK_IN_CONTAINER),
]
for upstream in gp.upstreams:
files.append((
expand_tilde(upstream.identity_file),
f"{GIT_GATE_CREDS_DIR_IN_CONTAINER}/{upstream.name}-key",
))
if upstream.known_hosts_file:
files.append((
str(upstream.known_hosts_file),
f"{GIT_GATE_CREDS_DIR_IN_CONTAINER}/{upstream.name}-known_hosts",
))
return tuple(files)
def _sidecar_run_argv(
plan: MacosContainerBottlePlan,
sidecar_name: str,
internal_network: str,
egress_network: str,
) -> list[str]:
argv = [
"container", "run",
"--name", sidecar_name,
"--detach",
"--rm",
"--network", egress_network,
"--network", internal_network,
"--dns", _sidecar_dns(),
"--env", f"BOT_BOTTLE_SIDECAR_DAEMONS={','.join(_sidecar_daemons(plan))}",
]
for entry in _sidecar_env_entries(plan):
argv += ["--env", entry]
for host_path, container_path, read_only in _sidecar_mounts(plan):
argv += ["--mount", _mount_spec(host_path, container_path, read_only)]
argv.append(SIDECAR_BUNDLE_IMAGE)
return argv
def _agent_run_argv(
plan: MacosContainerBottlePlan,
internal_network: str,
sidecar_ip: str,
) -> list[str]:
argv = [
"container", "run",
"--name", plan.container_name,
"--detach",
"--network", internal_network,
]
for entry in _agent_env_entries(plan, sidecar_ip):
argv += ["--env", entry]
argv += [plan.image, "sleep", _AGENT_SLEEP_SECONDS]
return argv
def _sidecar_dns() -> str:
return container_mod.dns_server()
def _sidecar_daemons(plan: MacosContainerBottlePlan) -> tuple[str, ...]:
daemons = ["egress"]
if plan.git_gate_plan.upstreams:
daemons += ["git-gate", "git-http"]
if plan.supervise_plan is not None:
daemons.append("supervise")
return tuple(daemons)
def _sidecar_env_entries(plan: MacosContainerBottlePlan) -> tuple[str, ...]:
env: list[str] = list(egress_sidecar_env_entries(plan.egress_plan))
if plan.git_gate_plan.upstreams:
env.append(f"BOT_BOTTLE_GIT_GATE_READY_FILE={_GIT_GATE_READY_FILE}")
if plan.supervise_plan is not None:
env += [
f"SUPERVISE_BOTTLE_SLUG={plan.slug}",
f"SUPERVISE_DB_PATH={DB_PATH_IN_CONTAINER}",
f"SUPERVISE_PORT={SUPERVISE_PORT}",
]
return tuple(env)
def _sidecar_mounts(
plan: MacosContainerBottlePlan,
) -> tuple[tuple[str, str, bool], ...]:
mounts: list[tuple[str, str, bool]] = []
ep = plan.egress_plan
mounts.append((
str(ep.mitmproxy_ca_host_path.parent),
str(Path(EGRESS_CA_IN_CONTAINER).parent),
False,
))
if ep.routes:
mounts.append((
str(ep.routes_path.parent),
str(Path(EGRESS_ROUTES_IN_CONTAINER).parent),
True,
))
sp = plan.supervise_plan
if sp is not None:
# `container run --mount type=bind` only accepts directory
# sources (a file source fails with "is not a directory") —
# mount db_path's dedicated parent dir instead of the file
# itself, same as the CA/routes mounts above.
mounts.append((
str(sp.db_path.parent),
str(Path(DB_PATH_IN_CONTAINER).parent),
False,
))
return tuple(mounts)
def _mount_spec(host_path: str, container_path: str, read_only: bool) -> str:
spec = f"type=bind,source={host_path},target={container_path}"
if read_only:
spec += ",readonly"
return spec
def _agent_env_entries(
plan: MacosContainerBottlePlan,
sidecar_ip: str,
) -> tuple[str, ...]:
proxy_url = f"http://{sidecar_ip}:{EGRESS_PORT}"
no_proxy = _agent_no_proxy(plan, sidecar_ip)
env = [
f"HTTPS_PROXY={proxy_url}",
f"HTTP_PROXY={proxy_url}",
f"https_proxy={proxy_url}",
f"http_proxy={proxy_url}",
f"NO_PROXY={no_proxy}",
f"no_proxy={no_proxy}",
f"NODE_EXTRA_CA_CERTS={AGENT_CA_PATH}",
f"SSL_CERT_FILE={AGENT_CA_BUNDLE}",
f"REQUESTS_CA_BUNDLE={AGENT_CA_BUNDLE}",
]
if plan.agent_git_gate_url:
env.append(f"GIT_GATE_URL={plan.agent_git_gate_url}")
if plan.agent_supervise_url:
env.append(f"MCP_SUPERVISE_URL={plan.agent_supervise_url}")
for name, value in sorted(plan.agent_provision.guest_env.items()):
env.append(f"{name}={value}")
for name in sorted(plan.forwarded_env.keys()):
env.append(name)
env.extend(egress_agent_env_entries(plan.egress_plan))
return tuple(env)
def _agent_no_proxy(plan: MacosContainerBottlePlan, sidecar_ip: str) -> str:
hosts = ["localhost", "127.0.0.1", sidecar_ip]
return ",".join(hosts)
@@ -1,79 +0,0 @@
"""Host setup + status for the macOS Apple Container backend.
Like Docker, this backend needs no privileged network-pool provisioning
— it wants Apple's `container` CLI installed and its system service
running. `setup()` points at the install/`container system start` steps;
`status()` reports readiness. Reached via
`MacosContainerBottleBackend.setup` / `.status`, dispatched from the
generic `./cli.py backend {setup,status}`.
"""
from __future__ import annotations
import shutil
import subprocess
import sys
from . import util as _container
def _service_running() -> bool:
if shutil.which("container") is None:
return False
return subprocess.run(
["container", "system", "status"],
stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL, check=False,
).returncode == 0
def setup() -> int:
if not _container.is_macos():
sys.stderr.write("macos-container backend requires macOS.\n")
return 1
if shutil.which("container") is None:
sys.stderr.write("Apple Container is required but was not found on PATH.\n")
sys.stderr.write("Install: https://github.com/apple/container/releases\n")
return 1
if not _service_running():
sys.stderr.write(
"Apple Container is installed but its system service isn't "
"running. Start it with: container system start\n"
)
return 1
sys.stderr.write(
"macos-container backend: ready — no privileged host setup required.\n"
)
return 0
def teardown() -> int:
sys.stderr.write(
"macos-container backend: nothing to undo — it provisions no "
"privileged host state. The Apple Container CLI and its system "
"service are left as-is (stop the service yourself with "
"`container system stop` if you want).\n"
)
return 0
def status() -> int:
ok = True
if _container.is_macos():
sys.stderr.write("host: macOS\n")
else:
sys.stderr.write("host: NOT macOS (backend unsupported here)\n")
ok = False
if shutil.which("container") is not None:
sys.stderr.write("container CLI on PATH: yes\n")
else:
sys.stderr.write("container CLI on PATH: NO\n")
ok = False
if ok:
sys.stderr.write(
f"container system service: {'running' if _service_running() else 'NOT running'}\n"
)
if not _service_running():
ok = False
if not ok:
sys.stderr.write("\nRun: ./cli.py backend setup --backend=macos-container\n")
return 0 if ok else 1
+1 -31
View File
@@ -60,21 +60,13 @@ def dns_server() -> str:
def build_image(ref: str, context: str, *, dockerfile: str = "") -> None:
"""Build an OCI image with Apple's BuildKit-backed `container build`.
Set `BOT_BOTTLE_NO_CACHE=1` (the `start --no-cache` flag) to force
`--no-cache`. The npm/curl installers some provider Dockerfiles
shell out to can silently no-op on a transient network failure —
e.g. an `optionalDependencies` fetch for a platform-native binary —
and the builder will then cache that broken layer indefinitely."""
"""Build an OCI image with Apple's BuildKit-backed `container build`."""
info(
f"building image {ref} from {context} with Apple Container "
"(layer cache keeps repeat builds fast)"
)
_ensure_builder_dns()
args = [_CONTAINER, "build", "-t", ref, "--dns", dns_server()]
if os.environ.get("BOT_BOTTLE_NO_CACHE") == "1":
args.append("--no-cache")
if dockerfile:
# `container build` resolves -f relative to the current working
# directory, not the build context. Anchor a relative Dockerfile to
@@ -86,28 +78,6 @@ def build_image(ref: str, context: str, *, dockerfile: str = "") -> None:
subprocess.run(args, check=True)
def verify_agent_image(image: str, argv: tuple[str, ...]) -> None:
"""Run `argv` inside a throwaway container of a freshly built agent
image and die loudly if it fails, instead of shipping an image
whose CLI only breaks at first real use. No-op when the provider
hasn't declared a smoke test (`AgentProviderRuntime.smoke_test`)."""
if not argv:
return
result = subprocess.run(
[_CONTAINER, "run", "--rm", "--entrypoint", argv[0], image, *argv[1:]],
capture_output=True,
text=True,
check=False,
)
if result.returncode != 0:
detail = (result.stderr or result.stdout or "").strip()
die(
f"agent image {image!r} failed its post-build smoke test "
f"({' '.join(argv)}): {detail}\n"
f"Try rebuilding from scratch: bot-bottle start --no-cache"
)
def commit_container(container_name: str, image_tag: str) -> None:
"""Snapshot a running Apple Container as a local image.
+4 -3
View File
@@ -1,8 +1,9 @@
"""Shared print helpers for BottlePlan.print implementations.
Lifts the multi-value label printer out of DockerBottlePlan so every
backend (and any future backend) renders the same two-column
scannable preflight without duplicating the indent math."""
Lifts the multi-value label printer out of DockerBottlePlan so the
smolmachines backend (and any future backend) renders the same
two-column scannable preflight without duplicating the indent
math."""
from __future__ import annotations
+3 -3
View File
@@ -1,7 +1,7 @@
"""Shared helpers used across backends' resolve_plan steps.
"""Shared helpers used by both backends' resolve_plan steps.
Each helper owns one well-defined step of the per-bottle plan
resolution so the backends don't repeat the same logic.
resolution so docker and smolmachines don't repeat the same logic.
Backend-specific steps (container names, env-file, per-bottle
Dockerfile overrides, subnet allocation) stay in the backend's own
resolve_plan.py.
@@ -94,7 +94,7 @@ def prepare_egress(
def prepare_supervise(bottle: ManifestBottle, slug: str) -> SupervisePlan | None:
"""Prepare the supervise daemon state dir. Returns None when
"""Prepare the supervise sidecar state dir. Returns None when
bottle.supervise is falsy."""
if not bottle.supervise:
return None
@@ -0,0 +1,15 @@
"""smolmachines bottle backend (PRD 0023).
Selectable via `BOT_BOTTLE_BACKEND=smolmachines`. Runs each
bottle inside a per-agent microVM (libkrun / Hypervisor.framework
on macOS) with a userspace gvproxy gateway as the egress
primitive. The sidecar bundle (PRD 0024) runs as a host-side
docker container reached only through gvproxy's port-forward list.
Chunk 1 (this commit) ships the backend skeleton + Smolfile +
gvproxy renderers + preflight check. VM lifecycle, sidecar
bringup, and provisioning land in later chunks."""
from .backend import SmolmachinesBottleBackend # noqa: F401
__all__ = ["SmolmachinesBottleBackend"]
@@ -1,10 +1,11 @@
"""FirecrackerBottleBackend — Linux microVM implementation.
"""SmolmachinesBottleBackend — the smolmachines implementation of
BottleBackend (PRD 0023).
Replaces smolmachines on Linux (issue #342): mature KVM-based
isolation via Firecracker, SSH control over a point-to-point TAP, and a
fail-closed nftables egress boundary. Selected by
`BOT_BOTTLE_BACKEND=firecracker` or `--backend=firecracker`.
"""
Per PRD 0050 the per-provider provisioning steps (prompt, skills,
the declarative provision-plan apply, supervise MCP registration)
live on the `AgentProvider` plugin under `bot_bottle/contrib/`. The
smolmachines backend only owns the steps that are about backend
infrastructure: CA install (no-op for now), workspace, git copy-in."""
from __future__ import annotations
@@ -16,50 +17,34 @@ from ...agent_provider import AgentProvisionPlan
from ...egress import EgressPlan
from ...env import ResolvedEnv
from ...git_gate import GitGatePlan
from ...manifest import Manifest
from ...supervise import SupervisePlan
from ...manifest import Manifest
from .. import ActiveAgent, BottleBackend, BottleSpec
from . import cleanup as _cleanup
from . import enumerate as _enumerate
from . import launch as _launch
from . import resolve_plan as _resolve_plan
from . import util as _util
from .bottle import FirecrackerBottle
from .bottle_cleanup_plan import FirecrackerBottleCleanupPlan
from .bottle_plan import FirecrackerBottlePlan
from . import smolvm as _smolvm
from .bottle import SmolmachinesBottle
from .bottle_cleanup_plan import SmolmachinesBottleCleanupPlan
from .bottle_plan import SmolmachinesBottlePlan
class FirecrackerBottleBackend(
BottleBackend["FirecrackerBottlePlan", "FirecrackerBottleCleanupPlan"]
class SmolmachinesBottleBackend(
BottleBackend["SmolmachinesBottlePlan", "SmolmachinesBottleCleanupPlan"]
):
name = "firecracker"
"""smolmachines backend. Selected by
`BOT_BOTTLE_BACKEND=smolmachines`."""
name = "smolmachines"
@classmethod
def is_available(cls) -> bool:
return _util.is_available()
@classmethod
def is_host_capable(cls) -> bool:
"""Linux + KVM, regardless of whether the `firecracker` binary
is installed. Drives default-backend selection so a capable host
without the binary still lands on firecracker and gets an
install pointer at launch."""
return _util.is_host_capable()
@classmethod
def setup(cls) -> int:
from . import setup as _setup
return _setup.setup()
@classmethod
def status(cls) -> int:
from . import setup as _setup
return _setup.status()
@classmethod
def teardown(cls) -> int:
from . import setup as _setup
return _setup.teardown()
"""`smolvm` on PATH. The backend additionally needs macOS
for libkrun + TSI, but `enumerate_active` / `cleanup` are
host-shell ops that gracefully no-op on Linux too the
runtime check happens at `prepare`."""
return _smolvm.is_available()
def _preflight(self) -> None:
_resolve_plan.preflight()
@@ -79,7 +64,7 @@ class FirecrackerBottleBackend(
git_gate_plan: GitGatePlan,
supervise_plan: SupervisePlan | None,
stage_dir: Path,
) -> FirecrackerBottlePlan:
) -> SmolmachinesBottlePlan:
return _resolve_plan.resolve_plan(
spec,
manifest=manifest,
@@ -94,19 +79,23 @@ class FirecrackerBottleBackend(
@contextmanager
def launch(
self, plan: FirecrackerBottlePlan
) -> Generator[FirecrackerBottle, None, None]:
self, plan: SmolmachinesBottlePlan
) -> Generator[SmolmachinesBottle, None, None]:
with _launch.launch(plan, provision=self.provision) as bottle:
yield bottle
def prepare_cleanup(self) -> FirecrackerBottleCleanupPlan:
def supervise_mcp_url(self, plan: SmolmachinesBottlePlan) -> str:
"""The smolmachines guest reaches the supervise sidecar via a
host-published random port the launch step pinned earlier
(`http://<loopback_ip>:<random_port>/`). `agent_supervise_url`
on the plan is "" when the bottle has no sidecar."""
return plan.agent_supervise_url
def prepare_cleanup(self) -> SmolmachinesBottleCleanupPlan:
return _cleanup.prepare_cleanup()
def cleanup(self, plan: FirecrackerBottleCleanupPlan) -> None:
def cleanup(self, plan: SmolmachinesBottleCleanupPlan) -> None:
_cleanup.cleanup(plan)
def enumerate_active(self) -> Sequence[ActiveAgent]:
return _enumerate.enumerate_active()
def supervise_mcp_url(self, plan: FirecrackerBottlePlan) -> str:
return plan.agent_supervise_url
+217
View File
@@ -0,0 +1,217 @@
"""SmolmachinesBottle — running-instance handle (PRD 0023 chunk 2d).
Routes `exec_agent` / `exec` / `cp_in` through `smolvm machine
exec` / `smolvm machine cp`. The handle is yielded by `launch`
and torn down via the surrounding ExitStack on context exit;
`close` is a no-op idempotent alias so the BottleBackend ABC's
context-manager contract is satisfied.
User context: `smolvm machine exec` runs commands as root in the
VM, but the agent image's USER is `node` and agent CLIs may refuse
to run as root in bypass modes. Both
`exec_agent` and `exec` switch to the requested user (default
`node`) via `runuser -u <user> --` and set `HOME` / `USER`
through `smolvm -e` avoiding `runuser -l`'s login-shell wiring
(PAM session setup, /etc/profile sourcing) which can hang on a
minimal Debian VM with no PAM session config."""
from __future__ import annotations
import subprocess
import sys
import time
import shlex
from typing import Mapping, cast
from ...agent_provider import PromptMode, prompt_args
from .. import Bottle, ExecResult
from ..terminal import exec_shell_script
from . import pty_resize as _pty_resize
from . import smolvm as _smolvm
# Absolute path to the pty_resize wrapper. Invoke as
# `python <path>` rather than `python -m <dotted-path>` so the
# wrapper runs regardless of cwd / sys.path — it has no
# bot_bottle.* imports, so it's self-contained.
_PTY_RESIZE_SCRIPT = _pty_resize.__file__
# Per-user env the agent image's USER (node) expects. Some providers
# write session state under the user's home directory;
# bare `runuser -u` inherits root's HOME=/root, which claude
# can't write to. Set HOME / USER explicitly through smolvm -e
# so the child process sees them.
_HOME_FOR = {
"node": "/home/node",
"root": "/root",
}
_DEFAULT_PATH_FOR = {
# Committed smolmachine snapshots are rebuilt from a rootfs tarball and
# lose Docker image ENV metadata. Restore the provider CLI path here so
# resumed Codex bottles can still find the per-user install.
"node": (
"/home/node/.local/bin:"
"/home/node/.codex/packages/standalone/current/bin:"
"/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"
),
"root": "/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
}
def _env_assignments_for(user: str, env: Mapping[str, str]) -> list[str]:
home = _HOME_FOR.get(user, f"/home/{user}")
out = [f"HOME={home}", f"USER={user}"]
if "PATH" not in env:
out.append(f"PATH={_DEFAULT_PATH_FOR.get(user, _DEFAULT_PATH_FOR['root'])}")
for k, v in env.items():
out.append(f"{k}={v}")
return out
class SmolmachinesBottle(Bottle):
"""Handle returned by `SmolmachinesBottleBackend.launch`. The
underlying VM lifecycle (create / start / stop / delete) lives
on the launch ExitStack this class only routes runtime
operations to the right `smolvm machine ...` subcommand."""
def __init__(
self,
machine_name: str,
*,
prompt_path: str | None = None,
guest_env: Mapping[str, str] | None = None,
agent_command: str = "claude",
agent_prompt_mode: PromptMode = "append_file",
agent_provider_template: str = "claude",
terminal_title: str = "",
terminal_color: str = "",
agent_workdir: str = "/home/node",
) -> None:
self.name = machine_name
# In-VM path to the agent's prompt file. None when the
# agent declared no prompt (file still exists; we just
# don't pass --append-system-prompt-file).
self.prompt_path = prompt_path
# Env vars the agent process needs (HTTPS_PROXY,
# CLAUDE_CODE_OAUTH_TOKEN, manifest-declared bottle env, …).
# Forwarded on every `smolvm machine exec` via `-e K=V`
# because exec doesn't inherit from machine_create's env.
self._guest_env = dict(guest_env or {})
self._agent_prompt_mode = agent_prompt_mode
self.agent_command = agent_command
self.terminal_title = terminal_title
self.terminal_color = terminal_color
self.agent_provider_template = agent_provider_template
self.agent_workdir = agent_workdir
def agent_argv(
self, argv: list[str], *, tty: bool = True,
) -> list[str]:
flags = ["smolvm", "machine", "exec", "--name", self.name]
if tty:
flags += ["-i", "-t"]
agent_tail = ["env", *_env_assignments_for("node", self._guest_env)]
if self.agent_workdir and self.agent_workdir != _HOME_FOR["node"]:
agent_tail += [
"sh", "-lc",
f"cd {shlex.quote(self.agent_workdir)} && exec \"$@\"",
"bot-bottle-agent",
]
agent_tail.append(self.agent_command)
provider_prompt_args = prompt_args(
cast(PromptMode, self._agent_prompt_mode), self.prompt_path, argv=argv,
)
if cast(PromptMode, self._agent_prompt_mode) == "read_prompt_file":
agent_tail += argv
agent_tail += provider_prompt_args
else:
agent_tail += provider_prompt_args
agent_tail += argv
flags += ["--", "runuser", "-u", "node", "--", *agent_tail]
if not tty:
# No PTY allocated — no SIGWINCH to forward, no resize
# bridge needed. Skip the wrapper so non-interactive
# exec paths (e.g., provisioning shell-outs that
# happen to go through this method) stay light.
return flags
return [
sys.executable, _PTY_RESIZE_SCRIPT,
self.name, "--", *flags,
]
def exec_agent(self, argv: list[str], *, tty: bool = True) -> int:
"""Run the selected agent interactively inside the VM as the `node`
user. Inherits the operator's terminal (stdin / stdout /
stderr) so the session feels native. Blocks until the agent
exits; returns the in-VM exit code.
We bypass the captured-output `machine_exec` helper here
because that one wraps stdout/stderr in pipes fine for
scripted exec, wrong for an interactive shell. Drop down
to `subprocess.run` with the TTY inherited.
UID switches via `runuser -u node --` (not `-l`) so we
avoid login-shell wiring. HOME / USER come from `smolvm
-e` instead, which sets them on the process env."""
agent_argv = self.agent_argv(argv, tty=tty)
script = exec_shell_script(agent_argv, self.terminal_title, self.terminal_color) if tty else None
if script is None:
return subprocess.run(agent_argv, check=False).returncode
# Use sh -c (not -lc) so the script inherits PATH from the calling
# process. sh -l sources login-shell init files (e.g. /etc/profile)
# which may NOT include smolvm's location when it was installed via
# homebrew. The calling process (./cli.py) already has smolvm on PATH
# (provision steps succeed), so -c is sufficient.
return subprocess.run(["sh", "-c", script], check=False).returncode
# smolvm/libkrun can SIGKILL an otherwise-normal exec during
# early-VM provisioning. Retry once after a short settle so
# callers (provision_ca, etc.) don't have to handle it themselves.
_SIGKILL_EXIT = 128 + 9
def exec(self, script: str, *, user: str = "node") -> ExecResult:
"""Run a POSIX shell script as `user` (default `node`) and
capture the result. Matches the docker backend's `exec`,
which defaults to the image's USER (also node) — so test
helpers / provision shell-outs run with the same identity
on both backends. Pass `user="root"` for tests that need
root.
`runuser -u <user> -- env ... /bin/sh -c <script>` switches UID
without invoking a login shell, then sets HOME / USER and the
bottle env in the child process.
Retries once on SIGKILL (exit 137) libkrun occasionally
kills short-lived execs during VM bring-up."""
r = self._exec_raw(script, user=user)
if r.returncode == self._SIGKILL_EXIT:
time.sleep(1.0)
r = self._exec_raw(script, user=user)
return r
def _exec_raw(self, script: str, *, user: str = "node") -> ExecResult:
argv = [
"--", "runuser", "-u", user, "--",
"env", *_env_assignments_for(user, self._guest_env),
"/bin/sh", "-c", script,
]
r = subprocess.run(
["smolvm", "machine", "exec", "--name", self.name] + argv,
capture_output=True, text=True, check=False,
)
return ExecResult(
returncode=r.returncode,
stdout=r.stdout or "",
stderr=r.stderr or "",
)
def cp_in(self, host_path: str, container_path: str) -> None:
"""Copy a host path into the guest at `container_path`."""
_smolvm.machine_cp(host_path, f"{self.name}:{container_path}")
def close(self) -> None:
# Real teardown lives on the launch ExitStack; this is just
# the idempotent alias the BottleBackend ABC expects.
pass
@@ -0,0 +1,55 @@
"""SmolmachinesBottleCleanupPlan — concrete BottleCleanupPlan (issue #77).
Tracks the resources `SmolmachinesBottleBackend.cleanup` will
remove:
- machines: smolvm machines whose name starts with
`bot-bottle-` (running or stopped). Stopped +
deleted via `smolvm machine stop` + `machine delete -f`.
- bundles: docker containers `bot-bottle-sidecars-<slug>`
left over from a smolmachines bottle (the bundle's
port-forwards stay published on lo0 aliases until
the container is gone). Removed via `docker rm -f`.
- networks: docker networks `bot-bottle-bundle-<slug>`
attached to the bundles. Removed via
`docker network rm`.
Smolmachines state dirs live under the same `~/.bot-bottle/state/`
path the docker backend uses; the docker backend's
`prepare_cleanup` already enumerates orphan state dirs and is the
single source of truth for that bucket (consults
`enumerate_active_bottles()` so it doesn't reap a live
smolmachines bottle's dir)."""
from __future__ import annotations
import sys
from dataclasses import dataclass
from ...log import info
from .. import BottleCleanupPlan
@dataclass(frozen=True)
class SmolmachinesBottleCleanupPlan(BottleCleanupPlan):
"""Resources SmolmachinesBottleBackend.cleanup will remove.
Produced by `prepare_cleanup`; sorted so the y/N output is
stable."""
machines: tuple[str, ...] = ()
bundles: tuple[str, ...] = ()
networks: tuple[str, ...] = ()
@property
def empty(self) -> bool:
return not self.machines and not self.bundles and not self.networks
def print(self) -> None:
print(file=sys.stderr)
for name in self.machines:
info(f"smolvm machine: {name}")
for name in self.bundles:
info(f"bundle container:{name}")
for name in self.networks:
info(f"bundle network: {name}")
print(file=sys.stderr)
@@ -0,0 +1,96 @@
"""SmolmachinesBottlePlan — concrete BottlePlan for the smolmachines
backend (PRD 0023).
Slug + legacy bundle network coordinates + smolvm machine name +
agent `.smolmachine` artifact + per-bottle guest env."""
from __future__ import annotations
from dataclasses import dataclass
from pathlib import Path
from ...agent_provider import PromptMode
from .. import BottlePlan
@dataclass(frozen=True)
class SmolmachinesBottlePlan(BottlePlan):
"""Resolved fields the launch step needs to bring up the bottle.
Inherits `spec`, `stage_dir`, `git_gate_plan`, `egress_plan`,
`supervise_plan`, and `agent_provision` from BottlePlan."""
slug: str
# Legacy per-bottle bundle network coordinates. These remain on
# the plan while BundleLaunchSpec still carries the original shape,
# but the smolmachines launch path exposes the sidecar VM through
# host-loopback forwarders instead of a Docker bridge IP.
bundle_subnet: str
bundle_gateway: str
bundle_ip: str
# In-guest env vars (HTTPS_PROXY etc) — IP-literal URLs since
# the guest has no DNS resolver inside the TSI allowlist.
# Passed to `smolvm machine create` as `-e K=V` flags.
# Smolfile-rendering is gone (smolvm 0.8.0's
# `--smolfile` is mutually exclusive with `--from`, and
# `--from` is the path that avoids the registry-pull race).
guest_env: dict[str, str]
# Agent-side endpoints. Empty at prepare time; launch populates
# these after sidecar VM bringup via `dataclasses.replace`.
# Format: a `host:port` for git-gate (insteadOf URL prefix) +
# full URLs for proxy / supervise.
agent_proxy_url: str = ""
agent_git_gate_host: str = ""
agent_supervise_url: str = ""
@property
def machine_name(self) -> str:
"""smolvm machine name. `machine_create` boots from a packed
`.smolmachine` artifact (pre-baked at prepare time via
`smolvm pack create`); using `--from` instead of `--image`
avoids the registry-pull race we hit when machine_start tried
to fetch on-demand and the libkrun agent's network attempt
got refused by macOS."""
return self.agent_provision.instance_name
@property
def agent_image(self) -> str:
"""Agent image ref (docker tag). `launch` runs the
build save registry push smolvm pack pipeline against
this and feeds the resulting `.smolmachine` artifact to
`machine_create --from`. The pipeline runs at launch time
(not prepare time) so the docker build output doesn't garble
the dashboard's preflight modal."""
return self.agent_provision.image
@property
def prompt_file(self) -> Path:
"""Path to the agent's prompt file on the host. Always written
(mode 0o600) so the in-VM path always exists; the file is
empty when the agent has no prompt claude-code reads it
via --append-system-prompt-file only when non-empty."""
return self.agent_provision.prompt_file
@property
def git_gate_insteadof_host(self) -> str:
return self.agent_git_gate_host
@property
def git_gate_insteadof_scheme(self) -> str:
return "http"
@property
def agent_command(self) -> str:
return self.agent_provision.command
@property
def agent_prompt_mode(self) -> PromptMode:
return self.agent_provision.prompt_mode
@property
def agent_provider_template(self) -> str:
return self.agent_provision.template
@property
def agent_dockerfile_path(self) -> str:
return self.agent_provision.dockerfile
+159
View File
@@ -0,0 +1,159 @@
"""Cleanup + active-listing for the smolmachines backend (issue #77).
`prepare_cleanup` enumerates leftover smolmachines resources:
- smolvm machines (`smolvm machine ls --json`) whose name starts
with `bot-bottle-`.
- bundle docker containers (`bot-bottle-sidecars-<slug>`).
- bundle docker networks (`bot-bottle-bundle-<slug>`).
State dirs live under `~/.bot-bottle/state/<identity>/`
shared layout with the docker backend, which has the single
orphan-state-dir enumerator (it already consults
`enumerate_active_agents()` so a live smolmachines bottle's dir
is preserved).
`cleanup` removes everything in the plan: stop + delete each VM,
force-rm each container, rm each network. Each step is
best-effort a failure on one resource doesn't block the others."""
from __future__ import annotations
import json
import subprocess
from ...log import info, warn
from . import sidecar_bundle as _bundle
from . import smolvm as _smolvm
from .bottle_cleanup_plan import SmolmachinesBottleCleanupPlan
# Both names start with the same prefix the launcher uses.
_VM_PREFIX = "bot-bottle-"
_BUNDLE_PREFIX = _bundle.bundle_container_name("") # `bot-bottle-sidecars-`
_NETWORK_PREFIX = _bundle.bundle_network_name("") # `bot-bottle-bundle-`
def prepare_cleanup() -> SmolmachinesBottleCleanupPlan:
"""Enumerate every smolmachines-owned resource on the host.
No side effects. Returns an empty plan when smolvm isn't on
PATH (no machines to reap) `cleanup` is a no-op in that
case too."""
machines = _list_bot_bottle_machines()
bundles = _list_bundle_containers()
networks = _list_bundle_networks()
return SmolmachinesBottleCleanupPlan(
machines=tuple(sorted(machines)),
bundles=tuple(sorted(bundles)),
networks=tuple(sorted(networks)),
)
def cleanup(plan: SmolmachinesBottleCleanupPlan) -> None:
"""Remove everything in the plan. Order matters: stop VMs
first (they hold ports on lo0 aliases via libkrun), then the
bundle containers (which hold the host port-forwards), then
the networks (which docker won't reap until the containers
are gone)."""
for name in plan.machines:
info(f"stopping smolvm machine {name}")
subprocess.run(
["smolvm", "machine", "stop", "--name", name],
stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL,
check=False,
)
info(f"deleting smolvm machine {name}")
r = subprocess.run(
["smolvm", "machine", "delete", "-f", name],
capture_output=True, text=True, check=False,
)
if r.returncode != 0:
warn(
f"smolvm machine delete -f {name} failed: "
f"{(r.stderr or '').strip()}"
)
for name in plan.bundles:
info(f"removing bundle container {name}")
subprocess.run(
["docker", "rm", "-f", name],
stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL,
check=False,
)
for name in plan.networks:
info(f"removing bundle network {name}")
r = subprocess.run(
["docker", "network", "rm", name],
capture_output=True, text=True, check=False,
)
if r.returncode != 0 and "no such network" not in (r.stderr or "").lower():
warn(
f"docker network rm {name} failed: "
f"{(r.stderr or '').strip()}"
)
def _list_bot_bottle_machines() -> list[str]:
"""All smolvm machines named `bot-bottle-*`, regardless of
state (running / stopped / created). Empty when smolvm isn't
installed."""
if not _smolvm.is_available():
return []
r = subprocess.run(
["smolvm", "machine", "ls", "--json"],
capture_output=True, text=True, check=False,
)
if r.returncode != 0:
return []
try:
machines = json.loads(r.stdout or "[]")
except json.JSONDecodeError:
return []
return [
m["name"] for m in machines
if isinstance(m, dict)
and m.get("name", "").startswith(_VM_PREFIX)
]
def _list_bundle_containers() -> list[str]:
"""All docker containers named `bot-bottle-sidecars-*`,
running or stopped. Empty when docker isn't installed."""
# Late import: `backend/__init__` imports this module
# transitively via the smolmachines backend.
from .. import has_backend
if not has_backend("docker"):
return []
r = subprocess.run(
["docker", "ps", "-a",
"--filter", f"name=^{_BUNDLE_PREFIX}",
"--format", "{{.Names}}"],
capture_output=True, text=True, check=False,
)
if r.returncode != 0:
return []
return [
line for line in (r.stdout or "").splitlines()
if line and line.startswith(_BUNDLE_PREFIX)
]
def _list_bundle_networks() -> list[str]:
"""All docker networks named `bot-bottle-bundle-*`. Empty
when docker isn't installed."""
from .. import has_backend
if not has_backend("docker"):
return []
r = subprocess.run(
["docker", "network", "ls",
"--filter", f"name={_NETWORK_PREFIX}",
"--format", "{{.Name}}"],
capture_output=True, text=True, check=False,
)
if r.returncode != 0:
return []
return [
line for line in (r.stdout or "").splitlines()
if line and line.startswith(_NETWORK_PREFIX)
]
@@ -0,0 +1,52 @@
"""Egress apply for the smolmachines backend.
The smolmachines sidecar bundle runs as a sidecar smolVM. Route-file
inspection and reload signalling therefore go through ``smolvm machine
exec`` instead of Docker.
"""
from __future__ import annotations
from ...egress import EGRESS_ROUTES_IN_CONTAINER
from ...log import warn
from ..egress_apply import EgressApplicator, EgressApplyError
from . import sidecar_bundle as _bundle
from . import smolvm as _smolvm
def fetch_current_routes(slug: str) -> str:
machine = _bundle.bundle_machine_name(slug)
result = _smolvm.machine_exec(machine, ["cat", EGRESS_ROUTES_IN_CONTAINER])
if result.returncode != 0:
raise EgressApplyError(
f"could not read routes.yaml from {machine}: "
f"{(result.stderr or '').strip() or 'sidecar VM not running?'}"
)
return result.stdout
class SmolmachinesEgressApplicator(EgressApplicator):
def _signal_bundle_reload(self, slug: str) -> None:
machine = _bundle.bundle_machine_name(slug)
result = _smolvm.machine_exec(machine, ["sh", "-c", "kill -HUP 1"])
if result.returncode != 0:
last_error = (result.stderr or "").strip() or (result.stdout or "").strip()
warn(
f"egress: routes updated on disk for {slug}, but bundle reload failed: "
f"{last_error or 'smolvm exec failed'}"
)
raise EgressApplyError(
f"could not reload egress bundle {machine}: "
f"{last_error or 'smolvm exec failed'}"
)
applicator = SmolmachinesEgressApplicator()
__all__ = [
"SmolmachinesEgressApplicator",
"EgressApplyError",
"applicator",
"fetch_current_routes",
]
@@ -0,0 +1,128 @@
"""Active-agent enumeration for the smolmachines backend (PRD
0023 chunk 4 follow-up + issue #77).
Returns a list of `ActiveAgent` records same shape the docker
backend produces so CLI `list active` and the dashboard agents
pane render both backends through one code path.
A smolmachines agent is "active" when its smolvm guest is
running. We cross-reference against the per-bottle sidecar
bundle container to populate the `services` field (which daemons
are up in the bundle); without a bundle we still surface the VM
so the operator can see + clean it up.
The cross-backend caller gates on `has_backend("smolmachines")`
and `has_backend("docker")`, so this module assumes both are
available when called. Both subprocess calls below still
tolerate "command not on PATH" defensively, but the gate is the
intended access pattern."""
from __future__ import annotations
import json
import subprocess
from .. import ActiveAgent
from ...bottle_state import read_metadata
from . import sidecar_bundle as _bundle
# Smolvm VM names produced by prepare are `bot-bottle-<slug>`,
# matching the bundle container name pattern. We use the prefix
# both as a filter and to strip back to the slug.
_VM_NAME_PREFIX = "bot-bottle-"
_SIDECAR_VM_PREFIX = _bundle.bundle_machine_name("")
def enumerate_active() -> list[ActiveAgent]:
"""All currently-running smolmachines-backed agents. Empty
list when no matching VMs are running. Caller is responsible
for gating on `has_backend('smolmachines')` if needed; if
smolvm is missing the `smolvm machine ls` call below returns
nothing silently."""
result = subprocess.run(
["smolvm", "machine", "ls", "--json"],
capture_output=True, text=True, check=False,
)
if result.returncode != 0:
return []
try:
machines = json.loads(result.stdout or "[]")
except json.JSONDecodeError:
return []
services_by_slug = _query_bundle_services()
out: list[ActiveAgent] = []
for m in machines:
name = m.get("name") or ""
state = m.get("state") or ""
if (
state != "running"
or not name.startswith(_VM_NAME_PREFIX)
or name.startswith(_SIDECAR_VM_PREFIX)
):
continue
slug = name[len(_VM_NAME_PREFIX):]
metadata = read_metadata(slug)
out.append(ActiveAgent(
backend_name="smolmachines",
slug=slug,
agent_name=metadata.agent_name if metadata else "?",
started_at=metadata.started_at if metadata else "",
services=services_by_slug.get(slug, ()),
label=metadata.label if metadata else "",
color=metadata.color if metadata else "",
))
return out
def _query_bundle_services() -> dict[str, tuple[str, ...]]:
"""`{slug: ('egress', ...)}` from each running bundle container's
`BOT_BOTTLE_SIDECAR_DAEMONS` env var.
Smolmachines bundles all run the PRD-0024 image with the
same daemon set declared via env, so one inspect per bundle
gets us the picture without exec'ing into the container.
Returns an empty mapping when the docker backend isn't
available the bundle services field on each ActiveAgent
just shows up empty, matching the docker backend's "starting"
state."""
# Late import: `has_backend` lives on the backend package's
# __init__, which imports this module transitively. Pulling
# the name in at call time sidesteps the cycle.
from .. import has_backend
if not has_backend("docker"):
return {}
ps = subprocess.run(
["docker", "ps",
"--filter", "name=" + _bundle.bundle_container_name(""),
"--format", "{{.Names}}"],
capture_output=True, text=True, check=False,
)
if ps.returncode != 0:
return {}
out: dict[str, tuple[str, ...]] = {}
for line in (ps.stdout or "").splitlines():
name = line.strip()
if not name:
continue
slug = name.removeprefix(_bundle.bundle_container_name(""))
if not slug:
continue
inspect = subprocess.run(
["docker", "inspect", name, "--format", "{{json .Config.Env}}"],
capture_output=True, text=True, check=False,
)
if inspect.returncode != 0:
continue
try:
env_list = json.loads(inspect.stdout or "[]")
except json.JSONDecodeError:
continue
for entry in env_list:
key, _, value = entry.partition("=")
if key == "BOT_BOTTLE_SIDECAR_DAEMONS":
out[slug] = tuple(sorted(
d for d in value.split(",") if d
))
break
return out
+145
View File
@@ -0,0 +1,145 @@
"""SmolmachinesFreezer — snapshot a smolmachines bottle.
`smolvm pack create --from-vm` requires the VM to be stopped, and smolvm
removes VMs when stopped (same issue as Apple Container). Instead, exec
into the running VM as root to write a gzip-compressed tar of the root
filesystem to /var/tmp, then copy it to the host with `smolvm machine cp`,
build a Docker image from the archive, convert it to a smolmachine artifact
via the existing registry pipeline, and record the sidecar path. The VM
stays running throughout."""
from __future__ import annotations
import tempfile
from pathlib import Path
from .. import ActiveAgent
from ..freeze import Freezer
from ..docker import util as docker_mod
from .local_registry import crane_push_tarball, ephemeral_registry
from .smolvm import machine_cp, machine_exec, pack_create
from ...bottle_state import bottle_state_dir
from ...log import die, info
# Temp file written inside the VM during commit. Lives in /var/tmp
# (on-disk, unlike tmpfs /tmp) to survive for machine_cp.
_VM_COMMIT_TAR = "/var/tmp/.bot-bottle-commit.tar.gz"
class SmolmachinesFreezer(Freezer):
"""Freezes a smolmachines bottle via exec-tar + Docker image + smolmachine pack.
The VM is NOT stopped. We exec into the running VM to write a compressed
tar of the root filesystem to /var/tmp, copy it to the host with
machine_cp, build a Docker image (Docker's ADD decompresses .tar.gz
automatically), then run the same imageregistrypack_create pipeline
that _ensure_smolmachine uses for fresh builds."""
backend_name = "smolmachines"
def _freeze(self, agent: ActiveAgent) -> str:
machine = f"bot-bottle-{agent.slug}"
image_ref = f"bot-bottle-committed-{agent.slug}:latest"
output_dir = bottle_state_dir(agent.slug)
output_dir.mkdir(parents=True, exist_ok=True)
binary = output_dir / "committed-smolmachine"
sidecar = output_dir / "committed-smolmachine.smolmachine"
_snapshot_running_vm(machine, image_ref, binary)
return str(sidecar)
def _export_hint(self, slug: str, image_ref: str) -> None:
info(f"to export for migration: cp {image_ref} {slug}.smolmachine")
def _snapshot_running_vm(machine: str, image_ref: str, binary: Path) -> None:
"""Exec-tar the running VM, build a Docker image, and pack to a smolmachine.
binary: destination for the launcher (sibling .smolmachine is the artifact
that machine_create --from consumes, same convention as pack_create).
"""
with tempfile.TemporaryDirectory(prefix="bot-bottle-vm-commit.") as tmp:
tmp_path = Path(tmp)
# Use .tar.gz — Docker ADD decompresses automatically and the
# compressed archive fits in the VM's /var/tmp more easily.
rootfs_tar_gz = tmp_path / "rootfs.tar.gz"
dockerfile = tmp_path / "Dockerfile"
_exec_tar_to_file(machine, rootfs_tar_gz)
dockerfile.write_text(
"FROM scratch\n"
"ADD rootfs.tar.gz /\n"
"USER node\n"
"WORKDIR /home/node\n"
)
docker_mod.build_image(image_ref, str(tmp_path), dockerfile=str(dockerfile))
image_tarball = binary.parent / "committed.image.tar"
docker_mod.save(image_ref, str(image_tarball))
try:
with ephemeral_registry() as handle:
digest = docker_mod.image_id(image_ref).split(":", 1)[-1][:16]
push_ref = f"{handle.push_endpoint}/bot-bottle-committed:{digest}"
pack_ref = f"{handle.pull_endpoint}/bot-bottle-committed:{digest}"
crane_push_tarball(handle, str(image_tarball), push_ref)
pack_create(pack_ref, binary)
finally:
image_tarball.unlink(missing_ok=True)
def _exec_tar_to_file(machine: str, dest: Path) -> None:
"""Snapshot the running VM's root filesystem to dest (.tar.gz).
Writes a gzip-compressed tar to _VM_COMMIT_TAR inside the VM via
machine_exec (same mechanism as provisioning), then copies it to the
host with machine_cp. This avoids binary-stdout piping through the
smolvm exec channel, which does not reliably handle large binary output.
A connectivity probe (machine_exec true) runs first so a concurrent-exec
limitation (smolvm may reject a second exec while -i -t is active) is
reported clearly rather than as a silent failure."""
# Connectivity probe — if smolvm rejects concurrent exec while an
# interactive session is running, fail clearly here.
probe = machine_exec(machine, ["true"])
if probe.returncode != 0:
die(
f"smolvm exec is not available for {machine!r} "
f"(exit {probe.returncode}: {probe.stderr.strip() or probe.stdout.strip() or '<no output>'}). "
f"If an interactive session is active, smolvm may not support concurrent exec."
)
# Create the compressed tar inside the VM.
# tar exits 1 when files change during archiving (normal for a live
# filesystem); only treat exit > 1 as fatal.
tar_result = machine_exec(
machine,
[
"tar", "--create", "--gzip",
"--exclude=./proc",
"--exclude=./sys",
"--exclude=./dev",
"--exclude=./run",
# /tmp and /var/tmp are ephemeral. Their stale contents
# (e.g. /tmp/claude-<uid>) have uid remapped by smolvm's
# pack process, causing Claude Code to refuse to use them
# on resume. Exclude both; _init_vm recreates them with
# mkdir -p + correct ownership on every boot.
"--exclude=./tmp",
"--exclude=./var/tmp",
f"--file={_VM_COMMIT_TAR}",
"--directory=/",
".",
],
)
if tar_result.returncode > 1:
die(
f"smolvm exec tar {machine!r} failed (exit {tar_result.returncode}): "
f"{tar_result.stderr.strip() or tar_result.stdout.strip() or '<no output>'}"
)
# Copy from VM to host, then clean up.
try:
machine_cp(f"{machine}:{_VM_COMMIT_TAR}", str(dest))
finally:
machine_exec(machine, ["rm", "-f", _VM_COMMIT_TAR])
+531
View File
@@ -0,0 +1,531 @@
"""End-to-end launch flow for the smolmachines backend.
Builds the sidecar bundle smolmachine, starts it as a sidecar VM
with real daemons + their config files, creates + starts the agent
smolVM, yields a `SmolmachinesBottle` handle, and tears everything
down on context exit.
The bundle's daemons consume the inner Plans the docker backend
already produces: egress reads routes + CAs from the EgressPlan.
Git-gate + supervise plumb through the same plans the docker
backend uses, minus the docker-network fields that don't apply here."""
from __future__ import annotations
import dataclasses
import os
from contextlib import ExitStack, contextmanager
from pathlib import Path
from typing import Callable, Generator
from ...egress import (
EGRESS_ROUTES_IN_CONTAINER,
egress_agent_env_entries,
egress_resolve_token_values,
egress_sidecar_env_entries,
)
from ...supervise import DB_PATH_IN_CONTAINER, SUPERVISE_PORT
from ...util import expand_tilde
from ..docker import util as docker_mod
from ..docker.egress import (
EGRESS_CA_IN_CONTAINER,
EGRESS_PORT as _EGRESS_PORT,
egress_tls_init,
)
from ..docker.git_gate import (
GIT_GATE_ACCESS_HOOK_IN_CONTAINER,
GIT_GATE_CREDS_DIR_IN_CONTAINER,
GIT_GATE_ENTRYPOINT_IN_CONTAINER,
GIT_GATE_HOOK_IN_CONTAINER,
)
from ...git_gate import (
provision_git_gate_dynamic_keys,
revoke_git_gate_provisioned_keys,
)
from ...log import info, warn
from ...bottle_state import (
egress_state_dir,
git_gate_state_dir,
read_committed_image,
)
from . import loopback_alias as _loopback
from . import port_forward as _forward
from . import sidecar_bundle as _bundle
from . import smolvm as _smolvm
from .bottle import SmolmachinesBottle
from .bottle_plan import SmolmachinesBottlePlan
from .local_registry import crane_push_tarball, ephemeral_registry
# Repo root, used as the `docker build` context for the agent image.
_REPO_DIR = str(Path(__file__).resolve().parent.parent.parent.parent)
# Per-host cache for `smolvm pack create` outputs. Keyed by the
# docker image ID so a Dockerfile change automatically invalidates
# the cache. `pack create` is idempotent on the smolvm side but
# takes several seconds even on a no-op rebuild.
_SMOLMACHINE_CACHE_DIR = Path.home() / ".cache" / "bot-bottle" / "smolmachines"
# Container-internal listening ports for each bundle daemon. The
# sidecar VM publishes each one on a random host loopback port, and
# the launch flow wraps those raw ports with per-bottle forwarders.
_GIT_HTTP_PORT = 9420
_SUPERVISE_PORT = SUPERVISE_PORT
@contextmanager
def launch(
plan: SmolmachinesBottlePlan,
*,
provision: Callable[[SmolmachinesBottlePlan, "SmolmachinesBottle"], str | None],
) -> Generator[SmolmachinesBottle, None, None]:
"""Build + run the bottle and yield a handle; tear everything
down on exit. Errors during bringup unwind any partial state
via the ExitStack."""
stack = ExitStack()
try:
loopback_ip, network = _allocate_resources(plan, stack)
plan = _mint_certs(plan)
proxy_host = loopback_ip
plan = _start_bundle(plan, network, proxy_host, stack)
agent_from_path = _agent_from_path(plan)
_launch_vm(plan, agent_from_path, proxy_host, stack)
_init_vm(plan)
bottle = SmolmachinesBottle(
plan.machine_name,
prompt_path=None,
guest_env=plan.guest_env,
agent_command=plan.agent_command,
agent_prompt_mode=plan.agent_prompt_mode,
agent_provider_template=plan.agent_provider_template,
terminal_title=f"{plan.spec.label} ({plan.spec.agent_name})" if plan.spec.label else plan.spec.agent_name,
terminal_color=plan.spec.color,
agent_workdir=plan.workspace_plan.workdir,
)
bottle.prompt_path = provision(plan, bottle)
yield bottle
finally:
_teardown_smolmachines(stack, plan)
def _teardown_smolmachines(
stack: ExitStack,
plan: SmolmachinesBottlePlan,
) -> None:
"""Unwind the ExitStack, then revoke any provisioned deploy keys.
ExitStack errors are caught and logged (non-fatal) so that key
revocation always runs. Revocation errors propagate a stranded
deploy key is a security concern the operator must address."""
teardown_exc: BaseException | None = None
try:
stack.close()
except BaseException as exc: # noqa: W0718 — teardown must not fail
teardown_exc = exc
warn(f"smolmachines teardown failed: {exc!r}")
bottle = plan.manifest.bottle
revoke_git_gate_provisioned_keys(bottle, git_gate_state_dir(plan.slug))
if teardown_exc is not None:
raise teardown_exc
def _allocate_resources(
plan: SmolmachinesBottlePlan,
stack: ExitStack,
) -> tuple[str, str]:
"""Reserve a per-bottle host address.
The per-bottle address scopes TSI's allowlist to this bottle's
forwarder-published ports so the agent can't reach other bottles'
or host services. The returned network name remains in the bundle
spec for compatibility with older helper tests; no Docker sidecar
container is launched."""
del stack
_loopback.ensure_pool()
loopback_ip = _loopback.allocate(plan.slug)
network = _bundle.bundle_network_name(plan.slug)
return loopback_ip, network
def _mint_certs(plan: SmolmachinesBottlePlan) -> SmolmachinesBottlePlan:
"""Mint the egress MITM CA and return the plan with CA paths filled."""
egress_ca_host, egress_ca_cert_only = egress_tls_init(
egress_state_dir(plan.slug),
)
egress_plan = dataclasses.replace(
plan.egress_plan,
mitmproxy_ca_host_path=egress_ca_host,
mitmproxy_ca_cert_only_host_path=egress_ca_cert_only,
)
return dataclasses.replace(plan, egress_plan=egress_plan)
def _start_bundle(
plan: SmolmachinesBottlePlan,
network: str,
proxy_host: str,
stack: ExitStack,
) -> SmolmachinesBottlePlan:
"""Build the BundleLaunchSpec, start the sidecar VM, wrap its raw
smolVM-published loopback ports with per-bottle forwarders, stamp
agent URLs from those forwarder ports, and register teardown."""
plan = _provision_git_gate_keys(plan)
bundle_spec = _bundle_launch_spec(plan, network, proxy_host)
token_env = _resolve_token_env(plan, dict(os.environ))
artifact = _ensure_smolmachine(
bundle_spec.image,
dockerfile=_bundle.SIDECAR_BUNDLE_DOCKERFILE,
)
launch = _bundle.start_bundle_vm(
bundle_spec,
from_path=artifact,
host_env={**os.environ, **token_env},
)
stack.callback(_bundle.stop_bundle_vm, plan.slug)
forward_specs = tuple(
_forward.ForwardSpec(
label=_label_for_port(container_port),
listen_host=proxy_host,
listen_port=0,
target_host="127.0.0.1",
target_port=host_port,
)
for container_port, host_port in launch.raw_ports.items()
)
handle = _forward.start_forwarder(forward_specs)
stack.callback(_forward.stop_forwarder, handle)
published_ports = {
_port_for_label(spec.label): spec.listen_port
for spec in handle.forwards
}
return _discover_urls(plan, proxy_host, published_ports)
def _provision_git_gate_keys(
plan: SmolmachinesBottlePlan,
) -> SmolmachinesBottlePlan:
if not plan.git_gate_plan.upstreams:
return plan
git_gate_plan = provision_git_gate_dynamic_keys(
plan.manifest.bottle,
plan.git_gate_plan,
git_gate_state_dir(plan.slug),
)
return dataclasses.replace(plan, git_gate_plan=git_gate_plan)
def _discover_urls(
plan: SmolmachinesBottlePlan,
proxy_host: str,
published_ports: dict[int, int],
) -> SmolmachinesBottlePlan:
"""Stamp URLs + guest_env from per-bottle forwarder ports.
`proxy_host` is the host IP that both TSI's allowlist and the
forwarder listeners are keyed to. The raw smolVM-published ports
are intentionally not advertised to the agent.
NO_PROXY includes `proxy_host` so supervise + git-gate URLs
bypass HTTPS_PROXY."""
agent_facing_host_port = published_ports[_EGRESS_PORT]
agent_proxy_url = f"http://{proxy_host}:{agent_facing_host_port}"
agent_git_gate_host = ""
if plan.git_gate_plan.upstreams:
git_gate_host_port = published_ports[_GIT_HTTP_PORT]
agent_git_gate_host = f"{proxy_host}:{git_gate_host_port}"
agent_supervise_url = ""
if plan.supervise_plan is not None:
supervise_host_port = published_ports[_SUPERVISE_PORT]
agent_supervise_url = f"http://{proxy_host}:{supervise_host_port}/"
existing_no_proxy = plan.guest_env.get("NO_PROXY", "localhost,127.0.0.1")
no_proxy = f"{existing_no_proxy},{proxy_host}"
guest_env = {
**plan.guest_env,
"HTTPS_PROXY": agent_proxy_url,
"HTTP_PROXY": agent_proxy_url,
"https_proxy": agent_proxy_url,
"http_proxy": agent_proxy_url,
"NO_PROXY": no_proxy,
"no_proxy": no_proxy,
}
if agent_git_gate_host:
guest_env["GIT_GATE_URL"] = f"http://{agent_git_gate_host}"
if agent_supervise_url:
guest_env["MCP_SUPERVISE_URL"] = agent_supervise_url
for entry in egress_agent_env_entries(plan.egress_plan):
name, value = entry.split("=", 1)
guest_env[name] = value
return dataclasses.replace(
plan,
guest_env=guest_env,
agent_proxy_url=agent_proxy_url,
agent_git_gate_host=agent_git_gate_host,
agent_supervise_url=agent_supervise_url,
)
def _launch_vm(
plan: SmolmachinesBottlePlan,
agent_from_path: Path,
proxy_host: str,
stack: ExitStack,
) -> None:
"""Create, patch, and start the smolvm VM; register teardown.
--allow-cidr is `proxy_host/32` the per-bottle loopback alias.
This ensures the guest can only reach sidecar forwarders published
on that IP, not host localhost or another bottle's alias.
force_allowlist confirms the allowlist persisted (patching smolvm
0.8.0's silent-drop of --allow-cidr when combined with --from) and
fails closed if it can't. Smolfile isn't usable here smolvm 0.8.0
makes --from and --smolfile mutually exclusive."""
tsi_cidr = f"{proxy_host}/32"
_smolvm.machine_create(
plan.machine_name,
from_path=agent_from_path,
allow_cidrs=[tsi_cidr],
env=plan.guest_env,
)
stack.callback(_smolvm.machine_delete, plan.machine_name)
# Confirm the booted VM's TSI allowlist will actually enforce the
# /32 before start (smolvm 0.8.0 silently drops `--allow-cidr`
# with `--from`, so the persisted state DB is patched if needed).
# Fails closed if enforcement can't be confirmed.
_loopback.force_allowlist(plan.machine_name, [tsi_cidr])
_smolvm.machine_start(plan.machine_name)
stack.callback(_smolvm.machine_stop, plan.machine_name)
def _init_vm(plan: SmolmachinesBottlePlan) -> None:
"""Repair filesystem ownership and wait for exec channel readiness.
Ownership repair: smolvm's pack process remaps files to the host
invoker's uid (e.g. 501 on macOS, 1000 on Linux). The chowns use
names not numbers so they're correct on either. /home/node must
be node:node so
Claude Code can write ~/.claude.json; /tmp + /var/tmp need root
mode 1777 so non-root processes can create per-uid scratch dirs.
All folded into one sh -c to avoid back-to-back exec calls
immediately after machine_start (libkrun exec-channel race).
mkdir -p guards: when booting from a committed snapshot, /tmp and
/var/tmp are excluded from the archive (they're ephemeral and their
stale contents would have wrong uid after smolvm's uid remap). The
directories must be created before chown/chmod can set permissions.
wait_exec_ready polls until the exec channel is ready for the
subsequent provision calls, replacing the empirical sleep."""
_smolvm.machine_exec(plan.machine_name, [
"sh", "-c",
"mkdir -p /tmp /var/tmp && "
"chown -R node:node /home/node && "
"chown root:root /tmp /var/tmp && "
"chmod 1777 /tmp /var/tmp",
])
_smolvm.wait_exec_ready(plan.machine_name)
def _label_for_port(port: int) -> str:
if port == _EGRESS_PORT:
return "egress"
if port == _GIT_HTTP_PORT:
return "git-http"
if port == _SUPERVISE_PORT:
return "supervise"
return f"port-{port}"
def _port_for_label(label: str) -> int:
if label == "egress":
return _EGRESS_PORT
if label == "git-http":
return _GIT_HTTP_PORT
if label == "supervise":
return _SUPERVISE_PORT
if label.startswith("port-"):
return int(label.removeprefix("port-"))
raise ValueError(f"unknown sidecar forward label: {label}")
def _bundle_launch_spec(
plan: SmolmachinesBottlePlan, network: str, proxy_host: str,
) -> _bundle.BundleLaunchSpec:
"""Build a BundleLaunchSpec from the resolved inner Plans.
Daemons in the CSV:
- egress is always present.
- git-gate + git-http are conditional on plan.git_gate_plan.upstreams.
- supervise is conditional on plan.supervise_plan.
Env + volumes are the union of the sidecar daemons' needs, with
daemon-private values only (HTTPS_PROXY is scoped to the
egress process by egress_entrypoint.sh see PRD 0024's bundle
bind-address PR)."""
daemons: list[str] = ["egress"]
env: list[str] = []
volumes: list[tuple[str, str, bool]] = []
# --- egress -----------------------------------------------
ep = plan.egress_plan
volumes.append((str(ep.mitmproxy_ca_host_path), EGRESS_CA_IN_CONTAINER, True))
if ep.routes:
volumes.append((str(ep.routes_path.parent), str(Path(EGRESS_ROUTES_IN_CONTAINER).parent), True))
env.extend(egress_sidecar_env_entries(ep))
# --- git-gate ---------------------------------------------
gp = plan.git_gate_plan
if gp.upstreams:
daemons += ["git-gate", "git-http"]
volumes += [
(str(gp.entrypoint_script), GIT_GATE_ENTRYPOINT_IN_CONTAINER, True),
(str(gp.hook_script), GIT_GATE_HOOK_IN_CONTAINER, True),
(str(gp.access_hook_script), GIT_GATE_ACCESS_HOOK_IN_CONTAINER, True),
]
for u in gp.upstreams:
keypath = expand_tilde(u.identity_file)
volumes.append((
keypath,
f"{GIT_GATE_CREDS_DIR_IN_CONTAINER}/{u.name}-key",
True,
))
if u.known_hosts_file:
volumes.append((
str(u.known_hosts_file),
f"{GIT_GATE_CREDS_DIR_IN_CONTAINER}/{u.name}-known_hosts",
True,
))
# --- supervise --------------------------------------------
sp = plan.supervise_plan
if sp is not None:
daemons.append("supervise")
env += [
f"SUPERVISE_BOTTLE_SLUG={plan.slug}",
f"SUPERVISE_DB_PATH={DB_PATH_IN_CONTAINER}",
f"SUPERVISE_PORT={SUPERVISE_PORT}",
]
volumes.append((str(sp.db_path), DB_PATH_IN_CONTAINER, False))
# Container ports the agent reaches from the smolvm guest —
# published on `proxy_host` so the TSI allowlist and the docker
# port-forward bindings point at the same IP. Egress is always
# the agent's HTTP/HTTPS proxy.
ports_to_publish: list[int] = [_EGRESS_PORT]
if gp.upstreams:
ports_to_publish.append(_GIT_HTTP_PORT)
if sp is not None:
ports_to_publish.append(_SUPERVISE_PORT)
return _bundle.BundleLaunchSpec(
slug=plan.slug,
network_name=network,
subnet=plan.bundle_subnet,
gateway=plan.bundle_gateway,
bundle_ip=plan.bundle_ip,
daemons_csv=",".join(daemons),
environment=tuple(env),
volumes=tuple(volumes),
ports_to_publish=tuple(ports_to_publish),
publish_host_ip=proxy_host,
)
def _resolve_token_env(
plan: SmolmachinesBottlePlan, host_env: dict[str, str],
) -> dict[str, str]:
"""Resolve the egress token env-var values from the host's
environ so they reach the bundle's process env via docker's
`-e NAME` inheritance. Empty when no routes declare auth."""
effective_env = {**host_env, **plan.agent_provision.provisioned_env}
return egress_resolve_token_values(plan.egress_plan.token_env_map, effective_env)
def _agent_from_path(plan: SmolmachinesBottlePlan) -> Path:
"""Return the `.smolmachine` artifact used for `machine create --from`.
Prefer a committed VM artifact when one is recorded and still
present. If the file was removed, fall back to the normal image
build + pack cache path.
"""
committed = read_committed_image(plan.slug)
if committed:
committed_path = Path(committed)
if committed_path.is_file():
info(f"using committed smolmachine {str(committed_path)!r}")
return committed_path
# Build the agent image and pack it into a `.smolmachine`
# artifact (or hit the per-Dockerfile-digest cache). Runs here,
# not in prepare, so the docker-build output doesn't garble the
# dashboard's preflight modal.
return _ensure_smolmachine(
plan.agent_image,
dockerfile=plan.agent_dockerfile_path,
)
def _ensure_smolmachine(image_ref: str, *, dockerfile: str = "") -> Path:
"""Build the agent docker image and convert it into a
`.smolmachine` artifact, caching the result under
`~/.cache/bot-bottle/smolmachines/` keyed by the docker image
ID (so a Dockerfile change automatically invalidates the cache).
Returns the `.smolmachine.smolmachine` sidecar path that's
the file `machine create --from` consumes (pack create produces
a launcher binary at `.smolmachine` plus the sidecar alongside
it; the sidecar is the actual artifact).
Conversion path: `docker build` (the existing layer cache
makes no-change rebuilds cheap) `docker save` to a tarball
spin up an ephemeral registry on a private docker network
`crane push --insecure` from a one-shot container on the same
network `smolvm pack create --image localhost:<host port>/...`
tear down the registry + network. The crane push detour
sidesteps the Docker-Desktop daemon's HTTPS preference for
non-loopback registries see the `local_registry` module
docstring for the gory details.
Each pack-create costs several seconds even on a hot cache,
so we skip the whole pipeline when the cached sidecar is
already on disk for this image ID."""
_SMOLMACHINE_CACHE_DIR.mkdir(parents=True, exist_ok=True)
docker_mod.build_image(image_ref, _REPO_DIR, dockerfile=dockerfile)
# `sha256:abcd...` -> `abcd...` first 16 chars: short enough to
# keep filenames manageable, long enough to make collisions
# astronomically unlikely.
digest = docker_mod.image_id(image_ref).split(":", 1)[-1][:16]
binary = _SMOLMACHINE_CACHE_DIR / f"{digest}.smolmachine"
sidecar = _SMOLMACHINE_CACHE_DIR / f"{digest}.smolmachine.smolmachine"
if sidecar.is_file():
return sidecar
tarball = _SMOLMACHINE_CACHE_DIR / f"{digest}.image.tar"
docker_mod.save(image_ref, str(tarball))
# On Linux, `docker save -o` writes the tarball with owner-only
# permissions (mode 600). The crane push container runs as UID
# 65532 (distroless nonroot) and can't read it through a bind
# mount unless world-read is set. The tarball is temporary and
# lives in ~/.cache, so 644 is safe.
tarball.chmod(0o644)
try:
with ephemeral_registry() as handle:
push_ref = f"{handle.push_endpoint}/bot-bottle:{digest}"
pack_ref = f"{handle.pull_endpoint}/bot-bottle:{digest}"
crane_push_tarball(handle, str(tarball), push_ref)
_smolvm.pack_create(pack_ref, binary)
finally:
# Tarball is ~500MB-1GB for the agent image; reclaim once
# the smolmachine artifact exists. The artifact itself is
# the long-lived cache entry.
tarball.unlink(missing_ok=True)
return sidecar
@@ -0,0 +1,236 @@
"""Ephemeral local OCI registry for the smolmachines agent-image
conversion path (PRD 0023 chunk 4c).
`smolvm pack create --image <ref>` only accepts OCI registry refs
it can't read the local docker daemon's image cache, an OCI
layout directory, or a `docker save` tarball. To convert the
agent's Dockerfile-built image into a `.smolmachine` artifact we
spin up a short-lived `registry:2.8.3` container alongside a
`crane` helper container on a private docker network, push via
`crane push --insecure <tarball> <registry-container>:5000/...`,
and let smolvm pull from the registry's published host port. The
network + both containers are torn down after the pack completes.
Why this two-container dance instead of plain `docker push`:
- Docker Desktop's daemon runs in its own Linux VM, so its
`localhost` is not the host's loopback. A registry bound to
the host's 127.0.0.1 is unreachable from the daemon side.
- `host.docker.internal` is reachable from the daemon but isn't
in Docker's default insecure-registries CIDRs (only `::1/128`
and `127.0.0.0/8` are), so `docker push` to it tries HTTPS,
hits a plain-HTTP registry, and dies with
`http: server gave HTTP response to HTTPS client`. Adding
`host.docker.internal` to daemon.json works but is a one-time
manual step the user has to do in Docker Desktop's UI.
- Going through a docker network sidesteps the host-vs-daemon
loopback mismatch (crane and registry containers see each
other on the network) AND the HTTPS preference (crane has an
`--insecure` flag that forces plain HTTP).
The registry is also published on a random host port so smolvm
a host process can pull from `localhost:<port>` via Docker's
port-forward. smolvm's bundled crane auto-falls-back to HTTP for
localhost addresses, so no insecure-registries config is needed
on that side either."""
from __future__ import annotations
import os
import socket
import subprocess
import time
import uuid
from contextlib import contextmanager
from dataclasses import dataclass
from typing import Generator
from ...log import die
# registry:2.8.3, pinned by digest. Same env-override pattern as the
# sidecar image pin in bot_bottle/backend/docker/sidecar_bundle.py.
REGISTRY_IMAGE = os.environ.get(
"BOT_BOTTLE_REGISTRY_IMAGE",
"registry@sha256:a3d8aaa63ed8681a604f1dea0aa03f100d5895b6a58ace528858a7b332415373",
)
# gcr.io/go-containerregistry/crane:latest, pinned by digest. ~10MB,
# stable upstream from Google; we only invoke `crane push --insecure`
# against a localhost-equivalent registry, so the trust surface is
# narrow.
CRANE_IMAGE = os.environ.get(
"BOT_BOTTLE_CRANE_IMAGE",
(
"gcr.io/go-containerregistry/crane@sha256:"
"0ae17ecb34315aa7cbff28f6eddee3b7adae0b2f90101260d990804db1eb0084"
),
)
# Internal port the registry binds to inside its container — fixed
# by the registry:2 image. The host-side mapping is random.
_REGISTRY_CONTAINER_PORT = "5000"
# How long to wait for the registry's HTTP layer to bind before
# giving up. Two seconds is empirically enough; 10s leaves headroom
# for slow CI runners without making the failure mode chatty.
_READY_TIMEOUT_S = 10.0
@dataclass(frozen=True)
class RegistryHandle:
"""Everything callers need to push to + pull from the ephemeral
registry.
`network` is the per-session docker network a `crane push`
container has to join it to reach the registry by name.
`push_endpoint` is the `<host>:<port>` form to embed in image
refs given to the crane push container (resolves via docker
network DNS). `pull_endpoint` is the `<host>:<port>` form a
host process (smolvm) uses; the registry's host port mapping
backs this."""
network: str
push_endpoint: str
pull_endpoint: str
@contextmanager
def ephemeral_registry() -> Generator[RegistryHandle, None, None]:
"""Bring up a per-session docker network + a `registry:2.8.3`
container on it (published on a random host port), yield a
`RegistryHandle`, force-remove both on exit.
The container is started with `--rm` so a clean exit cleans up
on its own; the `finally` block force-removes on abnormal exit
(the calling process crashes between yield and close)."""
session_id = uuid.uuid4().hex[:12]
network = f"bot-bottle-registry-net-{session_id}"
registry_name = f"bot-bottle-registry-{session_id}"
subprocess.run(
["docker", "network", "create", network],
check=True,
capture_output=True,
)
try:
subprocess.run(
[
"docker", "run", "-d", "--rm",
"--name", registry_name,
"--network", network,
# `-p :5000` (no IP prefix) binds the container's
# port 5000 on a random host port across all
# interfaces. The host side reaches the registry
# via this port — smolvm's `pack create` pulls from
# `localhost:<port>` and the docker port-forward
# routes there.
"-p", _REGISTRY_CONTAINER_PORT,
REGISTRY_IMAGE,
],
check=True,
capture_output=True,
)
try:
port = _host_port(registry_name)
_wait_ready(port)
yield RegistryHandle(
network=network,
push_endpoint=f"{registry_name}:{_REGISTRY_CONTAINER_PORT}",
pull_endpoint=f"localhost:{port}",
)
finally:
subprocess.run(
["docker", "rm", "-f", registry_name],
check=False,
capture_output=True,
)
finally:
subprocess.run(
["docker", "network", "rm", network],
check=False,
capture_output=True,
)
def crane_push_tarball(handle: RegistryHandle, tarball_path: str, ref: str) -> None:
"""Run `crane push --insecure <tarball> <ref>` inside a one-shot
container on the registry's docker network. `ref` should
reference the registry by `handle.push_endpoint` so the crane
container resolves it via docker network DNS.
Doesn't go through `docker push` to avoid the Docker-Desktop
daemon's HTTPS preference for non-loopback hostnames — crane's
`--insecure` flag forces plain HTTP, which is what the
registry container speaks."""
r = subprocess.run(
[
"docker", "run", "--rm",
"--network", handle.network,
"-v", f"{tarball_path}:/img.tar:ro",
CRANE_IMAGE,
"push", "--insecure", "/img.tar", ref,
],
capture_output=True,
text=True,
check=False,
)
if r.returncode != 0:
die(
f"crane push of {tarball_path!r} to {ref!r} failed: "
f"{(r.stderr or r.stdout or '').strip() or '<no output>'}"
)
def _host_port(name: str) -> int:
"""Resolve the host-side port docker mapped to the registry's
container port. `docker port <name> 5000/tcp` returns one or
more `host:port` lines (one per address family) we take the
first."""
r = subprocess.run(
["docker", "port", name, f"{_REGISTRY_CONTAINER_PORT}/tcp"],
capture_output=True,
text=True,
check=False,
)
if r.returncode != 0:
die(
f"docker port {name} {_REGISTRY_CONTAINER_PORT}/tcp failed: "
f"{(r.stderr or '').strip() or '<no stderr>'}"
)
# `0.0.0.0:54321\n[::]:54321\n` — split on the last colon to
# handle either IPv4 or IPv6 host syntax.
line = (r.stdout or "").splitlines()[0].strip()
_, _, port_str = line.rpartition(":")
try:
return int(port_str)
except ValueError:
die(f"unexpected `docker port` output: {line!r}")
def _wait_ready(port: int) -> None:
"""Block until the registry's HTTP layer accepts a TCP
connection on `127.0.0.1:<port>`, or `_READY_TIMEOUT_S`
elapses.
A successful TCP connect is sufficient registry:2.8.3 binds
after it's ready to serve `/v2/` requests, so the push that
follows will land on a working server. We probe loopback
specifically (not via the docker network) because this helper
runs on the host."""
deadline = time.monotonic() + _READY_TIMEOUT_S
last_err: Exception | None = None
while time.monotonic() < deadline:
try:
with socket.create_connection(("127.0.0.1", port), timeout=0.5):
return
except OSError as e:
last_err = e
time.sleep(0.1)
die(
f"local registry on 127.0.0.1:{port} did not accept "
f"connections within {_READY_TIMEOUT_S:.0f}s "
f"(last error: {last_err})"
)
@@ -0,0 +1,314 @@
"""Per-bottle loopback alias allocation + TSI allowlist
enforcement (PRD 0023, follow-up to PR #74).
After the pivot to host-loopback port-forwards, the smolmachines
TSI allowlist was `127.0.0.1/32` which meant the agent VM could
reach **any** service bound to macOS's loopback, not just the
bundle's published ports. Real downgrade from the docker
backend's `--internal` network isolation.
This module narrows the allowlist by allocating each bottle a
unique loopback alias (`127.0.0.16` .. `127.0.0.31`). The
bundle's port-forwards bind to that alias, and the alias's /32
is what TSI allows.
**Smolvm 0.8.0 quirk + workaround.** `smolvm machine create
--from <smolmachine> --net --allow-cidr X/32` silently drops the
flag verified empirically that the agent process's allowlist
ends up `null` in smolvm's persistent state DB (`~/Library/
Application Support/smolvm/server/smolvm.db`, `vms` table,
`data` BLOB), and the booted VM reaches all of `127.0.0.0/8`
regardless of what we passed. Workaround: after machine_create,
open the SQLite DB and patch the row's `allowed_cidrs` field
directly. Smolvm reads the DB at machine_start, so the patched
value takes effect on boot. Tested: enforcement is real the
guest's connect to a non-allowlisted IP fails with `Permission
denied`. Other paths we tried (machine update, stop-edit-
agent.config.json-restart, --smolfile, --image localhost:N/...)
were dead ends.
macOS only configures `127.0.0.1` on `lo0` by default; the
additional aliases require `sudo ifconfig lo0 alias`. We lazily
sudo-add the missing pool on first use per boot the aliases
persist on `lo0` until reboot, so subsequent launches don't
prompt.
On Linux the whole `127.0.0.0/8` is already routed to `lo`, so
docker can publish a bundle's ports directly on `127.0.0.<N>`
with no `ifconfig`/sudo step. `ensure_pool` is therefore a no-op
on Linux, but per-bottle alias *allocation* and the TSI allowlist
DB patch run on both platforms the isolation property is
identical, it's just cheaper to set up on Linux. The state-DB
path differs per platform (see `_smolvm_db_path`).
Allocation is coordinated by inspecting running bundle
containers' published host IPs — each bottle's bundle owns the
alias appearing in its port bindings. The lowest-numbered free
alias gets handed to a new bottle."""
from __future__ import annotations
import fcntl
import json
import os
import platform
import re
import sqlite3
import subprocess
from pathlib import Path
from typing import Iterable
from ...log import die, info
def _smolvm_db_path() -> Path:
"""smolvm's persistent VM state — a SQLite DB whose `vms` table
holds one JSON BLOB per machine. macOS stores it under
`Application Support`; Linux follows the XDG base-dir spec
(`$XDG_DATA_HOME`, default `~/.local/share`).
NOTE: the Linux location is inferred from smolvm's documented
`~/.local/share` install layout and must be confirmed against a
real Linux smolvm install. If it's wrong, `force_allowlist`'s
fail-closed check turns it into a clear launch-time error rather
than a silent escape."""
if platform.system() == "Darwin":
return (
Path.home()
/ "Library"
/ "Application Support"
/ "smolvm"
/ "server"
/ "smolvm.db"
)
xdg_data = os.environ.get("XDG_DATA_HOME")
base = Path(xdg_data) if xdg_data else Path.home() / ".local" / "share"
return base / "smolvm" / "server" / "smolvm.db"
# Resolved once at import: the host platform doesn't change within a
# process. Tests patch this attribute directly.
_SMOLVM_DB_PATH = _smolvm_db_path()
# Sixteen aliases by default. Tunable for hosts that want more
# concurrent bottles (each bottle reserves one alias for its
# bundle bringup). The range is chosen to avoid the reserved
# 127.0.0.1/2/3 ports (1 is the default, 2 is sometimes used by
# CUPS, 3 by other macOS services) and stay well clear of
# 127.0.0.53 (systemd-resolved) and 127.0.0.54 (libvirt).
_POOL_START = 16
_POOL_END = 31 # inclusive
# File lock that serialises concurrent allocate() calls so two
# simultaneous launches can't read the same docker state and claim
# the same alias. Narrowed to the allocate() call itself; docker run
# runs after the lock is released. Once the container is running it
# appears in docker state and future allocate() calls will see it.
_ALLOC_LOCK_PATH = Path.home() / ".cache" / "bot-bottle" / "smolmachines.lock"
# Loopback aliases pool: 127.0.0.<start>..127.0.0.<end>.
def _pool_addresses() -> list[str]:
return [f"127.0.0.{i}" for i in range(_POOL_START, _POOL_END + 1)]
def _is_macos() -> bool:
return platform.system() == "Darwin"
def ensure_pool() -> None:
"""Make sure each address in the pool is up on `lo0`. Lazily
runs `sudo ifconfig lo0 alias <ip>/32 up` for missing entries
(sudo prompts once, then the aliases persist on lo0 until
reboot). No-op on non-macOS hosts."""
if not _is_macos():
return
missing = [ip for ip in _pool_addresses() if not _alias_present(ip)]
if not missing:
return
info(
f"smolmachines needs {len(missing)} loopback alias(es) on lo0 "
f"({', '.join(missing[:3])}{', ...' if len(missing) > 3 else ''}) "
f"to scope per-bottle TSI allowlists. sudo will prompt once; "
f"aliases persist until reboot."
)
for ip in missing:
result = subprocess.run(
["sudo", "-p", "bot-bottle (loopback alias): ",
"ifconfig", "lo0", "alias", f"{ip}/32", "up"],
check=False,
)
if result.returncode != 0:
die(
f"sudo ifconfig lo0 alias {ip} failed (exit "
f"{result.returncode}). Re-run with sudo available, "
f"or add manually: sudo ifconfig lo0 alias {ip}/32 up"
)
def force_allowlist(machine_name: str, allowed_cidrs: list[str]) -> None:
"""Ensure the machine's persisted TSI allowlist equals
`allowed_cidrs`, failing **closed** if that can't be confirmed.
Runs on both macOS and Linux. It exists because smolvm 0.8.0
silently drops `--allow-cidr` when combined with `--from`, so
the allowlist has to be written into smolvm's persistent state
DB before `machine start`. Rather than assume the flag was
dropped, we read the persisted row and only patch when it
doesn't already match — so a newer smolvm that honors the flag
is left untouched.
Must run AFTER `smolvm machine create` (the row has to exist)
and BEFORE `smolvm machine start` (smolvm reads the row on
start; in-flight VMs don't pick up changes).
Fail-closed: if the state DB is missing, the row is missing, or
the allowlist still doesn't match after patching, we `die()`
rather than boot a VM whose egress confinement we can't verify
an unconfirmed allowlist is a sandbox-escape risk (the agent
VM could reach all of host loopback)."""
want = list(allowed_cidrs)
if not _SMOLVM_DB_PATH.is_file():
die(
f"smolvm state DB not found at {_SMOLVM_DB_PATH}; cannot "
f"confirm the TSI allowlist is enforced. Refusing to launch "
f"(fail-closed). Check `smolvm --version` and the DB "
f"location for your platform."
)
con = sqlite3.connect(str(_SMOLVM_DB_PATH))
try:
cfg = _read_machine_cfg(con, machine_name)
if cfg.get("allowed_cidrs") != want:
cfg["allowed_cidrs"] = want
# Write as BLOB (the column type smolvm uses) — passing a
# plain str makes sqlite store it as Text and smolvm then
# fails to read it.
con.execute(
"UPDATE vms SET data = ? WHERE name = ?",
(sqlite3.Binary(json.dumps(cfg).encode()), machine_name),
)
con.commit()
cfg = _read_machine_cfg(con, machine_name)
if cfg.get("allowed_cidrs") != want:
die(
f"could not enforce TSI allowlist {want!r} for machine "
f"{machine_name!r} (persisted value is "
f"{cfg.get('allowed_cidrs')!r}). Refusing to launch "
f"(fail-closed)."
)
finally:
con.close()
def _read_machine_cfg(con: sqlite3.Connection, machine_name: str) -> dict[str, object]:
"""Read + JSON-decode a machine's `data` BLOB from the smolvm
state DB. Dies (fail-closed) if the row is missing the caller
can't confirm enforcement without it."""
row = con.execute(
"SELECT data FROM vms WHERE name = ?", (machine_name,),
).fetchone()
if row is None:
die(
f"smolvm DB has no row for machine {machine_name!r}"
f"machine_create must run before force_allowlist."
)
return json.loads(row[0])
def allocate(_slug: str) -> str:
"""Pick the lowest-numbered alias from the pool not already
in use by a running smolmachines bundle. Bails when the pool
is exhausted the caller should report the limit to the
operator. `_slug` is logged for traceability; not otherwise
used (no on-disk reservation, allocation is purely
docker-state-driven).
Runs on both platforms: the allocation logic (docker-state
inspection + the file lock) is platform-independent. macOS
needs `ensure_pool` to have aliased the addresses on `lo0`
first; on Linux all of `127.0.0.0/8` is already loopback, so
docker can publish on the chosen `127.0.0.<N>` with no setup.
Per-bottle scoping (so the agent can't reach other bottles' or
host services' loopback ports) therefore holds on both.
An exclusive file lock serialises concurrent calls so two
simultaneous launches don't read the same docker state and
claim the same alias."""
_ALLOC_LOCK_PATH.parent.mkdir(parents=True, exist_ok=True)
with open(_ALLOC_LOCK_PATH, "w", encoding="utf-8") as lf:
fcntl.flock(lf, fcntl.LOCK_EX)
return _allocate_locked()
def _allocate_locked() -> str:
in_use = _aliases_in_use()
for ip in _pool_addresses():
if ip not in in_use:
return ip
die(
f"smolmachines loopback alias pool exhausted "
f"({_POOL_END - _POOL_START + 1} aliases, all in use). "
f"Stop a running bottle (`smolvm machine ls --json`) or "
f"raise _POOL_END in loopback_alias.py."
)
def _alias_present(ip: str) -> bool:
"""True iff `ifconfig lo0` shows `<ip>` as an inet address.
Exact-match `127.0.0.1` shouldn't match `127.0.0.16`."""
result = subprocess.run(
["/sbin/ifconfig", "lo0"],
capture_output=True, text=True, check=False,
)
if result.returncode != 0:
return False
pattern = re.compile(rf"\binet {re.escape(ip)}\b")
return bool(pattern.search(result.stdout or ""))
def _aliases_in_use() -> set[str]:
"""Aliases already bound by another smolmachines bundle's
published-port mappings. We inspect every container whose
name matches the smolmachines bundle prefix and pull the
`HostIp` out of its port bindings."""
result = subprocess.run(
["docker", "ps", "--format", "{{.Names}}",
"--filter", "name=bot-bottle-sidecars-"],
capture_output=True, text=True, check=False,
)
if result.returncode != 0:
return set()
names = [n.strip() for n in (result.stdout or "").splitlines() if n.strip()]
in_use: set[str] = set()
for name in names:
in_use.update(_host_ips_for_container(name))
return in_use
def _host_ips_for_container(name: str) -> Iterable[str]:
"""Yield the `HostIp` values across all port bindings on
container `name`. A bundle binds three or four ports and
they all share the same HostIp, so callers can take any."""
result = subprocess.run(
["docker", "inspect", name,
"--format", "{{json .HostConfig.PortBindings}}"],
capture_output=True, text=True, check=False,
)
if result.returncode != 0:
return ()
try:
bindings = json.loads(result.stdout or "{}")
except json.JSONDecodeError:
return ()
seen: set[str] = set()
for _port, mappings in (bindings or {}).items():
for m in mappings or []:
host_ip = m.get("HostIp") or ""
if host_ip:
seen.add(host_ip)
return seen
__all__ = ["allocate", "ensure_pool", "force_allowlist"]
@@ -0,0 +1,245 @@
"""Per-bottle TCP forwarder for smolmachines sidecar VMs.
smolVM currently publishes guest ports on host loopback as
``HOST:GUEST`` port pairs, without an address-scoped bind. The agent
VM's TSI allowlist is IP-only, so bot-bottle exposes a second, stricter
surface: listeners bound only to the bottle's allocated host address,
forwarding to the raw smolVM-published loopback ports.
"""
from __future__ import annotations
import argparse
import json
import os
import selectors
import signal
import socket
import subprocess
import sys
import threading
from dataclasses import asdict, dataclass
from typing import Any, Iterable, Sequence
from ...log import die, warn
_BUFFER_SIZE = 64 * 1024
_LOOPBACK_TARGETS = {"127.0.0.1", "::1"}
_FORBIDDEN_LISTEN_HOSTS = {"", "0.0.0.0", "::", "127.0.0.1", "::1"}
@dataclass(frozen=True)
class ForwardSpec:
label: str
listen_host: str
listen_port: int
target_host: str
target_port: int
@dataclass(frozen=True)
class ForwarderHandle:
process: Any
forwards: tuple[ForwardSpec, ...]
def start_forwarder(specs: Sequence[ForwardSpec]) -> ForwarderHandle:
"""Start one helper process for a bottle and wait until all
listeners are bound. ``listen_port`` may be 0; the returned specs
contain the kernel-assigned ports printed by the helper."""
if not specs:
return ForwarderHandle(process=_CompletedProcess(), forwards=())
_validate_specs(specs)
argv = [
sys.executable,
"-c",
"from bot_bottle.backend.smolmachines.port_forward import main; "
"raise SystemExit(main())",
"--spec-json",
json.dumps([asdict(s) for s in specs], separators=(",", ":")),
]
proc = subprocess.Popen(
argv,
stdin=subprocess.DEVNULL,
stdout=subprocess.PIPE,
stderr=None,
text=True,
env=os.environ,
)
assert proc.stdout is not None
line = proc.stdout.readline()
proc.stdout.close()
if not line:
proc.terminate()
proc.wait(timeout=5)
die("smolmachines forwarder failed before reporting readiness")
try:
payload = json.loads(line)
bound = tuple(ForwardSpec(**item) for item in payload["forwards"])
except (KeyError, TypeError, ValueError, json.JSONDecodeError) as exc:
proc.terminate()
proc.wait(timeout=5)
die(f"smolmachines forwarder returned invalid readiness payload: {exc}")
return ForwarderHandle(process=proc, forwards=bound)
def stop_forwarder(handle: ForwarderHandle) -> None:
proc = handle.process
if proc.poll() is not None:
return
proc.terminate()
try:
proc.wait(timeout=5)
except subprocess.TimeoutExpired:
proc.kill()
proc.wait(timeout=5)
def _validate_specs(specs: Iterable[ForwardSpec]) -> None:
for spec in specs:
if spec.listen_host in _FORBIDDEN_LISTEN_HOSTS:
die(f"refusing unsafe smolmachines forwarder bind: {spec.listen_host!r}")
if spec.target_host not in _LOOPBACK_TARGETS:
die(f"refusing non-loopback smolmachines forwarder target: {spec.target_host!r}")
if not 0 <= spec.listen_port <= 65535:
die(f"invalid smolmachines forwarder listen port: {spec.listen_port}")
if not 1 <= spec.target_port <= 65535:
die(f"invalid smolmachines forwarder target port: {spec.target_port}")
class _CompletedProcess:
"""Popen-like no-op used when a bottle has no forwards."""
def poll(self) -> int:
return 0
def terminate(self) -> None:
return None
def kill(self) -> None:
return None
def wait(self, timeout: float | None = None) -> int:
del timeout
return 0
class _Forwarder:
def __init__(self, specs: Sequence[ForwardSpec]):
_validate_specs(specs)
self.specs = tuple(specs)
self.selector = selectors.DefaultSelector()
self.stop_event = threading.Event()
self.listeners: list[tuple[socket.socket, ForwardSpec]] = []
self.bound: list[ForwardSpec] = []
def start(self) -> None:
for spec in self.specs:
listener = socket.create_server(
(spec.listen_host, spec.listen_port),
reuse_port=False,
backlog=64,
)
listener.setblocking(False)
host, port = listener.getsockname()[:2]
bound = ForwardSpec(
label=spec.label,
listen_host=str(host),
listen_port=int(port),
target_host=spec.target_host,
target_port=spec.target_port,
)
self.listeners.append((listener, bound))
self.bound.append(bound)
self.selector.register(listener, selectors.EVENT_READ, bound)
def serve(self) -> None:
while not self.stop_event.is_set():
try:
events = self.selector.select(timeout=0.25)
except OSError:
return
for key, _ in events:
listener = key.fileobj
if not isinstance(listener, socket.socket):
continue
spec = key.data
try:
client, _ = listener.accept()
except OSError:
continue
threading.Thread(
target=self._handle_client,
args=(client, spec),
daemon=True,
).start()
def stop(self) -> None:
self.stop_event.set()
for listener, _ in self.listeners:
try:
self.selector.unregister(listener)
except Exception: # noqa: BLE001 - best effort shutdown
pass
listener.close()
self.selector.close()
def _handle_client(self, client: socket.socket, spec: ForwardSpec) -> None:
with client:
try:
target = socket.create_connection((spec.target_host, spec.target_port))
except OSError as exc:
warn(f"smolmachines forwarder {spec.label}: target connect failed: {exc}")
return
with target:
left = threading.Thread(target=_pipe, args=(client, target), daemon=True)
right = threading.Thread(target=_pipe, args=(target, client), daemon=True)
left.start()
right.start()
left.join()
right.join()
def _pipe(src: socket.socket, dst: socket.socket) -> None:
while True:
try:
data = src.recv(_BUFFER_SIZE)
except OSError:
break
if not data:
break
try:
dst.sendall(data)
except OSError:
break
try:
dst.shutdown(socket.SHUT_WR)
except OSError:
pass
def main(argv: Sequence[str] | None = None) -> int:
parser = argparse.ArgumentParser()
parser.add_argument("--spec-json", required=True)
ns = parser.parse_args(argv)
specs = tuple(ForwardSpec(**item) for item in json.loads(ns.spec_json))
forwarder = _Forwarder(specs)
def request_stop(signum: int, _frame: object) -> None:
del signum
forwarder.stop()
signal.signal(signal.SIGTERM, request_stop)
signal.signal(signal.SIGINT, request_stop)
forwarder.start()
print(json.dumps({"forwards": [asdict(s) for s in forwarder.bound]}), flush=True)
try:
forwarder.serve()
finally:
forwarder.stop()
return 0
if __name__ == "__main__":
raise SystemExit(main())
@@ -0,0 +1,12 @@
"""Backend-infrastructure provisioners for the smolmachines backend.
Per PRD 0050 the per-provider provisioning steps (prompt, skills,
declarative provision-plan apply, supervise MCP registration) live on
the `AgentProvider` plugin under `bot_bottle/contrib/`. CA and git
provisioning also moved to the AgentProvider ABC (with Debian/node
defaults); user plugins override them for non-standard images.
No modules remain in this subpackage. Workspace copying now runs
through `BottleBackend.provision_workspace` against the running
bottle for every backend.
"""
@@ -0,0 +1,151 @@
"""Host-side SIGWINCH → in-VM PTY resize bridge (issue #82).
smolvm 0.8.0 `machine exec -t` allocates an in-VM PTY but never
forwards the host terminal's window size (TIOCSWINSZ) to it. The
PTY's initial size is `0 0`, and any host-side resize during the
session goes unnoticed the in-VM claude TUI keeps rendering for
whatever (typically tiny) box it last saw, ignoring the operator's
tmux pane resize. `docker exec -it` does this forwarding
automatically; smolvm doesn't.
This module wraps `smolvm machine exec` with a thin parent
process that:
1. Spawns the original argv as a child (it gets the inherited
TTY, so claude's stdin/stdout/stderr work unchanged).
2. On startup + every host SIGWINCH, reads the host terminal
size via TIOCGWINSZ on stdin (or stderr if stdin isn't a
TTY tmux respawn-pane gives us a TTY on stdout/stderr)
and pushes it into the VM with a side-channel
`smolvm machine exec -- sh -c 'for f in /dev/pts/*; do
stty -F $f cols X rows Y; done'`. The kernel delivers
SIGWINCH to the foreground process group on the slave end
automatically, so claude picks up the new size without
extra signalling.
3. Waits on the child and exits with its returncode.
The dashboard's tmux pane respawn calls `bottle.agent_argv`
which now prepends `[sys.executable, -m, ..., <machine>, --, ...]`
to the smolvm argv. Foreground handoff (curses endwin
subprocess.run) goes through the same path so behavior is
identical.
Removable once smolvm grows native SIGWINCH forwarding (upstream
follow-up tracked separately)."""
from __future__ import annotations
import fcntl
import signal
import struct
import subprocess
import sys
import termios
import threading
from types import FrameType
# How long to wait after the main exec starts before pushing the
# initial size. Concurrent `smolvm machine exec` invocations race
# libkrun's per-exec OCI config write during the main exec's
# bringup window; the side-channel firing immediately corrupts
# `config.json` and the main exec dies with SIGKILL (rc=137) or
# libkrun's "parse error: trailing garbage" depending on
# scheduling. Two seconds is well past the bringup window on a
# warm VM, well under the operator's "this is unresponsive"
# threshold, and short enough that claude's initial render
# almost always fires after the size has been set.
_STARTUP_SYNC_DELAY_SEC = 2.0
def _read_winsize() -> tuple[int, int] | None:
"""Return `(rows, cols)` from whichever of stdin / stdout /
stderr is a TTY, or None if none are. Different invocation
surfaces give us different TTYs:
- foreground handoff (curses endwin subprocess.run): all
three are the operator's terminal.
- tmux respawn-pane: tmux sets all three to the pane's PTY.
- non-TTY (someone piped stdin in tests): none are; the
sync just no-ops, which is the right behavior."""
for stream in (sys.stdin, sys.stdout, sys.stderr):
try:
fd = stream.fileno()
data = fcntl.ioctl(fd, termios.TIOCGWINSZ, b"\x00" * 8)
except OSError:
continue
rows, cols, _, _ = struct.unpack("hhhh", data)
if rows > 0 and cols > 0:
return rows, cols
return None
def _push_size(machine: str, rows: int, cols: int) -> None:
"""Side-channel `smolvm machine exec` that sets the size of
every PTY in the VM. The shell `for` loop covers the case of
multiple concurrent interactive sessions (rare but cheap to
handle); `stty -F` returns silently on PTYs that don't apply.
Best-effort: swallow failures. A failed resize doesn't break
the session it just leaves the in-VM PTY at its old size.
`stdin=DEVNULL` is load-bearing: under tmux, inheriting the
pane PTY here means two concurrent smolvm processes (this one
and the agent session the wrapper is shepherding) share the
PTY's foreground-process-group / input plumbing, and smolvm
bails with an internal config-parse error or SIGKILL within
~100ms of the side-channel firing. Outside tmux the same
pattern survived, presumably because iTerm's PTY plumbing is
more forgiving than tmux's, but the DEVNULL is the right
default either way the side-channel never needs stdin."""
subprocess.run(
["smolvm", "machine", "exec", "--name", machine, "--",
"sh", "-c",
f"for f in /dev/pts/*; do "
f"stty -F \"$f\" cols {cols} rows {rows} 2>/dev/null; "
f"done"],
stdin=subprocess.DEVNULL,
stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL,
check=False,
)
def main(argv: list[str]) -> int:
"""Entry point. `argv` shape: `<machine> -- <smolvm-argv...>`.
We don't use argparse — the `--` separator is the contract and
everything past it is forwarded verbatim. Keeps the wrapper
transparent for callers building argv programmatically."""
if len(argv) < 3 or argv[1] != "--":
sys.stderr.write(
"usage: python -m bot_bottle.backend.smolmachines.pty_resize "
"<machine> -- <smolvm-argv...>\n"
)
return 2
machine = argv[0]
inner = argv[2:]
def sync(_signum: int | None = None, _frame: FrameType | None = None) -> None:
size = _read_winsize()
if size is None:
return
_push_size(machine, *size)
signal.signal(signal.SIGWINCH, sync) # type: ignore[arg-type]
proc = subprocess.Popen(inner)
# Initial sync is deferred — see _STARTUP_SYNC_DELAY_SEC.
# daemon=True so the timer doesn't block exit when the child
# finishes before the delay elapses.
timer = threading.Timer(_STARTUP_SYNC_DELAY_SEC, sync)
timer.daemon = True
timer.start()
while True:
try:
return proc.wait()
except KeyboardInterrupt:
proc.send_signal(signal.SIGINT)
if __name__ == "__main__":
sys.exit(main(sys.argv[1:]))
@@ -0,0 +1,81 @@
"""smolmachines `_resolve_plan` (PRD 0023 chunks 2d + 4c).
Resolves the per-bottle docker subnet + bundle IP and assembles
the guest env. The agent's docker image build → smolmachine
pack pipeline runs in `launch.launch`, not here, so the
dashboard's preflight modal isn't garbled by docker-build output
before the operator has confirmed.
No VM bringup that's `launch.launch`'s job."""
from __future__ import annotations
from pathlib import Path
from .. import BottleSpec
from ...manifest import Manifest
from ...env import ResolvedEnv
from ...agent_provider import AgentProvisionPlan
from ...egress import EgressPlan
from ...supervise import SupervisePlan
from ...git_gate import GitGatePlan
from .bottle_plan import SmolmachinesBottlePlan
from .util import smolmachines_bundle_subnet, smolmachines_preflight
def preflight() -> None:
smolmachines_preflight()
def build_guest_env(resolved_env: ResolvedEnv) -> dict[str, str]:
# Agent's env: resolve through resolve_env() so ?prompt entries
# are prompted and ${HOST_VAR} entries are interpolated — matching
# the Docker backend's contract. Forwarded (secret/interpolated)
# values still reach the guest as -e K=V smolvm flags because
# smolvm 0.8.0 has no env-file or stdin injection path; this is
# the known argv-exposure gap documented in PRD 0038.
# HTTPS_PROXY / GIT_GATE_URL / MCP_SUPERVISE_URL are populated
# in launch.py after bundle bringup.
return {
**resolved_env.literals,
**resolved_env.forwarded,
"NO_PROXY": "localhost,127.0.0.1",
"NODE_EXTRA_CA_CERTS": "/etc/ssl/certs/ca-certificates.crt",
"SSL_CERT_FILE": "/etc/ssl/certs/ca-certificates.crt",
"REQUESTS_CA_BUNDLE": "/etc/ssl/certs/ca-certificates.crt",
}
def resolve_plan(
spec: BottleSpec,
manifest: Manifest,
slug: str,
resolved_env: ResolvedEnv,
agent_provision_plan: AgentProvisionPlan,
egress_plan: EgressPlan,
supervise_plan: SupervisePlan | None,
git_gate_plan: GitGatePlan,
stage_dir: Path,
) -> SmolmachinesBottlePlan:
"""Materialize the smolmachines plan. The agent `.smolmachine`
artifact is built (or cache-hit) here so launch's
`machine create --from` boots without a registry pull. Per-bottle
guest env lands on the plan for launch to pass straight through
to `machine create` flags."""
# ==== smolmachines specific setup ====
subnet, gateway, bundle_ip = smolmachines_bundle_subnet(slug)
return SmolmachinesBottlePlan(
spec=spec,
manifest=manifest,
stage_dir=stage_dir,
slug=slug,
bundle_subnet=subnet,
bundle_gateway=gateway,
bundle_ip=bundle_ip,
guest_env=agent_provision_plan.guest_env,
git_gate_plan=git_gate_plan,
egress_plan=egress_plan,
supervise_plan=supervise_plan,
agent_provision=agent_provision_plan,
)
@@ -0,0 +1,164 @@
"""Per-bottle sidecar bundle bringup for the smolmachines backend.
The sidecar bundle runs as its own smolVM. The agent VM reaches
bundle daemons through host-loopback ports published by that sidecar
VM and wrapped by per-bottle address-bound forwarders."""
from __future__ import annotations
import os
import socket
from dataclasses import dataclass, field
from pathlib import Path
from typing import Sequence
from ...log import warn
from ..docker import util as docker_mod
from ..docker.sidecar_bundle import (
SIDECAR_BUNDLE_DOCKERFILE,
SIDECAR_BUNDLE_IMAGE,
)
from . import smolvm as _smolvm
_REPO_DIR = str(Path(__file__).resolve().parent.parent.parent.parent)
def bundle_network_name(slug: str) -> str:
"""`bot-bottle-bundle-<slug>` — distinct from the docker
backend's `bot-bottle-net-<slug>` so a smolmachines bottle
and a docker bottle for the same agent don't collide on
network name."""
return f"bot-bottle-bundle-{slug}"
def bundle_container_name(slug: str) -> str:
"""`bot-bottle-sidecars-<slug>` — same name shape the docker
backend uses for the bundle (PRD 0024 chunk 5). The dashboard's
prefix-based discovery covers both backends with one filter."""
return f"bot-bottle-sidecars-{slug}"
def bundle_machine_name(slug: str) -> str:
"""Sidecar smolVM machine name."""
return bundle_container_name(slug)
@dataclass(frozen=True)
class BundleLaunchSpec:
"""Everything `start_bundle` needs to bring up one bundle
container. Populated by chunk-2d's launch flow from the inner
Plans the prepare step already produces."""
slug: str
network_name: str
subnet: str
gateway: str
bundle_ip: str
image: str = SIDECAR_BUNDLE_IMAGE
# Daemon subset CSV for BOT_BOTTLE_SIDECAR_DAEMONS. The
# supervisor inside the bundle reads it to skip
# bottle-irrelevant daemons (e.g. supervise=False bottles).
daemons_csv: str = "egress"
# Plain "KEY=VALUE" strings + "KEY" bare names. Bare names inherit
# from the host env passed to the sidecar VM launch.
environment: Sequence[str] = field(default_factory=tuple)
# (host_path, container_path, read_only) bind mounts.
volumes: Sequence[tuple[str, str, bool]] = field(default_factory=tuple)
# Container ports to publish on `publish_host_ip`, random
# host-side port per entry. The smolvm guest's TSI talks via
# macOS networking, so docker container IPs (192.168.x.x in
# the daemon's bridge) aren't directly reachable from the
# guest — host-loopback port-forwards are. Egress's port
# is bundle-internal and never published.
ports_to_publish: Sequence[int] = field(default_factory=tuple)
# Loopback IP to bind published ports against. Per-bottle
# loopback aliases (`127.0.0.16` etc., added via sudo
# ifconfig lo0 alias) narrow the TSI allowlist so a bottle
# can't reach other bottles' (or other host services') ports
# via 127.0.0.1.
publish_host_ip: str = "127.0.0.1"
@dataclass(frozen=True)
class BundleVmLaunch:
raw_ports: dict[int, int]
def ensure_bundle_image(image: str = SIDECAR_BUNDLE_IMAGE) -> None:
"""Build the sidecar bundle image before `docker run`.
The Docker backend gets this for free from compose's `build:`
stanza. smolmachines starts the bundle with plain `docker run`,
so without an explicit build a first launch tries to pull the
local-only `bot-bottle-sidecars:latest` tag from a registry.
"""
docker_mod.build_image(
image,
_REPO_DIR,
dockerfile=SIDECAR_BUNDLE_DOCKERFILE,
)
def allocate_raw_host_ports(container_ports: Sequence[int]) -> dict[int, int]:
"""Reserve candidate host loopback ports for smolVM `-p HOST:GUEST`.
The sockets are closed before smolVM binds them, so this remains
best-effort. smolVM failing to bind a selected port is fatal in
the caller's launch path."""
out: dict[int, int] = {}
for container_port in container_ports:
with socket.socket(socket.AF_INET, socket.SOCK_STREAM) as sock:
sock.bind(("127.0.0.1", 0))
out[container_port] = int(sock.getsockname()[1])
return out
def start_bundle_vm(
spec: BundleLaunchSpec,
*,
from_path: Path,
host_env: dict[str, str] | None = None,
) -> BundleVmLaunch:
"""Create and start the sidecar bundle as a smolVM.
smolVM's own published ports are raw host-loopback ports. The
launch flow wraps them with per-bottle address-bound forwarders
before exposing anything to the agent VM."""
raw_ports = allocate_raw_host_ports(spec.ports_to_publish)
effective_host_env = host_env if host_env is not None else os.environ
env: dict[str, str] = {"BOT_BOTTLE_SIDECAR_DAEMONS": spec.daemons_csv}
for entry in spec.environment:
name, sep, value = entry.partition("=")
if sep:
env[name] = value
elif entry in effective_host_env:
env[entry] = effective_host_env[entry]
name = bundle_machine_name(spec.slug)
_smolvm.machine_create(
name,
from_path=from_path,
net=True,
env=env,
volumes=spec.volumes,
ports=tuple(
(host_port, guest_port)
for guest_port, host_port in raw_ports.items()
),
)
_smolvm.machine_start(name)
_smolvm.wait_exec_ready(name)
return BundleVmLaunch(raw_ports=raw_ports)
def stop_bundle_vm(slug: str) -> None:
"""Best-effort sidecar VM teardown."""
name = bundle_machine_name(slug)
try:
_smolvm.machine_stop(name)
except _smolvm.SmolvmError as exc:
warn(f"smolvm machine stop {name} failed: {exc}")
try:
_smolvm.machine_delete(name)
except _smolvm.SmolvmError as exc:
warn(f"smolvm machine delete {name} failed: {exc}")
+276
View File
@@ -0,0 +1,276 @@
"""Thin subprocess wrapper around the `smolvm` CLI (PRD 0023).
One thin Python function per smolvm subcommand the launch flow
needs. Two design choices worth flagging:
- **No daemon, no SDK.** smolvm 0.8.0 ships a `smolvm serve`
HTTP API as the long-term-clean integration target. The
project's stdlib-first ethos + the lower-overhead CLI calls
push v1 to shell out via `subprocess.run`. If a future
smolvm release makes `serve` mandatory (or significantly
faster), revisit.
- **Two return shapes.** `SmolvmRunResult` (returncode + stdout
+ stderr captured) is returned by `machine_exec` because the
caller cares about the in-VM command's exit status, and by
test helpers that introspect output. The other calls
(`machine_start`, `machine_stop`, `pack_create`, etc.) raise
`SmolvmError` on non-zero exit failure to start a VM is
fatal to the launch flow, not something callers want to
branch on.
The wrapper is unit-tested with `subprocess.run` mocked; the
integration smoke test (chunk 2d) exercises against a real
smolvm binary."""
from __future__ import annotations
import json
import shutil
import subprocess
import time
from dataclasses import dataclass
from pathlib import Path
from typing import Mapping, Sequence
_SMOLVM = "smolvm"
@dataclass(frozen=True)
class SmolvmRunResult:
"""Captured result of an in-VM command. Mirrors the structure
`Bottle.exec` returns so callers can hand it straight through."""
returncode: int
stdout: str
stderr: str
class SmolvmError(RuntimeError):
"""Raised when a smolvm subprocess returns non-zero on a path
where the caller has no useful branch to take (start failed,
pack failed, etc.). Carries the captured stderr for the
operator-facing log line."""
def __init__(self, argv: Sequence[str], result: subprocess.CompletedProcess[str]):
self.argv = list(argv)
self.returncode = result.returncode
self.stdout = result.stdout
self.stderr = result.stderr
cmd = " ".join(self.argv)
super().__init__(
f"{cmd!r} failed (exit {result.returncode}): "
f"{(result.stderr or '').strip() or '<no stderr>'}"
)
def _smolvm(*args: str, env: Mapping[str, str] | None = None,
check: bool = True) -> subprocess.CompletedProcess[str]:
"""One subprocess call into the smolvm CLI. `check=True`
raises SmolvmError on non-zero; `check=False` returns the
CompletedProcess for the caller to inspect."""
argv = [_SMOLVM, *args]
result = subprocess.run(
argv,
capture_output=True,
text=True,
env=dict(env) if env is not None else None,
check=False,
)
if check and result.returncode != 0:
raise SmolvmError(argv, result)
return result
# --- Pack ----------------------------------------------------------------
def pack_create(image: str, output: Path) -> None:
"""`smolvm pack create --image <image> -o <output>`. Converts
an OCI image into a self-contained `.smolmachine` artifact
smolvm can boot via `machine create --from`. Idempotent on the
smolvm side re-running with the same image+output rebuilds
from layer cache."""
_smolvm("pack", "create", "--image", image, "-o", str(output))
def pack_create_from_vm(name: str, output: Path) -> None:
"""`smolvm pack create --from-vm <name> -o <output>`.
Snapshots an existing persistent VM into a pack artifact. As
with `pack_create`, smolvm writes a launcher at `output` and the
bootable sidecar at `output.smolmachine`.
"""
_smolvm("pack", "create", "--from-vm", name, "-o", str(output))
# --- Machine lifecycle ---------------------------------------------------
def machine_create(
name: str,
*,
image: str | None = None,
from_path: Path | None = None,
net: bool = False,
allow_cidrs: Sequence[str] = (),
env: Mapping[str, str] | None = None,
ports: Sequence[tuple[int, int]] = (),
volumes: Sequence[tuple[str, str, bool]] = (),
) -> None:
"""`smolvm machine create --name NAME [--image IMG | --from PATH]
[--allow-cidr CIDR ...] [-e K=V ...]`. NAME is passed as
`--name` (smolvm 1.4.7+; earlier versions took it positionally).
`image` (registry ref like `alpine:latest`) and `from_path`
(a `.smolmachine` artifact) are mutually exclusive one or
the other tells smolvm what to boot. The wrapper doesn't
enforce exclusivity; smolvm errors clearly enough.
`allow_cidrs` and `env` are passed as CLI flags instead of a
Smolfile because `--from` and `--smolfile` are themselves
mutually exclusive in smolvm 0.8.0 and we want `--from`'s
no-pull-at-start property. The flag form gives the same
result without the Smolfile complication.
`--net` is sent explicitly when `net=True` or `allow_cidrs` is
non-empty. `--allow-cidr` implies `--net` per the CLI help, but
sending `--net` explicitly is harmless and ensures the guest has
network access even if that implication changes across versions."""
args: list[str] = ["machine", "create", "--name", name]
if image is not None:
args += ["--image", image]
if from_path is not None:
args += ["--from", str(from_path)]
if net or allow_cidrs:
args.append("--net")
for cidr in allow_cidrs:
args += ["--allow-cidr", cidr]
for host_port, guest_port in ports:
args += ["-p", f"{host_port}:{guest_port}"]
for host_path, guest_path, read_only in volumes:
suffix = ":ro" if read_only else ""
args += ["-v", f"{host_path}:{guest_path}{suffix}"]
if env:
for k, v in env.items():
args += ["-e", f"{k}={v}"]
_smolvm(*args)
def machine_is_running(name: str) -> bool:
"""Return True if the named VM is in the 'running' state."""
result = _smolvm("machine", "ls", "--json", check=False)
if result.returncode != 0:
return False
try:
machines = json.loads(result.stdout or "[]")
except ValueError:
return False
return any(
isinstance(m, dict) and m.get("name") == name and m.get("state") == "running"
for m in machines
)
def machine_start(name: str) -> None:
"""`smolvm machine start --name NAME`."""
_smolvm("machine", "start", "--name", name)
def machine_stop(name: str) -> None:
"""`smolvm machine stop --name NAME`. Idempotent against
already-stopped machines: smolvm prints a notice and exits 0
in that case, so no special handling here."""
_smolvm("machine", "stop", "--name", name)
def machine_delete(name: str) -> None:
"""`smolvm machine delete --name NAME -f`. `-f` skips the
interactive confirmation required for non-interactive teardown."""
_smolvm("machine", "delete", "--name", name, "-f")
def machine_exec(
name: str,
argv: Sequence[str],
*,
env: Mapping[str, str] | None = None,
workdir: str | None = None,
timeout: str | None = None,
) -> SmolvmRunResult:
"""`smolvm machine exec --name NAME [-w DIR] [--timeout DUR]
[-e K=V ...] -- ARGV...`. Returns the captured result rather
than raising callers (including `Bottle.exec`) care about
the in-VM command's exit code, not just whether smolvm ran.
`env` here is in-VM env vars (`-e K=V`), not the host
subprocess env smolvm's own argv carries them through the
VMM."""
flags: list[str] = ["machine", "exec", "--name", name]
if workdir is not None:
flags += ["-w", workdir]
if timeout is not None:
flags += ["--timeout", timeout]
if env:
for k, v in env.items():
flags += ["-e", f"{k}={v}"]
# `--` separator before the command. smolvm's CLI requires it
# so its own flag parser doesn't grab argv items that look
# like flags.
flags.append("--")
flags += list(argv)
result = _smolvm(*flags, check=False)
return SmolvmRunResult(
returncode=result.returncode,
stdout=result.stdout or "",
stderr=result.stderr or "",
)
def wait_exec_ready(name: str, *, timeout: float = 5.0) -> None:
"""Poll `machine exec true` until exit 0 or `timeout` elapses.
Replaces `time.sleep(1.5)` after `machine_start`: libkrun's exec
channel needs a brief warm-up before back-to-back exec calls are
safe. Polling exits as soon as the channel is ready and fails
loudly if the VM never responds."""
deadline = time.monotonic() + timeout
delay = 0.1
while time.monotonic() < deadline:
r = machine_exec(name, ["true"])
if r.returncode == 0:
return
remaining = deadline - time.monotonic()
if remaining <= 0:
break
time.sleep(min(delay, remaining))
delay = min(delay * 2, 0.5)
argv = ["smolvm", "machine", "exec", "--name", name, "--", "true"]
raise SmolvmError(
argv,
subprocess.CompletedProcess(
args=argv, returncode=-1, stdout="",
stderr=f"exec channel not ready after {timeout:.0f}s — VM may have failed to boot.",
),
)
def machine_cp(src: str, dst: str) -> None:
"""`smolvm machine cp SRC DST`. Path syntax: `machine:path` to
reference a path inside the VM, bare path for the host. Both
SRC and DST are positional; either side can be machine: or
bare. Empty path is a no-op (returns immediately without
invoking smolvm)."""
if not src or not dst:
return
_smolvm("machine", "cp", src, dst)
# --- Discovery -----------------------------------------------------------
def is_available() -> bool:
"""True iff `smolvm` is on PATH. Used by the integration test
suite's skip-guards."""
return shutil.which(_SMOLVM) is not None
+82
View File
@@ -0,0 +1,82 @@
"""Slug / preflight / subnet helpers for the smolmachines backend
(PRD 0023). Kept in its own module so the renderers can be
unit-tested without importing the docker subprocess paths."""
from __future__ import annotations
import hashlib
import os
import platform
import shutil
from ...log import die
# libkrun's Linux backend drives the guest through KVM, so the host
# must expose `/dev/kvm` and the invoking user must be able to open
# it. macOS uses Hypervisor.framework and needs no device node.
_KVM_DEVICE = "/dev/kvm"
def smolmachines_preflight() -> None:
"""Ensure the host can run the smolmachines backend before the
launch flow starts. Called from `_resolve_plan`; surfaces a
clear, actionable error instead of a cryptic `smolvm` failure
deep in launch.
Checks `smolvm` is on PATH (both platforms) and, on Linux,
that `/dev/kvm` exists and is accessible. `gvproxy` is no
longer required see the PRD's design pivot section."""
if shutil.which("smolvm") is None:
die(
"BOT_BOTTLE_BACKEND=smolmachines requires `smolvm` on "
"PATH. Install with: "
"curl -sSL https://smolmachines.com/install.sh | sh. "
"To use the legacy Docker backend instead, set "
"BOT_BOTTLE_BACKEND=docker or pass --backend=docker."
)
if platform.system() == "Linux":
_preflight_kvm()
def _preflight_kvm() -> None:
"""Linux-only: libkrun needs `/dev/kvm`. Distinguish 'KVM not
enabled' from 'no permission' so the operator knows which to
fix."""
if not os.path.exists(_KVM_DEVICE):
die(
f"BOT_BOTTLE_BACKEND=smolmachines needs {_KVM_DEVICE} on "
"Linux but it is missing. Enable KVM: load the kvm-intel "
"or kvm-amd kernel module (and confirm virtualization is "
"enabled in BIOS/firmware). To use the legacy Docker "
"backend instead, set BOT_BOTTLE_BACKEND=docker."
)
if not os.access(_KVM_DEVICE, os.R_OK | os.W_OK):
die(
f"{_KVM_DEVICE} exists but is not readable/writable by the "
"current user. Add your user to the `kvm` group "
"(`sudo usermod -aG kvm \"$USER\"`) and re-login, or run "
"with access to the device."
)
def smolmachines_bundle_subnet(slug: str) -> tuple[str, str, str]:
"""Derive a per-bottle docker subnet + gateway IP + bundle IP
from the slug.
Returns `(subnet_cidr, gateway_ip, bundle_ip)`. The third
octet comes from SHA-256 of the slug mod 254 (skipping 17 to
avoid the docker-default bridge), so parallel bottles get
distinct /24s and `resume` reuses the same /24. The bundle
container always lands at `.2`; gateway is `.1`; the smolvm
Smolfile's `allow_cidrs` is `<bundle_ip>/32`."""
digest = hashlib.sha256(slug.encode("utf-8")).digest()
octet = (digest[0] % 254) + 1
# Skip the docker-default bridge to dodge the most common
# collision (operators with `docker0` at 172.17.x.x or a
# 192.168.17.x VPN client).
if octet == 17:
octet = 18
subnet = f"192.168.{octet}.0/24"
gateway = f"192.168.{octet}.1"
bundle_ip = f"192.168.{octet}.2"
return subnet, gateway, bundle_ip
+14 -14
View File
@@ -36,7 +36,7 @@ from dataclasses import dataclass
from pathlib import Path
from typing import cast
from .paths import bot_bottle_root
from . import supervise as _supervise
# Directory layout: ~/.bot-bottle/state/<identity>/...
@@ -44,16 +44,16 @@ _STATE_SUBDIR = "state"
_PER_BOTTLE_DOCKERFILE_NAME = "Dockerfile"
_COMMITTED_IMAGE_NAME = "committed-image"
_TRANSCRIPT_SUBDIR = "transcript"
# Per-daemon scratch subdirs. PRD 0018 chunk 2: bind-mount sources
# Per-sidecar scratch subdirs. PRD 0018 chunk 2: bind-mount sources
# live here so chunk 3's `docker compose up` can find them at stable
# paths. Each daemon's `prepare()` writes config + CAs into its own
# paths. Each sidecar's `prepare()` writes config + CAs into its own
# subdir; the launch step is unchanged today (still `docker cp`).
_EGRESS_SUBDIR = "egress"
_GIT_GATE_SUBDIR = "git-gate"
_SUPERVISE_SUBDIR = "supervise"
_AGENT_SUBDIR = "agent"
_METADATA_NAME = "metadata.json"
# Live-config dir bind-mounted into the supervise daemon (read-only).
# Live-config dir bind-mounted into the supervise sidecar (read-only).
# Host's apply paths keep these files fresh so supervise's
# `list-egress-routes` MCP tool returns the current state —
# not a snapshot from launch time.
@@ -105,9 +105,9 @@ class BottleMetadata:
# written before chunk 3 (resume / inspect should fall back to
# deriving from identity in that case).
compose_project: str = ""
# PRD 0040: backend name ("docker", "firecracker", "macos-container").
# Empty string for state dirs written before PRD 0040; callers default
# to "docker" for backward compatibility.
# PRD 0040: backend name ("docker" or "smolmachines"). Empty string
# for state dirs written before PRD 0040; callers default to "docker"
# for backward compatibility.
backend: str = ""
label: str = ""
color: str = ""
@@ -163,7 +163,7 @@ def read_metadata(identity: str) -> BottleMetadata | None:
def bottle_state_dir(identity: str) -> Path:
"""Per-bottle state directory on the host. Created lazily by the
write helpers; readers tolerate its absence."""
return bot_bottle_root() / _STATE_SUBDIR / identity
return _supervise.bot_bottle_root() / _STATE_SUBDIR / identity
def per_bottle_dockerfile_path(identity: str) -> Path:
@@ -222,7 +222,7 @@ def per_bottle_image_tag(identity: str) -> str:
def live_config_dir(identity: str) -> Path:
"""Per-bottle live-config dir. Bind-mounted read-only into the
supervise daemon; the host's apply paths refresh the files on
supervise sidecar; the host's apply paths refresh the files on
every operator approval so the agent's `list-*` MCP tools always
return current state."""
return bottle_state_dir(identity) / _LIVE_CONFIG_SUBDIR
@@ -260,9 +260,9 @@ def transcript_snapshot_dir(identity: str) -> Path:
return bottle_state_dir(identity) / _TRANSCRIPT_SUBDIR
# --- Per-daemon scratch subdirs (PRD 0018 chunk 2) ------------------------
# --- Per-sidecar scratch subdirs (PRD 0018 chunk 2) ------------------------
#
# Each daemon gets its own subdir under the bottle's state dir for
# Each sidecar gets its own subdir under the bottle's state dir for
# bind-mount sources (config, CAs, hooks, etc.). Prepare-time writes
# land here; the state dir's normal cleanup (`cleanup_state`) reaps
# them along with everything else when the bottle session ends and
@@ -270,20 +270,20 @@ def transcript_snapshot_dir(identity: str) -> Path:
def egress_state_dir(identity: str) -> Path:
"""State subdir for the egress daemon: routes.yaml + the
"""State subdir for the egress sidecar: routes.yaml + the
per-bottle mitmproxy CA. Bind-mount source from chunk 3 onward."""
return bottle_state_dir(identity) / _EGRESS_SUBDIR
def git_gate_state_dir(identity: str) -> Path:
"""State subdir for the git-gate daemon: entrypoint + hooks +
"""State subdir for the git-gate sidecar: entrypoint + hooks +
per-upstream known_hosts. Bind-mount source from chunk 3
onward."""
return bottle_state_dir(identity) / _GIT_GATE_SUBDIR
def supervise_state_dir(identity: str) -> Path:
"""State subdir reserved for supervise daemon bind-mount sources.
"""State subdir reserved for supervise sidecar bind-mount sources.
Runtime queue/audit rows live in the host-level bot-bottle SQLite
database, so they survive state-dir cleanup."""
return bottle_state_dir(identity) / _SUPERVISE_SUBDIR
+1 -4
View File
@@ -1,6 +1,6 @@
"""Main CLI dispatcher.
Commands: backend, cleanup, commit, edit, info, init, list, resume, start, supervise
Commands: cleanup, commit, edit, info, init, list, resume, start, supervise
"""
from __future__ import annotations
@@ -13,7 +13,6 @@ from ..manifest import ManifestError
from ..store_manager import StoreManager
from ._common import PROG
from . import list as _list_mod
from .backend import cmd_backend
from .cleanup import cmd_cleanup
from .commit import cmd_commit
from .edit import cmd_edit
@@ -26,7 +25,6 @@ from .supervise import cmd_supervise
cmd_list = _list_mod.cmd_list
COMMANDS = {
"backend": cmd_backend,
"cleanup": cmd_cleanup,
"commit": cmd_commit,
"edit": cmd_edit,
@@ -42,7 +40,6 @@ COMMANDS = {
def usage() -> None:
sys.stderr.write(f"usage: {PROG} <command> [args...]\n\n")
sys.stderr.write("Commands:\n")
sys.stderr.write(" backend set up / check / undo a backend's host prerequisites (setup|status|teardown)\n")
sys.stderr.write(" cleanup stop and remove all active bot-bottle containers\n")
sys.stderr.write(" commit snapshot a running bottle's container state to a Docker image\n")
sys.stderr.write(" edit open an agent in vim for editing\n")
-46
View File
@@ -1,46 +0,0 @@
"""`backend` CLI command — generic host setup/status across backends.
`./cli.py backend setup [--backend=NAME]` provisions (or points at how
to provision) the chosen backend's one-time host prerequisites.
`./cli.py backend status [--backend=NAME]` reports readiness.
`./cli.py backend teardown [--backend=NAME]` undoes setup (uninstall).
All dispatch to the backend's `setup()` / `status()` / `teardown()`
classmethods, so there are no backend-specific commands swapping
backends is just a different `--backend` (or `$BOT_BOTTLE_BACKEND`, or
the host default).
"""
from __future__ import annotations
import argparse
from ..backend import get_bottle_backend, known_backend_names
from ._common import PROG
def cmd_backend(args: list[str]) -> int:
parser = argparse.ArgumentParser(
prog=f"{PROG} backend",
description="Set up or check a backend's host prerequisites.",
)
parser.add_argument(
"action",
choices=("setup", "status", "teardown"),
help="setup: provision/print host prerequisites; status: report "
"readiness; teardown: undo setup (uninstall)",
)
parser.add_argument(
"--backend",
choices=known_backend_names(),
default=None,
help="backend to target (default: $BOT_BOTTLE_BACKEND or the host default)",
)
ns = parser.parse_args(args)
backend = get_bottle_backend(ns.backend)
if ns.action == "setup":
return backend.setup()
if ns.action == "teardown":
return backend.teardown()
return backend.status()
+5 -5
View File
@@ -1,14 +1,14 @@
"""cleanup: stop and remove all orphaned bot-bottle resources.
Walks every registered backend (docker, firecracker, macos-container)
so a single `./cli.py cleanup` reaps every backend's leftovers — a
firecracker bottle's VM processes and run dirs won't survive a
docker-only cleanup pass (issue addressed alongside #77).
Walks every registered backend (docker + smolmachines) so a single
`./cli.py cleanup` reaps both backends' leftovers — orphaned
smolvm machines won't survive a docker-only cleanup pass (issue
addressed alongside #77).
Each backend's `prepare_cleanup` enumerates its own resources;
docker's `_list_orphan_state_dirs` consults
`enumerate_active_agents()` for the union of live identities so
state dirs of running non-docker bottles aren't reaped. State
state dirs of running smolmachines bottles aren't reaped. State
dirs are shared layout, so docker is the single owner of that
bucket.
+4 -4
View File
@@ -2,10 +2,10 @@
Docker bottles are committed to a local Docker image. Macos-container
bottles are exported and rebuilt as a local Apple Container image.
Firecracker bottles stream the guest rootfs out over SSH and rebuild a
local Docker image. The resulting reference is stored in per-bottle
state so the next `./cli.py resume <slug>` boots from the snapshot
instead of rebuilding from the Dockerfile.
Smolmachines bottles are packed from the running VM into a
`.smolmachine` artifact. The resulting reference is stored in
per-bottle state so the next `./cli.py resume <slug>` boots from the
snapshot instead of rebuilding from the Dockerfile.
"""
from __future__ import annotations
+2 -3
View File
@@ -45,9 +45,8 @@ def cmd_list(argv: list[str]) -> int:
print(name)
return 0
# `active` enumerates every backend (docker, firecracker,
# macos-container) so non-docker bottles aren't hidden behind
# the env var.
# `active` enumerates every backend (docker + smolmachines)
# so smolmachines bottles aren't hidden behind the env var.
active = enumerate_active_agents()
if not active:
print("no active bot-bottle bottles", file=sys.stderr)
-16
View File
@@ -46,17 +46,6 @@ def cmd_start(argv: list[str]) -> int:
parser = argparse.ArgumentParser(prog=f"{PROG} start", add_help=True)
parser.add_argument("--dry-run", action="store_true")
parser.add_argument("--cwd", action="store_true", help="copy host cwd into the running bottle")
parser.add_argument(
"--no-cache",
action="store_true",
help=(
"rebuild agent/sidecar images from scratch, bypassing the "
"build layer cache. Use when an image looks broken after a "
"dependency bump — e.g. an installer's optionalDependencies "
"fetch silently no-op'd on a transient failure and got baked "
"into a cached layer."
),
)
parser.add_argument(
"--backend",
choices=known_backend_names(),
@@ -108,11 +97,6 @@ def cmd_start(argv: list[str]) -> int:
args = parser.parse_args(argv)
dry_run = args.dry_run or os.environ.get("BOT_BOTTLE_DRY_RUN") == "1"
if args.no_cache or os.environ.get("BOT_BOTTLE_NO_CACHE") == "1":
# Read by build_image() in each backend's util module — set here
# so both the interactive and --headless paths pick it up without
# threading a no_cache field through every backend's plan dataclass.
os.environ["BOT_BOTTLE_NO_CACHE"] = "1"
manifest = ManifestIndex.resolve(USER_CWD)
backend_name: str | None = args.backend
+7 -2
View File
@@ -19,7 +19,7 @@ from dataclasses import dataclass
from datetime import datetime, timezone
from pathlib import Path
from ..paths import bot_bottle_root
from .. import supervise as _supervise
from ..bottle_state import read_metadata
from ..backend.docker.egress_apply import (
EgressApplyError,
@@ -28,6 +28,9 @@ from ..backend.docker.egress_apply import (
from ..backend.macos_container.egress_apply import (
applicator as _macos_applicator,
)
from ..backend.smolmachines.egress_apply import (
applicator as _smolmachines_applicator,
)
from ..log import Die, error, info
from ..supervise import (
@@ -76,6 +79,8 @@ def apply_routes_change(slug: str, content: str) -> tuple[str, str]:
backend = meta.backend if meta is not None else ""
if backend == "macos-container":
return _macos_applicator.apply_routes_change(slug, content)
if backend == "smolmachines":
return _smolmachines_applicator.apply_routes_change(slug, content)
return _docker_applicator.apply_routes_change(slug, content)
@@ -276,7 +281,7 @@ def _write_crash_log(exc: BaseException) -> Path:
)
entry = f"=== supervise crash {stamp} ===\n{body}\n"
try:
log_dir = bot_bottle_root() / "logs"
log_dir = _supervise.bot_bottle_root() / "logs"
log_dir.mkdir(parents=True, exist_ok=True)
path = log_dir / "supervise-crash.log"
with path.open("a", encoding="utf-8") as fh:
+2 -3
View File
@@ -4,7 +4,7 @@ The Claude-specific behavior previously inlined under
`agent_provider.agent_provision_plan` (claude.json trust marker,
api.anthropic.com egress route, OAuth-token placeholder), plus
the `claude mcp add` invocation that registers the supervise
gateway in claude-code's user config (PRD 0013)."""
sidecar in claude-code's user config (PRD 0013)."""
from __future__ import annotations
@@ -91,7 +91,6 @@ _RUNTIME = AgentProviderRuntime(
prompt_mode="append_file",
bypass_args=("--dangerously-skip-permissions",),
resume_args=("--continue",),
smoke_test=("claude", "--version"),
)
@@ -294,7 +293,7 @@ class ClaudeAgentProvider(AgentProvider):
supervise_url: str,
) -> None:
"""Run `claude mcp add` inside the agent guest to register the
supervise daemon in claude-code's user config (~/.claude.json).
supervise sidecar in claude-code's user config (~/.claude.json).
Failure is logged but not fatal the bottle still works without
the entry; the operator can register it manually."""
+2 -3
View File
@@ -4,7 +4,7 @@ The Codex-specific behavior previously inlined under
`agent_provider.agent_provision_plan` (config.toml trust marker,
chatgpt.com / api.openai.com egress routes, optional host-credential
forwarding with dummy-auth.json + verify), plus the `codex mcp add`
invocation that registers the supervise daemon in Codex's
invocation that registers the supervise sidecar in Codex's
~/.codex/config.toml (PRD 0050)."""
from __future__ import annotations
@@ -61,7 +61,6 @@ _RUNTIME = AgentProviderRuntime(
prompt_mode="read_prompt_file",
bypass_args=("--dangerously-bypass-approvals-and-sandbox",),
resume_args=("resume", "--last"),
smoke_test=(_CODEX_CLI, "--version"),
)
@@ -267,7 +266,7 @@ class CodexAgentProvider(AgentProvider):
supervise_url: str,
) -> None:
"""Run `codex mcp add` inside the agent guest to register the
supervise daemon in Codex's user config (~/.codex/config.toml).
supervise sidecar in Codex's user config (~/.codex/config.toml).
Mirrors the Claude provider's `claude mcp add` flow — failure
is logged but not fatal."""
+1 -1
View File
@@ -3,7 +3,7 @@
Pure Python, no mitmproxy dependency. Each detector is a module-level
function returning `ScanResult | None`.
Ships flat into the gateway image alongside
Ships flat into the sidecar bundle image alongside
`egress_addon_core.py` both this file and the package source use
the same try/except import shim pattern.
"""
-27
View File
@@ -1,27 +0,0 @@
"""Lean, framework-free `docker` subprocess primitive.
Deliberately a top-level module with a single stdlib import so it can be
reused from anywhere without cost. It is intentionally *not* placed in
`backend.docker.util`: importing that module runs `backend/__init__.py`,
which eagerly loads all three bottle backends (docker + firecracker +
macos) plus the manifest/egress/git-gate/supervise framework ~76 modules
which would drag the whole backend layer into the deliberately-lean
orchestrator. This primitive stays free of that so both the orchestrator's
docker components and (in time) `backend.docker.util` can share it."""
from __future__ import annotations
import subprocess
def run_docker(argv: list[str]) -> subprocess.CompletedProcess[str]:
"""Run a `docker` command, capturing stdout/stderr as text. Never raises
on a non-zero exit callers inspect `returncode` / `stderr` so they can
stay fail-closed or tolerate idempotent no-ops (e.g. removing an
already-absent container)."""
return subprocess.run(
argv, stdout=subprocess.PIPE, stderr=subprocess.PIPE, text=True, check=False,
)
__all__ = ["run_docker"]
+6 -6
View File
@@ -2,7 +2,7 @@
This module defines the abstract proxy (`Egress`), its plan
dataclass (`EgressPlan`), and the resolved per-route shape
(`EgressRoute`). The gateway's start/stop lifecycle is backend-
(`EgressRoute`). The sidecar's start/stop lifecycle is backend-
specific and lives on concrete subclasses (see
`bot_bottle/backend/docker/egress.py`).
"""
@@ -63,8 +63,8 @@ def _random_canary_env() -> str:
return f"{first}_{second}_SECRET"
def egress_gateway_env_entries(plan: "EgressPlan") -> tuple[str, ...]:
"""Return gateway env entries needed by egress across all backends."""
def egress_sidecar_env_entries(plan: "EgressPlan") -> tuple[str, ...]:
"""Return sidecar env entries needed by egress across all backends."""
env: list[str] = []
if plan.routes:
env.extend(sorted(plan.token_env_map.keys()))
@@ -87,7 +87,7 @@ class EgressRoute(Route):
Inherits `host`, `matches`, `auth_scheme`, and `token_env`
from `egress_addon_core.Route` those are the fields that cross the
YAML wire into the gateway. The fields below are host-only and
YAML wire into the sidecar. The fields below are host-only and
are never serialised to the addon.
`token_ref` is the host env var the CLI reads at launch and forwards
@@ -386,7 +386,7 @@ class Egress(ABC):
routes_path.write_text(egress_render_routes(routes, log=log))
routes_path.chmod(0o600)
# Generate a per-session fake secret under a plausible random env name.
# The gateway marks that exact env name as sensitive for known-secret
# The sidecar marks that exact env name as sensitive for known-secret
# scanning; the agent receives the same name/value as exfil bait.
canary = secrets.token_urlsafe(32)
return EgressPlan(
@@ -412,6 +412,6 @@ __all__ = [
"egress_resolve_token_values",
"egress_routes_for_bottle",
"egress_agent_env_entries",
"egress_gateway_env_entries",
"egress_sidecar_env_entries",
"egress_token_env_map",
]
+41 -117
View File
@@ -1,4 +1,4 @@
"""mitmproxy addon entrypoint for the egress gateway (PRD 0017, PRD 0053).
"""mitmproxy addon entrypoint for the egress sidecar (PRD 0017, PRD 0053).
Loaded by `mitmdump -s /app/egress_addon.py` inside the
egress container."""
@@ -10,7 +10,6 @@ import json
import os
import signal
import sys
import typing
from pathlib import Path
from mitmproxy import http # type: ignore[import-not-found] # pylint: disable=import-error
@@ -33,7 +32,6 @@ from egress_addon_core import ( # type: ignore[import-not-found] # pylint: dis
is_git_push_request,
load_config,
match_route,
resolve_client_context,
outbound_scan_headers,
route_to_yaml_dict,
scan_inbound,
@@ -53,26 +51,11 @@ try:
except ImportError: # pragma: no cover - host-side path
from bot_bottle import supervise as _sv # type: ignore[import-not-found]
try:
from policy_resolver import PolicyResolver # type: ignore[import-not-found]
except ImportError: # pragma: no cover - host-side path
from bot_bottle.policy_resolver import PolicyResolver
DEFAULT_ROUTES_PATH = "/etc/egress/routes.yaml"
INTROSPECT_HOST = "_egress.local"
# Consolidated (multi-tenant) mode: when this points at the per-host
# orchestrator's control plane, the addon resolves each client's Config by
# source IP per request instead of using a single static routes file. Unset
# → legacy per-bottle single-tenant mode (unchanged).
ORCHESTRATOR_URL_ENV = "BOT_BOTTLE_ORCHESTRATOR_URL"
# App-layer identity token (defense-in-depth over the source-IP invariant);
# the agent injects it, the addon strips it so it never leaks upstream.
IDENTITY_HEADER = "x-bot-bottle-identity"
# Seconds the egress proxy holds a token-blocked request open waiting for the
# operator's supervisor decision (PRD 0062), overridable via env.
DEFAULT_TOKEN_ALLOW_TIMEOUT_SECONDS = 300.0
@@ -89,40 +72,20 @@ _TOKEN_ALLOW_JUSTIFICATION = (
class EgressAddon:
# Class default so addons built via __new__ (e.g. in tests) default to
# single-tenant; __init__ sets the instance attribute for real runs.
_resolver: "PolicyResolver | None" = None
def __init__(self) -> None:
self.routes_path = os.environ.get("EGRESS_ROUTES", DEFAULT_ROUTES_PATH)
self.config: Config = Config(routes=())
# Consolidated mode: resolve per-client Config from the orchestrator.
# Absent → single-tenant (static routes file); behaviour unchanged.
orch_url = os.environ.get(ORCHESTRATOR_URL_ENV, "").strip()
self._resolver = PolicyResolver(orch_url) if orch_url else None
# Tokens the operator has approved this session (PRD 0062), keyed by
# bottle so the shared gateway keeps each bottle's safelist separate —
# a global set would let bottle A's approved secret pass bottle B's DLP
# scan. In-memory only (a restart re-prompts); mutated only from the
# asyncio loop that runs the addon hooks, so no lock is needed.
self._safe_tokens: dict[str, set[str]] = {}
# Tokens the operator has approved this session (PRD 0062). In-memory
# only — a restart re-prompts. Mutated only from the asyncio loop that
# runs the addon hooks, so no lock is needed.
self.safe_tokens: set[str] = set()
self._supervise_slug = os.environ.get("SUPERVISE_BOTTLE_SLUG", "").strip()
self._token_allow_timeout = _token_allow_timeout_from_env(os.environ)
self._reload(initial=True)
self._install_sighup()
@staticmethod
def _supervise_available(slug: str) -> bool:
"""Supervise is reachable for this request iff we resolved a bottle to
attribute its proposals to (single-tenant env slug, or a source-IP
-attributed bottle id). Empty fail closed (no queue to write to)."""
return bool(slug)
def _safe_tokens_for(self, slug: str) -> set[str]:
"""This bottle's operator-approved DLP safelist (PRD 0062), created on
first use. Keyed by bottle so the shared gateway never leaks one
bottle's approved token into another's scan."""
return self._safe_tokens.setdefault(slug, set())
def _supervise_available(self) -> bool:
return bool(self._supervise_slug)
def _reload(self, *, initial: bool = False) -> None:
try:
@@ -231,28 +194,6 @@ class EgressAddon:
+ "\n"
)
def _resolve_flow(
self, flow: http.HTTPFlow,
) -> "tuple[Config, str, typing.Mapping[str, str]]":
"""The `(Config, supervise slug, env)` to apply to this request.
Single-tenant the static `self.config`, the env slug, and the process
env. Consolidated the calling bottle's Config + bottle id + auth
tokens, resolved by source IP in one round-trip (fail-closed to deny-all
+ empty slug if unattributed); `env` is the process env overlaid with
the bottle's tokens, so upstream-auth injection (and DLP) use *this*
bottle's credentials — exactly what the per-bottle gateway daemon's env did.
The identity token, if the agent injected one, is read then stripped so
it never leaks upstream."""
if self._resolver is None:
return self.config, self._supervise_slug, os.environ
conn = flow.client_conn
client_ip = conn.peername[0] if conn and conn.peername else ""
token = flow.request.headers.get(IDENTITY_HEADER, "")
flow.request.headers.pop(IDENTITY_HEADER, None)
config, slug, tokens = resolve_client_context(self._resolver, client_ip, token)
env = {**os.environ, **tokens} if tokens else os.environ
return config, slug, env
async def request(self, flow: http.HTTPFlow) -> None:
request_path, _, query = flow.request.path.partition("?")
@@ -260,14 +201,12 @@ class EgressAddon:
self._serve_introspection(flow, request_path)
return
config, slug, env = self._resolve_flow(flow)
# DLP outbound scan BEFORE stripping auth — catches tokens the
# agent tried to smuggle in any header, path, query param, or body.
# Hostname is included to catch DNS-tunnelling exfiltration attempts.
route = match_route(config.routes, flow.request.pretty_host)
route = match_route(self.config.routes, flow.request.pretty_host)
if route is not None:
if not await self._handle_outbound_dlp(flow, route, slug, env):
if not await self._handle_outbound_dlp(flow, route):
return
# The redact policy may have rewritten the request line; recompute
# the path/query the git checks below rely on.
@@ -285,7 +224,7 @@ class EgressAddon:
if is_git_fetch_request(request_path, query):
git_decision = decide_git_fetch(
config.routes, flow.request.pretty_host,
self.config.routes, flow.request.pretty_host,
)
if git_decision.action == "block":
self._block(
@@ -296,17 +235,17 @@ class EgressAddon:
return
# Strip agent-set Authorization after DLP scan so smuggled tokens
# are caught above; the route may inject gateway-owned auth below.
# are caught above; the route may inject sidecar-owned auth below.
flow.request.headers.pop("authorization", None)
# Build headers mapping for match evaluation
req_headers = {k.lower(): v for k, v in flow.request.headers.items()}
decision = decide(
config.routes,
self.config.routes,
flow.request.pretty_host,
request_path,
env,
os.environ,
request_method=flow.request.method,
request_headers=req_headers,
)
@@ -318,7 +257,7 @@ class EgressAddon:
if decision.inject_authorization is not None:
flow.request.headers["authorization"] = decision.inject_authorization
if config.log >= LOG_FULL:
if self.config.log >= LOG_FULL:
self._log_request(flow)
def _block_dlp(self, flow: http.HTTPFlow, result: ScanResult) -> None:
@@ -331,13 +270,10 @@ class EgressAddon:
self,
flow: http.HTTPFlow,
route: Route,
slug: str,
env: "typing.Mapping[str, str]",
) -> bool:
"""Scan the outbound request and apply the route's on-match policy
(PRD 0062). `env` is the per-bottle env overlay (process env + this
bottle's tokens) used for DLP detection. Returns True if the request may
be forwarded, False if a 403 response has been written to `flow`.
(PRD 0062). Returns True if the request may be forwarded, False if a
403 response has been written to `flow`.
Loops so the supervise policy can re-scan after each approval a
second, un-approved token in the same request is still caught."""
@@ -354,8 +290,8 @@ class EgressAddon:
flow.request.pretty_host, request_path, query, headers, "",
)
result = scan_outbound(
route, scan_text, env,
safe_tokens=self._safe_tokens_for(slug), crlf_text=crlf_text,
route, scan_text, os.environ,
safe_tokens=self.safe_tokens, crlf_text=crlf_text,
)
if result is None or result.severity != "block":
return True
@@ -365,7 +301,7 @@ class EgressAddon:
# redact scrubs every detection (tokens and structural CRLF) and
# forwards; it fails closed only if a match survives the scrub.
if policy == ON_MATCH_REDACT:
if self._redact_outbound(flow, route, env):
if self._redact_outbound(flow, route):
if self.config.log >= LOG_BLOCKS:
sys.stderr.write(json.dumps({
"event": "egress_redacted",
@@ -390,34 +326,32 @@ class EgressAddon:
# supervise (default): hold the request for operator approval.
# Fall back to a hard 403 when supervise isn't wired for the bottle.
if not self._supervise_available(slug):
if not self._supervise_available():
self._block_dlp(flow, result)
return False
approved = await self._supervise_token_block(flow, request_path, result, slug, env)
approved = await self._supervise_token_block(flow, request_path, result)
if not approved:
return False # _supervise_token_block wrote the 403 response
# loop: the approved value is now in safe_tokens; re-scan.
def _redact_outbound(
self, flow: http.HTTPFlow, route: Route, env: "typing.Mapping[str, str]",
) -> bool:
def _redact_outbound(self, flow: http.HTTPFlow, route: Route) -> bool:
"""Scrub detected tokens (and CRLF injection sequences) from the mutable
request surfaces (body, headers, path/query) and re-scan. `env` is the
per-bottle env overlay. Returns True if the request is now clean; False
if a block-severity match remains on a surface redaction cannot rewrite
(the hostname) so the caller fails closed."""
request surfaces (body, headers, path/query) and re-scan. Returns True
if the request is now clean; False if a block-severity match remains on
a surface redaction cannot rewrite (the hostname) so the caller fails
closed."""
body = flow.request.get_text(strict=False)
if body:
redacted_body = redact_tokens(body, env=env)
redacted_body = redact_tokens(body, env=os.environ)
if redacted_body != body:
flow.request.text = redacted_body
for name, value in list(flow.request.headers.items()):
if name.lower() == "host":
continue # routing-critical; never a legitimate token
redacted = strip_crlf(redact_tokens(value, env=env))
redacted = strip_crlf(redact_tokens(value, env=os.environ))
if redacted != value:
flow.request.headers[name] = redacted
redacted_path = strip_crlf(redact_tokens(flow.request.path, env=env))
redacted_path = strip_crlf(redact_tokens(flow.request.path, env=os.environ))
if redacted_path != flow.request.path:
flow.request.path = redacted_path
@@ -430,7 +364,7 @@ class EgressAddon:
crlf_text = build_outbound_scan_text(
flow.request.pretty_host, request_path, query, headers, "",
)
result = scan_outbound(route, scan_text, env, crlf_text=crlf_text)
result = scan_outbound(route, scan_text, os.environ, crlf_text=crlf_text)
return result is None or result.severity != "block"
async def _supervise_token_block(
@@ -438,27 +372,21 @@ class EgressAddon:
flow: http.HTTPFlow,
request_path: str,
result: ScanResult,
slug: str,
env: "typing.Mapping[str, str]",
) -> bool:
"""Route a token DLP block to the operator's supervisor queue and wait.
`slug` attributes the proposal to the calling bottle (its own queue +
safelist) in the shared gateway this is what keeps one bottle's
approval from unblocking another's request. `env` is the per-bottle env
overlay used to redact secrets from the proposal text. Returns True if
the operator approved (the matched value is added to that bottle's
safelist and the caller re-scans); False if the request must be blocked
(a 403 response has been written to `flow`)."""
Returns True if the operator approved (the matched value is added to
`self.safe_tokens` and the caller re-scans); False if the request must
be blocked (a 403 response has been written to `flow`)."""
host = flow.request.pretty_host
payload = build_token_allow_payload(
redact_tokens(host, env=env),
redact_tokens(host, env=os.environ),
flow.request.method,
redact_tokens(request_path, env=env),
redact_tokens(request_path, env=os.environ),
result,
)
proposal = _sv.Proposal.new(
bottle_slug=slug,
bottle_slug=self._supervise_slug,
tool=_sv.TOOL_EGRESS_TOKEN_ALLOW,
proposed_file=payload,
justification=_TOKEN_ALLOW_JUSTIFICATION,
@@ -481,13 +409,13 @@ class EgressAddon:
**self._req_ctx(flow),
}) + "\n")
response = await self._await_token_response(proposal.id, slug)
_sv.archive_proposal(slug, proposal.id)
response = await self._await_token_response(proposal.id)
_sv.archive_proposal(self._supervise_slug, proposal.id)
if response is not None and response.status in (
_sv.STATUS_APPROVED, _sv.STATUS_MODIFIED,
):
self._safe_tokens_for(slug).add(result.matched)
self.safe_tokens.add(result.matched)
if self.config.log >= LOG_BLOCKS:
sys.stderr.write(json.dumps({
"event": "egress_token_allowed",
@@ -510,7 +438,6 @@ class EgressAddon:
async def _await_token_response(
self,
proposal_id: str,
slug: str,
) -> "_sv.Response | None":
"""Poll the DB for the operator's response without blocking the
proxy event loop. Returns the Response, or None on timeout."""
@@ -518,7 +445,7 @@ class EgressAddon:
deadline = loop.time() + self._token_allow_timeout
while True:
try:
return _sv.read_response(slug, proposal_id)
return _sv.read_response(self._supervise_slug, proposal_id)
except (OSError, ValueError, KeyError):
# Not written yet, or a partial/malformed write — retry until
# the deadline, then fail closed.
@@ -572,9 +499,6 @@ class EgressAddon:
"""
if flow.websocket is None: # type: ignore[union-attr]
return
# WebSocket DLP runs against the static config only (single-tenant); in
# the consolidated gateway self.config has no routes, so this is inert
# until websocket routing is made source-IP-aware (a separate slice).
route = match_route(self.config.routes, flow.request.pretty_host)
if route is None:
return
@@ -585,7 +509,7 @@ class EgressAddon:
# not an injection vector here — scan only for credential leakage.
result = scan_outbound(
route, content, os.environ,
safe_tokens=self._safe_tokens_for(self._supervise_slug), crlf_text="",
safe_tokens=self.safe_tokens, crlf_text="",
)
if result is not None and result.severity == "block":
sys.stderr.write(f"egress DLP: {result.reason}\n")
+7 -75
View File
@@ -3,12 +3,12 @@
Split out of `egress_addon.py` so the host's unit tests can
exercise the parse + decision functions without depending on the
`mitmproxy` package. The companion module wraps these with the
`mitmproxy.http.HTTPFlow` API and is loaded inside the gateway
`mitmproxy.http.HTTPFlow` API and is loaded inside the sidecar
container.
Imports: stdlib + `yaml_subset` (which is itself stdlib-only and
ships flat into the gateway image alongside this file
see `Dockerfile.gateway`)."""
ships flat into the sidecar bundle image alongside this file
see `Dockerfile.sidecars`)."""
from __future__ import annotations
@@ -22,7 +22,7 @@ except ImportError: # pragma: no cover - host-side path
from .yaml_subset import YamlSubsetError, parse_yaml_subset
# DLP detector-config parsing lives in a sibling module (also flat-bundled
# into the gateway — see Dockerfile.gateway). Re-exported below so existing
# into the sidecar — see Dockerfile.sidecars). Re-exported below so existing
# `from egress_addon_core import ON_MATCH_*` callers keep working.
try:
from egress_dlp_config import ( # type: ignore[import-not-found]
@@ -120,7 +120,7 @@ class ScanResult:
reason: str
location: str = "" # where the match was found, e.g. "body", "authorization header"
context: str = "" # surrounding text with the match replaced by REDACT
# Raw substring the detector matched. Used inside the gateway to key the
# Raw substring the detector matched. Used inside the sidecar to key the
# supervisor-approved "safe tokens" set (PRD 0062); never logged or written
# to a proposal file. Empty for structural detectors (CRLF) that carry no
# safelist-able value.
@@ -413,70 +413,6 @@ def load_config(text: str) -> "Config":
return parse_config(payload)
class PolicyResolverLike(typing.Protocol):
"""The bit of `policy_resolver.PolicyResolver` this module needs — kept a
Protocol so egress_addon_core stays free of that import."""
def resolve(self, source_ip: str, identity_token: str = ...) -> "str | None":
...
def _config_from_policy(policy: "str | None") -> "Config":
"""Parse a resolved policy blob into a Config, fail-closed: None / empty /
unparseable all become a deny-all Config (no routes every request
blocked)."""
if not policy:
return Config(routes=()) # unattributed or empty → deny-all
try:
return load_config(policy)
except ValueError:
return Config(routes=()) # unparseable policy → deny
def resolve_client_config(
resolver: PolicyResolverLike, client_ip: str, identity_token: str = ""
) -> "Config":
"""The calling client's egress Config, resolved from the orchestrator via
`resolver` and parsed **fail-closed**. An unattributed client (None), a
resolver error, or an unparseable policy all yield a deny-all Config (no
routes every request blocked). A compromised, absent, or confused
orchestrator must never *widen* a bottle's egress."""
try:
policy = resolver.resolve(client_ip, identity_token)
except Exception: # noqa: BLE001 # pylint: disable=broad-exception-caught
return Config(routes=()) # orchestrator unreachable/errored → deny
return _config_from_policy(policy)
class ContextResolverLike(typing.Protocol):
"""The bit of `policy_resolver.PolicyResolver` `resolve_client_context`
needs one round-trip returning policy, bottle id, and auth tokens."""
def resolve_policy_and_bottle_id(
self, source_ip: str, identity_token: str = ...,
) -> "tuple[str | None, str | None, dict[str, str]]":
...
def resolve_client_context(
resolver: ContextResolverLike, client_ip: str, identity_token: str = "",
) -> "tuple[Config, str, dict[str, str]]":
"""The calling client's `(Config, bottle_id, tokens)` in one round-trip —
**fail-closed**. The Config follows `resolve_client_config`'s deny-all
rules; the bottle id is `""` whenever unattributed or the orchestrator
errored (caller treats as "supervise unavailable", never another bottle's
queue); `tokens` are the per-bottle upstream auth values the addon injects.
One `/resolve` keys the egress policy, the supervise queue + safelist, and
auth injection."""
try:
policy, bottle_id, tokens = resolver.resolve_policy_and_bottle_id(
client_ip, identity_token,
)
except Exception: # noqa: BLE001 # pylint: disable=broad-exception-caught
return Config(routes=()), "", {} # orchestrator unreachable/errored → deny
return _config_from_policy(policy), (bottle_id or ""), tokens
# ---------------------------------------------------------------------------
# Match evaluation
# ---------------------------------------------------------------------------
@@ -677,7 +613,7 @@ def outbound_scan_headers(
) -> dict[str, str]:
"""Return request headers that should be included in outbound DLP.
Routes that inject gateway-owned auth always strip the agent's
Routes that inject sidecar-owned auth always strip the agent's
Authorization header before forwarding. Scanning that header first
creates false positives for provider clients that insist on sending
their own bearer-shaped placeholder, while still not changing what
@@ -728,7 +664,7 @@ def scan_outbound(
crlf_text: str | None = None,
) -> ScanResult | None:
# Lazy import to avoid circular deps and keep dlp_detectors optional
# at import time (the gateway copies it flat alongside this file).
# at import time (the sidecar copies it flat alongside this file).
try:
from dlp_detectors import ( # type: ignore[import-not-found]
scan_crlf_injection,
@@ -868,10 +804,6 @@ __all__ = [
"is_git_push_request",
"is_git_fetch_request",
"load_config",
"resolve_client_config",
"resolve_client_context",
"PolicyResolverLike",
"ContextResolverLike",
"match_route",
"outbound_scan_headers",
"parse_config",
+2 -2
View File
@@ -6,8 +6,8 @@ and what the proxy does when an outbound detector matches a token
kept apart from the request-time scan/decision flow in `egress_addon_core`
so each half reads top-to-bottom without scrolling past the other.
Stdlib-only; ships flat into the gateway image alongside
`egress_addon_core.py` see `Dockerfile.gateway`."""
Stdlib-only; ships flat into the sidecar bundle image alongside
`egress_addon_core.py` see `Dockerfile.sidecars`."""
from __future__ import annotations
+5 -5
View File
@@ -1,8 +1,8 @@
#!/bin/sh
# Egress daemon entrypoint inside the gateway (PRD 0024).
# Egress daemon entrypoint inside the sidecar bundle (PRD 0024).
#
# Extracted verbatim from Dockerfile.egress's prior inline `sh -c`
# ENTRYPOINT so the supervisor in bot_bottle/gateway_init.py can
# ENTRYPOINT so the supervisor in bot_bottle/sidecar_init.py can
# call it as a normal child. Behavior is unchanged:
#
# * Upstream proxy: when EGRESS_UPSTREAM_PROXY is set, switch
@@ -22,7 +22,7 @@ set -e
# Pin mitmproxy's config dir to the bind-mount location of its CA
# regardless of which user mitmdump runs as. In the legacy
# four-daemon setup (Dockerfile.egress, USER mitmproxy) this
# four-sidecar setup (Dockerfile.egress, USER mitmproxy) this
# resolved naturally to `~mitmproxy/.mitmproxy`. In the PRD 0024
# bundle (USER root) `~root/.mitmproxy` is empty, so without this
# flag mitmdump would generate a fresh CA on the wrong path and
@@ -37,8 +37,8 @@ if [ -n "$EGRESS_UPSTREAM_PROXY" ]; then
fi
# Bind address. Docker backend wants `0.0.0.0` (agent dials egress
# directly via the docker network alias). A VM backend uses
# EGRESS_LISTEN_HOST when a non-default binding is needed.
# directly via the docker network alias). Smolmachines backend
# uses EGRESS_LISTEN_HOST when a non-default binding is needed.
LISTEN_HOST_FLAG=""
if [ -n "$EGRESS_LISTEN_HOST" ]; then
LISTEN_HOST_FLAG="--listen-host $EGRESS_LISTEN_HOST"
+4 -6
View File
@@ -1,6 +1,6 @@
"""Per-agent git-gate (PRD 0008).
A third per-agent daemon that fronts the bottle's declared git
A third per-agent sidecar that fronts the bottle's declared git
upstreams as a transparent mirror. Each `bottle.git` entry maps to
a bare repo on the gate; `git daemon` serves the bare repos over
`git://<gate>/<name>.git`. Two hooks make the mirror bidirectional:
@@ -15,7 +15,7 @@ a bare repo on the gate; `git daemon` serves the bare repos over
The agent never sees the upstream credential under either path.
Why a separate daemon (not folded into egress or ssh-gate): the
Why a separate sidecar (not folded into egress or ssh-gate): the
gate is the only one of the three that holds upstream push
credentials. Mixing it with egress would put push creds in the
same blast radius as internet-facing TLS interception; mixing it
@@ -23,7 +23,7 @@ with ssh-gate would force ssh-gate above L4 and into git-protocol
land. See `docs/prds/0008-git-gate.md`.
This module defines the abstract gate (`GitGate`) and its plan
dataclass (`GitGatePlan`). The gateway's start/stop lifecycle is
dataclass (`GitGatePlan`). The sidecar's start/stop lifecycle is
backend-specific and lives on concrete subclasses (see
`bot_bottle/backend/docker/git_gate.py`)."""
@@ -46,7 +46,6 @@ from .git_gate_render import (
git_gate_known_hosts_line,
git_gate_render_access_hook,
git_gate_render_entrypoint,
git_gate_render_provision,
git_gate_render_gitconfig,
git_gate_render_hook,
git_gate_upstreams_for_bottle,
@@ -86,7 +85,7 @@ class GitGatePlan:
class GitGate(ABC):
"""The per-agent git-gate. Encapsulates the host-side prepare
(upstream lift + entrypoint/hook render); the gateway's
(upstream lift + entrypoint/hook render); the sidecar's
start/stop lifecycle is backend-specific and lives on concrete
subclasses."""
@@ -156,7 +155,6 @@ __all__ = [
"git_gate_render_gitconfig",
"git_gate_known_hosts_line",
"git_gate_render_entrypoint",
"git_gate_render_provision",
"git_gate_render_hook",
"git_gate_render_access_hook",
"provision_git_gate_dynamic_keys",
+27 -63
View File
@@ -1,7 +1,7 @@
"""Pure host-side rendering for the per-agent git-gate (PRD 0008).
Builds the agent's `.gitconfig` insteadOf rewrites, the known_hosts
line, and the entrypoint / pre-receive / access-hook scripts the gateway
line, and the entrypoint / pre-receive / access-hook scripts the sidecar
runs. No docker or forge calls exposed for tests and reuse across
backends. Split out of `git_gate.py` so the control surface (`GitGate`)
and the deploy-key lifecycle (`git_gate_provision`) each read on their
@@ -9,14 +9,13 @@ own; `git_gate` re-exports these names for API stability."""
from __future__ import annotations
import re
import shlex
from dataclasses import dataclass
from pathlib import Path
from .manifest import ManifestBottle, ManifestGitEntry
# Short network alias for git-gate inside the gateway. The
# Short network alias for git-gate inside the sidecar bundle. The
# agent's `.gitconfig` insteadOf rewrites resolve through this name.
GIT_GATE_HOSTNAME = "git-gate"
# Shared timeout (seconds) for all git-gate subprocess and CGI calls:
@@ -38,7 +37,7 @@ class GitGateUpstream:
KnownHostKey string from the manifest; the gate's start step
materialises it into a known_hosts file if non-empty.
the gate credential paths inside the running gateway."""
the gate credential paths inside the running sidecar."""
name: str
upstream_url: str
@@ -77,14 +76,14 @@ def git_gate_render_gitconfig(
entries: tuple[ManifestGitEntry, ...], gate_host: str, *, scheme: str = "git",
) -> str:
"""Render the agent's ~/.gitconfig content for git-gate
`insteadOf` rewrites. Pure host-side, no docker / VM;
`insteadOf` rewrites. Pure host-side, no docker / smolvm;
exposed for tests + reuse across backends.
`gate_host` is the part of the URL between `<scheme>://` and the
repo path backends differ here:
- docker: `git-gate` (the short network alias)
- firecracker: `<bundle_ip>:<port>` (no DNS on the
point-to-point TAP link)
- smolmachines: `<bundle_ip>:<port>` (no DNS in the
TSI-allowlisted guest)
Empty `entries` returns an empty string so callers can no-op
cleanly without conditional formatting at the call site."""
@@ -126,18 +125,22 @@ def git_gate_known_hosts_line(host: str, port: str, key: str) -> str:
return f"{target} {key}\n"
def _git_gate_init_repo_fn(repo_root: str, creds_dir: str) -> list[str]:
"""The `init_repo` shell function, parameterized by the bare-repo root
and the per-bottle creds dir. Single source of the credential-wiring
logic, shared by the single-tenant daemon entrypoint (`/git`,
`/git-gate/creds`) and the consolidated per-bottle provisioning
(`/git/<bottle_id>`, `/git-gate/creds/<bottle_id>`)."""
return [
def git_gate_render_entrypoint(upstreams: tuple[GitGateUpstream, ...]) -> str:
"""Posix-sh entrypoint. One `init_repo` call per upstream, then
`exec git daemon`. The function reads
`/git-gate/creds/<name>-{key,known_hosts}` (bind-mounted into
the bundle by the renderer) and wires them into each bare repo's
config; the access-hook + pre-receive hook pick those paths up
at fetch / push time."""
lines = [
"#!/bin/sh",
"set -eu",
"",
"init_repo() {",
" name=$1",
" upstream_url=$2",
f" keyfile={creds_dir}/${{name}}-key",
f" hostsfile={creds_dir}/${{name}}-known_hosts",
" keyfile=/git-gate/creds/${name}-key",
" hostsfile=/git-gate/creds/${name}-known_hosts",
"",
# `|| true`: PRD 0018 chunk 3+ bind-mounts these RO from the
# host, so chmod-syscalls fail with EROFS. The files already
@@ -150,14 +153,14 @@ def _git_gate_init_repo_fn(repo_root: str, creds_dir: str) -> list[str]:
" chmod 600 \"$hostsfile\" 2>/dev/null || true",
" fi",
"",
f" repo={repo_root}/${{name}}.git",
" repo=/git/${name}.git",
" if [ ! -d \"$repo\" ]; then",
" git init --bare \"$repo\" >/dev/null",
# --mirror=fetch sets remote.origin.fetch = +refs/*:refs/* so a later
# `git fetch origin` mirrors the upstream's full ref graph (heads,
# tags, notes) into the bare repo at canonical paths. It does NOT set
# remote.origin.mirror=true, so an explicit `git push origin
# <ref>:<ref>` still pushes one ref.
# --mirror=fetch sets remote.origin.fetch = +refs/*:refs/* so",
# a later `git fetch origin` mirrors the upstream's full ref",
# graph (heads, tags, notes) into the bare repo at canonical",
# paths. It does NOT set remote.origin.mirror=true, so an",
# explicit `git push origin <ref>:<ref>` still pushes one ref.",
" git -C \"$repo\" remote add --mirror=fetch origin \"$upstream_url\"",
" fi",
" git -C \"$repo\" config git-gate.identityFile \"$keyfile\"",
@@ -167,19 +170,9 @@ def _git_gate_init_repo_fn(repo_root: str, creds_dir: str) -> list[str]:
" git -C \"$repo\" config http.receivepack true",
" install -m 755 /etc/git-gate/pre-receive \"$repo/hooks/pre-receive\"",
"}",
"",
"mkdir -p /git",
]
def git_gate_render_entrypoint(upstreams: tuple[GitGateUpstream, ...]) -> str:
"""Posix-sh entrypoint. One `init_repo` call per upstream, then
`exec git daemon`. The function reads
`/git-gate/creds/<name>-{key,known_hosts}` (bind-mounted into
the bundle by the renderer) and wires them into each bare repo's
config; the access-hook + pre-receive hook pick those paths up
at fetch / push time."""
lines = ["#!/bin/sh", "set -eu", ""]
lines += _git_gate_init_repo_fn("/git", "/git-gate/creds")
lines += ["", "mkdir -p /git"]
for u in upstreams:
lines.append(f"init_repo {shlex.quote(u.name)} {shlex.quote(u.upstream_url)}")
lines.extend([
@@ -197,35 +190,6 @@ def git_gate_render_entrypoint(upstreams: tuple[GitGateUpstream, ...]) -> str:
return "\n".join(lines) + "\n"
# A bottle id namespaces the consolidated gateway's repo + creds dirs; it is
# embedded unquoted in the provisioning script, so restrict it to a shell- and
# path-safe alphabet (registry ids are token_hex — this is defense in depth).
_SAFE_BOTTLE_ID = re.compile(r"[A-Za-z0-9_-]+")
def git_gate_render_provision(
bottle_id: str, upstreams: tuple[GitGateUpstream, ...],
) -> str:
"""Posix-sh script that provisions ONE bottle's bare repos into the
consolidated gateway (PRD 0070), under `/git/<bottle_id>/` with creds
read from `/git-gate/creds/<bottle_id>/`. Init-only no `git daemon`,
since the shared gateway already serves every bottle; run inside the
running gateway when the bottle is registered.
Isolating each bottle's repo root and creds dir by id is what keeps one
bottle's push credentials out of another's repos on the shared gateway."""
if not _SAFE_BOTTLE_ID.fullmatch(bottle_id):
raise ValueError(f"git-gate: unsafe bottle id {bottle_id!r}")
repo_root = f"/git/{bottle_id}"
creds_dir = f"/git-gate/creds/{bottle_id}"
lines = ["#!/bin/sh", "set -eu", ""]
lines += _git_gate_init_repo_fn(repo_root, creds_dir)
lines += ["", f"mkdir -p {shlex.quote(repo_root)}"]
for u in upstreams:
lines.append(f"init_repo {shlex.quote(u.name)} {shlex.quote(u.upstream_url)}")
return "\n".join(lines) + "\n"
def git_gate_render_hook() -> str:
"""The shared pre-receive hook: gitleaks-scan all incoming refs,
then forward each accepted ref to the real upstream (`origin`)
+10 -122
View File
@@ -1,20 +1,10 @@
"""Tiny smart-HTTP wrapper for git-gate repos.
Used where `git://` push traffic over a host-published Docker port can
hang before receive-pack reaches hooks (e.g. the firecracker backend,
where the guest reaches the gateway over the point-to-point TAP). The
wrapper serves the same `/git/*.git` bare repos through
Used by the smolmachines backend where `git://` push traffic over the
host-published Docker port can hang before receive-pack reaches hooks.
The wrapper serves the same `/git/*.git` bare repos through
`git http-backend`, so pre-receive and upstream forwarding remain the
git-gate enforcement point.
Consolidated (PRD 0070): when `BOT_BOTTLE_ORCHESTRATOR_URL` is set, one
shared gateway serves every bottle, and each request is served from the
calling bottle's repo namespace (`<root>/<bottle_id>`), attributed from
the unspoofable source IP via the orchestrator. Per-repo credentials +
hooks scope by repo directory, so isolating the *root* per bottle isolates
its creds too. Unattributed clients fail closed (404). Unset the legacy
per-bottle single-tenant flat root, unchanged a transitional path that
gets stripped out once every backend runs the consolidated gateway.
"""
from __future__ import annotations
@@ -22,89 +12,16 @@ from __future__ import annotations
import os
import subprocess
import sys
import typing
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
from pathlib import Path
from urllib.parse import urlsplit
# policy_resolver ships flat alongside this file in the gateway
# image (see Dockerfile.gateway); the bot_bottle.* fallback is the
# host-side / test path. Mirrors egress_addon's import shape.
try:
from policy_resolver import ( # type: ignore[import-not-found]
PolicyResolveError,
PolicyResolver,
)
except ImportError: # pragma: no cover - host-side path
from bot_bottle.policy_resolver import PolicyResolveError, PolicyResolver
DEFAULT_PORT = 9420
# Consolidated (multi-tenant) mode: when this points at the per-host
# orchestrator's control plane, the backend serves each request from the
# *calling* bottle's repo namespace, selected by source IP, instead of a
# single flat repo root. Unset → legacy per-bottle single-tenant mode
# (unchanged). Same env the egress addon reads, so one orchestrator setting
# flips the whole shared gateway multi-tenant.
ORCHESTRATOR_URL_ENV = "BOT_BOTTLE_ORCHESTRATOR_URL"
# App-layer identity token (defense-in-depth over the source-IP invariant);
# the agent injects it, the backend reads it for attribution and never
# forwards it to `git http-backend`. Mirrors egress_addon.IDENTITY_HEADER
# (duplicated, not imported: egress_addon pulls in mitmproxy).
IDENTITY_HEADER = "x-bot-bottle-identity"
# Default flat repo root (single-tenant, and the base under which
# consolidated mode nests each sandbox's namespace).
DEFAULT_REPO_ROOT = "/git"
class ResolverLike(typing.Protocol):
"""Structural type for the resolver `resolve_sandbox_root` needs — just
`resolve_bottle_id`. A Protocol so tests can pass a fake without
importing PolicyResolver (mirrors egress_addon_core.PolicyResolverLike)."""
def resolve_bottle_id(
self, source_ip: str, identity_token: str = ...,
) -> "str | None": ...
def resolve_sandbox_root(
resolver: "ResolverLike | None",
base_root: Path,
source_ip: str,
identity_token: str = "",
) -> Path | None:
"""The per-sandbox repo root to serve this request from, or None to
deny (404).
Single-tenant (`resolver is None`): the flat `base_root`, unchanged.
NOTE: this legacy per-bottle single-tenant path is transitional it
will be stripped out once every backend runs the consolidated gateway
(PRD 0070), leaving only the source-IP-attributed path below.
Consolidated: `base_root/<bottle_id>`, where the sandbox is attributed
from the source IP via the orchestrator. Fail-closed an unattributed
client, a resolver error, or a namespace that would escape `base_root`
all deny, so one sandbox can never reach another's repos."""
if resolver is None:
return base_root
try:
bottle_id = resolver.resolve_bottle_id(source_ip, identity_token)
except PolicyResolveError:
return None # orchestrator unreachable/errored → deny
if not bottle_id:
return None # unattributed → deny
base = base_root.resolve()
namespace = (base / bottle_id).resolve()
if base not in namespace.parents:
return None # bottle_id tried to escape the root → deny
return namespace
# Mirrors git_gate_render.GIT_GATE_TIMEOUT_SECS. Duplicated rather than
# imported: this module ships as a flat top-level sibling in the gateway
# bundle image (see Dockerfile.gateway), not as part of the bot_bottle
# imported: this module ships as a flat top-level sibling in the sidecar
# bundle image (see Dockerfile.sidecars), not as part of the bot_bottle
# package, so `bot_bottle.git_gate` and its dependency chain aren't
# available at runtime.
GIT_GATE_TIMEOUT_SECS = 15
@@ -122,25 +39,10 @@ class GitHttpHandler(BaseHTTPRequestHandler):
def do_POST(self) -> None:
self._run_backend()
def _sandbox_root(self) -> Path | None:
"""This request's per-sandbox repo root, or None to deny. Single-tenant
unless the server was started with a resolver (consolidated mode), in
which case the root is the calling sandbox's source-IP-selected
namespace. `GIT_PROJECT_ROOT` keeps git's own env-var name."""
base = Path(os.environ.get("GIT_PROJECT_ROOT", DEFAULT_REPO_ROOT))
resolver = getattr(self.server, "policy_resolver", None)
token = self.headers.get(IDENTITY_HEADER, "")
return resolve_sandbox_root(resolver, base, self.client_address[0], token)
def _run_backend(self) -> None:
sandbox_root = self._sandbox_root()
if sandbox_root is None:
# Unattributed / resolver error: deny before touching any repo.
self.send_error(404)
return
parsed = urlsplit(self.path)
if self._is_upload_pack(parsed.path, parsed.query):
repo_dir = self._repo_dir(sandbox_root, parsed.path)
repo_dir = self._repo_dir(parsed.path)
if repo_dir is None:
self.send_error(404)
return
@@ -174,7 +76,7 @@ class GitHttpHandler(BaseHTTPRequestHandler):
return
env = os.environ.copy()
env.update({
"GIT_PROJECT_ROOT": str(sandbox_root),
"GIT_PROJECT_ROOT": os.environ.get("GIT_PROJECT_ROOT", "/git"),
"GIT_HTTP_EXPORT_ALL": "1",
"REQUEST_METHOD": self.command,
"PATH_INFO": parsed.path,
@@ -188,14 +90,6 @@ class GitHttpHandler(BaseHTTPRequestHandler):
"SERVER_PORT": str(self.server.server_port), # type: ignore
"SERVER_PROTOCOL": self.request_version,
})
# Consolidated mode: attribute the gitleaks-allow supervise proposal
# (written by receive-pack's pre-receive hook, a child of the CGI we
# spawn below) to the calling bottle. The namespaced root is
# `<base>/<bottle_id>`, so its final component is the bottle id — the
# same per-bottle key egress uses. Single-tenant leaves the hook's
# container-stamped SUPERVISE_BOTTLE_SLUG untouched.
if getattr(self.server, "policy_resolver", None) is not None:
env["SUPERVISE_BOTTLE_SLUG"] = sandbox_root.name
for header, variable in (
("accept", "HTTP_ACCEPT"),
("content-encoding", "HTTP_CONTENT_ENCODING"),
@@ -228,8 +122,8 @@ class GitHttpHandler(BaseHTTPRequestHandler):
)
self._write_cgi_response(proc.stdout)
def _repo_dir(self, sandbox_root: Path, path: str) -> Path | None:
root = sandbox_root.resolve()
def _repo_dir(self, path: str) -> Path | None:
root = Path(os.environ.get("GIT_PROJECT_ROOT", "/git")).resolve()
relative = path.lstrip("/").split(".git", 1)[0] + ".git"
candidate = (root / relative).resolve()
if root not in (candidate, *candidate.parents):
@@ -286,13 +180,7 @@ class GitHttpHandler(BaseHTTPRequestHandler):
def main() -> int:
port = int(os.environ.get("GIT_HTTP_PORT", str(DEFAULT_PORT)))
server = ThreadingHTTPServer(("0.0.0.0", port), GitHttpHandler)
orch_url = os.environ.get(ORCHESTRATOR_URL_ENV, "").strip()
# Consolidated mode: resolve each request's sandbox namespace by source
# IP. Absent → single-tenant (flat repo root); behaviour unchanged.
resolver = PolicyResolver(orch_url) if orch_url else None
server.policy_resolver = resolver # type: ignore[attr-defined]
mode = "multi-tenant" if orch_url else "single-tenant"
sys.stdout.write(f"git-http listening on 0.0.0.0:{port} ({mode})\n")
sys.stdout.write(f"git-http listening on 0.0.0.0:{port}\n")
sys.stdout.flush()
server.serve_forever()
return 0
+2 -2
View File
@@ -17,7 +17,7 @@ class ManifestAgentProvider:
`template` selects a built-in launch/runtime contract. `dockerfile`
optionally points at a custom agent-image Dockerfile while leaving
bot-bottle's gateway infrastructure intact.
bot-bottle's sidecar infrastructure intact.
`auth_token` names the host env var that holds the provider's OAuth
token (Claude only). The provisioner injects a provider-owned egress
@@ -26,7 +26,7 @@ class ManifestAgentProvider:
so the Claude Code CLI starts.
`forward_host_credentials` forwards the host Codex auth token into
the egress daemon (Codex only).
the egress sidecar (Codex only).
"""
template: str = "claude"
+4 -4
View File
@@ -39,10 +39,10 @@ class ManifestBottle:
# identity without any git-gate.repos upstreams, and vice versa.
git_user: ManifestGitUser = field(default_factory=ManifestGitUser)
egress: ManifestEgressConfig = field(default_factory=ManifestEgressConfig)
# Per-bottle stuck-recovery daemon (PRD 0013). When true (the
# Per-bottle stuck-recovery sidecar (PRD 0013). When true (the
# default, issue #249), the launch step brings up a supervise
# daemon that exposes egress MCP tools to the agent. Set
# `supervise: false` to skip the gateway.
# sidecar that exposes egress MCP tools to the agent. Set
# `supervise: false` to skip the sidecar.
supervise: bool = True
@classmethod
@@ -61,7 +61,7 @@ class ManifestBottle:
raise ManifestError(
f"bottle '{name}' has an 'ssh' field, which has been removed "
f"(PRD 0009). Declare upstreams under 'git-gate.repos' with "
f"url + identity + host_key; the git-gate daemon (PRD 0008) "
f"url + identity + host_key; the git-gate sidecar (PRD 0008) "
f"holds the credential and gitleaks-scans pushes."
)
-64
View File
@@ -1,64 +0,0 @@
"""Per-host orchestrator service (PRD 0070).
A single persistent per-host service that will run the gateway functions
(egress / git-gate / supervise), coordinate with the console, and broker
agent launches. This package is being built bottom-up, starting with the
backend-neutral "consolidation core" that needs no VM packaging:
* `registry` the SQLite runtime-state store + fail-closed
attribution (source IP + per-bottle identity token).
* `broker` the signed, structured launch-request contract + a
`LaunchBroker` (stub for the harness) that verifies
provenance before acting.
* `gateway` the consolidated per-host gateway: a `Gateway`
lifecycle contract (idempotent singleton) + a
`DockerGateway` impl. One gateway shared by all
bottles instead of one per bottle.
* `service` the `Orchestrator`: owns the registry, brokers the
launch lifecycle (launch/teardown), manages the
shared gateway, attributes.
* `control_plane` the HTTP control-plane RPC (launch / teardown /
list / attribute / gateway / health).
The actual backend-native launch (a real docker/firecracker broker) and
the egress/git/supervise data plane land in later slices once this core is
proven (see the PRD sequencing: plain-process dev-harness -> docker
orchestrator -> firecracker).
"""
from __future__ import annotations
from .registry import BottleRecord, RegistryStore, new_identity_token
from .broker import (
BrokerAuthError,
LaunchBroker,
LaunchRequest,
StubBroker,
sign_request,
verify_request,
)
from .docker_broker import DockerBroker, DockerBrokerError
from .gateway import DockerGateway, Gateway, GatewayError
from .service import Orchestrator
from .control_plane import ControlPlaneServer, dispatch, make_server
__all__ = [
"BottleRecord",
"RegistryStore",
"new_identity_token",
"BrokerAuthError",
"LaunchBroker",
"LaunchRequest",
"StubBroker",
"DockerBroker",
"DockerBrokerError",
"Gateway",
"DockerGateway",
"GatewayError",
"sign_request",
"verify_request",
"Orchestrator",
"ControlPlaneServer",
"dispatch",
"make_server",
]
-85
View File
@@ -1,85 +0,0 @@
"""Run the orchestrator control plane as a plain process (PRD 0070 dev-harness).
python -m bot_bottle.orchestrator [--host H] [--port P] [--db PATH]
The PRD sequences the orchestrator as a plain-process dev-harness first, so
the consolidation core (registry + attribution + HTTP control plane + live
reload) can be exercised with fast iteration, decoupled from any VM /
container packaging. Wrapping this exact service in a backend-native unit
(docker container, then Firecracker VM) comes later.
"""
from __future__ import annotations
import argparse
import secrets
from pathlib import Path
from .. import log
from .broker import LaunchBroker, StubBroker
from .control_plane import make_server
from .docker_broker import DockerBroker
from .registry import RegistryStore, default_db_path
from .service import Orchestrator
from .gateway import DockerGateway, Gateway
def main(argv: list[str] | None = None) -> int:
"""Parse args, migrate the registry, and serve the control plane."""
parser = argparse.ArgumentParser(prog="bot_bottle.orchestrator")
parser.add_argument("--host", default="127.0.0.1", help="bind address")
parser.add_argument("--port", type=int, default=8080, help="bind port (0 = ephemeral)")
parser.add_argument(
"--db", type=Path, default=None,
help=f"registry DB path (default: {default_db_path()})",
)
parser.add_argument(
"--broker", choices=("stub", "docker"), default="stub",
help="launch broker: 'stub' records requests; 'docker' runs containers",
)
parser.add_argument(
"--gateway", action="store_true",
help="run one consolidated per-host gateway (build-if-missing)",
)
args = parser.parse_args(argv)
registry = RegistryStore(args.db)
registry.migrate()
# An ephemeral signing secret ties the orchestrator (signer) to its
# broker (verifier). 'stub' records launches instead of starting
# anything; 'docker' runs real containers (firecracker drops in later).
secret = secrets.token_bytes(32)
broker: LaunchBroker = DockerBroker(secret) if args.broker == "docker" else StubBroker(secret)
# Standalone `--gateway` (not the consolidated flow, where the host
# lifecycle runs the gateway). The gateway resolves against this same
# process; the URL is only reachable when they share a docker network.
gateway: Gateway | None = (
DockerGateway(orchestrator_url=f"http://{args.host}:{args.port}")
if args.gateway else None
)
orchestrator = Orchestrator(registry, broker, secret, gateway)
# One persistent per-host gateway, shared by every bottle: build the
# bundle image if missing, then bring the singleton up (idempotent).
if gateway is not None:
orchestrator.ensure_gateway()
log.info("consolidated gateway ensured", context={"name": gateway.name})
server = make_server(orchestrator, host=args.host, port=args.port)
bound_host, bound_port = server.server_address[0], server.server_address[1]
log.info(
"orchestrator control plane listening",
context={"host": bound_host, "port": bound_port, "db": str(registry.db_path)},
)
try:
server.serve_forever()
except KeyboardInterrupt:
log.info("orchestrator shutting down")
finally:
server.server_close()
return 0
if __name__ == "__main__":
raise SystemExit(main())
-176
View File
@@ -1,176 +0,0 @@
"""Launch broker contract (PRD 0070).
A VM/container can't spawn its own host-network siblings, so the
orchestrator brokers agent launches through a small privileged component.
The request it sends is:
* **structured** static flags + ids only (bottle id, pool slot, a
content-addressed image ref), never a free-form path or argv, so a
request can't be coerced into launching an arbitrary payload; and
* **signed** wrapped as a compact JWS/JWT the broker verifies before
acting, so a *compromised co-located component* (an agent-facing
gateway, say) can't forge a launch. This is the concrete form of the
"structured requests only" rule in the PRD security review.
Signing is **HS256** over a secret shared by the orchestrator (signer) and
the broker (verifier) appropriate here because both are trusted host
components provisioned together; the secret is exactly what an
agent-facing component does not have. (Stdlib only the project takes no
runtime deps; a cross-host deployment could swap in an asymmetric alg.)
"""
from __future__ import annotations
import abc
import base64
import hashlib
import hmac
import json
import secrets
import time
from dataclasses import dataclass
_JWT_HEADER = {"alg": "HS256", "typ": "JWT"}
_ALLOWED_OPS = ("launch", "teardown")
class BrokerAuthError(Exception):
"""A broker request failed provenance or schema verification —
bad/absent signature, malformed token, or a payload that doesn't match
the fixed launch-request shape. Fail-closed: the broker must not act."""
@dataclass(frozen=True)
class LaunchRequest:
"""The structured, un-coercible launch/teardown request. Ids + flags
only `image_ref` is a content-addressed id, never a path/argv."""
op: str # one of _ALLOWED_OPS
bottle_id: str
source_ip: str = ""
image_ref: str = ""
slot: int | None = None
# --- minimal JWS/JWT (HS256), stdlib only ----------------------------------
def _b64url(data: bytes) -> str:
return base64.urlsafe_b64encode(data).rstrip(b"=").decode("ascii")
def _b64url_decode(text: str) -> bytes:
return base64.urlsafe_b64decode(text + "=" * (-len(text) % 4))
def _mac(signing_input: str, secret: bytes) -> str:
digest = hmac.new(secret, signing_input.encode("ascii"), hashlib.sha256).digest()
return _b64url(digest)
def sign_request(req: LaunchRequest, secret: bytes) -> str:
"""Serialize `req` to signed-JWT form. Adds a per-signature `jti`
(replay id) and `iat` (issued-at)."""
claims: dict[str, object] = {
"op": req.op,
"bottle_id": req.bottle_id,
"source_ip": req.source_ip,
"image_ref": req.image_ref,
"slot": req.slot,
"jti": secrets.token_hex(8),
"iat": int(time.time()),
}
header = _b64url(json.dumps(_JWT_HEADER, sort_keys=True, separators=(",", ":")).encode())
payload = _b64url(json.dumps(claims, sort_keys=True, separators=(",", ":")).encode())
signing_input = f"{header}.{payload}"
return f"{signing_input}.{_mac(signing_input, secret)}"
def verify_request(token: str, secret: bytes) -> LaunchRequest:
"""Verify the signature and the request shape; return the parsed
request. Raises `BrokerAuthError` on any failure (fail-closed)."""
parts = token.split(".")
if len(parts) != 3:
raise BrokerAuthError("malformed token")
header_b, payload_b, sig = parts
if not hmac.compare_digest(_mac(f"{header_b}.{payload_b}", secret), sig):
raise BrokerAuthError("bad signature")
try:
header = json.loads(_b64url_decode(header_b))
claims = json.loads(_b64url_decode(payload_b))
except (ValueError, json.JSONDecodeError) as e:
raise BrokerAuthError("malformed token payload") from e
if not isinstance(header, dict) or header.get("alg") != "HS256":
raise BrokerAuthError("unexpected header/alg")
if not isinstance(claims, dict):
raise BrokerAuthError("claims are not an object")
op = claims.get("op")
bottle_id = claims.get("bottle_id")
source_ip = claims.get("source_ip", "")
image_ref = claims.get("image_ref", "")
slot = claims.get("slot")
if (
not isinstance(op, str) or op not in _ALLOWED_OPS
or not isinstance(bottle_id, str) or not bottle_id
or not isinstance(source_ip, str) or not isinstance(image_ref, str)
or not (slot is None or isinstance(slot, int))
):
raise BrokerAuthError("request does not match the launch-request schema")
return LaunchRequest(
op=op, bottle_id=bottle_id, source_ip=source_ip, image_ref=image_ref, slot=slot
)
# --- the broker itself ------------------------------------------------------
class LaunchBroker(abc.ABC):
"""Verifies a signed request came from the orchestrator, then performs
the backend-native launch/teardown. Subclasses implement `_launch` /
`_teardown`; verification is shared and fail-closed."""
def __init__(self, secret: bytes) -> None:
self._secret = secret
def submit(self, token: str) -> LaunchRequest:
"""Verify `token` and perform its op. Returns the verified request;
raises `BrokerAuthError` if provenance/shape fails."""
req = verify_request(token, self._secret)
if req.op == "launch":
self._launch(req)
else:
self._teardown(req)
return req
@abc.abstractmethod
def _launch(self, req: LaunchRequest) -> None:
...
@abc.abstractmethod
def _teardown(self, req: LaunchRequest) -> None:
...
class StubBroker(LaunchBroker):
"""Dev-harness broker: records verified requests without launching
anything. Exercises the full sign -> verify -> act contract in-process."""
def __init__(self, secret: bytes) -> None:
super().__init__(secret)
self.launched: list[LaunchRequest] = []
self.torn_down: list[LaunchRequest] = []
def _launch(self, req: LaunchRequest) -> None:
self.launched.append(req)
def _teardown(self, req: LaunchRequest) -> None:
self.torn_down.append(req)
__all__ = [
"BrokerAuthError",
"LaunchRequest",
"LaunchBroker",
"StubBroker",
"sign_request",
"verify_request",
]
-144
View File
@@ -1,144 +0,0 @@
"""Host-side control-plane client (PRD 0070).
The launch path talks to the orchestrator over its HTTP control plane to
register, re-policy, and tear down bottles the counterpart to the
gateway-side `PolicyResolver` (which only reads `/resolve`). Where
`PolicyResolver` is fail-closed and lives in the untrusted data plane, this
is the trusted control-plane caller: a non-success response is an error the
launch path must surface, not silently swallow.
Stdlib-only, so the CLI can drive the orchestrator without any dependency.
"""
from __future__ import annotations
import json
import urllib.error
import urllib.request
from dataclasses import dataclass
DEFAULT_TIMEOUT_SECONDS = 5.0
class OrchestratorClientError(RuntimeError):
"""A control-plane call failed (unreachable, or an unexpected status)."""
@dataclass(frozen=True)
class RegisteredBottle:
"""What `POST /bottles` returns: the minted bottle id and the per-bottle
identity token the agent presents for app-layer attribution."""
bottle_id: str
identity_token: str
class OrchestratorClient:
"""Trusted host-side client for the orchestrator control plane."""
def __init__(self, base_url: str, *, timeout: float = DEFAULT_TIMEOUT_SECONDS) -> None:
self._base = base_url.rstrip("/")
self._timeout = timeout
def _request(
self, method: str, path: str, body: dict[str, object] | None = None,
) -> tuple[int, dict[str, object]]:
"""Send one request; return `(status, payload)`. Raises
`OrchestratorClientError` only when the orchestrator can't be reached
or returns malformed data HTTP *status* codes are returned so
callers can treat 404 as a meaningful "no such bottle"."""
data = json.dumps(body).encode() if body is not None else None
headers = {"Content-Type": "application/json"} if data is not None else {}
req = urllib.request.Request(
f"{self._base}{path}", data=data, method=method, headers=headers,
)
try:
with urllib.request.urlopen(req, timeout=self._timeout) as resp:
raw = resp.read()
payload = json.loads(raw) if raw else {}
return resp.status, payload if isinstance(payload, dict) else {}
except urllib.error.HTTPError as e:
# A structured error response still carries a usable status.
try:
payload = json.loads(e.read() or b"{}")
except (ValueError, OSError):
payload = {}
return e.code, payload if isinstance(payload, dict) else {}
except (urllib.error.URLError, TimeoutError, OSError, ValueError) as e:
raise OrchestratorClientError(f"{method} {path}: {e}") from e
def _ok(self, method: str, path: str, body: dict[str, object] | None = None) -> dict[str, object]:
"""`_request` that requires a 2xx, raising otherwise."""
status, payload = self._request(method, path, body)
if not 200 <= status < 300:
detail = payload.get("error", "")
raise OrchestratorClientError(f"{method} {path}: HTTP {status} {detail}".rstrip())
return payload
def health(self) -> bool:
"""True iff the control plane answers `GET /health` with 200."""
try:
status, _ = self._request("GET", "/health")
except OrchestratorClientError:
return False
return status == 200
def register_bottle(
self,
source_ip: str,
*,
image_ref: str = "",
metadata: str = "",
policy: str = "",
tokens: dict[str, str] | None = None,
) -> RegisteredBottle:
"""Register a bottle and broker its launch (`POST /bottles`). `tokens`
are the per-bottle egress auth values (env_name -> value) the
orchestrator holds in memory for the gateway to inject. Returns the
minted id + identity token."""
payload = self._ok("POST", "/bottles", {
"source_ip": source_ip,
"image_ref": image_ref,
"metadata": metadata,
"policy": policy,
"tokens": tokens or {},
})
bottle_id = payload.get("bottle_id")
token = payload.get("identity_token")
if not isinstance(bottle_id, str) or not isinstance(token, str):
raise OrchestratorClientError("register: response missing bottle_id/identity_token")
return RegisteredBottle(bottle_id=bottle_id, identity_token=token)
def teardown_bottle(self, bottle_id: str) -> bool:
"""Tear a bottle down (`DELETE /bottles/<id>`). False if the
orchestrator didn't know it (404) — idempotent for cleanup paths."""
status, _ = self._request("DELETE", f"/bottles/{bottle_id}")
if status == 404:
return False
if not 200 <= status < 300:
raise OrchestratorClientError(f"teardown {bottle_id}: HTTP {status}")
return True
def set_policy(self, bottle_id: str, policy: str) -> bool:
"""Live-reload a bottle's policy (`PUT /bottles/<id>/policy`). False on
404 (unknown bottle)."""
status, _ = self._request("PUT", f"/bottles/{bottle_id}/policy", {"policy": policy})
if status == 404:
return False
if not 200 <= status < 300:
raise OrchestratorClientError(f"set_policy {bottle_id}: HTTP {status}")
return True
def list_bottles(self) -> list[dict[str, object]]:
"""Every registered bottle's redacted record (`GET /bottles`)."""
payload = self._ok("GET", "/bottles")
bottles = payload.get("bottles")
return bottles if isinstance(bottles, list) else []
__all__ = [
"OrchestratorClient",
"OrchestratorClientError",
"RegisteredBottle",
"DEFAULT_TIMEOUT_SECONDS",
]

Some files were not shown because too many files have changed in this diff Show More