From a2ac124d5c245e42fff49f8f25cacd0c50ddae4c Mon Sep 17 00:00:00 2001 From: didericis Date: Tue, 26 May 2026 23:19:08 -0400 Subject: [PATCH 1/3] docs(prd-0023): smolmachines bottle backend Specs a second concrete BottleBackend selectable via CLAUDE_BOTTLE_BACKEND=smolmachines: per-agent libkrun microVM on macOS, sidecars relocated to host-side loopback ports plumbed via Smolfile env, PRD 0022's sandbox-escape suite as the acceptance gate (the env-var flip is the only change required). Docker backend ships unchanged and remains default. Co-Authored-By: Claude Opus 4.7 --- docs/prds/0023-smolmachines-backend.md | 427 +++++++++++++++++++++++++ 1 file changed, 427 insertions(+) create mode 100644 docs/prds/0023-smolmachines-backend.md diff --git a/docs/prds/0023-smolmachines-backend.md b/docs/prds/0023-smolmachines-backend.md new file mode 100644 index 0000000..dd602ff --- /dev/null +++ b/docs/prds/0023-smolmachines-backend.md @@ -0,0 +1,427 @@ +# PRD 0023: smolmachines bottle backend + +- **Status:** Draft +- **Author:** didericis +- **Created:** 2026-05-26 + +## Summary + +Ship a second concrete `BottleBackend` — `SmolmachinesBottleBackend`, +selected via `CLAUDE_BOTTLE_BACKEND=smolmachines` — that runs a +bottle inside a per-agent libkrun microVM on macOS (and KVM on Linux, +opportunistically). The egress topology moves out of an internal +Docker network and onto libkrun's TSI ("Transport Socket Interface") +allowlist plus a host-side pipelock/egress/git-gate/supervise stack +listening on per-bottle loopback ports. The Docker backend ships +unchanged; this is opt-in via the existing env-var selector. + +The acceptance gate is PRD 0022's `tests/integration/test_sandbox_escape.py` +running green against `CLAUDE_BOTTLE_BACKEND=smolmachines`. + +## Problem + +`agent-vm-isolation.md` argues for hardware-isolated microVMs over +container-based bottles on macOS; `smolmachines-as-vm-backend.md` +concludes that smolmachines is the most plausible concrete VMM for +this project. Today, the only backend in the registry is Docker +(`claude_bottle/backend/__init__.py:_BACKENDS = {"docker": ...}`), +and three things motivate a second one now: + +- **Isolation ceiling.** On macOS the Docker backend's agent + container shares Docker Desktop's host VM with every other bottle. + Container escape from claude-code lands the agent inside that + shared VM. A per-bottle libkrun microVM gets hardware page tables + via `Hypervisor.framework`; cross-bottle isolation becomes + enforced by the CPU's MMU instead of namespace bookkeeping. +- **PRD 0022 is backend-agnostic by design** but currently only + exercises the Docker backend. The suite was written with + `CLAUDE_BOTTLE_BACKEND` selection in mind precisely so the + smolmachines path could be validated against the same five + attacks. Until a second backend exists, the abstraction is + unproven. +- **CI carve-outs.** Most bottle-bringup integration tests skip + under `GITEA_ACTIONS=true` because act_runner shares the host + Docker socket but not the host filesystem. A smolmachines path + doesn't share that constraint shape (it has its own, but + different), so adding the backend forces the abstraction to be + clean in places where Docker-specific assumptions have been + tolerated. + +The smolmachines research note's `## Recommendation` ("adopt +smolmachines as the bottle VM backend on macOS; keep pipelock DIY") +is the design hypothesis under test here. + +## Goals / Success Criteria + +The feature works when all of the following are observable on a +macOS host with smolmachines installed: + +- `CLAUDE_BOTTLE_BACKEND=smolmachines python3 cli.py start ` + brings up a microVM, runs claude-code inside it, and tears it + down on exit. Same y/N preflight UX as Docker — only the + resolved-runtime line differs. +- The sandbox-escape suite in `tests/integration/test_sandbox_escape.py` + runs green against the smolmachines backend (all five attack + categories blocked). +- Selecting the backend on a host without `smolvm` installed dies + at startup with an install pointer; no silent fall-through to + Docker. +- Active bottles show up under + `python3 cli.py list-bottles` regardless of backend. +- `python3 cli.py stop ` and orphan cleanup work for both + Docker bottles and smolmachines bottles via the same CLI surface. + +The feature is **done** when all of the following ship: + +- A new `claude_bottle/backend/smolmachines/` subpackage exists, + mirroring the layout of `claude_bottle/backend/docker/` + (`backend.py`, `bottle.py`, `bottle_plan.py`, + `bottle_cleanup_plan.py`, `prepare.py`, `launch.py`, + `cleanup.py`, `util.py`, and a `provision/` subpackage for the + five `provision_*` methods). +- `SmolmachinesBottleBackend` registered under the + `"smolmachines"` key in `claude_bottle/backend/__init__.py:_BACKENDS`. +- Per-bottle Smolfile generation: a runtime-rendered TOML written + to the bottle's stage dir, analogous to the compose file the + Docker backend writes today. The Smolfile pins `command`, + `env`, `--outbound-localhost-only`, and the per-bottle DNS + allowlist. +- Host-side sidecar relocation: pipelock, egress, git-gate, and + supervise each run as host processes (one set per bottle), + bound to `127.0.0.1` on per-bottle dynamically-allocated ports. + The agent's environment carries the resolved URLs (e.g. + `HTTPS_PROXY=http://127.0.0.1:`). +- The agent guest image is produced from the existing `Dockerfile` + (or a thin variant), exported as an OCI archive, and consumed by + `smolvm machine create`. The image build step is part of `prepare`, + analogous to `docker_mod.build_image`. +- The PRD 0022 sandbox-escape suite, run with + `CLAUDE_BOTTLE_BACKEND=smolmachines`, passes locally on a + smolmachines-capable host. The suite is updated to skip cleanly + on hosts that can't reach smolmachines (same shape as the + existing `GITEA_ACTIONS == "true"` skip), not to fail. +- README + `CLAUDE.md` updated to document the env-var selection, + the macOS-only scope for v1, and the `smolvm` install + prerequisite. + +## Non-goals + +- **No Linux KVM support shipped in this PRD.** smolmachines works + on Linux via KVM, but the abstraction win is biggest on macOS + where Docker's shared-VM topology hurts most. Linux can come + later behind the same selector. +- **No removal of the Docker backend.** Both backends ship side by + side. Selection stays env-driven; the manifest does not gain a + `backend` field. +- **No default-backend change.** `docker` remains the default + value of `CLAUDE_BOTTLE_BACKEND`; smolmachines is strictly + opt-in until it has been load-bearing on at least one operator's + workflow for a release cycle. +- **No host bind mounts.** The smolmachines research note flagged + that `-v HOST:GUEST` mounts via virtiofs would defeat the + isolation goal. The manifest already has no concept of host + mounts; this PRD does not introduce one. If a future PRD wants + agent-side access to host files, it must come through a + controlled channel (vsock relay, OCI overlay, supervise sidecar + endpoint). +- **No HTTP API mode.** `smolvm serve` is the long-term-clean + control plane, but v1 drives smolmachines via CLI subprocess + invocations — the lower-overhead first iteration the research + note already endorses. +- **No custom kernel / initrd.** smolmachines uses libkrunfw + only; the agent image is an OCI ref, not a kernel + rootfs pair. +- **No warm-pool or snapshot/restore.** Each bottle gets a fresh + microVM; cold-start cost is paid up front. +- **No supervise/agent-credential rewrites for the new backend.** + Provisioning logic ports as-is; only the *transport* (host-side + port URLs instead of in-network DNS names) changes. + +## Scope + +### In scope + +- New `claude_bottle/backend/smolmachines/` subpackage with the + full set of `BottleBackend` overrides. +- Smolfile generator (TOML), analogous to + `backend/docker/compose.py`'s `bottle_plan_to_compose`. +- A host-side sidecar process manager that owns the lifecycle of + pipelock + egress + git-gate + supervise for one bottle, binding + them to per-bottle loopback ports and tearing them down with the + bottle. This is the smolmachines-specific replacement for + `docker compose up`/`down`. +- Per-bottle CA install path: the egress sidecar's CA cert lands + inside the microVM via `smolvm machine exec` after start + (analogous to the existing `provision_ca` for Docker). +- DNS allowlist plumbing: every host in `bottle.egress.allowlist` + goes into the Smolfile's DNS filter section (vsock port 6002), + so the VMM-layer DNS filter and the bottle's policy stay in + sync — agent can't `dig` its way out via raw IP literals (TSI + + CIDR allowlist enforces this; DNS filter denies hostname + resolution). +- Preflight `smolvm` check: if the user selects this backend and + `smolvm` isn't on `$PATH`, die with an install pointer (brew tap + + version pin TBD in implementation; see open question 3). +- Manifest validation: refuse any bottle field this backend can't + honor (today there are none, since the Docker backend already + rejects host mounts; this is a forward-compat check). +- Tests: + - Smoke unit-level test: Smolfile renderer produces the + expected TOML for a fixture bottle. + - Integration test: `prepare → launch → exec("echo hi") → + teardown` on a smolmachines-capable host (skips otherwise + via the same env/platform gate the Docker integration tests + use). + - PRD 0022 suite, re-run with the env var flipped, passes. + +### Out of scope + +- VM image caching across bottles (each prepare rebuilds from the + OCI archive; layer reuse is whatever smolmachines provides). +- Cross-host bottle relocation (the OCI archive is local-only). +- Operator-facing knobs for vCPU / memory / overlay size (use + sensible defaults; expose as manifest fields in a later PRD if + needed). +- Integration with the `supervise` plane's permission-prompt UX + beyond port plumbing — supervise already speaks HTTP and binds + to whatever loopback the backend hands it. + +## Proposed Design + +### Backend layout + +``` +claude_bottle/backend/smolmachines/ + __init__.py re-exports SmolmachinesBottleBackend + backend.py SmolmachinesBottleBackend façade + bottle.py SmolmachinesBottle (exec_claude / exec / cp_in / close) + bottle_plan.py SmolmachinesBottlePlan + .print() + bottle_cleanup_plan.py SmolmachinesBottleCleanupPlan + prepare.py resolve_plan(spec, stage_dir, ...) -> SmolmachinesBottlePlan + launch.py @contextmanager launch(plan) -> SmolmachinesBottle + cleanup.py prepare_cleanup / cleanup / list_active + smolfile.py bottle_plan_to_smolfile(...) -> dict + render + sidecars.py host-side pipelock/egress/git-gate/supervise lifecycle + smolvm.py thin subprocess wrapper: machine create/start/exec/stop + util.py slugify, port allocation, OCI archive helpers + provision/ ca.py, prompt.py, skills.py, git.py, supervise.py +``` + +### Network + egress topology + +``` + ┌── macOS host ─────────────────────────────────────────────┐ + │ │ + │ ┌── per-bottle host sidecars (one set per microVM) ─┐ │ + │ │ pipelock 127.0.0.1: │ │ + │ │ egress 127.0.0.1: │ │ + │ │ git-gate 127.0.0.1: │ │ + │ │ supervise 127.0.0.1: │ │ + │ └───────────────────────────────────────────────────┘ │ + │ ▲ │ + │ │ TSI passthrough (localhost) │ + │ │ │ + │ ┌── libkrun microVM (per bottle) ───────────────────┐ │ + │ │ env: HTTPS_PROXY=http://127.0.0.1: │ │ + │ │ EGRESS_URL=http://127.0.0.1: │ │ + │ │ GIT_GATE_URL=http://127.0.0.1: │ │ + │ │ MCP_SUPERVISE_URL=http://127.0.0.1: │ │ + │ │ --outbound-localhost-only │ │ + │ │ DNS filter (vsock:6002) → host allowlist │ │ + │ └───────────────────────────────────────────────────┘ │ + │ │ + └───────────────────────────────────────────────────────────┘ +``` + +Two changes vs. the Docker backend: + +1. **Sidecars are host processes, not sibling containers.** No + internal Docker network; isolation comes from TSI plus the + per-bottle loopback port set. +2. **The "internal" allowlist becomes localhost-only.** Egress out + to the public internet still happens through pipelock + egress + — the same scanning + DLP + auth-injection chain — but the + agent's first hop is `127.0.0.1:` reached via TSI, not a + sidecar's IP on a Docker-managed bridge. + +### Lifecycle + +`SmolmachinesBottleBackend.prepare(spec, stage_dir)`: + +1. Cross-backend validation via `BottleBackend._validate` (skills, + git identity files). +2. Allocate four loopback ports (bind, get free port, release; + record on plan). +3. Resolve the agent OCI archive path (build if missing, cache by + Dockerfile + agent-name hash). +4. Render the per-bottle Smolfile to `stage_dir/smolfile.toml`, + pinning command/env/`--outbound-localhost-only` + DNS allowlist. +5. Resolve the in-VM CA paths so launch knows where to copy + pipelock's CA after start. +6. Return a `SmolmachinesBottlePlan` carrying the slug, port map, + OCI archive path, Smolfile path, and host sidecar specs. + +`SmolmachinesBottleBackend.launch(plan)`: + +1. Start the four host sidecars in dependency order (pipelock → + egress → git-gate → supervise), bound to the plan's allocated + ports. Register teardown callbacks in reverse order. +2. `smolvm machine create --smolfile ` and + `smolvm machine start `. +3. Provisioning: CA install → prompt → skills → git → supervise + config, each via `smolvm machine exec` (analogous to + `docker exec`). +4. Yield a `SmolmachinesBottle` whose `exec_claude` / `exec` / + `cp_in` all funnel through `smolvm machine exec` / + `smolvm machine cp`. +5. Teardown: stop and remove the VM, then stop the sidecars (in + reverse start order). + +### Data model + +No manifest schema change. `bottles[]` continues to carry +`egress.allowlist`, `env`, `git`, `skills` references, etc.; the +smolmachines backend reads the same fields as the docker backend. +The DNS allowlist plumbed into the Smolfile is just +`bottle.egress.allowlist` re-encoded as TOML. + +The `BottleSpec` dataclass and the `Bottle` ABC do not change. + +### Selection wiring + +In `claude_bottle/backend/__init__.py`: + +```python +from .docker import DockerBottleBackend +from .smolmachines import SmolmachinesBottleBackend + +_BACKENDS: dict[str, BottleBackend[Any, Any]] = { + "docker": DockerBottleBackend(), + "smolmachines": SmolmachinesBottleBackend(), +} +``` + +The existing "unknown backend" `die()` path stays as-is. + +### External dependencies + +- `smolvm` CLI binary on `$PATH` (one new external dep, gated by + the preflight check). Pinned version policy is deferred to the + open questions; v1 reads `smolvm --version` and refuses to launch + outside a known-good range. +- No new Python packages. Subprocess + stdlib `tomllib`/`tomli_w` + for Smolfile authoring. (`tomli_w` is the only candidate + module; if it's not stdlib in the target Python, render TOML + by hand from a `dict[str, Any]` — Smolfile shape is small.) + +### Acceptance test plan + +- **Unit:** `tests/unit/test_smolfile.py` verifies the renderer + produces the expected TOML for a fixture bottle (allowlist → + DNS rules, env → `env =`, command line, outbound-localhost + flag). +- **Integration smoke:** `tests/integration/test_smolmachines_smoke.py` + with `prepare → launch → exec → teardown`, guarded by a + `smolvm` presence check + macOS / KVM platform check. +- **PRD 0022 re-run:** with `CLAUDE_BOTTLE_BACKEND=smolmachines`, + all five attack categories return sandbox-block markers and the + suite passes. The test code does not change beyond the env-var + flip — that's the contract the PRD 0022 abstraction was + designed for. + +## Sizing — into chunks + +1. **Backend skeleton + selection + Smolfile renderer.** Subpackage + layout, `_resolve_plan` stub that emits a TOML file but doesn't + launch anything, `_BACKENDS` registration, preflight `smolvm` + check. Unit test on the renderer. No VM bringup yet. +2. **VM lifecycle + OCI archive build.** `smolvm.py` subprocess + wrapper, prepare-time image build (existing Dockerfile → OCI + archive), launch path that creates + starts + stops a VM with + no sidecars wired. Smoke integration test: `exec("echo hi")` + inside a started VM. +3. **Host-side sidecar relocation.** `sidecars.py`: per-bottle + pipelock + egress + git-gate + supervise as host processes on + loopback. Port allocator. Teardown ordering. No provisioning + yet beyond what the sidecars need. +4. **Provisioning parity with Docker.** CA install via + `smolvm machine exec`, prompt/skills/.git copy-in, supervise + MCP config. End-to-end `start` works for a real agent manifest. +5. **PRD 0022 sandbox-escape suite green.** Skip-guard update, + small adjustments to test helpers if any (the test uses + `bottle.exec(script)` and inspects `returncode` + body for + sandbox markers — should be transport-agnostic, but verify). + Document the macOS-only scope in README. + +## Open questions + +1. **Sidecar locality: host process vs in-VM init.** This PRD + defaults to host-process sidecars (proposed design above). The + alternative — bake pipelock + egress + git-gate + supervise + into the OCI image and start them via init in the same VM — + would simplify port plumbing (the agent reaches sidecars over + localhost inside the VM, not over TSI) but expands the trust + boundary of the agent VM. Default A unless someone identifies + a TSI loopback edge case during chunk 3. +2. **`smolvm` install policy.** Pin via brew formula version, or + build-from-source step, or vendored binary checked into the + repo. v1 most likely runs `smolvm --version` at preflight and + accepts a documented range; vendoring is heavier but reduces + "works on my Mac" drift. +3. **CA install inside the OCI overlay.** Two paths: bake at + prepare time (one OCI archive per CA fingerprint, big cache + key) vs. inject at start time via `smolvm machine exec` after + the VM is up. PRD 0006 chose the runtime path for Docker + (docker-cp + `update-ca-certificates`); smolvm has the same + shape via `machine exec`. Default to runtime injection unless + it conflicts with `--outbound-localhost-only` start order. +4. **DNS filter granularity.** smolmachines's vsock-6002 filter + accepts an allowlist of hostnames; we want to enforce both + "agent can only resolve names on the bottle's allowlist" *and* + "agent can only egress via TSI to 127.0.0.1." Confirm + empirically (smoke test in chunk 2) that the allowlist applies + to *guest-initiated* DNS only and doesn't accidentally NXDOMAIN + the host-side pipelock's upstream lookups. +5. **`bottle.exec(script)` exit-code fidelity.** The PRD 0022 test + suite reads `returncode` + stdout + stderr from + `ExecResult`. Confirm `smolvm machine exec` propagates exit + codes and separated streams — the research note's + "external integration is the CLI" implies yes, but the + embedded SDK bug it flagged suggests we should verify before + coding around it. +6. **CI gating.** Gitea's act_runner is Linux without nested KVM, + so smolmachines integration tests will skip there for the same + structural reason the Docker bringup tests do (no real + isolation primitive available on the runner). The skip + predicate becomes `not (smolvm_available() and + (platform.system() == "Darwin" or kvm_available()))`. CI + coverage for this backend will come from local runs on the + maintainer's macOS host until a Darwin runner is wired up; + ack that as a known gap. +7. **Active bottle discovery.** Docker uses container labels to + enumerate active bottles (`list_active` queries the daemon). + smolmachines's enumeration story is `smolvm machine list`; the + plan is to mirror the label scheme via Smolfile metadata + (`labels = { "claude-bottle" = "1" }`-style entries, if the + format supports it; otherwise via a deterministic name prefix + `claude-bottle-`). + +## References + +- `docs/research/smolmachines-as-vm-backend.md` — primary research + note recommending this adoption; PRD 0023's design hypothesis. +- `docs/research/agent-vm-isolation.md` — the broader microVM / + gvproxy / pipelock landscape this PRD lands inside of. +- `docs/research/agent-sandbox-landscape.md` — identifies + `"runtime": "microvm"`-style opt-in as the borrowable idea; + smolmachines is the concrete implementation. +- PRD 0003 (`docs/prds/0003-bottle-backend-abstraction.md`) — the + backend abstraction this PRD is the first non-Docker consumer + of. +- PRD 0017 (`docs/prds/0017-egress-proxy-via-mitmproxy.md`) — the + egress sidecar the host-side relocation reuses verbatim, only + with a different transport. +- PRD 0022 + (`docs/prds/0022-sandbox-escape-integration-test.md`) — the + acceptance gate for this PRD; the suite already runs through + `get_bottle_backend()` so the env-var flip is the only change + needed to exercise the smolmachines path. From 041da1d7af99712599a0b91f8a28c5a5be47a346 Mon Sep 17 00:00:00 2001 From: didericis Date: Tue, 26 May 2026 23:41:32 -0400 Subject: [PATCH 2/3] docs(prd-0023): make gvproxy the network primitive; reject TSI TSI's --outbound-localhost-only is permissive on all of 127.0.0.0/8 with no destination-port filter, so any host loopback service (local Postgres, IDE plugins, another bottle's sidecar) is reachable from the guest. That's the wrong default for the malicious-agent threat model. Reworked the network design around gvproxy + VFKT unixgram attachment: the guest gets a virtio-net device, gvproxy is the userspace TCP/IP stack on the host side, and the only thing reachable from the guest is the explicit port-forward list (typically just pipelock). Host LAN, host loopback, and the public internet directly are gone by construction. VMM choice (smolmachines vs PyObjC + Virtualization.framework) is an open question contingent on whether libkrun's virtio-net mode lets us point at a custom unixgram socket. Backend name stays "smolmachines" either way per the original spec. Co-Authored-By: Claude Opus 4.7 --- docs/prds/0023-smolmachines-backend.md | 421 +++++++++++++++++-------- 1 file changed, 286 insertions(+), 135 deletions(-) diff --git a/docs/prds/0023-smolmachines-backend.md b/docs/prds/0023-smolmachines-backend.md index dd602ff..441005b 100644 --- a/docs/prds/0023-smolmachines-backend.md +++ b/docs/prds/0023-smolmachines-backend.md @@ -6,31 +6,54 @@ ## Summary -Ship a second concrete `BottleBackend` — `SmolmachinesBottleBackend`, -selected via `CLAUDE_BOTTLE_BACKEND=smolmachines` — that runs a -bottle inside a per-agent libkrun microVM on macOS (and KVM on Linux, -opportunistically). The egress topology moves out of an internal -Docker network and onto libkrun's TSI ("Transport Socket Interface") -allowlist plus a host-side pipelock/egress/git-gate/supervise stack -listening on per-bottle loopback ports. The Docker backend ships -unchanged; this is opt-in via the existing env-var selector. +Ship a second concrete `BottleBackend` — +`SmolmachinesBottleBackend`, selected via +`CLAUDE_BOTTLE_BACKEND=smolmachines` — that runs each bottle inside +a per-agent microVM on macOS. The egress topology is enforced by +**gvproxy** (gvisor-tap-vsock), a userspace TCP/IP stack the guest's +virtio-net device is wired into via `VZFileHandleNetworkDeviceAttachment`. +gvproxy's only outbound configuration is an explicit per-bottle port +forward to a host-side pipelock; everything else — the host's LAN, +the host's loopback services, the public internet — is unreachable +from the guest by construction. pipelock + egress + git-gate + +supervise stay as host-side processes on per-bottle loopback ports, +reached *only* through gvproxy's forwarded ports. -The acceptance gate is PRD 0022's `tests/integration/test_sandbox_escape.py` -running green against `CLAUDE_BOTTLE_BACKEND=smolmachines`. +This explicitly rejects libkrun's TSI ("Transport Socket Interface") +allowlist as the network primitive. TSI's `--outbound-localhost-only` +is permissive on the entire `127.0.0.0/8` range with no +destination-port filter — the agent can dial any host-side service +bound to loopback (a local Postgres, an IDE plugin, a different +bottle's pipelock). That's the wrong default for a malicious-agent +threat model; see "Why gvproxy, not TSI" below. + +The Docker backend ships unchanged; this is opt-in via the existing +env-var selector. The acceptance gate is PRD 0022's +`tests/integration/test_sandbox_escape.py` running green against +`CLAUDE_BOTTLE_BACKEND=smolmachines`. ## Problem `agent-vm-isolation.md` argues for hardware-isolated microVMs over container-based bottles on macOS; `smolmachines-as-vm-backend.md` -concludes that smolmachines is the most plausible concrete VMM for -this project. Today, the only backend in the registry is Docker +evaluates smolmachines as the lifecycle wrapper. Today, the only +backend in the registry is Docker (`claude_bottle/backend/__init__.py:_BACKENDS = {"docker": ...}`), -and three things motivate a second one now: +and four things motivate a second one now: +- **Network reach beyond pipelock.** The threat model is a malicious + agent attempting to dial something on the operator's *local + network* (`192.168.x.x` services, the home router, a coworker's + laptop on the same Wi-Fi) or *host's loopback* (a local database, + an editor plugin, another bottle's sidecar) without traversing + pipelock. The Docker backend's `--internal` network blocks the + first; nothing in the current stack blocks the second cleanly. + This PRD's gvproxy-based design closes both gaps: the guest can + only reach the explicit port-forward list, period. - **Isolation ceiling.** On macOS the Docker backend's agent - container shares Docker Desktop's host VM with every other bottle. - Container escape from claude-code lands the agent inside that - shared VM. A per-bottle libkrun microVM gets hardware page tables + container shares Docker Desktop's host VM with every other + bottle. Container escape from claude-code lands the agent inside + that shared VM. A per-bottle microVM gets hardware page tables via `Hypervisor.framework`; cross-bottle isolation becomes enforced by the CPU's MMU instead of namespace bookkeeping. - **PRD 0022 is backend-agnostic by design** but currently only @@ -41,15 +64,36 @@ and three things motivate a second one now: unproven. - **CI carve-outs.** Most bottle-bringup integration tests skip under `GITEA_ACTIONS=true` because act_runner shares the host - Docker socket but not the host filesystem. A smolmachines path + Docker socket but not the host filesystem. A microVM path doesn't share that constraint shape (it has its own, but different), so adding the backend forces the abstraction to be clean in places where Docker-specific assumptions have been tolerated. -The smolmachines research note's `## Recommendation` ("adopt -smolmachines as the bottle VM backend on macOS; keep pipelock DIY") -is the design hypothesis under test here. +## Why gvproxy, not TSI + +libkrun's TSI hijacks guest socket syscalls inside the VMM and +opens the actual sockets from the host process, with a CIDR +allowlist gate. That works fine for blocking LAN reach (don't +allowlist `192.168.0.0/16`, agent can't dial it). But TSI's +`--outbound-localhost-only` permits the *entire* `127.0.0.0/8` +range across all ports — there is no destination-port filter at +the TSI layer (`smolmachines-as-vm-backend.md` flags this in the +"`--allow-host` semantics" caveat). For our threat model that +means any host-loopback service is reachable from the guest. + +gvproxy implements a full userspace TCP/IP stack on the host side +of a `VZFileHandleNetworkDeviceAttachment` unixgram socket. The +guest has a real virtio-net device; gvproxy is its gateway. The +guest can only reach what gvproxy is configured to forward — +typically a single port forward to the per-bottle pipelock — +and DNS resolves NXDOMAIN by default. There is no "permissive +loopback" mode to mis-configure; if it's not in `port_forwards`, +the guest cannot reach it. + +That property — *explicit allowlist by port forward, not CIDR* — +is the load-bearing reason this PRD chooses gvproxy. TSI shows up +once more in this doc, under Non-goals, where it is closed off. ## Goals / Success Criteria @@ -84,13 +128,25 @@ The feature is **done** when all of the following ship: - Per-bottle Smolfile generation: a runtime-rendered TOML written to the bottle's stage dir, analogous to the compose file the Docker backend writes today. The Smolfile pins `command`, - `env`, `--outbound-localhost-only`, and the per-bottle DNS - allowlist. + `env`, and a virtio-net device backed by a unixgram socket + pointed at the per-bottle gvproxy. There is no TSI + `--allow-cidr` / `--outbound-localhost-only` / `--allow-host` + in the Smolfile — TSI is not used. +- Per-bottle gvproxy: one `gvproxy` process per bottle, started + before the VM, listening on a unixgram socket the VM's + virtio-net device hooks into. The gvproxy config has exactly + one `port_forwards` entry — gateway-port to the per-bottle + pipelock's host port — and a DNS section that resolves only + `proxy.internal`. Every other hostname returns NXDOMAIN; every + other destination is unreachable. - Host-side sidecar relocation: pipelock, egress, git-gate, and supervise each run as host processes (one set per bottle), bound to `127.0.0.1` on per-bottle dynamically-allocated ports. The agent's environment carries the resolved URLs (e.g. - `HTTPS_PROXY=http://127.0.0.1:`). + `HTTPS_PROXY=http://proxy.internal:`). + Only pipelock is exposed through gvproxy; egress / git-gate / + supervise are chained *behind* pipelock on the host side and + are not reachable directly from the guest. - The agent guest image is produced from the existing `Dockerfile` (or a thin variant), exported as an OCI archive, and consumed by `smolvm machine create`. The image build step is part of `prepare`, @@ -117,6 +173,15 @@ The feature is **done** when all of the following ship: value of `CLAUDE_BOTTLE_BACKEND`; smolmachines is strictly opt-in until it has been load-bearing on at least one operator's workflow for a release cycle. +- **No TSI for network policy.** libkrun's TSI mode is rejected + for this backend — it lacks per-port filtering on `127.0.0.0/8` + and would expose every host-loopback service to the guest. The + Smolfile must select libkrun's virtio-net mode and attach to + the per-bottle gvproxy unixgram socket; if that combination is + not supported by the pinned smolmachines version (see open + question 1), the implementation falls back to driving + Virtualization.framework directly via PyObjC and reuses the + same gvproxy attachment. - **No host bind mounts.** The smolmachines research note flagged that `-v HOST:GUEST` mounts via virtiofs would defeat the isolation goal. The manifest already has no concept of host @@ -152,12 +217,18 @@ The feature is **done** when all of the following ship: - Per-bottle CA install path: the egress sidecar's CA cert lands inside the microVM via `smolvm machine exec` after start (analogous to the existing `provision_ca` for Docker). -- DNS allowlist plumbing: every host in `bottle.egress.allowlist` - goes into the Smolfile's DNS filter section (vsock port 6002), - so the VMM-layer DNS filter and the bottle's policy stay in - sync — agent can't `dig` its way out via raw IP literals (TSI - + CIDR allowlist enforces this; DNS filter denies hostname - resolution). +- gvproxy lifecycle: per-bottle `gvproxy` started by the backend + before VM bringup, torn down after VM teardown, configured with + one `port_forwards` entry (gateway → host pipelock port) and a + DNS section that resolves only `proxy.internal`. Subnet and + gateway IP are derived from the bottle slug so two concurrent + bottles don't collide. +- DNS policy: the bottle's `egress.allowlist` does *not* go into + gvproxy's DNS — the agent resolves only `proxy.internal`, and + pipelock on the host enforces the egress allowlist against + the actual upstream connect target. This keeps the DNS-exfil + attack (PRD 0022 test 4) blocked because gvproxy answers + NXDOMAIN for every name except `proxy.internal`. - Preflight `smolvm` check: if the user selects this backend and `smolvm` isn't on `$PATH`, die with an install pointer (brew tap + version pin TBD in implementation; see open question 3). @@ -200,8 +271,10 @@ claude_bottle/backend/smolmachines/ launch.py @contextmanager launch(plan) -> SmolmachinesBottle cleanup.py prepare_cleanup / cleanup / list_active smolfile.py bottle_plan_to_smolfile(...) -> dict + render + gvproxy.py per-bottle gvproxy config render + process lifecycle sidecars.py host-side pipelock/egress/git-gate/supervise lifecycle smolvm.py thin subprocess wrapper: machine create/start/exec/stop + vfkit_attach.py VZFileHandleNetworkDeviceAttachment + VFKT handshake util.py slugify, port allocation, OCI archive helpers provision/ ca.py, prompt.py, skills.py, git.py, supervise.py ``` @@ -209,39 +282,65 @@ claude_bottle/backend/smolmachines/ ### Network + egress topology ``` - ┌── macOS host ─────────────────────────────────────────────┐ - │ │ - │ ┌── per-bottle host sidecars (one set per microVM) ─┐ │ - │ │ pipelock 127.0.0.1: │ │ - │ │ egress 127.0.0.1: │ │ - │ │ git-gate 127.0.0.1: │ │ - │ │ supervise 127.0.0.1: │ │ - │ └───────────────────────────────────────────────────┘ │ - │ ▲ │ - │ │ TSI passthrough (localhost) │ - │ │ │ - │ ┌── libkrun microVM (per bottle) ───────────────────┐ │ - │ │ env: HTTPS_PROXY=http://127.0.0.1: │ │ - │ │ EGRESS_URL=http://127.0.0.1: │ │ - │ │ GIT_GATE_URL=http://127.0.0.1: │ │ - │ │ MCP_SUPERVISE_URL=http://127.0.0.1: │ │ - │ │ --outbound-localhost-only │ │ - │ │ DNS filter (vsock:6002) → host allowlist │ │ - │ └───────────────────────────────────────────────────┘ │ - │ │ - └───────────────────────────────────────────────────────────┘ + ┌── macOS host ─────────────────────────────────────────────────────┐ + │ │ + │ ┌── per-bottle sidecar chain (one set per microVM) ────┐ │ + │ │ agent ──HTTPS_PROXY──► pipelock ──► egress ──► internet │ + │ │ 127.0.0.1:p1 (DLP) (MITM, │ + │ │ auth-inject) │ + │ │ │ + │ │ git push ──► git-gate ──► upstream │ + │ │ 127.0.0.1:p3 (gitleaks) │ + │ │ │ + │ │ MCP ──► supervise 127.0.0.1:p4 │ + │ └────────────────────────────────────────────────────────────────┘ + │ ▲ host TCP, reached via gvproxy port-forward │ + │ │ │ + │ ┌── gvproxy (per bottle) ─────────────────────────────┐ │ + │ │ subnet: 192.168.127.X/24 (X derived from slug) │ │ + │ │ gateway: 192.168.127.X.1 │ │ + │ │ port_forwards: │ │ + │ │ - gateway 8888 → host 127.0.0.1: │ │ + │ │ # nothing else │ │ + │ │ DNS: proxy.internal → gateway IP; * → NXDOMAIN │ │ + │ └─────────────────────────────────────────────────────┘ │ + │ ▲ unixgram socket (VFKT handshake) │ + │ │ │ + │ ┌── microVM (per bottle) ─────────────────────────────┐ │ + │ │ virtio-net device backed by VZFileHandle... │ │ + │ │ env: HTTPS_PROXY=http://proxy.internal:8888 │ │ + │ │ GIT_GATE_URL=http://proxy.internal:8889 │ │ + │ │ MCP_SUPERVISE_URL=http://proxy.internal:8890 │ │ + │ │ no other host visible │ │ + │ └─────────────────────────────────────────────────────┘ │ + │ │ + └───────────────────────────────────────────────────────────────────┘ ``` +What the guest can reach, exhaustively: **only `proxy.internal` +on the gateway-port set we configured.** Everything else — +host LAN, host loopback (Postgres, IDE plugins, other bottles' +sidecars), public internet directly — is gone, enforced at the +gvproxy userspace stack rather than relying on guest cooperation. + Two changes vs. the Docker backend: 1. **Sidecars are host processes, not sibling containers.** No - internal Docker network; isolation comes from TSI plus the - per-bottle loopback port set. -2. **The "internal" allowlist becomes localhost-only.** Egress out - to the public internet still happens through pipelock + egress - — the same scanning + DLP + auth-injection chain — but the - agent's first hop is `127.0.0.1:` reached via TSI, not a - sidecar's IP on a Docker-managed bridge. + internal Docker network. The isolation primitive is gvproxy's + explicit port-forward list, not Docker's `--internal` flag. +2. **The agent's first hop is `proxy.internal`, not a sidecar's + container hostname.** Egress out to the public internet still + happens through pipelock + egress — same scanning + DLP + + auth-injection chain — but the first hop crosses a userspace + TCP/IP stack we own, not a Docker-managed bridge. + +The chain `agent → pipelock → egress → internet` collapses on +the host side: pipelock listens on 127.0.0.1:p1, makes its +upstream connect against egress at 127.0.0.1:p2, which makes its +upstream connect against the public internet. git-gate and +supervise are separate gateway ports if and only if the bottle +uses them — otherwise they're omitted from gvproxy's +`port_forwards`, narrowing the attack surface further. ### Lifecycle @@ -249,40 +348,56 @@ Two changes vs. the Docker backend: 1. Cross-backend validation via `BottleBackend._validate` (skills, git identity files). -2. Allocate four loopback ports (bind, get free port, release; - record on plan). +2. Allocate host loopback ports for each sidecar the bottle uses + (pipelock always; egress / git-gate / supervise conditional on + manifest). 3. Resolve the agent OCI archive path (build if missing, cache by Dockerfile + agent-name hash). -4. Render the per-bottle Smolfile to `stage_dir/smolfile.toml`, - pinning command/env/`--outbound-localhost-only` + DNS allowlist. -5. Resolve the in-VM CA paths so launch knows where to copy +4. Pick a per-bottle gvproxy subnet (e.g. `192.168.127.X/24` where + `X` is derived from the slug) and render + `stage_dir/gvproxy.yaml`: one DNS entry for `proxy.internal`, + one `port_forwards` entry per active sidecar (gateway port → + host loopback port). +5. Render the per-bottle Smolfile to `stage_dir/smolfile.toml`, + pinning command / env / a virtio-net device backed by the + gvproxy unixgram socket path. No TSI flags. +6. Resolve the in-VM CA paths so launch knows where to copy pipelock's CA after start. -6. Return a `SmolmachinesBottlePlan` carrying the slug, port map, - OCI archive path, Smolfile path, and host sidecar specs. +7. Return a `SmolmachinesBottlePlan` carrying the slug, port map, + OCI archive path, Smolfile path, gvproxy config path, and + host sidecar specs. `SmolmachinesBottleBackend.launch(plan)`: -1. Start the four host sidecars in dependency order (pipelock → - egress → git-gate → supervise), bound to the plan's allocated - ports. Register teardown callbacks in reverse order. -2. `smolvm machine create --smolfile ` and - `smolvm machine start `. -3. Provisioning: CA install → prompt → skills → git → supervise +1. Start host sidecars in dependency order (egress → pipelock → + git-gate → supervise — egress before pipelock so pipelock's + upstream resolves; pipelock is the only one exposed through + gvproxy). Register teardown callbacks in reverse order. +2. Start the per-bottle `gvproxy` against the unixgram socket + path the Smolfile references. Wait for the socket to appear + (the spike-style poll loop from `agent-vm-isolation.md`). +3. `smolvm machine create --smolfile ` and + `smolvm machine start `. The Smolfile's virtio-net + device handshakes (`VFKT` magic) with gvproxy on start. +4. Provisioning: CA install → prompt → skills → git → supervise config, each via `smolvm machine exec` (analogous to `docker exec`). -4. Yield a `SmolmachinesBottle` whose `exec_claude` / `exec` / +5. Yield a `SmolmachinesBottle` whose `exec_claude` / `exec` / `cp_in` all funnel through `smolvm machine exec` / `smolvm machine cp`. -5. Teardown: stop and remove the VM, then stop the sidecars (in - reverse start order). +6. Teardown: stop and remove the VM → stop gvproxy → stop + sidecars (in reverse start order). ### Data model No manifest schema change. `bottles[]` continues to carry `egress.allowlist`, `env`, `git`, `skills` references, etc.; the smolmachines backend reads the same fields as the docker backend. -The DNS allowlist plumbed into the Smolfile is just -`bottle.egress.allowlist` re-encoded as TOML. +`egress.allowlist` is enforced by pipelock on the host side +(unchanged from the docker backend); gvproxy's DNS resolves only +`proxy.internal` regardless of the allowlist's contents, so an +agent that bypasses pipelock by raw IP cannot resolve any name +gvproxy doesn't know about. The `BottleSpec` dataclass and the `Bottle` ABC do not change. @@ -308,20 +423,36 @@ The existing "unknown backend" `die()` path stays as-is. the preflight check). Pinned version policy is deferred to the open questions; v1 reads `smolvm --version` and refuses to launch outside a known-good range. -- No new Python packages. Subprocess + stdlib `tomllib`/`tomli_w` - for Smolfile authoring. (`tomli_w` is the only candidate - module; if it's not stdlib in the target Python, render TOML - by hand from a `dict[str, Any]` — Smolfile shape is small.) +- `gvproxy` binary on `$PATH` + (`go install github.com/containers/gvisor-tap-vsock/cmd/gvproxy@latest`, + or vendored). Same preflight pattern as `smolvm`. +- `pyobjc-framework-Virtualization` *only* if smolmachines does + not expose a way to attach virtio-net to a unixgram socket and + we fall back to driving Virtualization.framework directly (see + open question 1). Default path is "no PyObjC needed." +- No new pure-Python packages. Subprocess + stdlib `tomllib` for + Smolfile authoring; the gvproxy YAML is small enough to render + by hand from a `dict[str, Any]`. ### Acceptance test plan -- **Unit:** `tests/unit/test_smolfile.py` verifies the renderer - produces the expected TOML for a fixture bottle (allowlist → - DNS rules, env → `env =`, command line, outbound-localhost - flag). +- **Unit (smolfile):** `tests/unit/test_smolfile.py` verifies the + renderer produces the expected TOML for a fixture bottle — + command line, env entries, virtio-net device referencing the + expected unixgram socket path, no TSI flags. +- **Unit (gvproxy config):** `tests/unit/test_gvproxy_config.py` + verifies the per-bottle YAML has exactly one DNS entry + (`proxy.internal`), one `port_forwards` entry per active + sidecar pointed at the resolved host loopback port, and a + per-bottle subnet/gateway derived from the slug. - **Integration smoke:** `tests/integration/test_smolmachines_smoke.py` with `prepare → launch → exec → teardown`, guarded by a - `smolvm` presence check + macOS / KVM platform check. + `smolvm` + `gvproxy` presence check + macOS / KVM platform check. +- **Localhost-reach probe:** a focused integration test that + brings up a bottle, has the host bind a test service on + `127.0.0.1:`, and asserts the in-bottle agent + cannot connect to it. This is the regression test for the + exact gap that motivated choosing gvproxy over TSI. - **PRD 0022 re-run:** with `CLAUDE_BOTTLE_BACKEND=smolmachines`, all five attack categories return sandbox-block markers and the suite passes. The test code does not change beyond the env-var @@ -330,19 +461,24 @@ The existing "unknown backend" `die()` path stays as-is. ## Sizing — into chunks -1. **Backend skeleton + selection + Smolfile renderer.** Subpackage - layout, `_resolve_plan` stub that emits a TOML file but doesn't - launch anything, `_BACKENDS` registration, preflight `smolvm` - check. Unit test on the renderer. No VM bringup yet. -2. **VM lifecycle + OCI archive build.** `smolvm.py` subprocess - wrapper, prepare-time image build (existing Dockerfile → OCI - archive), launch path that creates + starts + stops a VM with - no sidecars wired. Smoke integration test: `exec("echo hi")` - inside a started VM. +1. **Backend skeleton + selection + Smolfile + gvproxy renderers.** + Subpackage layout, `_resolve_plan` stub that emits both a + TOML Smolfile and a gvproxy YAML but doesn't launch anything, + `_BACKENDS` registration, preflight `smolvm` + `gvproxy` + checks. Unit tests on both renderers. No VM bringup yet. +2. **gvproxy + VM lifecycle + OCI archive build.** `smolvm.py` + and `gvproxy.py` subprocess wrappers, prepare-time image + build (existing Dockerfile → OCI archive), launch path that + starts gvproxy, brings up the VM attached to gvproxy's socket + via VFKT handshake, exec into the VM, tear everything down. + Smoke integration test: `exec("echo hi")` inside a started + VM. Includes the localhost-reach probe test from the + acceptance plan. 3. **Host-side sidecar relocation.** `sidecars.py`: per-bottle pipelock + egress + git-gate + supervise as host processes on - loopback. Port allocator. Teardown ordering. No provisioning - yet beyond what the sidecars need. + loopback, with gvproxy `port_forwards` wired only for the + sidecars the bottle actually uses. Port allocator. Teardown + ordering. No provisioning yet beyond what the sidecars need. 4. **Provisioning parity with Docker.** CA install via `smolvm machine exec`, prompt/skills/.git copy-in, supervise MCP config. End-to-end `start` works for a real agent manifest. @@ -354,63 +490,78 @@ The existing "unknown backend" `die()` path stays as-is. ## Open questions -1. **Sidecar locality: host process vs in-VM init.** This PRD - defaults to host-process sidecars (proposed design above). The - alternative — bake pipelock + egress + git-gate + supervise - into the OCI image and start them via init in the same VM — - would simplify port plumbing (the agent reaches sidecars over - localhost inside the VM, not over TSI) but expands the trust - boundary of the agent VM. Default A unless someone identifies - a TSI loopback edge case during chunk 3. -2. **`smolvm` install policy.** Pin via brew formula version, or - build-from-source step, or vendored binary checked into the - repo. v1 most likely runs `smolvm --version` at preflight and - accepts a documented range; vendoring is heavier but reduces - "works on my Mac" drift. +1. **VMM choice: smolmachines vs PyObjC + Virtualization.framework.** + The network design requires libkrun's virtio-net mode attached + to a unixgram socket (so gvproxy is the gateway). The + smolmachines research note says libkrun *has* a virtio-net + mode but says it "does not support policy" — meaning libkrun + itself enforces no allowlist in that mode, which is exactly + what we want (gvproxy is the policy). What's unverified is + whether the Smolfile surface lets us point virtio-net at a + custom unixgram socket. If yes: this is a smolmachines backend + verbatim. If no: chunk 2 drops `smolvm` and drives + `Virtualization.framework` via PyObjC directly (the recipe in + `agent-vm-isolation.md` § "gvisor-tap-vsock + PyObjC + + Pipelock"), keeping the backend name "smolmachines" because + the operator-facing UX is unchanged. Resolve in chunk 1 via a + spike against the pinned smolmachines version. +2. **`smolvm` + `gvproxy` install policy.** Pin via brew / + `go install` versions, or vendor binaries in the repo. v1 + likely runs `smolvm --version` / `gvproxy --help` at preflight + and accepts a documented range; vendoring is heavier but + reduces "works on my Mac" drift. 3. **CA install inside the OCI overlay.** Two paths: bake at prepare time (one OCI archive per CA fingerprint, big cache key) vs. inject at start time via `smolvm machine exec` after the VM is up. PRD 0006 chose the runtime path for Docker (docker-cp + `update-ca-certificates`); smolvm has the same shape via `machine exec`. Default to runtime injection unless - it conflicts with `--outbound-localhost-only` start order. -4. **DNS filter granularity.** smolmachines's vsock-6002 filter - accepts an allowlist of hostnames; we want to enforce both - "agent can only resolve names on the bottle's allowlist" *and* - "agent can only egress via TSI to 127.0.0.1." Confirm - empirically (smoke test in chunk 2) that the allowlist applies - to *guest-initiated* DNS only and doesn't accidentally NXDOMAIN - the host-side pipelock's upstream lookups. + it conflicts with VM start order. +4. **gvproxy subnet collision.** Two concurrent bottles must not + land on the same `192.168.127.X/24` subnet — they'd both want + the same gateway IP. Derive the third octet from a hash of + the slug (mod 254, skip the docker-default 17), and at launch + time confirm the subnet isn't already in use by another + bottle's gvproxy. Resolve the hash-collision policy in + chunk 2. 5. **`bottle.exec(script)` exit-code fidelity.** The PRD 0022 test suite reads `returncode` + stdout + stderr from - `ExecResult`. Confirm `smolvm machine exec` propagates exit - codes and separated streams — the research note's - "external integration is the CLI" implies yes, but the - embedded SDK bug it flagged suggests we should verify before - coding around it. + `ExecResult`. Confirm the VM-exec path (`smolvm machine exec` + or its PyObjC equivalent) propagates exit codes and separated + streams. The research note's "external integration is the CLI" + implies yes, but the embedded SDK bug it flagged suggests we + should verify before coding around it. 6. **CI gating.** Gitea's act_runner is Linux without nested KVM, - so smolmachines integration tests will skip there for the same - structural reason the Docker bringup tests do (no real + so this backend's integration tests will skip there for the + same structural reason the Docker bringup tests do (no real isolation primitive available on the runner). The skip - predicate becomes `not (smolvm_available() and - (platform.system() == "Darwin" or kvm_available()))`. CI - coverage for this backend will come from local runs on the - maintainer's macOS host until a Darwin runner is wired up; - ack that as a known gap. + predicate becomes `not (smolvm_available() and gvproxy_available() + and platform.system() == "Darwin")`. CI coverage for this + backend will come from local runs on the maintainer's macOS + host until a Darwin runner is wired up; ack that as a known + gap. 7. **Active bottle discovery.** Docker uses container labels to enumerate active bottles (`list_active` queries the daemon). - smolmachines's enumeration story is `smolvm machine list`; the - plan is to mirror the label scheme via Smolfile metadata + The microVM enumeration story is `smolvm machine list` + (or the PyObjC backend's own bookkeeping); the plan is to + mirror the label scheme via Smolfile metadata (`labels = { "claude-bottle" = "1" }`-style entries, if the format supports it; otherwise via a deterministic name prefix - `claude-bottle-`). + `claude-bottle-` + on-disk metadata under + `state//`). ## References -- `docs/research/smolmachines-as-vm-backend.md` — primary research - note recommending this adoption; PRD 0023's design hypothesis. -- `docs/research/agent-vm-isolation.md` — the broader microVM / - gvproxy / pipelock landscape this PRD lands inside of. +- `docs/research/agent-vm-isolation.md` — primary reference for + the gvproxy + `VZFileHandleNetworkDeviceAttachment` network + attachment used here. The "Full Setup: gvisor-tap-vsock + + PyObjC + Pipelock" section is the recipe the PyObjC fallback + in open question 1 would adopt verbatim. +- `docs/research/smolmachines-as-vm-backend.md` — evaluation of + smolmachines as the VM lifecycle wrapper. This PRD diverges + from its conclusion on the *network* primitive (rejecting TSI + in favor of gvproxy) but keeps its VM-lifecycle conclusion + conditional on the libkrun-virtio-net spike in open question 1. - `docs/research/agent-sandbox-landscape.md` — identifies `"runtime": "microvm"`-style opt-in as the borrowable idea; smolmachines is the concrete implementation. From 4e00430c6e5b8a4c4359f3cef9d2830422365022 Mon Sep 17 00:00:00 2001 From: didericis Date: Tue, 26 May 2026 23:51:57 -0400 Subject: [PATCH 3/3] docs(prd-0023): consume PRD 0024's bundle as the single sidecar Replace the four host-side sidecar processes (pipelock + egress + git-gate + supervise) with a single bundled container per bottle, defined in PRD 0024 and consumed here. egress is internal to the bundle as pipelock's upstream; only pipelock, git-gate, and supervise are externally addressable, and only when the bottle uses them. gvproxy port_forwards collapse from one-per-process to one-per- external-port, all pointing into the one bundle container. Sizing: chunk 3 becomes "sidecar bundle lifecycle" and depends on PRD 0024 having landed. Co-Authored-By: Claude Opus 4.7 --- docs/prds/0023-smolmachines-backend.md | 180 ++++++++++++++----------- 1 file changed, 104 insertions(+), 76 deletions(-) diff --git a/docs/prds/0023-smolmachines-backend.md b/docs/prds/0023-smolmachines-backend.md index 441005b..502cc4f 100644 --- a/docs/prds/0023-smolmachines-backend.md +++ b/docs/prds/0023-smolmachines-backend.md @@ -12,12 +12,19 @@ Ship a second concrete `BottleBackend` — a per-agent microVM on macOS. The egress topology is enforced by **gvproxy** (gvisor-tap-vsock), a userspace TCP/IP stack the guest's virtio-net device is wired into via `VZFileHandleNetworkDeviceAttachment`. -gvproxy's only outbound configuration is an explicit per-bottle port -forward to a host-side pipelock; everything else — the host's LAN, -the host's loopback services, the public internet — is unreachable -from the guest by construction. pipelock + egress + git-gate + -supervise stay as host-side processes on per-bottle loopback ports, -reached *only* through gvproxy's forwarded ports. +gvproxy's only outbound configuration is an explicit per-bottle +port-forward set into a **single per-bottle sidecar container** that +bundles pipelock + egress + git-gate + supervise behind one supervised +init. Everything else — the host's LAN, the host's loopback +services, the public internet — is unreachable from the guest by +construction. + +The sidecar bundle is the same image PRD 0024 introduces for the +docker backend; this PRD consumes it. Inside the bundle, egress is +pipelock's internal upstream over localhost and is not exposed +externally. gvproxy port-forwards three external ports into the +bundle: pipelock (for `HTTPS_PROXY`), git-gate (for git push), and +supervise (for MCP). This explicitly rejects libkrun's TSI ("Transport Socket Interface") allowlist as the network primitive. TSI's `--outbound-localhost-only` @@ -134,19 +141,21 @@ The feature is **done** when all of the following ship: in the Smolfile — TSI is not used. - Per-bottle gvproxy: one `gvproxy` process per bottle, started before the VM, listening on a unixgram socket the VM's - virtio-net device hooks into. The gvproxy config has exactly - one `port_forwards` entry — gateway-port to the per-bottle - pipelock's host port — and a DNS section that resolves only - `proxy.internal`. Every other hostname returns NXDOMAIN; every - other destination is unreachable. -- Host-side sidecar relocation: pipelock, egress, git-gate, and - supervise each run as host processes (one set per bottle), - bound to `127.0.0.1` on per-bottle dynamically-allocated ports. - The agent's environment carries the resolved URLs (e.g. + virtio-net device hooks into. The gvproxy config has up to + three `port_forwards` entries (pipelock / git-gate / supervise + — git-gate and supervise only when the bottle uses them) all + pointing at the per-bottle sidecar bundle's exposed ports, plus + a DNS section that resolves only `proxy.internal`. Every other + hostname returns NXDOMAIN; every other destination is + unreachable. +- Per-bottle sidecar bundle: one container per bottle running the + bundle image defined in PRD 0024. The bundle exposes up to + three host ports (pipelock for `HTTPS_PROXY`, git-gate for git + push, supervise for MCP), bound to `127.0.0.1` on dynamically + allocated ports. egress runs *inside* the bundle as pipelock's + upstream over localhost and is not exposed externally. The + agent's environment carries the resolved URLs (e.g. `HTTPS_PROXY=http://proxy.internal:`). - Only pipelock is exposed through gvproxy; egress / git-gate / - supervise are chained *behind* pipelock on the host side and - are not reachable directly from the guest. - The agent guest image is produced from the existing `Dockerfile` (or a thin variant), exported as an OCI archive, and consumed by `smolvm machine create`. The image build step is part of `prepare`, @@ -209,17 +218,19 @@ The feature is **done** when all of the following ship: full set of `BottleBackend` overrides. - Smolfile generator (TOML), analogous to `backend/docker/compose.py`'s `bottle_plan_to_compose`. -- A host-side sidecar process manager that owns the lifecycle of - pipelock + egress + git-gate + supervise for one bottle, binding - them to per-bottle loopback ports and tearing them down with the - bottle. This is the smolmachines-specific replacement for - `docker compose up`/`down`. -- Per-bottle CA install path: the egress sidecar's CA cert lands - inside the microVM via `smolvm machine exec` after start +- A host-side sidecar-bundle lifecycle manager that brings up + one container per bottle (the bundle image defined in PRD 0024), + publishes its one to three host ports, waits for readiness, + and tears it down with the bottle. This backend depends on + PRD 0024's bundle image; it does not own the bundle's + Dockerfile or init. +- Per-bottle CA install path: the bundle's CA cert lands inside + the microVM via `smolvm machine exec` after start (analogous to the existing `provision_ca` for Docker). - gvproxy lifecycle: per-bottle `gvproxy` started by the backend before VM bringup, torn down after VM teardown, configured with - one `port_forwards` entry (gateway → host pipelock port) and a + up to three `port_forwards` entries (gateway port → host + bundle port for each of pipelock / git-gate / supervise) and a DNS section that resolves only `proxy.internal`. Subnet and gateway IP are derived from the bottle slug so two concurrent bottles don't collide. @@ -272,7 +283,7 @@ claude_bottle/backend/smolmachines/ cleanup.py prepare_cleanup / cleanup / list_active smolfile.py bottle_plan_to_smolfile(...) -> dict + render gvproxy.py per-bottle gvproxy config render + process lifecycle - sidecars.py host-side pipelock/egress/git-gate/supervise lifecycle + sidecar_bundle.py host-side lifecycle for the PRD 0024 bundle container smolvm.py thin subprocess wrapper: machine create/start/exec/stop vfkit_attach.py VZFileHandleNetworkDeviceAttachment + VFKT handshake util.py slugify, port allocation, OCI archive helpers @@ -284,16 +295,19 @@ claude_bottle/backend/smolmachines/ ``` ┌── macOS host ─────────────────────────────────────────────────────┐ │ │ - │ ┌── per-bottle sidecar chain (one set per microVM) ────┐ │ - │ │ agent ──HTTPS_PROXY──► pipelock ──► egress ──► internet │ - │ │ 127.0.0.1:p1 (DLP) (MITM, │ - │ │ auth-inject) │ - │ │ │ - │ │ git push ──► git-gate ──► upstream │ - │ │ 127.0.0.1:p3 (gitleaks) │ - │ │ │ - │ │ MCP ──► supervise 127.0.0.1:p4 │ - │ └────────────────────────────────────────────────────────────────┘ + │ ┌── per-bottle sidecar bundle (one container per microVM) ─┐ │ + │ │ init.py (Python supervisor) │ │ + │ │ ├─ pipelock (binds 0.0.0.0:8888 in container) │ │ + │ │ ├─ egress (mitmproxy) (binds 127.0.0.1:p_internal) │ │ + │ │ ├─ git-gate (binds 0.0.0.0:8889) │ │ + │ │ └─ supervise (MCP) (binds 0.0.0.0:8890) │ │ + │ │ pipelock's upstream is 127.0.0.1:p_internal (egress); │ │ + │ │ egress is not exposed outside the bundle. │ │ + │ └─────────────────────────────────────────────────────┬─────┘ │ + │ Host ports published (loopback, dynamic): │ │ + │ pipelock 127.0.0.1: │ │ + │ git-gate 127.0.0.1: (conditional) │ │ + │ supervise 127.0.0.1: (conditional) │ │ │ ▲ host TCP, reached via gvproxy port-forward │ │ │ │ │ ┌── gvproxy (per bottle) ─────────────────────────────┐ │ @@ -301,6 +315,8 @@ claude_bottle/backend/smolmachines/ │ │ gateway: 192.168.127.X.1 │ │ │ │ port_forwards: │ │ │ │ - gateway 8888 → host 127.0.0.1: │ │ + │ │ - gateway 8889 → host 127.0.0.1: (cond) │ │ + │ │ - gateway 8890 → host 127.0.0.1: (cond) │ │ │ │ # nothing else │ │ │ │ DNS: proxy.internal → gateway IP; * → NXDOMAIN │ │ │ └─────────────────────────────────────────────────────┘ │ @@ -323,24 +339,23 @@ host LAN, host loopback (Postgres, IDE plugins, other bottles' sidecars), public internet directly — is gone, enforced at the gvproxy userspace stack rather than relying on guest cooperation. -Two changes vs. the Docker backend: +Three changes vs. the Docker backend: -1. **Sidecars are host processes, not sibling containers.** No - internal Docker network. The isolation primitive is gvproxy's - explicit port-forward list, not Docker's `--internal` flag. -2. **The agent's first hop is `proxy.internal`, not a sidecar's - container hostname.** Egress out to the public internet still - happens through pipelock + egress — same scanning + DLP + - auth-injection chain — but the first hop crosses a userspace - TCP/IP stack we own, not a Docker-managed bridge. +1. **One sidecar container per bottle, not four.** The bundle + defined in PRD 0024 is the unit of sidecar lifecycle on both + backends. egress is internal to the bundle as pipelock's + upstream, never directly addressed. +2. **Sidecar container is on the host, not a sibling on a Docker + internal network.** Isolation primitive is gvproxy's explicit + port-forward list, not Docker's `--internal` flag. +3. **The agent's first hop is `proxy.internal`, not a sidecar's + container hostname.** Same scanning + DLP + auth-injection + chain, but the first hop crosses a userspace TCP/IP stack we + own, not a Docker-managed bridge. -The chain `agent → pipelock → egress → internet` collapses on -the host side: pipelock listens on 127.0.0.1:p1, makes its -upstream connect against egress at 127.0.0.1:p2, which makes its -upstream connect against the public internet. git-gate and -supervise are separate gateway ports if and only if the bottle -uses them — otherwise they're omitted from gvproxy's -`port_forwards`, narrowing the attack surface further. +git-gate and supervise are conditional port forwards: only +emitted into gvproxy's config when the bottle actually uses +them, narrowing the attack surface for bottles that don't. ### Lifecycle @@ -348,16 +363,19 @@ uses them — otherwise they're omitted from gvproxy's 1. Cross-backend validation via `BottleBackend._validate` (skills, git identity files). -2. Allocate host loopback ports for each sidecar the bottle uses - (pipelock always; egress / git-gate / supervise conditional on - manifest). +2. Allocate one to three host loopback ports for the sidecar + bundle (pipelock always; git-gate and supervise conditional on + manifest — egress is internal to the bundle and gets no host + port). 3. Resolve the agent OCI archive path (build if missing, cache by - Dockerfile + agent-name hash). + Dockerfile + agent-name hash). The sidecar-bundle image + (`claude-bottle-sidecars:`) is pulled or built per + PRD 0024; this backend does not own its build. 4. Pick a per-bottle gvproxy subnet (e.g. `192.168.127.X/24` where `X` is derived from the slug) and render - `stage_dir/gvproxy.yaml`: one DNS entry for `proxy.internal`, - one `port_forwards` entry per active sidecar (gateway port → - host loopback port). + `stage_dir/gvproxy.yaml`: one DNS entry for `proxy.internal` + and one `port_forwards` entry per active sidecar port + (gateway port → host loopback port on the bundle). 5. Render the per-bottle Smolfile to `stage_dir/smolfile.toml`, pinning command / env / a virtio-net device backed by the gvproxy unixgram socket path. No TSI flags. @@ -365,17 +383,19 @@ uses them — otherwise they're omitted from gvproxy's pipelock's CA after start. 7. Return a `SmolmachinesBottlePlan` carrying the slug, port map, OCI archive path, Smolfile path, gvproxy config path, and - host sidecar specs. + the bundle's container/run spec. `SmolmachinesBottleBackend.launch(plan)`: -1. Start host sidecars in dependency order (egress → pipelock → - git-gate → supervise — egress before pipelock so pipelock's - upstream resolves; pipelock is the only one exposed through - gvproxy). Register teardown callbacks in reverse order. +1. Start the sidecar bundle container with `docker run` (still + using the local Docker daemon for sidecars; the VM is what's + moving off Docker). Wait for its three readiness signals: + pipelock listening, git-gate listening (if enabled), supervise + listening (if enabled). Register the teardown callback. 2. Start the per-bottle `gvproxy` against the unixgram socket - path the Smolfile references. Wait for the socket to appear - (the spike-style poll loop from `agent-vm-isolation.md`). + path the Smolfile references, with `port_forwards` pointed at + the bundle's published host ports. Wait for the socket to + appear (the spike-style poll loop from `agent-vm-isolation.md`). 3. `smolvm machine create --smolfile ` and `smolvm machine start `. The Smolfile's virtio-net device handshakes (`VFKT` magic) with gvproxy on start. @@ -385,8 +405,8 @@ uses them — otherwise they're omitted from gvproxy's 5. Yield a `SmolmachinesBottle` whose `exec_claude` / `exec` / `cp_in` all funnel through `smolvm machine exec` / `smolvm machine cp`. -6. Teardown: stop and remove the VM → stop gvproxy → stop - sidecars (in reverse start order). +6. Teardown: stop and remove the VM → stop gvproxy → stop + + remove the sidecar bundle container. ### Data model @@ -461,6 +481,9 @@ The existing "unknown backend" `die()` path stays as-is. ## Sizing — into chunks +PRD 0024's bundle image is a prerequisite — this PRD assumes +`claude-bottle-sidecars:` is available when chunk 3 lands. + 1. **Backend skeleton + selection + Smolfile + gvproxy renderers.** Subpackage layout, `_resolve_plan` stub that emits both a TOML Smolfile and a gvproxy YAML but doesn't launch anything, @@ -474,11 +497,12 @@ The existing "unknown backend" `die()` path stays as-is. Smoke integration test: `exec("echo hi")` inside a started VM. Includes the localhost-reach probe test from the acceptance plan. -3. **Host-side sidecar relocation.** `sidecars.py`: per-bottle - pipelock + egress + git-gate + supervise as host processes on - loopback, with gvproxy `port_forwards` wired only for the - sidecars the bottle actually uses. Port allocator. Teardown - ordering. No provisioning yet beyond what the sidecars need. +3. **Sidecar bundle lifecycle.** `sidecar_bundle.py`: per-bottle + bundle container brought up via `docker run`, with one to + three published host ports, gvproxy `port_forwards` pointed + at them, and teardown integrated into the bottle's lifecycle. + Port allocator. No provisioning yet beyond what the bundle + needs. 4. **Provisioning parity with Docker.** CA install via `smolvm machine exec`, prompt/skills/.git copy-in, supervise MCP config. End-to-end `start` works for a real agent manifest. @@ -569,10 +593,14 @@ The existing "unknown backend" `die()` path stays as-is. backend abstraction this PRD is the first non-Docker consumer of. - PRD 0017 (`docs/prds/0017-egress-proxy-via-mitmproxy.md`) — the - egress sidecar the host-side relocation reuses verbatim, only - with a different transport. + egress sidecar the bundle reuses verbatim as pipelock's internal + upstream. - PRD 0022 (`docs/prds/0022-sandbox-escape-integration-test.md`) — the acceptance gate for this PRD; the suite already runs through `get_bottle_backend()` so the env-var flip is the only change needed to exercise the smolmachines path. +- PRD 0024 + (`docs/prds/0024-consolidate-sidecar-bundle.md`) — defines the + single bundle image (`claude-bottle-sidecars`) this PRD + consumes. Prerequisite for chunk 3 of this PRD.