From 041da1d7af99712599a0b91f8a28c5a5be47a346 Mon Sep 17 00:00:00 2001 From: didericis Date: Tue, 26 May 2026 23:41:32 -0400 Subject: [PATCH] docs(prd-0023): make gvproxy the network primitive; reject TSI TSI's --outbound-localhost-only is permissive on all of 127.0.0.0/8 with no destination-port filter, so any host loopback service (local Postgres, IDE plugins, another bottle's sidecar) is reachable from the guest. That's the wrong default for the malicious-agent threat model. Reworked the network design around gvproxy + VFKT unixgram attachment: the guest gets a virtio-net device, gvproxy is the userspace TCP/IP stack on the host side, and the only thing reachable from the guest is the explicit port-forward list (typically just pipelock). Host LAN, host loopback, and the public internet directly are gone by construction. VMM choice (smolmachines vs PyObjC + Virtualization.framework) is an open question contingent on whether libkrun's virtio-net mode lets us point at a custom unixgram socket. Backend name stays "smolmachines" either way per the original spec. Co-Authored-By: Claude Opus 4.7 --- docs/prds/0023-smolmachines-backend.md | 421 +++++++++++++++++-------- 1 file changed, 286 insertions(+), 135 deletions(-) diff --git a/docs/prds/0023-smolmachines-backend.md b/docs/prds/0023-smolmachines-backend.md index dd602ff..441005b 100644 --- a/docs/prds/0023-smolmachines-backend.md +++ b/docs/prds/0023-smolmachines-backend.md @@ -6,31 +6,54 @@ ## Summary -Ship a second concrete `BottleBackend` — `SmolmachinesBottleBackend`, -selected via `CLAUDE_BOTTLE_BACKEND=smolmachines` — that runs a -bottle inside a per-agent libkrun microVM on macOS (and KVM on Linux, -opportunistically). The egress topology moves out of an internal -Docker network and onto libkrun's TSI ("Transport Socket Interface") -allowlist plus a host-side pipelock/egress/git-gate/supervise stack -listening on per-bottle loopback ports. The Docker backend ships -unchanged; this is opt-in via the existing env-var selector. +Ship a second concrete `BottleBackend` — +`SmolmachinesBottleBackend`, selected via +`CLAUDE_BOTTLE_BACKEND=smolmachines` — that runs each bottle inside +a per-agent microVM on macOS. The egress topology is enforced by +**gvproxy** (gvisor-tap-vsock), a userspace TCP/IP stack the guest's +virtio-net device is wired into via `VZFileHandleNetworkDeviceAttachment`. +gvproxy's only outbound configuration is an explicit per-bottle port +forward to a host-side pipelock; everything else — the host's LAN, +the host's loopback services, the public internet — is unreachable +from the guest by construction. pipelock + egress + git-gate + +supervise stay as host-side processes on per-bottle loopback ports, +reached *only* through gvproxy's forwarded ports. -The acceptance gate is PRD 0022's `tests/integration/test_sandbox_escape.py` -running green against `CLAUDE_BOTTLE_BACKEND=smolmachines`. +This explicitly rejects libkrun's TSI ("Transport Socket Interface") +allowlist as the network primitive. TSI's `--outbound-localhost-only` +is permissive on the entire `127.0.0.0/8` range with no +destination-port filter — the agent can dial any host-side service +bound to loopback (a local Postgres, an IDE plugin, a different +bottle's pipelock). That's the wrong default for a malicious-agent +threat model; see "Why gvproxy, not TSI" below. + +The Docker backend ships unchanged; this is opt-in via the existing +env-var selector. The acceptance gate is PRD 0022's +`tests/integration/test_sandbox_escape.py` running green against +`CLAUDE_BOTTLE_BACKEND=smolmachines`. ## Problem `agent-vm-isolation.md` argues for hardware-isolated microVMs over container-based bottles on macOS; `smolmachines-as-vm-backend.md` -concludes that smolmachines is the most plausible concrete VMM for -this project. Today, the only backend in the registry is Docker +evaluates smolmachines as the lifecycle wrapper. Today, the only +backend in the registry is Docker (`claude_bottle/backend/__init__.py:_BACKENDS = {"docker": ...}`), -and three things motivate a second one now: +and four things motivate a second one now: +- **Network reach beyond pipelock.** The threat model is a malicious + agent attempting to dial something on the operator's *local + network* (`192.168.x.x` services, the home router, a coworker's + laptop on the same Wi-Fi) or *host's loopback* (a local database, + an editor plugin, another bottle's sidecar) without traversing + pipelock. The Docker backend's `--internal` network blocks the + first; nothing in the current stack blocks the second cleanly. + This PRD's gvproxy-based design closes both gaps: the guest can + only reach the explicit port-forward list, period. - **Isolation ceiling.** On macOS the Docker backend's agent - container shares Docker Desktop's host VM with every other bottle. - Container escape from claude-code lands the agent inside that - shared VM. A per-bottle libkrun microVM gets hardware page tables + container shares Docker Desktop's host VM with every other + bottle. Container escape from claude-code lands the agent inside + that shared VM. A per-bottle microVM gets hardware page tables via `Hypervisor.framework`; cross-bottle isolation becomes enforced by the CPU's MMU instead of namespace bookkeeping. - **PRD 0022 is backend-agnostic by design** but currently only @@ -41,15 +64,36 @@ and three things motivate a second one now: unproven. - **CI carve-outs.** Most bottle-bringup integration tests skip under `GITEA_ACTIONS=true` because act_runner shares the host - Docker socket but not the host filesystem. A smolmachines path + Docker socket but not the host filesystem. A microVM path doesn't share that constraint shape (it has its own, but different), so adding the backend forces the abstraction to be clean in places where Docker-specific assumptions have been tolerated. -The smolmachines research note's `## Recommendation` ("adopt -smolmachines as the bottle VM backend on macOS; keep pipelock DIY") -is the design hypothesis under test here. +## Why gvproxy, not TSI + +libkrun's TSI hijacks guest socket syscalls inside the VMM and +opens the actual sockets from the host process, with a CIDR +allowlist gate. That works fine for blocking LAN reach (don't +allowlist `192.168.0.0/16`, agent can't dial it). But TSI's +`--outbound-localhost-only` permits the *entire* `127.0.0.0/8` +range across all ports — there is no destination-port filter at +the TSI layer (`smolmachines-as-vm-backend.md` flags this in the +"`--allow-host` semantics" caveat). For our threat model that +means any host-loopback service is reachable from the guest. + +gvproxy implements a full userspace TCP/IP stack on the host side +of a `VZFileHandleNetworkDeviceAttachment` unixgram socket. The +guest has a real virtio-net device; gvproxy is its gateway. The +guest can only reach what gvproxy is configured to forward — +typically a single port forward to the per-bottle pipelock — +and DNS resolves NXDOMAIN by default. There is no "permissive +loopback" mode to mis-configure; if it's not in `port_forwards`, +the guest cannot reach it. + +That property — *explicit allowlist by port forward, not CIDR* — +is the load-bearing reason this PRD chooses gvproxy. TSI shows up +once more in this doc, under Non-goals, where it is closed off. ## Goals / Success Criteria @@ -84,13 +128,25 @@ The feature is **done** when all of the following ship: - Per-bottle Smolfile generation: a runtime-rendered TOML written to the bottle's stage dir, analogous to the compose file the Docker backend writes today. The Smolfile pins `command`, - `env`, `--outbound-localhost-only`, and the per-bottle DNS - allowlist. + `env`, and a virtio-net device backed by a unixgram socket + pointed at the per-bottle gvproxy. There is no TSI + `--allow-cidr` / `--outbound-localhost-only` / `--allow-host` + in the Smolfile — TSI is not used. +- Per-bottle gvproxy: one `gvproxy` process per bottle, started + before the VM, listening on a unixgram socket the VM's + virtio-net device hooks into. The gvproxy config has exactly + one `port_forwards` entry — gateway-port to the per-bottle + pipelock's host port — and a DNS section that resolves only + `proxy.internal`. Every other hostname returns NXDOMAIN; every + other destination is unreachable. - Host-side sidecar relocation: pipelock, egress, git-gate, and supervise each run as host processes (one set per bottle), bound to `127.0.0.1` on per-bottle dynamically-allocated ports. The agent's environment carries the resolved URLs (e.g. - `HTTPS_PROXY=http://127.0.0.1:`). + `HTTPS_PROXY=http://proxy.internal:`). + Only pipelock is exposed through gvproxy; egress / git-gate / + supervise are chained *behind* pipelock on the host side and + are not reachable directly from the guest. - The agent guest image is produced from the existing `Dockerfile` (or a thin variant), exported as an OCI archive, and consumed by `smolvm machine create`. The image build step is part of `prepare`, @@ -117,6 +173,15 @@ The feature is **done** when all of the following ship: value of `CLAUDE_BOTTLE_BACKEND`; smolmachines is strictly opt-in until it has been load-bearing on at least one operator's workflow for a release cycle. +- **No TSI for network policy.** libkrun's TSI mode is rejected + for this backend — it lacks per-port filtering on `127.0.0.0/8` + and would expose every host-loopback service to the guest. The + Smolfile must select libkrun's virtio-net mode and attach to + the per-bottle gvproxy unixgram socket; if that combination is + not supported by the pinned smolmachines version (see open + question 1), the implementation falls back to driving + Virtualization.framework directly via PyObjC and reuses the + same gvproxy attachment. - **No host bind mounts.** The smolmachines research note flagged that `-v HOST:GUEST` mounts via virtiofs would defeat the isolation goal. The manifest already has no concept of host @@ -152,12 +217,18 @@ The feature is **done** when all of the following ship: - Per-bottle CA install path: the egress sidecar's CA cert lands inside the microVM via `smolvm machine exec` after start (analogous to the existing `provision_ca` for Docker). -- DNS allowlist plumbing: every host in `bottle.egress.allowlist` - goes into the Smolfile's DNS filter section (vsock port 6002), - so the VMM-layer DNS filter and the bottle's policy stay in - sync — agent can't `dig` its way out via raw IP literals (TSI - + CIDR allowlist enforces this; DNS filter denies hostname - resolution). +- gvproxy lifecycle: per-bottle `gvproxy` started by the backend + before VM bringup, torn down after VM teardown, configured with + one `port_forwards` entry (gateway → host pipelock port) and a + DNS section that resolves only `proxy.internal`. Subnet and + gateway IP are derived from the bottle slug so two concurrent + bottles don't collide. +- DNS policy: the bottle's `egress.allowlist` does *not* go into + gvproxy's DNS — the agent resolves only `proxy.internal`, and + pipelock on the host enforces the egress allowlist against + the actual upstream connect target. This keeps the DNS-exfil + attack (PRD 0022 test 4) blocked because gvproxy answers + NXDOMAIN for every name except `proxy.internal`. - Preflight `smolvm` check: if the user selects this backend and `smolvm` isn't on `$PATH`, die with an install pointer (brew tap + version pin TBD in implementation; see open question 3). @@ -200,8 +271,10 @@ claude_bottle/backend/smolmachines/ launch.py @contextmanager launch(plan) -> SmolmachinesBottle cleanup.py prepare_cleanup / cleanup / list_active smolfile.py bottle_plan_to_smolfile(...) -> dict + render + gvproxy.py per-bottle gvproxy config render + process lifecycle sidecars.py host-side pipelock/egress/git-gate/supervise lifecycle smolvm.py thin subprocess wrapper: machine create/start/exec/stop + vfkit_attach.py VZFileHandleNetworkDeviceAttachment + VFKT handshake util.py slugify, port allocation, OCI archive helpers provision/ ca.py, prompt.py, skills.py, git.py, supervise.py ``` @@ -209,39 +282,65 @@ claude_bottle/backend/smolmachines/ ### Network + egress topology ``` - ┌── macOS host ─────────────────────────────────────────────┐ - │ │ - │ ┌── per-bottle host sidecars (one set per microVM) ─┐ │ - │ │ pipelock 127.0.0.1: │ │ - │ │ egress 127.0.0.1: │ │ - │ │ git-gate 127.0.0.1: │ │ - │ │ supervise 127.0.0.1: │ │ - │ └───────────────────────────────────────────────────┘ │ - │ ▲ │ - │ │ TSI passthrough (localhost) │ - │ │ │ - │ ┌── libkrun microVM (per bottle) ───────────────────┐ │ - │ │ env: HTTPS_PROXY=http://127.0.0.1: │ │ - │ │ EGRESS_URL=http://127.0.0.1: │ │ - │ │ GIT_GATE_URL=http://127.0.0.1: │ │ - │ │ MCP_SUPERVISE_URL=http://127.0.0.1: │ │ - │ │ --outbound-localhost-only │ │ - │ │ DNS filter (vsock:6002) → host allowlist │ │ - │ └───────────────────────────────────────────────────┘ │ - │ │ - └───────────────────────────────────────────────────────────┘ + ┌── macOS host ─────────────────────────────────────────────────────┐ + │ │ + │ ┌── per-bottle sidecar chain (one set per microVM) ────┐ │ + │ │ agent ──HTTPS_PROXY──► pipelock ──► egress ──► internet │ + │ │ 127.0.0.1:p1 (DLP) (MITM, │ + │ │ auth-inject) │ + │ │ │ + │ │ git push ──► git-gate ──► upstream │ + │ │ 127.0.0.1:p3 (gitleaks) │ + │ │ │ + │ │ MCP ──► supervise 127.0.0.1:p4 │ + │ └────────────────────────────────────────────────────────────────┘ + │ ▲ host TCP, reached via gvproxy port-forward │ + │ │ │ + │ ┌── gvproxy (per bottle) ─────────────────────────────┐ │ + │ │ subnet: 192.168.127.X/24 (X derived from slug) │ │ + │ │ gateway: 192.168.127.X.1 │ │ + │ │ port_forwards: │ │ + │ │ - gateway 8888 → host 127.0.0.1: │ │ + │ │ # nothing else │ │ + │ │ DNS: proxy.internal → gateway IP; * → NXDOMAIN │ │ + │ └─────────────────────────────────────────────────────┘ │ + │ ▲ unixgram socket (VFKT handshake) │ + │ │ │ + │ ┌── microVM (per bottle) ─────────────────────────────┐ │ + │ │ virtio-net device backed by VZFileHandle... │ │ + │ │ env: HTTPS_PROXY=http://proxy.internal:8888 │ │ + │ │ GIT_GATE_URL=http://proxy.internal:8889 │ │ + │ │ MCP_SUPERVISE_URL=http://proxy.internal:8890 │ │ + │ │ no other host visible │ │ + │ └─────────────────────────────────────────────────────┘ │ + │ │ + └───────────────────────────────────────────────────────────────────┘ ``` +What the guest can reach, exhaustively: **only `proxy.internal` +on the gateway-port set we configured.** Everything else — +host LAN, host loopback (Postgres, IDE plugins, other bottles' +sidecars), public internet directly — is gone, enforced at the +gvproxy userspace stack rather than relying on guest cooperation. + Two changes vs. the Docker backend: 1. **Sidecars are host processes, not sibling containers.** No - internal Docker network; isolation comes from TSI plus the - per-bottle loopback port set. -2. **The "internal" allowlist becomes localhost-only.** Egress out - to the public internet still happens through pipelock + egress - — the same scanning + DLP + auth-injection chain — but the - agent's first hop is `127.0.0.1:` reached via TSI, not a - sidecar's IP on a Docker-managed bridge. + internal Docker network. The isolation primitive is gvproxy's + explicit port-forward list, not Docker's `--internal` flag. +2. **The agent's first hop is `proxy.internal`, not a sidecar's + container hostname.** Egress out to the public internet still + happens through pipelock + egress — same scanning + DLP + + auth-injection chain — but the first hop crosses a userspace + TCP/IP stack we own, not a Docker-managed bridge. + +The chain `agent → pipelock → egress → internet` collapses on +the host side: pipelock listens on 127.0.0.1:p1, makes its +upstream connect against egress at 127.0.0.1:p2, which makes its +upstream connect against the public internet. git-gate and +supervise are separate gateway ports if and only if the bottle +uses them — otherwise they're omitted from gvproxy's +`port_forwards`, narrowing the attack surface further. ### Lifecycle @@ -249,40 +348,56 @@ Two changes vs. the Docker backend: 1. Cross-backend validation via `BottleBackend._validate` (skills, git identity files). -2. Allocate four loopback ports (bind, get free port, release; - record on plan). +2. Allocate host loopback ports for each sidecar the bottle uses + (pipelock always; egress / git-gate / supervise conditional on + manifest). 3. Resolve the agent OCI archive path (build if missing, cache by Dockerfile + agent-name hash). -4. Render the per-bottle Smolfile to `stage_dir/smolfile.toml`, - pinning command/env/`--outbound-localhost-only` + DNS allowlist. -5. Resolve the in-VM CA paths so launch knows where to copy +4. Pick a per-bottle gvproxy subnet (e.g. `192.168.127.X/24` where + `X` is derived from the slug) and render + `stage_dir/gvproxy.yaml`: one DNS entry for `proxy.internal`, + one `port_forwards` entry per active sidecar (gateway port → + host loopback port). +5. Render the per-bottle Smolfile to `stage_dir/smolfile.toml`, + pinning command / env / a virtio-net device backed by the + gvproxy unixgram socket path. No TSI flags. +6. Resolve the in-VM CA paths so launch knows where to copy pipelock's CA after start. -6. Return a `SmolmachinesBottlePlan` carrying the slug, port map, - OCI archive path, Smolfile path, and host sidecar specs. +7. Return a `SmolmachinesBottlePlan` carrying the slug, port map, + OCI archive path, Smolfile path, gvproxy config path, and + host sidecar specs. `SmolmachinesBottleBackend.launch(plan)`: -1. Start the four host sidecars in dependency order (pipelock → - egress → git-gate → supervise), bound to the plan's allocated - ports. Register teardown callbacks in reverse order. -2. `smolvm machine create --smolfile ` and - `smolvm machine start `. -3. Provisioning: CA install → prompt → skills → git → supervise +1. Start host sidecars in dependency order (egress → pipelock → + git-gate → supervise — egress before pipelock so pipelock's + upstream resolves; pipelock is the only one exposed through + gvproxy). Register teardown callbacks in reverse order. +2. Start the per-bottle `gvproxy` against the unixgram socket + path the Smolfile references. Wait for the socket to appear + (the spike-style poll loop from `agent-vm-isolation.md`). +3. `smolvm machine create --smolfile ` and + `smolvm machine start `. The Smolfile's virtio-net + device handshakes (`VFKT` magic) with gvproxy on start. +4. Provisioning: CA install → prompt → skills → git → supervise config, each via `smolvm machine exec` (analogous to `docker exec`). -4. Yield a `SmolmachinesBottle` whose `exec_claude` / `exec` / +5. Yield a `SmolmachinesBottle` whose `exec_claude` / `exec` / `cp_in` all funnel through `smolvm machine exec` / `smolvm machine cp`. -5. Teardown: stop and remove the VM, then stop the sidecars (in - reverse start order). +6. Teardown: stop and remove the VM → stop gvproxy → stop + sidecars (in reverse start order). ### Data model No manifest schema change. `bottles[]` continues to carry `egress.allowlist`, `env`, `git`, `skills` references, etc.; the smolmachines backend reads the same fields as the docker backend. -The DNS allowlist plumbed into the Smolfile is just -`bottle.egress.allowlist` re-encoded as TOML. +`egress.allowlist` is enforced by pipelock on the host side +(unchanged from the docker backend); gvproxy's DNS resolves only +`proxy.internal` regardless of the allowlist's contents, so an +agent that bypasses pipelock by raw IP cannot resolve any name +gvproxy doesn't know about. The `BottleSpec` dataclass and the `Bottle` ABC do not change. @@ -308,20 +423,36 @@ The existing "unknown backend" `die()` path stays as-is. the preflight check). Pinned version policy is deferred to the open questions; v1 reads `smolvm --version` and refuses to launch outside a known-good range. -- No new Python packages. Subprocess + stdlib `tomllib`/`tomli_w` - for Smolfile authoring. (`tomli_w` is the only candidate - module; if it's not stdlib in the target Python, render TOML - by hand from a `dict[str, Any]` — Smolfile shape is small.) +- `gvproxy` binary on `$PATH` + (`go install github.com/containers/gvisor-tap-vsock/cmd/gvproxy@latest`, + or vendored). Same preflight pattern as `smolvm`. +- `pyobjc-framework-Virtualization` *only* if smolmachines does + not expose a way to attach virtio-net to a unixgram socket and + we fall back to driving Virtualization.framework directly (see + open question 1). Default path is "no PyObjC needed." +- No new pure-Python packages. Subprocess + stdlib `tomllib` for + Smolfile authoring; the gvproxy YAML is small enough to render + by hand from a `dict[str, Any]`. ### Acceptance test plan -- **Unit:** `tests/unit/test_smolfile.py` verifies the renderer - produces the expected TOML for a fixture bottle (allowlist → - DNS rules, env → `env =`, command line, outbound-localhost - flag). +- **Unit (smolfile):** `tests/unit/test_smolfile.py` verifies the + renderer produces the expected TOML for a fixture bottle — + command line, env entries, virtio-net device referencing the + expected unixgram socket path, no TSI flags. +- **Unit (gvproxy config):** `tests/unit/test_gvproxy_config.py` + verifies the per-bottle YAML has exactly one DNS entry + (`proxy.internal`), one `port_forwards` entry per active + sidecar pointed at the resolved host loopback port, and a + per-bottle subnet/gateway derived from the slug. - **Integration smoke:** `tests/integration/test_smolmachines_smoke.py` with `prepare → launch → exec → teardown`, guarded by a - `smolvm` presence check + macOS / KVM platform check. + `smolvm` + `gvproxy` presence check + macOS / KVM platform check. +- **Localhost-reach probe:** a focused integration test that + brings up a bottle, has the host bind a test service on + `127.0.0.1:`, and asserts the in-bottle agent + cannot connect to it. This is the regression test for the + exact gap that motivated choosing gvproxy over TSI. - **PRD 0022 re-run:** with `CLAUDE_BOTTLE_BACKEND=smolmachines`, all five attack categories return sandbox-block markers and the suite passes. The test code does not change beyond the env-var @@ -330,19 +461,24 @@ The existing "unknown backend" `die()` path stays as-is. ## Sizing — into chunks -1. **Backend skeleton + selection + Smolfile renderer.** Subpackage - layout, `_resolve_plan` stub that emits a TOML file but doesn't - launch anything, `_BACKENDS` registration, preflight `smolvm` - check. Unit test on the renderer. No VM bringup yet. -2. **VM lifecycle + OCI archive build.** `smolvm.py` subprocess - wrapper, prepare-time image build (existing Dockerfile → OCI - archive), launch path that creates + starts + stops a VM with - no sidecars wired. Smoke integration test: `exec("echo hi")` - inside a started VM. +1. **Backend skeleton + selection + Smolfile + gvproxy renderers.** + Subpackage layout, `_resolve_plan` stub that emits both a + TOML Smolfile and a gvproxy YAML but doesn't launch anything, + `_BACKENDS` registration, preflight `smolvm` + `gvproxy` + checks. Unit tests on both renderers. No VM bringup yet. +2. **gvproxy + VM lifecycle + OCI archive build.** `smolvm.py` + and `gvproxy.py` subprocess wrappers, prepare-time image + build (existing Dockerfile → OCI archive), launch path that + starts gvproxy, brings up the VM attached to gvproxy's socket + via VFKT handshake, exec into the VM, tear everything down. + Smoke integration test: `exec("echo hi")` inside a started + VM. Includes the localhost-reach probe test from the + acceptance plan. 3. **Host-side sidecar relocation.** `sidecars.py`: per-bottle pipelock + egress + git-gate + supervise as host processes on - loopback. Port allocator. Teardown ordering. No provisioning - yet beyond what the sidecars need. + loopback, with gvproxy `port_forwards` wired only for the + sidecars the bottle actually uses. Port allocator. Teardown + ordering. No provisioning yet beyond what the sidecars need. 4. **Provisioning parity with Docker.** CA install via `smolvm machine exec`, prompt/skills/.git copy-in, supervise MCP config. End-to-end `start` works for a real agent manifest. @@ -354,63 +490,78 @@ The existing "unknown backend" `die()` path stays as-is. ## Open questions -1. **Sidecar locality: host process vs in-VM init.** This PRD - defaults to host-process sidecars (proposed design above). The - alternative — bake pipelock + egress + git-gate + supervise - into the OCI image and start them via init in the same VM — - would simplify port plumbing (the agent reaches sidecars over - localhost inside the VM, not over TSI) but expands the trust - boundary of the agent VM. Default A unless someone identifies - a TSI loopback edge case during chunk 3. -2. **`smolvm` install policy.** Pin via brew formula version, or - build-from-source step, or vendored binary checked into the - repo. v1 most likely runs `smolvm --version` at preflight and - accepts a documented range; vendoring is heavier but reduces - "works on my Mac" drift. +1. **VMM choice: smolmachines vs PyObjC + Virtualization.framework.** + The network design requires libkrun's virtio-net mode attached + to a unixgram socket (so gvproxy is the gateway). The + smolmachines research note says libkrun *has* a virtio-net + mode but says it "does not support policy" — meaning libkrun + itself enforces no allowlist in that mode, which is exactly + what we want (gvproxy is the policy). What's unverified is + whether the Smolfile surface lets us point virtio-net at a + custom unixgram socket. If yes: this is a smolmachines backend + verbatim. If no: chunk 2 drops `smolvm` and drives + `Virtualization.framework` via PyObjC directly (the recipe in + `agent-vm-isolation.md` § "gvisor-tap-vsock + PyObjC + + Pipelock"), keeping the backend name "smolmachines" because + the operator-facing UX is unchanged. Resolve in chunk 1 via a + spike against the pinned smolmachines version. +2. **`smolvm` + `gvproxy` install policy.** Pin via brew / + `go install` versions, or vendor binaries in the repo. v1 + likely runs `smolvm --version` / `gvproxy --help` at preflight + and accepts a documented range; vendoring is heavier but + reduces "works on my Mac" drift. 3. **CA install inside the OCI overlay.** Two paths: bake at prepare time (one OCI archive per CA fingerprint, big cache key) vs. inject at start time via `smolvm machine exec` after the VM is up. PRD 0006 chose the runtime path for Docker (docker-cp + `update-ca-certificates`); smolvm has the same shape via `machine exec`. Default to runtime injection unless - it conflicts with `--outbound-localhost-only` start order. -4. **DNS filter granularity.** smolmachines's vsock-6002 filter - accepts an allowlist of hostnames; we want to enforce both - "agent can only resolve names on the bottle's allowlist" *and* - "agent can only egress via TSI to 127.0.0.1." Confirm - empirically (smoke test in chunk 2) that the allowlist applies - to *guest-initiated* DNS only and doesn't accidentally NXDOMAIN - the host-side pipelock's upstream lookups. + it conflicts with VM start order. +4. **gvproxy subnet collision.** Two concurrent bottles must not + land on the same `192.168.127.X/24` subnet — they'd both want + the same gateway IP. Derive the third octet from a hash of + the slug (mod 254, skip the docker-default 17), and at launch + time confirm the subnet isn't already in use by another + bottle's gvproxy. Resolve the hash-collision policy in + chunk 2. 5. **`bottle.exec(script)` exit-code fidelity.** The PRD 0022 test suite reads `returncode` + stdout + stderr from - `ExecResult`. Confirm `smolvm machine exec` propagates exit - codes and separated streams — the research note's - "external integration is the CLI" implies yes, but the - embedded SDK bug it flagged suggests we should verify before - coding around it. + `ExecResult`. Confirm the VM-exec path (`smolvm machine exec` + or its PyObjC equivalent) propagates exit codes and separated + streams. The research note's "external integration is the CLI" + implies yes, but the embedded SDK bug it flagged suggests we + should verify before coding around it. 6. **CI gating.** Gitea's act_runner is Linux without nested KVM, - so smolmachines integration tests will skip there for the same - structural reason the Docker bringup tests do (no real + so this backend's integration tests will skip there for the + same structural reason the Docker bringup tests do (no real isolation primitive available on the runner). The skip - predicate becomes `not (smolvm_available() and - (platform.system() == "Darwin" or kvm_available()))`. CI - coverage for this backend will come from local runs on the - maintainer's macOS host until a Darwin runner is wired up; - ack that as a known gap. + predicate becomes `not (smolvm_available() and gvproxy_available() + and platform.system() == "Darwin")`. CI coverage for this + backend will come from local runs on the maintainer's macOS + host until a Darwin runner is wired up; ack that as a known + gap. 7. **Active bottle discovery.** Docker uses container labels to enumerate active bottles (`list_active` queries the daemon). - smolmachines's enumeration story is `smolvm machine list`; the - plan is to mirror the label scheme via Smolfile metadata + The microVM enumeration story is `smolvm machine list` + (or the PyObjC backend's own bookkeeping); the plan is to + mirror the label scheme via Smolfile metadata (`labels = { "claude-bottle" = "1" }`-style entries, if the format supports it; otherwise via a deterministic name prefix - `claude-bottle-`). + `claude-bottle-` + on-disk metadata under + `state//`). ## References -- `docs/research/smolmachines-as-vm-backend.md` — primary research - note recommending this adoption; PRD 0023's design hypothesis. -- `docs/research/agent-vm-isolation.md` — the broader microVM / - gvproxy / pipelock landscape this PRD lands inside of. +- `docs/research/agent-vm-isolation.md` — primary reference for + the gvproxy + `VZFileHandleNetworkDeviceAttachment` network + attachment used here. The "Full Setup: gvisor-tap-vsock + + PyObjC + Pipelock" section is the recipe the PyObjC fallback + in open question 1 would adopt verbatim. +- `docs/research/smolmachines-as-vm-backend.md` — evaluation of + smolmachines as the VM lifecycle wrapper. This PRD diverges + from its conclusion on the *network* primitive (rejecting TSI + in favor of gvproxy) but keeps its VM-lifecycle conclusion + conditional on the libkrun-virtio-net spike in open question 1. - `docs/research/agent-sandbox-landscape.md` — identifies `"runtime": "microvm"`-style opt-in as the borrowable idea; smolmachines is the concrete implementation.