bot-bottle/docs/research/smolmachines-as-vm-backend.md

smolmachines as a VM backend for bot-bottle

Evaluation of whether smolmachines would simplify the macOS agent-VM-isolation work spelled out in agent-vm-isolation.md.

Research conducted 2026-05-11.

Summary

smolmachines replaces four of the six subsystems in agent-vm-isolation.md cleanly, including the two hardest ones — the VZFileHandleNetworkDeviceAttachment + gvproxy wiring and the PyObjC lifecycle wrapper. Pipelock stays. The disk image story changes from "sealed .img" to "OCI image + writable overlay," which is fine for the isolation goal as long as -v host mounts are forbidden in any bottle that maps to a smolmachine.

Recommendation: adopt smolmachines as the macOS VM backend; keep pipelock DIY; wire the two via --outbound-localhost-only plus HTTPS_PROXY in the Smolfile, after smoke-testing that TSI passes through 127.0.0.1 traffic to a host-side pipelock.

What smolmachines actually is

• libkrun VMM linked as a library (no daemon); rides directly on Apple Hypervisor.framework on macOS and KVM on Linux.
• Custom kernel is not supported — you get libkrunfw only. Day-to-day knobs are command and env in a TOML Smolfile.
• Networking model: libkrun TSI ("Transport Socket Interface") — userspace socket hijacking inside the VMM library itself. DNS filtering is built in via vsock port 6002 — the guest's /etc/resolv.conf points at 127.0.0.1 and a guest-side DNS proxy tunnels queries over vsock to the host, which returns NXDOMAIN for anything not allow-listed.
• vsock control plane is fully implemented with well-known ports: 5000 workload control, 5001 log streaming, 6000 agent OCI ops, 6001 SSH agent, 6002 DNS filter.
• External integration is the CLI (smolvm machine create/start/stop/exec) or the HTTP API (smolvm serve). No Python SDK yet; Node.js embedded SDK exists but has a known bug where machines aren't visible to the CLI.

Subsystem-by-subsystem comparison

#	Subsystem	DIY recipe (today)	smolmachines	Verdict	Caveats
1	MicroVM runtime	vfkit or PyObjC + Virtualization.framework, minimal device model	libkrun (library, no daemon) over Hypervisor.framework / KVM. libkrunfw kernel only.	Replaces	No custom kernel/initrd.
2	Network attachment	VZFileHandleNetworkDeviceAttachment + unixgram socket → gvproxy userspace stack; DNS NXDOMAIN by default	libkrun TSI — userspace socket hijacking inside the VMM. CIDR allowlist enforced at the VMM layer; guest cannot bypass by dialing IPs. DNS filter via vsock port 6002.	Replaces	TSI is enabled when --allow-cidr / --allow-host is used; the alternative virtio-net backend does not support policy.
3	Egress proxy (pipelock)	pipelock at 127.0.0.1:8888, HTTPS MITM + DLP + allowlist	No analogue. Integration: --outbound-localhost-only + env = ["HTTPS_PROXY=http://127.0.0.1:8888"] in the Smolfile.	Irrelevant — keep pipelock	Whether TSI passes 127.0.0.1 traffic through to a host-side proxy is unverified; smoke test required.
4	Control plane (vsock)	VZVirtioSocketDeviceConfiguration + AF_VSOCK in guest, Unix socket on host	Full vsock plane built in. External use via smolvm machine exec or the smolvm serve HTTP API.	Replaces	The well-known vsock ports are internal to smolmachines. Custom task protocols must use the HTTP API or open a fresh vsock port inside the guest.
5	Disk image	Sealed virtio-blk raw image, no host mounts	OCI image + writable overlay (default 2 GiB, --overlay to tune). -v HOST:GUEST mounts use virtiofs. .smolmachine packs the whole rootfs.	Partial	Overlay is writable and lives on the host. For "no host filesystem visible to the guest," forbid -v mounts in bottles that map to smolmachines.
6	Lifecycle wrapper	~100 lines PyObjC + subprocess.Popen for gvproxy	CLI or smolvm serve HTTP API.	Replaces	No Python SDK yet. Drive via httpx to the HTTP API, or shell out to the CLI. Embedded Node.js SDK has a known bug (machines invisible to CLI) — avoid for now.

Caveats worth flagging before commitment

• No custom kernel. If the agent-vm-isolation work assumed a hand-rolled kernel cmdline, that flexibility goes away. Smolfile env and command cover the everyday cases.
• --allow-host semantics. Hostnames are resolved at VM start time and stored as /32 CIDRs. All ports on resolved IPs are permitted — there is no destination-port filtering at the smolmachines layer. For the pipelock integration path this is acceptable because the right flag is --outbound-localhost-only, not --allow-host.
• TSI passthrough to 127.0.0.1. The TSI code path for localhost isn't explicitly documented. Validate with a pipelock instance before building around it: curl-from-guest → pipelock-on-host should succeed; curl to any other host should be blocked.
• Embedded SDK bug. Machines created via the Node.js embedded SDK are currently invisible to the CLI. Use the HTTP API instead.
• Volume policy. "No host filesystem visible to the guest" needs to be a manifest-validation rule (no -v mounts in microvm-backed bottles), not just a documentation note.

Recommendation

Adopt smolmachines as the bottle VM backend on macOS; keep pipelock DIY.

The work in agent-vm-isolation.md is mostly the network-attachment plumbing and the PyObjC wrapper — exactly the parts smolmachines eliminates. What remains (pipelock integration, picking the right networking flag, deciding on volumes vs. sealed overlay) is the work that needs doing regardless of the VM backend.

This aligns with the borrowable idea identified in agent-sandbox-landscape.md — a "runtime": "microvm" opt-in field on a bottle. smolmachines is the most plausible concrete implementation of that field on macOS today.

Prereqs before this becomes more than a research note

1. Smoke-test TSI → pipelock on localhost. Confirm the guest can reach 127.0.0.1:8888 on the host through TSI when launched with --outbound-localhost-only, and that all other hosts are blocked.
2. Decide volume policy. Add a manifest-validation rule disallowing -v mounts in any bottle with "runtime": "microvm".
3. Decide control-plane shape. Either drive smolmachines via the HTTP API (smolvm serve as a long-running sidecar) or via CLI subprocess invocation per bottle. The HTTP API is the cleaner long-term path; CLI subprocesses are the lower-overhead first iteration.