didericis
aba9a823ba
Transcript-style notes on running an agent in a hardware-isolated microVM on macOS. Covers Virtualization.framework / vfkit / libkrun choices, hardware-isolation guarantees, driving VMs from Python (subprocess or PyObjC), pipelock as the egress proxy, vsock for the control channel, and egress enforcement via VZFileHandleNetworkDeviceAttachment + gvisor-tap-vsock.
536 lines
17 KiB
Markdown
536 lines
17 KiB
Markdown
# Agent VM Isolation on macOS with Pipelock and gvproxy
|
||
|
||
A transcript covering microVM options for macOS, hardware-isolated agent sandboxing
|
||
with Virtualization.framework + PyObjC, network egress enforcement via
|
||
`VZFileHandleNetworkDeviceAttachment` + gvisor-tap-vsock, and Pipelock as a
|
||
scanning proxy.
|
||
|
||
---
|
||
|
||
## MicroVMs Available for macOS
|
||
|
||
**Apple's Native Framework**
|
||
- **Virtualization.framework** — Apple's first-party API (macOS 11+) for lightweight VMs. Most other tools build on top of this.
|
||
|
||
**Built on Virtualization.framework**
|
||
- **vfkit** — CLI tool from Red Hat, very lightweight, used heavily in container tooling (Podman Desktop, etc.)
|
||
- **libkrun** — library for running microVMs, focused on minimal overhead; also from Red Hat
|
||
- **QEMU** (with `hvf` accelerator) — not a true microVM but can be configured lightweight; uses Apple Hypervisor.framework directly
|
||
|
||
**Higher-Level Tools (microVM-oriented)**
|
||
- **OrbStack** — fast, lightweight Linux VM runner; uses Virtualization.framework under the hood; very low overhead compared to Docker Desktop
|
||
- **Lima** — lightweight Linux VMs for macOS, supports both QEMU and vfkit backends; popular for running containerd/nerdctl
|
||
- **Podman Desktop / Podman Machine** — uses vfkit for fast Linux VMs
|
||
- **Colima** — container runtimes on macOS via Lima; supports Docker and containerd
|
||
|
||
**Lower-Level / Embedded**
|
||
- **CrosVM** — Chrome OS's VM monitor, has some macOS support via the Hypervisor.framework backend
|
||
- **Cloud Hypervisor** — has experimental macOS support via `hvf`
|
||
|
||
**Key constraint:** macOS doesn't support KVM, so everything either goes through **Apple Hypervisor.framework** (for hardware acceleration) or **Virtualization.framework** (higher-level abstraction on top of it). True Linux-native microVMs like Firecracker don't run on macOS since Firecracker requires KVM.
|
||
|
||
---
|
||
|
||
## Hardware Isolation for AI Agents
|
||
|
||
Anything using **Apple Hypervisor.framework** gets real hardware-enforced isolation —
|
||
separate page tables, MMU/IOMMU enforcement, no shared memory with the host unless
|
||
explicitly mapped. This is comparable to KVM-level isolation on Linux.
|
||
|
||
Firecracker (the gold standard for agent sandboxing, used by AWS Lambda) requires
|
||
KVM, so it's out on macOS natively.
|
||
|
||
### Best Options
|
||
|
||
**vfkit + Virtualization.framework** — probably the best path:
|
||
- Thinnest wrapper over Virtualization.framework
|
||
- Hardware-isolated via Hypervisor.framework underneath
|
||
- Fast boot (~1–2s for a minimal Linux guest)
|
||
- Control networking (virtio-net with NAT or isolated bridge), block devices, and resource limits directly
|
||
- No extra abstractions that could introduce shared state
|
||
|
||
**Virtualization.framework directly (Swift/ObjC)** — if programmatic control is needed:
|
||
- Full control over VM lifecycle, device model, memory limits
|
||
- Snapshot/restore, enforce CPU/mem caps, configure virtio devices precisely
|
||
- Best option for building infrastructure (not just running VMs ad-hoc)
|
||
|
||
### What to Watch For
|
||
|
||
| Concern | Mitigation |
|
||
|---|---|
|
||
| Network isolation | Use isolated virtio-net; don't bridge to host NIC |
|
||
| Filesystem | Don't use virtio-fs/9p shares; use a sealed disk image instead |
|
||
| Shared memory (virtiomem) | Don't enable it |
|
||
| Resource exhaustion | Set hard vCPU and memory caps at VM creation |
|
||
| Agent escape via device emulation | vfkit/Virt.framework has a minimal device model — less attack surface than QEMU |
|
||
|
||
### What to Avoid
|
||
|
||
- **Lima / Colima / OrbStack** — they add layers for convenience (shared mounts, host socket bridging) that work against isolation
|
||
- **QEMU** — larger attack surface due to full device emulation; more CVE history
|
||
- **Docker Desktop** — lots of host sharing by design
|
||
|
||
---
|
||
|
||
## Driving VMs from Python
|
||
|
||
### Option 1: vfkit via subprocess
|
||
|
||
Drive it with `subprocess.Popen`, configure via CLI flags or a JSON config file.
|
||
|
||
```python
|
||
import subprocess, json, tempfile, os
|
||
|
||
config = {
|
||
"vcpus": 2,
|
||
"memory": 512, # MB
|
||
"bootloader": {
|
||
"kind": "linux",
|
||
"kernel": "/path/to/vmlinuz",
|
||
"initrd": "/path/to/initrd",
|
||
"cmdline": "console=ttyS0 root=/dev/vda"
|
||
},
|
||
"devices": [
|
||
{"kind": "virtio-blk", "path": "/path/to/agent.img"},
|
||
{"kind": "virtio-net", "nat": {}}, # NAT only — no bridge to host
|
||
{"kind": "virtio-serial", "logFile": "/tmp/vm-serial.log"},
|
||
{"kind": "virtio-vsock", "port": 1024, "socketURL": "/tmp/agent.sock"}
|
||
]
|
||
}
|
||
|
||
with tempfile.NamedTemporaryFile("w", suffix=".json", delete=False) as f:
|
||
json.dump(config, f)
|
||
config_path = f.name
|
||
|
||
proc = subprocess.Popen(["vfkit", "--config", config_path])
|
||
```
|
||
|
||
**vsock** gives you a Unix socket on the host that maps to a virtio socket in the
|
||
guest — a clean communication channel without opening any network ports.
|
||
|
||
### Option 2: PyObjC + Virtualization.framework
|
||
|
||
More verbose but gives full programmatic lifecycle management — snapshots, resource
|
||
adjustment, event callbacks. Requires `pip install pyobjc-framework-Virtualization`.
|
||
|
||
```python
|
||
from Virtualization import (
|
||
VZVirtualMachineConfiguration,
|
||
VZLinuxBootLoader,
|
||
VZVirtioNetworkDeviceConfiguration,
|
||
VZNATNetworkDeviceAttachment,
|
||
VZVirtualMachine,
|
||
)
|
||
|
||
bootloader = VZLinuxBootLoader.alloc().initWithKernelURL_(kernel_url)
|
||
bootloader.setCommandLine_("root=/dev/vda console=ttyS0")
|
||
|
||
config = VZVirtualMachineConfiguration.alloc().init()
|
||
config.setBootLoader_(bootloader)
|
||
config.setCPUCount_(2)
|
||
config.setMemorySize_(512 * 1024 * 1024)
|
||
|
||
# ... add devices ...
|
||
|
||
config.validateWithError_(None)
|
||
vm = VZVirtualMachine.alloc().initWithConfiguration_(config)
|
||
vm.startWithCompletionHandler_(lambda err: print("started"))
|
||
```
|
||
|
||
### Comparison
|
||
|
||
| | vfkit + subprocess | PyObjC + Virt.framework |
|
||
|---|---|---|
|
||
| Setup effort | Low | Medium |
|
||
| Programmatic control | Medium | High |
|
||
| Snapshot support | No | Yes (Apple Silicon) |
|
||
| Async lifecycle events | Via poll/log | Native callbacks |
|
||
| Best for | ≤10 concurrent VMs | VM pool / scale |
|
||
|
||
---
|
||
|
||
## Pipelock as Egress Proxy
|
||
|
||
Pipelock's design is capability separation: the agent holds secrets but has no direct
|
||
network access, the proxy holds network access but no agent secrets, and all traffic
|
||
crosses a scanning boundary between the two.
|
||
|
||
### Architecture
|
||
|
||
```
|
||
┌─────────────────────────────────┐ ┌──────────┐
|
||
│ macOS Host │ │ │
|
||
│ │ │ Internet │
|
||
│ ┌──────────┐ virtio-net NAT │ │ │
|
||
│ │ VM │──────────────────►│ pf └──────────┘
|
||
│ │ (agent) │ 192.168.64.0/24 │ rules ▲
|
||
│ └──────────┘ │ │
|
||
│ │ │ ┌──────────────────┐
|
||
│ │ HTTPS_PROXY │ │ Pipelock │
|
||
│ └────────────────────────►│ │ :8888 │
|
||
│ │ └──────────────────┘
|
||
└─────────────────────────────────┘
|
||
```
|
||
|
||
### Pipelock Config
|
||
|
||
```bash
|
||
pipelock generate config --preset balanced > pipelock.yaml
|
||
```
|
||
|
||
```yaml
|
||
fetch_proxy:
|
||
listen: "192.168.64.1:8888"
|
||
mcp_proxy:
|
||
listen: "192.168.64.1:9999"
|
||
```
|
||
|
||
### Enforce with `pf` (optional — see next section for better approach)
|
||
|
||
```
|
||
# /etc/pf.anchors/agent-isolation
|
||
block out quick from 192.168.64.0/24 to any
|
||
pass out quick from 192.168.64.0/24 to 192.168.64.1 port 8888
|
||
pass out quick from 192.168.64.0/24 to 192.168.64.1 port 9999
|
||
```
|
||
|
||
```bash
|
||
sudo pfctl -a agent-isolation -f /etc/pf.anchors/agent-isolation
|
||
sudo pfctl -e
|
||
```
|
||
|
||
### PyObjC VM Setup with Pipelock
|
||
|
||
```python
|
||
import objc
|
||
from Foundation import NSURL
|
||
from Virtualization import (
|
||
VZVirtualMachineConfiguration,
|
||
VZLinuxBootLoader,
|
||
VZVirtioNetworkDeviceConfiguration,
|
||
VZNATNetworkDeviceAttachment,
|
||
VZVirtioBlockDeviceConfiguration,
|
||
VZDiskImageStorageDeviceAttachment,
|
||
VZVirtioSocketDeviceConfiguration,
|
||
VZVirtualMachine,
|
||
)
|
||
|
||
KERNEL = "/path/to/vmlinuz"
|
||
INITRD = "/path/to/initrd.img"
|
||
DISK = "/path/to/agent.img"
|
||
PIPELOCK_HOST = "192.168.64.1"
|
||
PIPELOCK_PORT = 8888
|
||
|
||
def make_vm_config():
|
||
config = VZVirtualMachineConfiguration.alloc().init()
|
||
|
||
bootloader = VZLinuxBootLoader.alloc().initWithKernelURL_(
|
||
NSURL.fileURLWithPath_(KERNEL)
|
||
)
|
||
bootloader.setInitialRamdiskURL_(NSURL.fileURLWithPath_(INITRD))
|
||
bootloader.setCommandLine_(
|
||
f"root=/dev/vda console=ttyS0 "
|
||
f"https_proxy=http://{PIPELOCK_HOST}:{PIPELOCK_PORT} "
|
||
f"http_proxy=http://{PIPELOCK_HOST}:{PIPELOCK_PORT} "
|
||
f"no_proxy=localhost,127.0.0.1"
|
||
)
|
||
config.setBootLoader_(bootloader)
|
||
config.setCPUCount_(2)
|
||
config.setMemorySize_(512 * 1024 * 1024)
|
||
|
||
disk_attachment = VZDiskImageStorageDeviceAttachment.alloc().initWithURL_readOnly_error_(
|
||
NSURL.fileURLWithPath_(DISK), False, None
|
||
)
|
||
disk = VZVirtioBlockDeviceConfiguration.alloc().initWithAttachment_(disk_attachment)
|
||
config.setStorageDevices_([disk])
|
||
|
||
net_device = VZVirtioNetworkDeviceConfiguration.alloc().init()
|
||
net_device.setAttachment_(VZNATNetworkDeviceAttachment.alloc().init())
|
||
config.setNetworkDevices_([net_device])
|
||
|
||
vsock = VZVirtioSocketDeviceConfiguration.alloc().init()
|
||
config.setSocketDevices_([vsock])
|
||
|
||
config.validateWithError_(None)
|
||
return config
|
||
|
||
|
||
class AgentVM:
|
||
def __init__(self):
|
||
self.config = make_vm_config()
|
||
self.vm = VZVirtualMachine.alloc().initWithConfiguration_(self.config)
|
||
|
||
def start(self):
|
||
self.vm.startWithCompletionHandler_(
|
||
lambda err: print(f"VM start error: {err}") if err else print("VM started")
|
||
)
|
||
|
||
def stop(self):
|
||
self.vm.stopWithCompletionHandler_(lambda err: None)
|
||
```
|
||
|
||
### vsock Control Channel
|
||
|
||
Use vsock for control plane (sending tasks, receiving results) — entirely separate
|
||
from HTTP, never egressing to the internet.
|
||
|
||
**Host side:**
|
||
```python
|
||
import socket
|
||
|
||
VSOCK_PATH = "/tmp/agent-control.sock"
|
||
|
||
server = socket.socket(socket.AF_UNIX, socket.SOCK_STREAM)
|
||
server.bind(VSOCK_PATH)
|
||
server.listen(1)
|
||
|
||
conn, _ = server.accept()
|
||
conn.sendall(b'{"task": "run", "prompt": "..."}')
|
||
result = conn.recv(65536)
|
||
```
|
||
|
||
**Guest side:**
|
||
```python
|
||
import socket, json
|
||
|
||
sock = socket.socket(socket.AF_VSOCK, socket.SOCK_STREAM)
|
||
sock.connect((socket.VMADDR_CID_HOST, 1024))
|
||
task = json.loads(sock.recv(65536))
|
||
# run agent...
|
||
sock.sendall(json.dumps(result).encode())
|
||
```
|
||
|
||
---
|
||
|
||
## Egress Enforcement Without pf: VZFileHandleNetworkDeviceAttachment
|
||
|
||
Instead of NAT (which gives the VM real network access), give the VM a **raw
|
||
ethernet interface backed by a file handle pair**. You own the other end — every
|
||
packet flows through your process.
|
||
|
||
```
|
||
VM (virtio-net) ──── raw frames ────► your Python process ──► Pipelock only
|
||
(file handle pair) (userspace stack)
|
||
```
|
||
|
||
### Option A: gvisor-tap-vsock (recommended)
|
||
|
||
gvproxy implements a full userspace TCP/IP stack and is designed exactly for this.
|
||
It handles ARP, DHCP, and DNS in userspace, and you control what it routes.
|
||
|
||
### Option B: No NIC at all
|
||
|
||
Give the VM no network device and use vsock only. Zero network attack surface.
|
||
|
||
```
|
||
VM ──vsock──► host relay ──HTTP──► Pipelock ──► internet
|
||
```
|
||
|
||
### Isolation Comparison
|
||
|
||
| Approach | Isolation | Complexity |
|
||
|---|---|---|
|
||
| `pf` rules | OS firewall (bypassable by root in VM) | Low |
|
||
| `VZFileHandleNetworkDeviceAttachment` + gvproxy | Virtualization layer | Medium |
|
||
| `VZFileHandleNetworkDeviceAttachment` + custom forwarder | Virtualization layer | High |
|
||
| No NIC + vsock relay | Hardest possible | Low–Medium |
|
||
|
||
---
|
||
|
||
## Full Setup: gvisor-tap-vsock + PyObjC + Pipelock
|
||
|
||
### Install
|
||
|
||
```bash
|
||
brew install go
|
||
go install github.com/containers/gvisor-tap-vsock/cmd/gvproxy@latest
|
||
go install github.com/luckyPipewrench/pipelock@latest
|
||
```
|
||
|
||
### Architecture
|
||
|
||
```
|
||
VM (virtio-net, unixgram)
|
||
│ VFKT handshake + raw ethernet frames
|
||
▼
|
||
gvproxy (192.168.127.1 gateway)
|
||
│ DNS: NXDOMAIN for everything except Pipelock
|
||
│ NAT: all outbound → host
|
||
▼
|
||
Pipelock (127.0.0.1:8888)
|
||
│ scans, blocks, logs
|
||
▼
|
||
Internet
|
||
```
|
||
|
||
### Pipelock Config
|
||
|
||
```yaml
|
||
# pipelock.yaml
|
||
fetch_proxy:
|
||
listen: "127.0.0.1:8888"
|
||
mcp_proxy:
|
||
listen: "127.0.0.1:9999"
|
||
```
|
||
|
||
### gvproxy Config
|
||
|
||
```yaml
|
||
# gvproxy.yaml
|
||
subnet: "192.168.127.0/24"
|
||
gateway: "192.168.127.1"
|
||
|
||
dns:
|
||
- zone: "."
|
||
records:
|
||
- name: "proxy.internal"
|
||
ip: "192.168.127.1"
|
||
# Everything else gets NXDOMAIN
|
||
|
||
port_forwards:
|
||
- gateway_port: 8888
|
||
host: "127.0.0.1"
|
||
host_port: 8888
|
||
```
|
||
|
||
### Python: Wiring VZFileHandleNetworkDeviceAttachment to gvproxy
|
||
|
||
The connection protocol is unixgram + a `VFKT` magic byte handshake.
|
||
|
||
```python
|
||
import os, socket, subprocess, time
|
||
import objc
|
||
from Foundation import NSFileHandle, NSURL
|
||
from Virtualization import (
|
||
VZVirtualMachineConfiguration,
|
||
VZLinuxBootLoader,
|
||
VZVirtioNetworkDeviceConfiguration,
|
||
VZFileHandleNetworkDeviceAttachment,
|
||
VZVirtioBlockDeviceConfiguration,
|
||
VZDiskImageStorageDeviceAttachment,
|
||
VZVirtioSocketDeviceConfiguration,
|
||
VZVirtualMachine,
|
||
)
|
||
|
||
GVPROXY_SOCK = "/tmp/agent-gvproxy.sock"
|
||
VM_CLIENT_SOCK = "/tmp/agent-vm-client.sock"
|
||
KERNEL = "/path/to/vmlinuz"
|
||
INITRD = "/path/to/initrd.img"
|
||
DISK = "/path/to/agent.img"
|
||
|
||
def start_gvproxy():
|
||
for p in [GVPROXY_SOCK, VM_CLIENT_SOCK]:
|
||
try: os.unlink(p)
|
||
except FileNotFoundError: pass
|
||
|
||
proc = subprocess.Popen([
|
||
"gvproxy",
|
||
"--listen-vfkit", f"unixgram://{GVPROXY_SOCK}",
|
||
"--listen", f"unix://{GVPROXY_SOCK}.api",
|
||
"-config", "gvproxy.yaml",
|
||
])
|
||
for _ in range(20):
|
||
if os.path.exists(GVPROXY_SOCK):
|
||
break
|
||
time.sleep(0.1)
|
||
else:
|
||
raise RuntimeError("gvproxy socket never appeared")
|
||
return proc
|
||
|
||
def make_vfkit_socket():
|
||
sock = socket.socket(socket.AF_UNIX, socket.SOCK_DGRAM)
|
||
try: os.unlink(VM_CLIENT_SOCK)
|
||
except FileNotFoundError: pass
|
||
sock.bind(VM_CLIENT_SOCK)
|
||
sock.sendto(b"VFKT", GVPROXY_SOCK)
|
||
return sock
|
||
|
||
def make_network_attachment(sock):
|
||
fd = sock.fileno()
|
||
read_fd = os.dup(fd)
|
||
write_fd = os.dup(fd)
|
||
read_handle = NSFileHandle.alloc().initWithFileDescriptor_closeOnDealloc_(read_fd, True)
|
||
write_handle = NSFileHandle.alloc().initWithFileDescriptor_closeOnDealloc_(write_fd, True)
|
||
return VZFileHandleNetworkDeviceAttachment.alloc().initWithFileHandleForReading_fileHandleForWriting_(
|
||
read_handle, write_handle
|
||
)
|
||
|
||
def make_vm_config(net_attachment):
|
||
config = VZVirtualMachineConfiguration.alloc().init()
|
||
|
||
bootloader = VZLinuxBootLoader.alloc().initWithKernelURL_(
|
||
NSURL.fileURLWithPath_(KERNEL)
|
||
)
|
||
bootloader.setInitialRamdiskURL_(NSURL.fileURLWithPath_(INITRD))
|
||
bootloader.setCommandLine_(
|
||
"root=/dev/vda console=ttyS0 "
|
||
"https_proxy=http://proxy.internal:8888 "
|
||
"http_proxy=http://proxy.internal:8888 "
|
||
"no_proxy=192.168.127.1"
|
||
)
|
||
config.setBootLoader_(bootloader)
|
||
config.setCPUCount_(2)
|
||
config.setMemorySize_(512 * 1024 * 1024)
|
||
|
||
disk_attach = VZDiskImageStorageDeviceAttachment.alloc()\
|
||
.initWithURL_readOnly_error_(NSURL.fileURLWithPath_(DISK), False, None)
|
||
disk = VZVirtioBlockDeviceConfiguration.alloc().initWithAttachment_(disk_attach)
|
||
config.setStorageDevices_([disk])
|
||
|
||
net_device = VZVirtioNetworkDeviceConfiguration.alloc().init()
|
||
net_device.setAttachment_(net_attachment)
|
||
config.setNetworkDevices_([net_device])
|
||
|
||
config.setSocketDevices_([VZVirtioSocketDeviceConfiguration.alloc().init()])
|
||
|
||
config.validateWithError_(None)
|
||
return config
|
||
|
||
|
||
class AgentVM:
|
||
def __init__(self):
|
||
self.gvproxy = start_gvproxy()
|
||
self._sock = make_vfkit_socket()
|
||
attachment = make_network_attachment(self._sock)
|
||
cfg = make_vm_config(attachment)
|
||
self.vm = VZVirtualMachine.alloc().initWithConfiguration_(cfg)
|
||
|
||
def start(self):
|
||
self.vm.startWithCompletionHandler_(
|
||
lambda err: print("VM error:", err) if err else print("VM started")
|
||
)
|
||
|
||
def stop(self):
|
||
self.vm.stopWithCompletionHandler_(lambda err: None)
|
||
self.gvproxy.terminate()
|
||
self._sock.close()
|
||
```
|
||
|
||
### Starting Everything
|
||
|
||
```python
|
||
import subprocess, time
|
||
|
||
pipelock = subprocess.Popen(["pipelock", "run", "--config", "pipelock.yaml"])
|
||
time.sleep(0.5)
|
||
|
||
vm = AgentVM()
|
||
vm.start()
|
||
```
|
||
|
||
---
|
||
|
||
## Egress Enforcement: Honest Assessment
|
||
|
||
| Layer | What it stops | What it doesn't stop |
|
||
|---|---|---|
|
||
| gvproxy DNS (NXDOMAIN) | Agent resolving arbitrary hostnames | Agent dialing IPs directly |
|
||
| HTTPS_PROXY in VM env | Well-behaved agents | Agent unsetting the env var |
|
||
| `pf` rules | Everything including direct IP dials | Nothing (kernel-enforced) |
|
||
| Custom Python frame filter | Everything | Nothing, but complex to write |
|
||
|
||
For most agent isolation use cases, DNS + HTTPS_PROXY is sufficient. For a
|
||
potentially hostile agent that might actively attempt to bypass the proxy, add `pf`
|
||
on top, or replace the gvproxy attachment with a custom frame filter that acts as a
|
||
proper TCP-level allowlist.
|