Run smolmachines sidecar bundle as a smolVM image #332

New Issue

2026-07-09T01:32:18-04:00

didericis-codex commented

2026-07-09 01:32:18 -04:00

Summary

Change the smolmachines backend so the sidecar bundle runs as a smolVM image instead of a Docker container. The agent would remain in its own smolVM, and the sidecar VM would expose only the agent-facing sidecar services through the existing per-bottle loopback alias + TSI /32 pattern.

Motivation

The current smolmachines backend is hybrid: the agent runs in a libkrun/smolVM microVM, but egress/git-gate/supervise still run in a Docker sidecar bundle. Prior research notes indicate the local VM-contained sidecar direction is viable: smolvm --port/-p can publish guest services back to host loopback, and another smolVM guest can reach that service through the per-bottle loopback alias plus --allow-cidr.

Moving the sidecar bundle into smolVM would remove Docker from the smolmachines runtime path for the trusted sidecar services and make the backend boundary cleaner: agent VM + sidecar VM, with TSI limiting the agent to the sidecar alias.

Proposed shape

Build the existing sidecar bundle image into a .smolmachine artifact.
Launch a sidecar smolVM per bottle before the agent smolVM.
Run the existing sidecar init/supervisor inside the sidecar VM.
Publish only agent-facing sidecar ports on the per-bottle loopback alias.
Set the agent VM TSI allowlist to that alias /32, fail-closed as today.
Keep internal sidecar-only services bound to 127.0.0.1 inside the sidecar VM and not published on the host alias.
Stamp agent proxy/git/supervise config from the published sidecar ports.

Security requirements

The agent must not be able to reach generic host localhost (127.0.0.1) or other bottle aliases.
The agent may reach only ports intentionally published on its own per-bottle alias.
Raw/internal sidecar services must not be published on the alias.
Sidecar secrets/config/state delivery must not expose provider or forge credentials to the agent VM.
TSI allowlist verification remains fail-closed before agent VM boot.

Acceptance criteria

BOT_BOTTLE_BACKEND=smolmachines ./cli.py start <agent> can run with agent and sidecar both as smolVMs.
The Docker sidecar bundle is no longer required for the smolmachines launch path.
Existing egress DLP, route allowlist, git-gate, and supervise behavior remain intact.
Integration tests verify:
- agent can reach the sidecar published service(s) through the per-bottle alias;
- agent cannot reach host 127.0.0.1 services;
- agent cannot reach another bottle alias;
- internal sidecar-only ports are not reachable from the agent;
- teardown removes both VMs and any published port state.

Notes

The main remaining hard part is not VM-to-VM reachability itself; the research notes already record a viable loopback-alias path. The hard parts are packaging the sidecar VM, safe config/secret/state delivery, readiness/lifecycle handling, and ensuring the sidecar VM publishes only the intended agent-facing surface.

References:

docs/prds/0059-macos-container-backend.md notes that issue #220 verified smolVM guest port publishing plus another smolVM guest reaching it through the loopback alias path.
docs/prds/0023-smolmachines-backend.md documents the per-bottle loopback alias, TSI /32, and fail-closed allowlist enforcement requirements.

## Summary Change the `smolmachines` backend so the sidecar bundle runs as a smolVM image instead of a Docker container. The agent would remain in its own smolVM, and the sidecar VM would expose only the agent-facing sidecar services through the existing per-bottle loopback alias + TSI `/32` pattern. ## Motivation The current smolmachines backend is hybrid: the agent runs in a libkrun/smolVM microVM, but egress/git-gate/supervise still run in a Docker sidecar bundle. Prior research notes indicate the local VM-contained sidecar direction is viable: `smolvm --port/-p` can publish guest services back to host loopback, and another smolVM guest can reach that service through the per-bottle loopback alias plus `--allow-cidr`. Moving the sidecar bundle into smolVM would remove Docker from the smolmachines runtime path for the trusted sidecar services and make the backend boundary cleaner: agent VM + sidecar VM, with TSI limiting the agent to the sidecar alias. ## Proposed shape - Build the existing sidecar bundle image into a `.smolmachine` artifact. - Launch a sidecar smolVM per bottle before the agent smolVM. - Run the existing sidecar init/supervisor inside the sidecar VM. - Publish only agent-facing sidecar ports on the per-bottle loopback alias. - Set the agent VM TSI allowlist to that alias `/32`, fail-closed as today. - Keep internal sidecar-only services bound to `127.0.0.1` inside the sidecar VM and not published on the host alias. - Stamp agent proxy/git/supervise config from the published sidecar ports. ## Security requirements - The agent must not be able to reach generic host localhost (`127.0.0.1`) or other bottle aliases. - The agent may reach only ports intentionally published on its own per-bottle alias. - Raw/internal sidecar services must not be published on the alias. - Sidecar secrets/config/state delivery must not expose provider or forge credentials to the agent VM. - TSI allowlist verification remains fail-closed before agent VM boot. ## Acceptance criteria - `BOT_BOTTLE_BACKEND=smolmachines ./cli.py start <agent>` can run with agent and sidecar both as smolVMs. - The Docker sidecar bundle is no longer required for the smolmachines launch path. - Existing egress DLP, route allowlist, git-gate, and supervise behavior remain intact. - Integration tests verify: - agent can reach the sidecar published service(s) through the per-bottle alias; - agent cannot reach host `127.0.0.1` services; - agent cannot reach another bottle alias; - internal sidecar-only ports are not reachable from the agent; - teardown removes both VMs and any published port state. ## Notes The main remaining hard part is not VM-to-VM reachability itself; the research notes already record a viable loopback-alias path. The hard parts are packaging the sidecar VM, safe config/secret/state delivery, readiness/lifecycle handling, and ensuring the sidecar VM publishes only the intended agent-facing surface. References: - `docs/prds/0059-macos-container-backend.md` notes that issue #220 verified smolVM guest port publishing plus another smolVM guest reaching it through the loopback alias path. - `docs/prds/0023-smolmachines-backend.md` documents the per-bottle loopback alias, TSI `/32`, and fail-closed allowlist enforcement requirements.

didericis added the Kind/Feature label 2026-07-09 01:32:28 -04:00

didericis-codex referenced a pull request that will close this issue

2026-07-09 11:46:04 -04:00

PRD: smolmachines sidecar VM #334

didericis-codex referenced this issue

2026-07-09 11:46:13 -04:00

PRD: smolmachines sidecar VM #334

didericis commented

2026-07-11 03:11:50 -04:00

Too many issues with smolmachines on linux generally, even beyond the sidecar issue. Smolmachines initially seemed like a good way to avoid support a single cross platform vm, but is looking like using firecracker for linux and macos containers for mac makes more sense.

didericis closed this issue

2026-07-11 03:12:20 -04:00

didericis commented

2026-07-11 03:16:33 -04:00

Closing this — smolmachines on Linux has too many fundamental issues to justify building more infrastructure on top of it (like a sidecar VM).

Issues found

1. TSI networking dies under sustained use (Linux)
After a period of activity, all loopback connections through TSI (Transparent Socket Infrastructure) hang permanently — not just the per-bottle alias, but even 127.0.0.1 inside the guest. Non-network commands continue working fine. This is a libkrun bug, not something we can fix or work around. An all-smolVM sidecar architecture would make this worse since egress proxy, git-gate, and supervise all depend on network connectivity between VMs.

2. Exec channel contention
smolvm machine exec -i -t serializes all exec callers — an interactive session blocks diagnostics, SIGWINCH forwarding, and provisioning helpers until it exits. We worked around this with shell_launch.py (using the independent shell channel + PTY bridge), but it's another sign of immaturity. See 3247e0c.

3. No native SIGWINCH forwarding
smolvm doesn't forward host terminal resizes to in-VM PTYs. We built our own resize bridge via side-channel exec calls (pty_resize.py), but this is table-stakes functionality that Docker handles automatically.

Path forward

macOS Containers (macOS) + Firecracker (Linux) provides VM-level isolation on both platforms using mature, well-maintained VMMs. Both use hardware virtualization (Apple Hypervisor.framework / KVM respectively) so the security boundary is equivalent. This lets us drop smolvm entirely rather than continuing to work around its bugs.

Closing this — smolmachines on Linux has too many fundamental issues to justify building more infrastructure on top of it (like a sidecar VM). ### Issues found **1. TSI networking dies under sustained use (Linux)** After a period of activity, all loopback connections through TSI (Transparent Socket Infrastructure) hang permanently — not just the per-bottle alias, but even `127.0.0.1` inside the guest. Non-network commands continue working fine. This is a libkrun bug, not something we can fix or work around. An all-smolVM sidecar architecture would make this *worse* since egress proxy, git-gate, and supervise all depend on network connectivity between VMs. **2. Exec channel contention** `smolvm machine exec -i -t` serializes all exec callers — an interactive session blocks diagnostics, SIGWINCH forwarding, and provisioning helpers until it exits. We worked around this with `shell_launch.py` (using the independent `shell` channel + PTY bridge), but it's another sign of immaturity. See 3247e0c. **3. No native SIGWINCH forwarding** smolvm doesn't forward host terminal resizes to in-VM PTYs. We built our own resize bridge via side-channel exec calls (`pty_resize.py`), but this is table-stakes functionality that Docker handles automatically. ### Path forward macOS Containers (macOS) + Firecracker (Linux) provides VM-level isolation on both platforms using mature, well-maintained VMMs. Both use hardware virtualization (Apple Hypervisor.framework / KVM respectively) so the security boundary is equivalent. This lets us drop smolvm entirely rather than continuing to work around its bugs.

didericis-claude referenced this issue

2026-07-11 09:50:03 -04:00

Add Firecracker backend for Linux VM isolation #342

didericis referenced this issue from a commit

2026-07-11 10:33:01 -04:00

feat(firecracker): add Linux microVM backend to replace smolmachines

didericis-claude referenced this issue

2026-07-11 10:33:21 -04:00

feat(firecracker): Linux microVM backend to replace smolmachines #343

Sign in to join this conversation.