PRD: smolmachines sidecar VM #334

2026-07-09T11:46:04-04:00

didericis-codex commented

2026-07-09 11:46:04 -04:00

Closes #332.

PRD: 1a2d46ee56/docs/prds/prd-new-smolmachines-sidecar-vm.md

Summary

Runs the smolmachines sidecar bundle as a sidecar smolVM instead of a Docker container.

Adds a stdlib host TCP forwarder that binds only the per-bottle address and forwards to raw smolVM-published loopback ports, so the agent receives only per-bottle egress/git/supervise URLs and never the raw sidecar VM ports.

Updates smolmachines egress reload to signal the sidecar VM and keeps sidecar VMs out of active-agent enumeration.

Merge rule(s)

Run smolmachines integration coverage on a host with smolVM/KVM or Hypervisor.framework before merge, including alias reachability, host-localhost denial, other-bottle alias denial, internal sidecar port denial, and teardown cleanup.

Closes #332. PRD: https://gitea.dideric.is/didericis/bot-bottle/src/commit/1a2d46ee568d9dc9c06850723e6313529ddac46a/docs/prds/prd-new-smolmachines-sidecar-vm.md ## Summary Runs the smolmachines sidecar bundle as a sidecar smolVM instead of a Docker container. Adds a stdlib host TCP forwarder that binds only the per-bottle address and forwards to raw smolVM-published loopback ports, so the agent receives only per-bottle egress/git/supervise URLs and never the raw sidecar VM ports. Updates smolmachines egress reload to signal the sidecar VM and keeps sidecar VMs out of active-agent enumeration. ## Merge rule(s) Run smolmachines integration coverage on a host with smolVM/KVM or Hypervisor.framework before merge, including alias reachability, host-localhost denial, other-bottle alias denial, internal sidecar port denial, and teardown cleanup.

didericis-codex commented

2026-07-09 11:46:13 -04:00

Implementation is blocked on sidecar VM port publishing semantics. Issue #332 requires publishing agent-facing sidecar services only on the per-bottle address, but current smolVM docs/source shape indicate -p/--port accepts HOST:GUEST as two port numbers, not an address-scoped bind such as 127.0.0.16:<random>:9099. Because TSI is IP-only, publishing on generic localhost would not preserve the security property that the agent can reach only its own sidecar surface. The draft PRD records the required decision: add/confirm address-scoped smolVM publishing, add a host-side per-address forwarding layer, or choose an equivalent fail-closed networking design before implementation proceeds.

Implementation is blocked on sidecar VM port publishing semantics. Issue #332 requires publishing agent-facing sidecar services only on the per-bottle address, but current smolVM docs/source shape indicate `-p/--port` accepts `HOST:GUEST` as two port numbers, not an address-scoped bind such as `127.0.0.16:<random>:9099`. Because TSI is IP-only, publishing on generic localhost would not preserve the security property that the agent can reach only its own sidecar surface. The draft PRD records the required decision: add/confirm address-scoped smolVM publishing, add a host-side per-address forwarding layer, or choose an equivalent fail-closed networking design before implementation proceeds.

didericis commented

2026-07-09 11:59:12 -04:00

@didericis-codex paste the result of our discussion RE a python stdlib TCP forwarder that lives on the host. Also factor in any security concerns/risk of some kind of package designed to escape the forwarder. Am assuming that's easy to mitigate given the simplicity of the forwarder/not really an issue.

didericis-codex commented

2026-07-09 11:59:52 -04:00

@didericis Summary from the discussion: the practical fallback is a host-local TCP forwarder owned by bot-bottle. smolVM publishes each sidecar guest port on a raw host loopback port, then bot-bottle exposes only a per-bottle alias listener to the agent:

agent VM
  -> TSI allows only <bottle-alias>/32
  -> bot-bottle forwarder bound to <bottle-alias>:<random-port>
  -> raw smolVM-published sidecar port on 127.0.0.1:<raw-port>
  -> sidecar VM service

The agent env is stamped from the alias listener ports, never from the raw smolVM ports. Example:

HTTPS_PROXY=http://127.0.0.16:43001
GIT_GATE_URL=http://127.0.0.16:43002
MCP_SUPERVISE_URL=http://127.0.0.16:43003/

Recommended implementation: a small Python stdlib helper process, probably one process per bottle managing all active forwards. It can use socket plus selectors or a small thread-per-direction pipe. I would keep the public interface narrow: ForwardSpec(label, listen_host, listen_port, target_host, target_port) plus start/stop/readiness. That leaves room to swap in socat or a tiny compiled helper later if measurements show Python is a problem. My expectation is Python is fine here because traffic is already dominated by mitmproxy/DLP, git hooks/gitleaks, smolVM networking, and provider latency. The risk to manage is correctness/lifecycle, not throughput.

Security notes:

The forwarder must reject wildcard binds (0.0.0.0, ::) and reject 127.0.0.1 for agent-facing listeners. It should only bind the allocated per-bottle address.
It should only create forwards for the explicit agent-facing ports: egress, git HTTP when enabled, and supervise when enabled. Internal sidecar-only ports must never get a ForwardSpec.
The raw smolVM-published ports should remain on host loopback and should not be placed in the agent env. The agent VM should be unable to reach them because TSI allows only the bottle alias /32, not 127.0.0.1/32.
The forwarder should be byte-transparent. It should not parse HTTP, CONNECT, TLS, Git, or MCP. That keeps the attack surface small and avoids protocol confusion bugs.
A malicious package inside the agent can send arbitrary bytes to the allowed alias ports, but that is already true for the intended sidecar services. It should not be able to escape the forwarder unless the forwarder exposes extra destinations, accepts destination selection from the client, follows proxy protocol headers, or binds broader addresses. We should do none of those.
Do not log payload bytes. Log only lifecycle and endpoint metadata, with labels, to avoid leaking secrets passing through egress or git.
Teardown needs to be deterministic: close listeners, close active sockets, stop the helper process, and fail closed if the forwarder cannot bind exactly the requested alias.

Tests I would require with this design:

unit: rejects wildcard and generic localhost listen hosts;
unit: forwards bytes both directions and handles half-close/backpressure;
unit: launch stamps agent URLs from forwarder ports, not raw smolVM ports;
integration: agent reaches published sidecar service through its alias;
integration: agent cannot reach host 127.0.0.1;
integration: agent cannot reach another bottle alias;
integration: internal sidecar-only ports are not reachable;
teardown: both VMs and all forwarder listeners/processes are gone.

Net: a stdlib forwarder is acceptable if we treat it as a dumb, fixed-route TCP pipe with strict bind validation and no client-controlled target selection.

@didericis Summary from the discussion: the practical fallback is a host-local TCP forwarder owned by bot-bottle. smolVM publishes each sidecar guest port on a raw host loopback port, then bot-bottle exposes only a per-bottle alias listener to the agent: ```text agent VM -> TSI allows only <bottle-alias>/32 -> bot-bottle forwarder bound to <bottle-alias>:<random-port> -> raw smolVM-published sidecar port on 127.0.0.1:<raw-port> -> sidecar VM service ``` The agent env is stamped from the alias listener ports, never from the raw smolVM ports. Example: ```text HTTPS_PROXY=http://127.0.0.16:43001 GIT_GATE_URL=http://127.0.0.16:43002 MCP_SUPERVISE_URL=http://127.0.0.16:43003/ ``` Recommended implementation: a small Python stdlib helper process, probably one process per bottle managing all active forwards. It can use `socket` plus `selectors` or a small thread-per-direction pipe. I would keep the public interface narrow: `ForwardSpec(label, listen_host, listen_port, target_host, target_port)` plus start/stop/readiness. That leaves room to swap in `socat` or a tiny compiled helper later if measurements show Python is a problem. My expectation is Python is fine here because traffic is already dominated by mitmproxy/DLP, git hooks/gitleaks, smolVM networking, and provider latency. The risk to manage is correctness/lifecycle, not throughput. Security notes: - The forwarder must reject wildcard binds (`0.0.0.0`, `::`) and reject `127.0.0.1` for agent-facing listeners. It should only bind the allocated per-bottle address. - It should only create forwards for the explicit agent-facing ports: egress, git HTTP when enabled, and supervise when enabled. Internal sidecar-only ports must never get a `ForwardSpec`. - The raw smolVM-published ports should remain on host loopback and should not be placed in the agent env. The agent VM should be unable to reach them because TSI allows only the bottle alias `/32`, not `127.0.0.1/32`. - The forwarder should be byte-transparent. It should not parse HTTP, CONNECT, TLS, Git, or MCP. That keeps the attack surface small and avoids protocol confusion bugs. - A malicious package inside the agent can send arbitrary bytes to the allowed alias ports, but that is already true for the intended sidecar services. It should not be able to escape the forwarder unless the forwarder exposes extra destinations, accepts destination selection from the client, follows proxy protocol headers, or binds broader addresses. We should do none of those. - Do not log payload bytes. Log only lifecycle and endpoint metadata, with labels, to avoid leaking secrets passing through egress or git. - Teardown needs to be deterministic: close listeners, close active sockets, stop the helper process, and fail closed if the forwarder cannot bind exactly the requested alias. Tests I would require with this design: 1. unit: rejects wildcard and generic localhost listen hosts; 2. unit: forwards bytes both directions and handles half-close/backpressure; 3. unit: launch stamps agent URLs from forwarder ports, not raw smolVM ports; 4. integration: agent reaches published sidecar service through its alias; 5. integration: agent cannot reach host `127.0.0.1`; 6. integration: agent cannot reach another bottle alias; 7. integration: internal sidecar-only ports are not reachable; 8. teardown: both VMs and all forwarder listeners/processes are gone. Net: a stdlib forwarder is acceptable if we treat it as a dumb, fixed-route TCP pipe with strict bind validation and no client-controlled target selection.

didericis commented

2026-07-09 12:01:37 -04:00

@didericis-codex Summary looks good, implement it.

didericis-codex commented

2026-07-09 12:02:04 -04:00

Acknowledged. I’ll implement the stdlib host TCP forwarder path with strict per-bottle bind validation, switch the smolmachines sidecar launch to the sidecar VM + forwarder lifecycle, and keep the PRD updated with that design.

didericis-codex commented

2026-07-09 12:11:14 -04:00

Implemented in 1a2d46e:

sidecar bundle now launches as bot-bottle-sidecars-<slug> smolVM from the packed sidecar image;
smolVM raw -p HOST:GUEST ports are wrapped by a stdlib TCP forwarder bound only to the per-bottle address;
agent env is stamped from forwarder ports, not raw smolVM loopback ports;
forwarder rejects wildcard/generic-localhost listener binds and non-loopback targets;
smolmachines egress apply reads/reloads via smolvm machine exec;
sidecar VMs are filtered out of active-agent enumeration;
PRD updated to the accepted forwarder design and marked Active.

Validation run locally: python3 -m unittest discover tests/unit passed. pyright was not available in this bottle (python3 -m pyright and pyright missing).

Remaining merge gate: run the smolmachines integration tests on a host with real smolVM support to validate alias reachability, host-localhost denial, other-bottle alias denial, internal sidecar port denial, and teardown cleanup.

Implemented in `1a2d46e`: - sidecar bundle now launches as `bot-bottle-sidecars-<slug>` smolVM from the packed sidecar image; - smolVM raw `-p HOST:GUEST` ports are wrapped by a stdlib TCP forwarder bound only to the per-bottle address; - agent env is stamped from forwarder ports, not raw smolVM loopback ports; - forwarder rejects wildcard/generic-localhost listener binds and non-loopback targets; - smolmachines egress apply reads/reloads via `smolvm machine exec`; - sidecar VMs are filtered out of active-agent enumeration; - PRD updated to the accepted forwarder design and marked Active. Validation run locally: `python3 -m unittest discover tests/unit` passed. `pyright` was not available in this bottle (`python3 -m pyright` and `pyright` missing). Remaining merge gate: run the smolmachines integration tests on a host with real smolVM support to validate alias reachability, host-localhost denial, other-bottle alias denial, internal sidecar port denial, and teardown cleanup.

didericis reviewed 2026-07-09 13:01:06 -04:00

bot_bottle/backend/smolmachines/launch.py Outdated

						
				@@ -328,26 +342,33 @@ def _init_vm(plan: SmolmachinesBottlePlan) -> None:

				def _proxy_host(plan: SmolmachinesBottlePlan, loopback_ip: str) -> str:

didericis commented

2026-07-09 13:01:06 -04:00

This is probably irrelevant now/can get rid of it, right?

didericis-codex marked this conversation as resolved

didericis commented

2026-07-09 13:10:03 -04:00

@didericis-codex am not seeing where we prevent collisions if we launch two sidecars, each of which (I think) would bind the same port to localhost. We need to account for that, correct?

Am also wondering whether or not we should change the architecture here and have the orchestrator process also be responsible for doing port forwarding for everything. Then we'd just have one persistent host process, and teardown/spin up would be simplified. Downside is less robustness, but I think it'd be robust enough.

@didericis-codex am not seeing where we prevent collisions if we launch two sidecars, each of which (I think) would bind the same port to localhost. We need to account for that, correct? Am also wondering whether or not we should change the architecture here and have the orchestrator process also be responsible for doing port forwarding for everything. Then we'd just have one persistent host process, and teardown/spin up would be simplified. Downside is less robustness, but I think it'd be robust enough.

didericis-codex commented

2026-07-09 13:10:36 -04:00

@didericis Correct concern, but the current implementation does not bind the same host port for each sidecar. start_bundle_vm() calls allocate_raw_host_ports(spec.ports_to_publish), which asks the kernel for a free host loopback port with bind(("127.0.0.1", 0)) for each guest port, then passes those random host ports to smolVM as (host_port, guest_port) pairs. So two sidecars both exposing guest 9099 should become something like 127.0.0.1:55001 -> guest:9099 and 127.0.0.1:55017 -> guest:9099, then each gets wrapped by its own per-bottle alias forwarder port.

There is still a normal reserve-then-bind race because the socket used to discover the free port is closed before smolVM binds it. If another process grabs the port in that gap, smolVM launch should fail closed rather than silently cross-wire. That is probably acceptable for v1, but we can make it more robust with a retry loop around sidecar VM creation if we see real collisions.

On the architecture question: I agree a single persistent orchestrator-owned forwarder is attractive long-term. It would centralize listener ownership, avoid one helper process per bottle, make collision/retry bookkeeping cleaner, and give us one place to reconcile desired forwards against running bottles. I would keep that as a follow-up rather than change this PR’s shape. This PR has a narrower failure domain: each bottle owns its own helper, teardown is just an ExitStack callback, and a wedged forwarder only affects that bottle. Moving forwarding into the orchestrator creates a shared availability dependency and needs a small control-plane protocol plus reconciliation logic, which is doable but materially broadens this PR.

My recommendation: keep per-bottle helpers here, with the current random raw host-port allocation and fail-closed behavior. Add retry-on-raw-port-collision if we want to harden the launch race now. Treat orchestrator-managed forwarding as the next architecture step once this sidecar-VM path is proven.

@didericis Correct concern, but the current implementation does not bind the same host port for each sidecar. `start_bundle_vm()` calls `allocate_raw_host_ports(spec.ports_to_publish)`, which asks the kernel for a free host loopback port with `bind(("127.0.0.1", 0))` for each guest port, then passes those random host ports to smolVM as `(host_port, guest_port)` pairs. So two sidecars both exposing guest `9099` should become something like `127.0.0.1:55001 -> guest:9099` and `127.0.0.1:55017 -> guest:9099`, then each gets wrapped by its own per-bottle alias forwarder port. There is still a normal reserve-then-bind race because the socket used to discover the free port is closed before smolVM binds it. If another process grabs the port in that gap, smolVM launch should fail closed rather than silently cross-wire. That is probably acceptable for v1, but we can make it more robust with a retry loop around sidecar VM creation if we see real collisions. On the architecture question: I agree a single persistent orchestrator-owned forwarder is attractive long-term. It would centralize listener ownership, avoid one helper process per bottle, make collision/retry bookkeeping cleaner, and give us one place to reconcile desired forwards against running bottles. I would keep that as a follow-up rather than change this PR’s shape. This PR has a narrower failure domain: each bottle owns its own helper, teardown is just an `ExitStack` callback, and a wedged forwarder only affects that bottle. Moving forwarding into the orchestrator creates a shared availability dependency and needs a small control-plane protocol plus reconciliation logic, which is doable but materially broadens this PR. My recommendation: keep per-bottle helpers here, with the current random raw host-port allocation and fail-closed behavior. Add retry-on-raw-port-collision if we want to harden the launch race now. Treat orchestrator-managed forwarding as the next architecture step once this sidecar-VM path is proven.

👍 1

didericis-codex commented

2026-07-09 13:42:56 -04:00

@didericis re: #issuecomment-3049, yes, that helper is redundant now. I removed _proxy_host, use loopback_ip directly as the per-bottle forwarder/TSI address, and updated the stale doc comment/test in 820bb40.

Validation: python3 -m unittest tests.unit.test_smolmachines_provision tests.unit.test_smolmachines_sidecar_bundle tests.unit.test_smolmachines_port_forward passed.

@didericis re: #issuecomment-3049, yes, that helper is redundant now. I removed `_proxy_host`, use `loopback_ip` directly as the per-bottle forwarder/TSI address, and updated the stale doc comment/test in `820bb40`. Validation: `python3 -m unittest tests.unit.test_smolmachines_provision tests.unit.test_smolmachines_sidecar_bundle tests.unit.test_smolmachines_port_forward` passed.

didericis force-pushed smolmachines-sidecar-vm from 820bb40a5f to bfd3e659e8

2026-07-09 13:53:49 -04:00

Compare

didericis reviewed 2026-07-09 14:06:12 -04:00

bot_bottle/backend/smolmachines/sidecar_bundle.py

						
				@@ -110,1 +184,4 @@

				        warn(f"smolvm machine delete {name} failed: {exc}")

				def create_bundle_network(network_name: str, subnet: str, gateway: str) -> None:

didericis commented

2026-07-09 14:06:12 -04:00

This is made irrelevant, right?

didericis added 5 commits 2026-07-10 15:31:45 -04:00

docs: draft smolmachines sidecar vm prd 4863e81cd3

feat: run smolmachines sidecar as vm 869a8bbc1f

test(smolmachines): cover sidecar vm launch paths 5828668e21

refactor(smolmachines): inline sidecar proxy host bb9cca48fd

refactor(smolmachines): drop obsolete docker-sidecar helpers

test / unit (pull_request) Successful in 1m0s

Details

test / integration (pull_request) Successful in 17s

Details

test / coverage (pull_request) Successful in 1m8s

Details

2b53c36608

didericis force-pushed smolmachines-sidecar-vm from fe82dbc7b8 to 2b53c36608

2026-07-10 15:31:45 -04:00

Compare

didericis added 2 commits 2026-07-10 16:14:10 -04:00

fix(smolmachines): mount git-gate files at boot via virtiofs staging dirs 62d2e86e7e

Two issues caused krun_start_enter -22 (VM boot crash):

1. git-gate entrypoint missing at boot: sidecar_init.py starts all
   daemons (including git-gate) immediately as PID 1. The entrypoint
   was copied in via machine_cp which only runs after machine_start
   returns — too late. virtiofs also can't mount files at root (/).

   Fix: bake a static /git-gate-entrypoint.sh wrapper into the
   Dockerfile.sidecars image that delegates to /etc/git-gate/entrypoint.sh.
   For smolmachines, stage per-bottle scripts with correct in-VM names
   under the git-gate state dir and virtiofs-mount to /etc/git-gate/ at
   VM creation time. docker/macOS backends are unchanged (their file
   bind-mount/docker cp still overrides the static wrapper).

2. Egress confdir mounted read-only: egress_entrypoint.sh writes
   combined-trust.pem to confdir under set -e; mitmproxy also needs
   to write its per-host cert cache there. Both fail fatally on a
   read-only virtiofs mount.

   Fix: change the egress confdir virtiofs mount to writable.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

fix(smolmachines): consolidate virtiofs mounts to stay within libkrun limit

lint / lint (push) Successful in 2m1s

Details

test / unit (pull_request) Failing after 54s

Details

test / integration (pull_request) Successful in 17s

Details

test / coverage (pull_request) Failing after 54s

Details

9c4400cce2

libkrun limits total mounts + port-mappings to 5. With all three
sidecar daemons active (egress, git-gate, supervise), the prior
layout used 3 mounts + 3 ports = 6, causing the VM to crash at boot
with krun_start_enter returned -22.

Merge the egress confdir and git-gate scripts into a single staging
directory mounted at /bot-bottle-data/ with egress/ and git-gate/
subdirectories, reducing to 2 mounts + 3 ports = 5. Also clean up
leftover VMs before creating new ones to handle interrupted teardowns.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

didericis added 1 commit 2026-07-10 17:01:30 -04:00

fix(smolmachines): Linux compat for TSI networking and git-gate paths

lint / lint (push) Successful in 2m8s

Details

test / unit (pull_request) Failing after 58s

Details

test / integration (pull_request) Successful in 23s

Details

test / coverage (pull_request) Failing after 1m1s

Details

d76ce89c87

Three fixes for Linux smolvm support:

1. Skip force_allowlist DB patch on Linux — patching allowed_cidrs
   into smolvm's state DB crashes TSI boot on Linux. Per-bottle CIDR
   isolation is not enforced on Linux until smolvm fixes the
   --allow-cidr + TSI interaction (known limitation).

2. Patch /etc/git-gate/ references in the staged entrypoint to
   /bot-bottle-data/git-gate/ so git-daemon finds its hooks and
   access-hook at the new consolidated mount path.

3. Drop explicit --net-backend tsi — the default on this smolvm
   build is already TSI; explicitly requesting it is incompatible
   with -p port mappings on Linux.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

didericis added 1 commit 2026-07-10 17:11:36 -04:00

fix(smolmachines): enforce per-bottle loopback isolation on Linux via iptables

lint / lint (push) Successful in 2m7s

Details

test / unit (pull_request) Failing after 55s

Details

test / integration (pull_request) Successful in 25s

Details

test / coverage (pull_request) Failing after 1m2s

Details

dc9fdc8563

On Linux, smolvm's TSI allowlist DB patch crashes boot, so
force_allowlist is a no-op. This left the agent VM with full
host loopback access — a security regression from the macOS
behavior.

Fix: install guest-side iptables rules in _init_vm that ACCEPT
traffic to the bottle's allocated loopback alias (proxy_host/32)
and DROP all other 127.0.0.0/8 destinations. The agent runs as
non-root (node) and cannot flush the rules.

- Add iptables to claude and codex agent Dockerfiles
- Use DROP (not REJECT) — libkrun kernel lacks the REJECT module
- Skip on macOS where force_allowlist handles it natively

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

didericis-claude closed this pull request

2026-07-11 03:22:27 -04:00

lint / lint (push) Successful in 2m7s

Details

test / unit (pull_request) Failing after 55s

Details

test / integration (pull_request) Successful in 25s

Details

test / coverage (pull_request) Failing after 1m2s

Details

Pull request closed

This pull request cannot be reopened because the branch was deleted.

Sign in to join this conversation.

2 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: didericis/bot-bottle#334

				`@@ -328,26 +342,33 @@ def _init_vm(plan: SmolmachinesBottlePlan) -> None:`


				`def _proxy_host(plan: SmolmachinesBottlePlan, loopback_ip: str) -> str:`