The agent-image build ran over ssh with output fully captured, shown
only as a 20-line tail on failure — so a successful (or in-progress)
first build was a long silent wait through the base pull + apt/npm
installs. Stream the `buildah build` step's stdout/stderr straight to
our own (like the docker backend's `docker build`) via a non-capturing
_ssh_streamed helper, so the operator sees `STEP i/n` progress live.
Failure now points at the streamed output above instead of a captured
tail. The smoke test and rootfs export stay captured (short / piped).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WBMWTEtQdJ4W5UrWuLHCck
Codex flagged that parallel `start`s race on shared state: two cold
launches could both stop/build/boot the singleton on the same rootfs/PID,
concurrent builds share the infra VM's buildah store, the cleanup did
`buildah rm -a` (nuking a peer build's container), and the rootfs cache
was populated non-atomically.
- `infra_vm.ensure_running`: a host flock (`singleton.lock`) around the
cold stop/build/boot path, with a double-checked health re-test under
the lock so a second launcher adopts rather than re-boots. The healthy
fast-path stays lock-free.
- `image_builder.build_agent_rootfs_dir`: a host flock (`.build.lock`)
around cache-lookup + build + publish; build into `.building-<digest>`
and publish by atomic `os.rename`, so a partial build never appears as
`agent-<digest>`.
- scoped cleanup: per-build named working containers
(`<tag>-smoke`/`-export`) removed by name instead of `buildah rm -a`;
also cleared before the build to recover from a crashed prior run.
Verified: agent image builds via the locked/atomic path, infra VM stays
healthy, agent boots, `claude --version` = 2.1.172. Full unit suite +
pyright green.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WBMWTEtQdJ4W5UrWuLHCck
Addresses the review finding that buildah lived only in the orchestrator
image, so the persistent infra VM wasn't the builder — a separate throwaway
builder VM contended with it for the orchestrator TAP. Consolidate:
- **Rebase the gateway (and thus infra) on `python:3.12-slim` = Debian
trixie**, pip-installing mitmproxy instead of `FROM mitmproxy/mitmproxy`
(Debian bookworm). trixie ships buildah 1.39, which can build agent
Dockerfiles that use heredocs; bookworm's 1.28 can't (`Unknown
instruction: "{"`). CA path is unchanged (set via `--set confdir=`).
- **buildah lives only in `Dockerfile.infra`** now (removed from the
orchestrator image, which is lean/stdlib-only again).
- **Shared orchestrator content**: `Dockerfile.orchestrator` is the single
definition of the control-plane payload; the infra image `COPY --from`s
it (same trixie base → clean copy, and future deps like iroh are added
once). The docker backend runs the orchestrator image directly.
- **`image_builder` builds inside the infra VM** (which now has buildah)
over SSH — no throwaway builder VM, so the `bborch0` contention is gone.
`ensure_built` builds orchestrator + gateway before infra (FROM gateway,
COPY --from orchestrator).
Verified on a KVM host: images build (buildah 1.39 in infra), the agent
image builds *inside* the infra VM (heredoc Dockerfile and all), the infra
VM stays healthy, the agent boots and `claude --version` = 2.1.172. The
rebased gateway still starts as a docker container and generates its CA
(docker backend unaffected).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WBMWTEtQdJ4W5UrWuLHCck
Replace the host `docker build` + `docker export` behind the Firecracker
agent rootfs with an in-VM buildah build. `image_builder.build_agent_rootfs_dir`
boots a throwaway builder VM (the orchestrator image, which carries
buildah) on the NAT'd orchestrator link, sends the Dockerfile over SSH,
`buildah build`s it, smoke-tests the result with `buildah run` (the
image's own PATH, so it catches an npm silent-failure stub), and streams
the rootfs tar back into the content-addressed cache dir — the same base
dir `util.build_rootfs_ext4` already turns into a bootable ext4 with
`mke2fs -d`. No host Docker daemon, no root-equivalent `docker` group;
an untrusted Dockerfile runs in a confined microVM, not on the host.
- new firecracker/image_builder.py (boot → build → smoke → stream).
- launch.py: `_build_agent_image` → `_build_agent_base`, returning the
base dir from the builder VM. The committed-snapshot (freeze/migrate)
path still exports via host docker until it too is ported.
- util: `_inject_guest_boot` → public `inject_guest_boot` (shared with
the builder). unit tests for the cache decision + smoke-test paths.
Verified on a KVM host end-to-end: builds the real claude Dockerfile
(node:22-slim + npm claude-code) in-VM in ~60s, the produced agent VM
boots and `claude --version` returns 2.1.172; the content cache skips
the rebuild on a repeat launch.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WBMWTEtQdJ4W5UrWuLHCck
didericis