fix(firecracker): stop leaking per-bottle run dirs #459

2026-07-21T21:38:27-04:00

didericis-claude commented

2026-07-21 21:38:27 -04:00

Closes #461

Fixes the CI-runner disk exhaustion: leaked Firecracker run/<slug> dirs (each ~1G rootfs.ext4) piled up until / hit 97% and jobs failed "no space left" (187 stale sandbox-tester-* dirs = 141 GB in one incident).

Two commits

1. Remove the run dir on clean teardown (`launch.py`)

Each bottle created its run dir but only registered stack.callback(vm.terminate) — never a callback to remove the dir. Every clean exit still leaked its rootfs. Now a shutil.rmtree callback is registered before vm.terminate so, ExitStack being LIFO, the VM is gone before its rootfs is removed.

2. Make `cleanup()` reap orphans only, never live VMs (`cleanup.py`)

cleanup() enumerated every firecracker process under the run root and every run dir, so ./cli.py cleanup would kill running bottles and delete their rootfs — despite being described as orphan cleanup (the enumerate_active registry is still a stub, #354). Now the running firecracker processes are the live/dead signal: a run dir with a live VM is protected; only dirs with no live process (leaked by a hard-killed launch) are reaped, plus firecracker pids whose dir is already gone. This makes cleanup safe to run alongside live bottles and lets it back a periodic GC for hard-killed jobs the teardown fix can't cover.

Testing

tests/unit/test_firecracker_cleanup.py rewritten for the process-aware behavior, incl. explicit coverage that a live VM's dir is never in the plan (70 firecracker unit tests pass).
The launch.py generator has no unit harness (needs real KVM VMs); exercised end-to-end by the firecracker integration job in CI.

Follow-up (host, not this repo)

A NixOS systemd timer backstop (in didi-scripts) sweeps /var/cache/bot-bottle-fc/run + docker system prune --volumes for jobs hard-killed before teardown runs.

🤖 Generated with Claude Code

Closes #461 Fixes the CI-runner disk exhaustion: leaked Firecracker `run/<slug>` dirs (each ~1G rootfs.ext4) piled up until `/` hit 97% and jobs failed "no space left" (187 stale `sandbox-tester-*` dirs = 141 GB in one incident). ## Two commits ### 1. Remove the run dir on clean teardown (`launch.py`) Each bottle created its run dir but only registered `stack.callback(vm.terminate)` — never a callback to remove the dir. Every clean exit still leaked its rootfs. Now a `shutil.rmtree` callback is registered **before** `vm.terminate` so, ExitStack being LIFO, the VM is gone before its rootfs is removed. ### 2. Make `cleanup()` reap orphans only, never live VMs (`cleanup.py`) `cleanup()` enumerated *every* firecracker process under the run root and *every* run dir, so `./cli.py cleanup` would **kill running bottles and delete their rootfs** — despite being described as orphan cleanup (the `enumerate_active` registry is still a stub, #354). Now the running firecracker processes are the live/dead signal: a run dir with a live VM is protected; only dirs with no live process (leaked by a hard-killed launch) are reaped, plus firecracker pids whose dir is already gone. This makes cleanup safe to run alongside live bottles and lets it back a periodic GC for hard-killed jobs the teardown fix can't cover. ## Testing - `tests/unit/test_firecracker_cleanup.py` rewritten for the process-aware behavior, incl. explicit coverage that a live VM's dir is never in the plan (70 firecracker unit tests pass). - The `launch.py` generator has no unit harness (needs real KVM VMs); exercised end-to-end by the firecracker integration job in CI. ## Follow-up (host, not this repo) A NixOS systemd timer backstop (in didi-scripts) sweeps `/var/cache/bot-bottle-fc/run` + `docker system prune --volumes` for jobs hard-killed before teardown runs. 🤖 Generated with [Claude Code](https://claude.com/claude-code)

didericis commented

2026-07-21 21:58:29 -04:00

@didericis-claude ideally we’d have some sort of secure remote store for firecracker images that would allow users to restore snapshotted agent images easily. That way they don’t have to know what agent runs they want snapshotted beforehand and don’t need to worry about this as much.

Would want some sort of sane policy that stores up to a certain amount in a certain window (maybe retain for a week by default if below a given threshold we warn about, then evict?). Exact number will depend on storage cost. Do some research for good candidates, and see if there’s a way to reduce these image sizes.

@didericis-claude ideally we’d have some sort of secure remote store for firecracker images that would allow users to restore snapshotted agent images easily. That way they don’t have to know what agent runs they want snapshotted beforehand and don’t need to worry about this as much. Would want some sort of sane policy that stores up to a certain amount in a certain window (maybe retain for a week by default if below a given threshold we warn about, then evict?). Exact number will depend on storage cost. Do some research for good candidates, and see if there’s a way to reduce these image sizes.

didericis-claude changed title from ~~fix(firecracker): remove per-bottle run dir on teardown~~ to fix(firecracker): stop leaking per-bottle run dirs

2026-07-21 22:28:47 -04:00

didericis referenced this issue from a commit

2026-07-21 22:34:47 -04:00

docs(research): firecracker image remote store

didericis-claude commented

2026-07-21 22:35:15 -04:00

Good call. I’ve written up a research doc covering the two directions you flagged — storage candidates and image size reduction.

1fb524cc1a/docs/research/firecracker-image-remote-store.md

tl;dr

The committed artifact from FirecrackerFreezer is a tar (not the ext4 — launch.py rebuilds ext4 from it at resume), so the object to store and restore is already reasonably shaped for remote storage.
Best storage fit: Cloudflare R2 for hosted deployments (zero egress, no config overhead); MinIO for private/on-prem. Backblaze B2 + Cloudflare CDN is cheaper on storage ($0.006/GB) if the Bandwidth Alliance pairing is acceptable. R2’s missing native lifecycle support pushes the 7-day eviction logic into bot-bottle itself, which is probably the right place for it anyway since the threshold/warn/evict logic is richer than what provider lifecycle rules express.
Size reduction: zstd-compressing the tar before upload gets a typical 1 GB rootfs down to ~330–360 MB (65–67% reduction) with near-instantaneous decompression. That’s the highest-leverage single change. OverlayFS for concurrent in-flight bottles (replace the per-run mke2fs copy with a sparse overlay) is the bigger win on local disk but more involved — noted in the doc as a follow-up.
Open questions at the bottom of the doc: whether upload should be automatic on commit vs an explicit push step, and what restore latency is acceptable (bounds whether we keep a local copy after uploading).

Good call. I’ve written up a research doc covering the two directions you flagged — storage candidates and image size reduction. https://gitea.dideric.is/didericis/bot-bottle/src/commit/1fb524cc1ae367647ac7649d6c09d094009334c1/docs/research/firecracker-image-remote-store.md **tl;dr** - The committed artifact from `FirecrackerFreezer` is a tar (not the ext4 — `launch.py` rebuilds ext4 from it at resume), so the object to store and restore is already reasonably shaped for remote storage. - **Best storage fit:** Cloudflare R2 for hosted deployments (zero egress, no config overhead); MinIO for private/on-prem. Backblaze B2 + Cloudflare CDN is cheaper on storage ($0.006/GB) if the Bandwidth Alliance pairing is acceptable. R2’s missing native lifecycle support pushes the 7-day eviction logic into bot-bottle itself, which is probably the right place for it anyway since the threshold/warn/evict logic is richer than what provider lifecycle rules express. - **Size reduction:** zstd-compressing the tar before upload gets a typical 1 GB rootfs down to ~330–360 MB (65–67% reduction) with near-instantaneous decompression. That’s the highest-leverage single change. OverlayFS for concurrent in-flight bottles (replace the per-run mke2fs copy with a sparse overlay) is the bigger win on local disk but more involved — noted in the doc as a follow-up. - **Open questions** at the bottom of the doc: whether upload should be automatic on `commit` vs an explicit push step, and what restore latency is acceptable (bounds whether we keep a local copy after uploading).

didericis added 3 commits 2026-07-21 22:47:52 -04:00

fix(firecracker): remove per-bottle run dir on teardown 2cd06814e6

Each Firecracker bottle created its run dir (holding an ~1G
rootfs.ext4) but never removed it — launch teardown only terminated
the VM. Every run leaked its rootfs, filling the CI runner's disk
(187 stale sandbox-tester dirs / 141G in one incident).

Register a run-dir rmtree callback on the ExitStack before the
vm.terminate callback so, LIFO, the VM is gone before its rootfs is
removed. ignore_errors keeps teardown resilient.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

fix(firecracker): make cleanup reap orphans only, never live VMs cfb2284b99

`cleanup()` enumerated *every* firecracker process under the run root and
*every* run dir, so `./cli.py cleanup` would kill running bottles and
delete their rootfs — despite being described as orphan cleanup. The
backend's `enumerate_active` registry is still a stub (#354), so there
was no live/dead signal.

Use the running firecracker processes themselves as that signal: a run
dir with a live VM is protected (never killed, never removed); only run
dirs with no live process (leaked by a hard-killed launch) are reaped,
plus firecracker pids whose run dir is already gone (lingering VMMs).

This makes cleanup safe to run alongside live bottles and lets it back a
periodic GC for the run-dir leak the launch.py teardown fix closes on the
clean-exit path.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

docs(research): firecracker image remote store

test / integration-docker (pull_request) Successful in 17s

Details

test / unit (pull_request) Successful in 40s

Details

tracker-policy-pr / check-pr (pull_request) Successful in 6s

Details

test / integration-firecracker (pull_request) Successful in 3m33s

Details

test / coverage (pull_request) Successful in 16s

Details

test / publish-infra (pull_request) Has been skipped

Details

test / integration-docker (push) Successful in 35s

Details

test / unit (push) Successful in 39s

Details

Update Quality Badges / update-badges (push) Successful in 47s

Details

lint / lint (push) Successful in 51s

Details

test / integration-firecracker (push) Successful in 5m8s

Details

test / coverage (push) Successful in 44s

Details

test / publish-infra (push) Successful in 1m53s

Details

182a28d724

Investigates storage candidates, retention policy design, and
image-size reduction techniques in response to the PR #459 comment
on a secure remote store for committed Firecracker snapshots.

didericis force-pushed fix/firecracker-run-dir-leak from 1fb524cc1a to 182a28d724

2026-07-21 22:47:52 -04:00

Compare

didericis referenced this issue from a commit

2026-07-21 22:47:53 -04:00

docs(research): firecracker image remote store

didericis-claude referenced this pull request

2026-07-21 22:50:41 -04:00

Firecracker per-bottle run dirs leak → CI runner disk exhaustion #461

didericis approved these changes 2026-07-21 22:54:00 -04:00

didericis merged commit 182a28d724 into main

2026-07-21 22:54:09 -04:00