Firecracker per-bottle run dirs leak → CI runner disk exhaustion #461

New Issue

2026-07-21T22:50:40-04:00

didericis-claude commented

2026-07-21 22:50:40 -04:00

The CI runner (delphi) fills its root filesystem with leaked Firecracker per-bottle run/<slug> directories (each a ~1G rootfs.ext4), until / hits ~97% and jobs start failing with "no space left on device". One incident had 187 stale sandbox-tester-* dirs = 141 GB.

Root causes

Clean teardown leaks the run dir (launch.py). Each bottle creates its run/<slug> dir but only registers stack.callback(vm.terminate) — there is no callback to remove the dir, so every clean exit leaves its rootfs behind.
cleanup() is unsafe, so it cannot be used as a GC (cleanup.py). It enumerates every firecracker process + every run dir under the run root, so ./cli.py cleanup would kill running bottles and delete their rootfs — despite being described as orphan cleanup (enumerate_active is still a stub, #354). Because it is unsafe to run alongside live bottles, it cannot back a periodic sweep.

Impact

Root FS exhaustion → CI jobs fail no space left; requires manual rm -rf of stale run dirs to recover.

Fix

Register a shutil.rmtree callback before vm.terminate (ExitStack LIFO → VM gone before rootfs removed) so clean teardown reclaims its own dir.
Make cleanup() process-aware: a run dir with a live firecracker process is protected; only dirs with no live process (leaked by a hard-killed launch) are reaped. Safe to run alongside live bottles → can back a periodic GC.
Host backstop (didi-scripts, not this repo): a NixOS systemd timer sweeps /var/cache/bot-bottle-fc/run + docker system prune --volumes for jobs hard-killed before teardown runs.

Fixed by #459.

The CI runner (delphi) fills its root filesystem with leaked Firecracker per-bottle `run/<slug>` directories (each a ~1G `rootfs.ext4`), until `/` hits ~97% and jobs start failing with "no space left on device". One incident had **187 stale `sandbox-tester-*` dirs = 141 GB**. ## Root causes 1. **Clean teardown leaks the run dir (`launch.py`).** Each bottle creates its `run/<slug>` dir but only registers `stack.callback(vm.terminate)` — there is no callback to remove the dir, so *every* clean exit leaves its rootfs behind. 2. **`cleanup()` is unsafe, so it cannot be used as a GC (`cleanup.py`).** It enumerates *every* firecracker process + *every* run dir under the run root, so `./cli.py cleanup` would kill running bottles and delete their rootfs — despite being described as orphan cleanup (`enumerate_active` is still a stub, #354). Because it is unsafe to run alongside live bottles, it cannot back a periodic sweep. ## Impact Root FS exhaustion → CI jobs fail `no space left`; requires manual `rm -rf` of stale run dirs to recover. ## Fix - Register a `shutil.rmtree` callback **before** `vm.terminate` (ExitStack LIFO → VM gone before rootfs removed) so clean teardown reclaims its own dir. - Make `cleanup()` process-aware: a run dir with a live firecracker process is protected; only dirs with no live process (leaked by a hard-killed launch) are reaped. Safe to run alongside live bottles → can back a periodic GC. - Host backstop (didi-scripts, not this repo): a NixOS systemd timer sweeps `/var/cache/bot-bottle-fc/run` + `docker system prune --volumes` for jobs hard-killed before teardown runs. Fixed by #459.

gitea-actions bot added the Status/Needs Triage label 2026-07-21 22:50:48 -04:00

didericis-claude referenced a pull request that will close this issue

2026-07-21 22:50:56 -04:00

fix(firecracker): stop leaking per-bottle run dirs #459

didericis closed this issue

2026-07-21 22:54:09 -04:00

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: didericis/bot-bottle#461