fix(firecracker): stop leaking per-bottle run dirs #459
@@ -0,0 +1,178 @@
|
||||
# Firecracker Image Remote Store
|
||||
|
||||
**Date:** 2026-07-22
|
||||
**Context:** PR #459 (run-dir leak fix) surfaced that committed Firecracker snapshots
|
||||
currently live only on the host machine. Once a host is wiped or a run-dir is
|
||||
evicted, a user's preserved bottle is gone. This note investigates a secure remote
|
||||
store so committed images survive host turnover and can be restored without the
|
||||
user having to pre-flag which sessions to keep.
|
||||
|
||||
## Verdict
|
||||
|
||||
Backblaze B2 + Cloudflare CDN is the cost-optimal choice for most deployments.
|
||||
Cloudflare R2 is the simpler zero-config option at slightly higher storage cost.
|
||||
Self-hosted MinIO is the right call for air-gapped or on-premises installs.
|
||||
|
||||
On the image-size front, zstd-compressing the committed tar before upload
|
||||
produces roughly a 60–70% reduction with negligible impact on restore latency.
|
||||
OverlayFS (for in-flight working rootfs, not for the committed artifact) cuts
|
||||
per-instance disk use to ~10–50 MB per extra bottle sharing the same base.
|
||||
|
||||
---
|
||||
|
||||
## What Gets Stored
|
||||
|
||||
The Firecracker backend produces two artifact types:
|
||||
|
||||
| Artifact | Created by | Size | Lifetime |
|
||||
|----------|-----------|------|---------|
|
||||
| `rootfs/agent-<digest>/` (dir) | `image_builder.py` → `mke2fs -d` | ~1 GB as ext4 | Cached per Dockerfile hash; evictable |
|
||||
| `committed/<slug>/rootfs.tar` | `FirecrackerFreezer._freeze` via SSH tar | 500 MB–1 GB | User-preserved; must survive host wipe |
|
||||
|
||||
The committed artifact is a tar of the guest's live filesystem streamed out over
|
||||
SSH (`freezer.py:58–90`). At resume time `launch.py` calls `mke2fs -d` to
|
||||
rebuild a fresh ext4 from this tar. The tar — not the ext4 — is what needs to be
|
||||
pushed to remote storage and pulled back at restore time.
|
||||
|
||||
The base image cache (`rootfs/agent-<digest>/`) is derivable from the Dockerfile
|
||||
and can be rebuilt on demand; it is lower priority for remote storage.
|
||||
|
||||
---
|
||||
|
||||
## Storage Candidates
|
||||
|
||||
### Object storage
|
||||
|
||||
| Provider | Storage | Egress | Notes |
|
||||
|----------|---------|--------|-------|
|
||||
| **Backblaze B2** | $0.006/GB | Free (via Cloudflare Bandwidth Alliance) | Cheapest storage; pairs with Cloudflare CDN to eliminate egress |
|
||||
| **Cloudflare R2** | $0.015/GB | $0 always | Zero-config egress; no lifecycle transitions (limitation) |
|
||||
| **Wasabi** | $0.0069/GB | Free (1:1 ratio) | 90-day minimum retention; good for archival; lifecycle evaluated daily |
|
||||
| **AWS S3** | $0.023/GB | $0.09/GB | Richest lifecycle support; expensive at scale; avoid unless already in AWS |
|
||||
| **MinIO** (self-hosted) | Host cost only | None | S3-compatible; best for private/on-prem deployments |
|
||||
|
||||
**B2 + Cloudflare CDN** is effectively $0.006/GB with zero egress — about 18×
|
||||
cheaper than S3 for restore-heavy workloads. **R2** is the zero-config choice
|
||||
($0 egress by default, no Bandwidth Alliance pairing needed) at a slightly
|
||||
higher storage rate.
|
||||
|
||||
**Cloudflare R2's missing lifecycle support** is the main caveat: auto-eviction
|
||||
rules (evict images older than N days) cannot currently be expressed natively in
|
||||
R2. Wasabi and S3 both support declarative lifecycle policies.
|
||||
|
||||
### Retention policy recommendation
|
||||
|
||||
The comment proposes:
|
||||
- Retain images for ~1 week by default
|
||||
- Warn when approaching a capacity threshold
|
||||
- Auto-evict oldest images once threshold is exceeded
|
||||
|
||||
This maps cleanly to an application-level policy (not a provider lifecycle rule),
|
||||
which avoids the R2 limitation and works consistently across providers:
|
||||
|
||||
1. On `commit`: upload tar, record `(slug, size_bytes, uploaded_at)` in a local
|
||||
or remote manifest file.
|
||||
2. On startup / on `list`: scan the manifest, warn if total stored size exceeds
|
||||
e.g. 80% of the configured threshold.
|
||||
3. On eviction run (CLI or cron): delete objects older than `retention_days`
|
||||
(default 7) that push total over `max_capacity`; oldest-first.
|
||||
|
||||
This keeps the policy logic in bot-bottle and the storage provider as a dumb
|
||||
object store — no vendor-specific lifecycle API required.
|
||||
|
||||
---
|
||||
|
||||
## Image Size Reduction
|
||||
|
||||
### Current artifact sizes
|
||||
|
||||
A typical committed tar for a Claude Code agent image is 500 MB–1 GB uncompressed.
|
||||
The per-run ext4 (copy of the base, written at `start`) adds another ~1 GB of
|
||||
local disk. The leak fix in this PR addresses the ext4 copies; the remote store
|
||||
addresses the committed tars.
|
||||
|
||||
### Compression
|
||||
|
||||
zstd compression of the committed tar before upload is the highest-leverage
|
||||
single change:
|
||||
|
||||
| Codec | Typical size (1 GB rootfs) | Compress speed | Decompress speed |
|
||||
|-------|---------------------------|---------------|-----------------|
|
||||
| gzip | 350–430 MB | ~100 MB/s | ~500 MB/s |
|
||||
| **zstd (default)** | **330–360 MB** | **~400 MB/s** | **~2 GB/s** |
|
||||
| xz | 290–320 MB | ~20 MB/s | ~200 MB/s |
|
||||
|
||||
**zstd is the best trade-off**: 65–67% size reduction, near-instantaneous
|
||||
decompression. The `tar` call in `freezer.py` could pipe through `zstd` before
|
||||
writing to disk and to the remote; `resume` decompresses on the way back. A
|
||||
`.tar.zst` suffix marks compressed artifacts so old tars remain restorable
|
||||
without the codec.
|
||||
|
||||
### SquashFS for the base image cache
|
||||
|
||||
The `rootfs/agent-<digest>/` directory (the buildah-exported tree) is rebuilt by
|
||||
`image_builder.py` and turned into per-run ext4 by `mke2fs -d`. Storing the
|
||||
base as a SquashFS image instead of a flat directory tree would reduce it from
|
||||
~1 GB to ~330–360 MB and make the cache remote-friendly. Firecracker does not
|
||||
directly boot SquashFS, but the existing `mke2fs -d` path reads a directory tree
|
||||
— a SquashFS mount could serve as the source. This is a larger change and lower
|
||||
priority than tar compression.
|
||||
|
||||
### OverlayFS for per-run rootfs
|
||||
|
||||
Multiple simultaneous bottles sharing the same agent image today each get a full
|
||||
`mke2fs -d` copy (~1 GB). OverlayFS (read-only base + writable sparse overlay)
|
||||
would reduce this to ~10–50 MB per instance beyond the first:
|
||||
|
||||
- Mount the base image directory as read-only lower layer
|
||||
- Attach a sparse ext4 or tmpfs writable layer per bottle
|
||||
- Pass the merged overlay to Firecracker as the block device
|
||||
|
||||
E2B's public write-up on Firecracker + OverlayFS confirms this approach works
|
||||
at scale. The `launch.py` changes would be non-trivial (device mapper or
|
||||
`fuse-overlayfs` plumbing), so this is a follow-up rather than a prerequisite
|
||||
for the remote store.
|
||||
|
||||
---
|
||||
|
||||
## Recommended Approach
|
||||
|
||||
**Phase 1 — remote store with zstd (tight scope, actionable now)**
|
||||
|
||||
1. Add `--zstd` to the `tar` call in `FirecrackerFreezer._freeze`; name the
|
||||
artifact `rootfs.tar.zst`. Keep uncompressed restore path for legacy tars.
|
||||
2. Add a `bb firecracker upload <slug>` / `bb firecracker pull <slug>` pair that
|
||||
pushes/fetches the compressed tar to the configured object store (S3-compatible
|
||||
API, so B2, R2, MinIO, and Wasabi all work with the same client).
|
||||
3. Store a `manifest.json` in the bucket (or a local mirror) tracking slug →
|
||||
`{size, uploaded_at}`. Use it for threshold warnings and eviction.
|
||||
4. Default retention: 7 days, configurable via `firecracker.image_retention_days`
|
||||
in `~/.config/bot-bottle/config.toml` (or equivalent).
|
||||
5. Warn at 80% of `max_capacity` (default e.g. 50 GB); evict oldest on commit
|
||||
once at 100%.
|
||||
|
||||
**Storage recommendation:** Cloudflare R2 for hosted deployments (zero egress,
|
||||
zero config), MinIO for private/on-premises.
|
||||
|
||||
**Phase 2 — base image cache compression**
|
||||
|
||||
Compress the `agent-<digest>` cache dir as a `.tar.zst` to save ~65% on repeated
|
||||
image uploads. Low urgency since the base image is rebuildable.
|
||||
|
||||
**Phase 3 — OverlayFS per-run disk**
|
||||
|
||||
Replace the full per-run ext4 copy with an OverlayFS sparse layer. Largest disk
|
||||
impact (~90–95% savings per concurrent bottle) but highest implementation
|
||||
complexity. Track as a separate PRD.
|
||||
|
||||
---
|
||||
|
||||
## Open Questions
|
||||
|
||||
- Does the host have a configured object-store credential path, or should the
|
||||
remote store be an opt-in with an explicit `bb config set image-store.url ...`?
|
||||
- Should `commit` automatically upload, or should upload be an explicit step to
|
||||
avoid surprise egress?
|
||||
- What is the acceptable cold-start latency for a restore from remote? A 330 MB
|
||||
zstd tar at 100 Mbit/s takes ~26 s; at 1 Gbit/s, ~2.6 s. This bounds the
|
||||
retention strategy (evict from local after successful upload vs keep local copy).
|
||||
Reference in New Issue
Block a user