bot-bottle/docs/research/firecracker-image-remote-store.md

Firecracker Image Remote Store

Date: 2026-07-22
Context: PR #459 (run-dir leak fix) surfaced that committed Firecracker snapshots currently live only on the host machine. Once a host is wiped or a run-dir is evicted, a user's preserved bottle is gone. This note investigates a secure remote store so committed images survive host turnover and can be restored without the user having to pre-flag which sessions to keep.

Verdict

Backblaze B2 + Cloudflare CDN is the cost-optimal choice for most deployments. Cloudflare R2 is the simpler zero-config option at slightly higher storage cost. Self-hosted MinIO is the right call for air-gapped or on-premises installs.

On the image-size front, zstd-compressing the committed tar before upload produces roughly a 60–70% reduction with negligible impact on restore latency. OverlayFS (for in-flight working rootfs, not for the committed artifact) cuts per-instance disk use to ~10–50 MB per extra bottle sharing the same base.

---

What Gets Stored

The Firecracker backend produces two artifact types:

Artifact	Created by	Size	Lifetime
rootfs/agent-<digest>/ (dir)	image_builder.py → mke2fs -d	~1 GB as ext4	Cached per Dockerfile hash; evictable
committed/<slug>/rootfs.tar	FirecrackerFreezer._freeze via SSH tar	500 MB–1 GB	User-preserved; must survive host wipe

The committed artifact is a tar of the guest's live filesystem streamed out over SSH (freezer.py:58–90). At resume time launch.py calls mke2fs -d to rebuild a fresh ext4 from this tar. The tar — not the ext4 — is what needs to be pushed to remote storage and pulled back at restore time.

The base image cache (rootfs/agent-<digest>/) is derivable from the Dockerfile and can be rebuilt on demand; it is lower priority for remote storage.

---

Storage Candidates

Object storage

Provider	Storage	Egress	Notes
Backblaze B2	$0.006/GB	Free (via Cloudflare Bandwidth Alliance)	Cheapest storage; pairs with Cloudflare CDN to eliminate egress
Cloudflare R2	$0.015/GB	$0 always	Zero-config egress; no lifecycle transitions (limitation)
Wasabi	$0.0069/GB	Free (1:1 ratio)	90-day minimum retention; good for archival; lifecycle evaluated daily
AWS S3	$0.023/GB	$0.09/GB	Richest lifecycle support; expensive at scale; avoid unless already in AWS
MinIO (self-hosted)	Host cost only	None	S3-compatible; best for private/on-prem deployments

B2 + Cloudflare CDN is effectively $0.006/GB with zero egress — about 18× cheaper than S3 for restore-heavy workloads. R2 is the zero-config choice ($0 egress by default, no Bandwidth Alliance pairing needed) at a slightly higher storage rate.

Cloudflare R2's missing lifecycle support is the main caveat: auto-eviction rules (evict images older than N days) cannot currently be expressed natively in R2. Wasabi and S3 both support declarative lifecycle policies.

Retention policy recommendation

The comment proposes:

• Retain images for ~1 week by default
• Warn when approaching a capacity threshold
• Auto-evict oldest images once threshold is exceeded

This maps cleanly to an application-level policy (not a provider lifecycle rule), which avoids the R2 limitation and works consistently across providers:

1. On commit: upload tar, record (slug, size_bytes, uploaded_at) in a local or remote manifest file.
2. On startup / on list: scan the manifest, warn if total stored size exceeds e.g. 80% of the configured threshold.
3. On eviction run (CLI or cron): delete objects older than retention_days (default 7) that push total over max_capacity; oldest-first.

This keeps the policy logic in bot-bottle and the storage provider as a dumb object store — no vendor-specific lifecycle API required.

---

Image Size Reduction

Current artifact sizes

A typical committed tar for a Claude Code agent image is 500 MB–1 GB uncompressed. The per-run ext4 (copy of the base, written at start) adds another ~1 GB of local disk. The leak fix in this PR addresses the ext4 copies; the remote store addresses the committed tars.

Compression

zstd compression of the committed tar before upload is the highest-leverage single change:

Codec	Typical size (1 GB rootfs)	Compress speed	Decompress speed
gzip	350–430 MB	~100 MB/s	~500 MB/s
zstd (default)	330–360 MB	~400 MB/s	~2 GB/s
xz	290–320 MB	~20 MB/s	~200 MB/s

zstd is the best trade-off: 65–67% size reduction, near-instantaneous decompression. The tar call in freezer.py could pipe through zstd before writing to disk and to the remote; resume decompresses on the way back. A .tar.zst suffix marks compressed artifacts so old tars remain restorable without the codec.

SquashFS for the base image cache

The rootfs/agent-<digest>/ directory (the buildah-exported tree) is rebuilt by image_builder.py and turned into per-run ext4 by mke2fs -d. Storing the base as a SquashFS image instead of a flat directory tree would reduce it from ~1 GB to ~330–360 MB and make the cache remote-friendly. Firecracker does not directly boot SquashFS, but the existing mke2fs -d path reads a directory tree — a SquashFS mount could serve as the source. This is a larger change and lower priority than tar compression.

OverlayFS for per-run rootfs

Multiple simultaneous bottles sharing the same agent image today each get a full mke2fs -d copy (~1 GB). OverlayFS (read-only base + writable sparse overlay) would reduce this to ~10–50 MB per instance beyond the first:

• Mount the base image directory as read-only lower layer
• Attach a sparse ext4 or tmpfs writable layer per bottle
• Pass the merged overlay to Firecracker as the block device

E2B's public write-up on Firecracker + OverlayFS confirms this approach works at scale. The launch.py changes would be non-trivial (device mapper or fuse-overlayfs plumbing), so this is a follow-up rather than a prerequisite for the remote store.

---

Recommended Approach

Phase 1 — remote store with zstd (tight scope, actionable now)

1. Add --zstd to the tar call in FirecrackerFreezer._freeze; name the artifact rootfs.tar.zst. Keep uncompressed restore path for legacy tars.
2. Add a bb firecracker upload <slug> / bb firecracker pull <slug> pair that pushes/fetches the compressed tar to the configured object store (S3-compatible API, so B2, R2, MinIO, and Wasabi all work with the same client).
3. Store a manifest.json in the bucket (or a local mirror) tracking slug → {size, uploaded_at}. Use it for threshold warnings and eviction.
4. Default retention: 7 days, configurable via firecracker.image_retention_days in ~/.config/bot-bottle/config.toml (or equivalent).
5. Warn at 80% of max_capacity (default e.g. 50 GB); evict oldest on commit once at 100%.

Storage recommendation: Cloudflare R2 for hosted deployments (zero egress, zero config), MinIO for private/on-premises.

Phase 2 — base image cache compression

Compress the agent-<digest> cache dir as a .tar.zst to save ~65% on repeated image uploads. Low urgency since the base image is rebuildable.

Phase 3 — OverlayFS per-run disk

Replace the full per-run ext4 copy with an OverlayFS sparse layer. Largest disk impact (~90–95% savings per concurrent bottle) but highest implementation complexity. Track as a separate PRD.

---

Open Questions

• Does the host have a configured object-store credential path, or should the remote store be an opt-in with an explicit bb config set image-store.url ...?
• Should commit automatically upload, or should upload be an explicit step to avoid surprise egress?
• What is the acceptable cold-start latency for a restore from remote? A 330 MB zstd tar at 100 Mbit/s takes ~26 s; at 1 Gbit/s, ~2.6 s. This bounds the retention strategy (evict from local after successful upload vs keep local copy).