Firecracker gateway CA is not persisted across rebuilds — breaks running bottles (issue #450, firecracker half) #510

New Issue

2026-07-26T17:15:06-04:00

didericis-claude commented

2026-07-26 17:15:06 -04:00

Summary

Rebuilding/restarting the Firecracker gateway mints a brand-new mitmproxy CA, breaking every already-running bottle with SSL certificate verification failed on egress.

The #450 fix ("the gateway CA must survive recreation") was only implemented for the Docker backend, via the persistent host_gateway_ca_dir() bind-mount. The Firecracker gateway had no equivalent: its CA lived only in the ephemeral per-boot rootfs.ext4, so every rebuild regenerated it.

Evidence (observed)

CA	SHA-256 fingerprint	Captured
Live gateway (rebuilt)	`66:63:6F:06:94:…`	after rebuild
claude-agent-1 (relaunched with gateway)	`66:63:6F:06:94:…` ✅	after rebuild
claude-agent-3 (running since prior day)	`C6:B1:80:C0:22:…` ❌	day before

agent-3 → rebuilt gateway bumps TLS with a leaf signed by 66:63:6F… → agent-3's trust store only has C6:B1:80… → handshake fails.

Root cause

backend/firecracker/gateway.py booted the VM via infra_vm.boot_vm(...) with no data_drive — contrast the orchestrator VM, which gets data_drive=self._ensure_registry_volume() (persistent /dev/vdb).
The mitmproxy CA lives at /home/mitmproxy/.mitmproxy/ inside the rootfs, which boot_vm re-materializes fresh each boot.
Each bottle only captures a point-in-time CA snapshot at launch (launch.py → egress/gateway-ca/gateway-ca.pem); it is never re-synced when the gateway rotates.

Fix

Give the Firecracker gateway a persistent CA volume, mirroring the orchestrator's registry volume: boot the gateway VM with a small ext4 data_drive and mount it at mitmproxy's confdir in the gateway guest init, so the CA is reused across rebuilds.

Operational caveat

The first gateway restart after the fix still mints one final CA (into the new, initially-empty volume); it persists from then on. To avoid that one-time break, seed the volume with the current gateway's CA material before restarting.

Related / out of scope

Docker bot-bottle-orch-gateway is attached to a stale CI CA volume (bot-bottle-ci-ca-<run>) because CI reuses the production gateway singleton name — separate latent issue.
orchestrator/rotate_ca.py is docker-only.

## Summary Rebuilding/restarting the **Firecracker** gateway mints a brand-new mitmproxy CA, breaking every already-running bottle with `SSL certificate verification failed` on egress. The #450 fix ("the gateway CA must survive recreation") was only implemented for the **Docker** backend, via the persistent `host_gateway_ca_dir()` bind-mount. The Firecracker gateway had no equivalent: its CA lived only in the ephemeral per-boot `rootfs.ext4`, so every rebuild regenerated it. ## Evidence (observed) | CA | SHA-256 fingerprint | Captured | |---|---|---| | Live gateway (rebuilt) | `66:63:6F:06:94:…` | after rebuild | | claude-agent-1 (relaunched with gateway) | `66:63:6F:06:94:…` ✅ | after rebuild | | claude-agent-3 (running since prior day) | `C6:B1:80:C0:22:…` ❌ | day before | agent-3 → rebuilt gateway bumps TLS with a leaf signed by `66:63:6F…` → agent-3's trust store only has `C6:B1:80…` → handshake fails. ## Root cause - `backend/firecracker/gateway.py` booted the VM via `infra_vm.boot_vm(...)` with **no `data_drive`** — contrast the orchestrator VM, which gets `data_drive=self._ensure_registry_volume()` (persistent `/dev/vdb`). - The mitmproxy CA lives at `/home/mitmproxy/.mitmproxy/` inside the rootfs, which `boot_vm` re-materializes fresh each boot. - Each bottle only captures a point-in-time CA snapshot at launch (`launch.py` → `egress/gateway-ca/gateway-ca.pem`); it is never re-synced when the gateway rotates. ## Fix Give the Firecracker gateway a persistent CA volume, mirroring the orchestrator's registry volume: boot the gateway VM with a small ext4 `data_drive` and mount it at mitmproxy's confdir in the gateway guest init, so the CA is reused across rebuilds. ## Operational caveat The first gateway restart after the fix still mints one final CA (into the new, initially-empty volume); it persists from then on. To avoid that one-time break, seed the volume with the current gateway's CA material before restarting. ## Related / out of scope - Docker `bot-bottle-orch-gateway` is attached to a stale CI CA volume (`bot-bottle-ci-ca-<run>`) because CI reuses the production gateway singleton name — separate latent issue. - `orchestrator/rotate_ca.py` is docker-only.

gitea-actions bot added the Status/Needs Triage label 2026-07-26 17:15:12 -04:00

didericis referenced this issue from a commit

2026-07-26 17:15:21 -04:00

fix(firecracker): persist the gateway mitmproxy CA across rebuilds

didericis-claude referenced a pull request that will close this issue

2026-07-26 17:15:36 -04:00

fix(firecracker): persist the gateway mitmproxy CA across rebuilds #511

didericis-claude referenced this issue

2026-07-26 17:24:49 -04:00

git-gate per-bottle state does not survive a gateway restart — running bottles 404 on fetch/push #512

didericis referenced this issue from a commit

2026-07-26 17:34:26 -04:00