didericis/bot-bottle
1
0
Fork 0
Code Issues 34 Pull Requests 11 Actions 4 Packages Releases Wiki Activity

fix(firecracker): persist the gateway mitmproxy CA across rebuilds #511

Closed
didericis-claude wants to merge 1 commits from fix/firecracker-gateway-ca-persist-510 into main
pull from: fix/firecracker-gateway-ca-persist-510
merge into: didericis:main
Conversation 1 Commits 1 Files Changed 4
+93 -6
didericis-claude commented 2026-07-26 17:15:36 -04:00
Collaborator
Copy Link
Copy Source

Problem

Rebuilding/restarting the Firecracker gateway minted a fresh mitmproxy CA, breaking every already-running bottle with SSL certificate verification failed on egress. The #450 CA-persistence fix existed only for the docker backend (host_gateway_ca_dir() bind-mount); the firecracker gateway had no equivalent, so its CA lived only in the ephemeral per-boot rootfs.

Fix

Give the gateway VM a persistent CA volume, mirroring the orchestrator's registry volume:

  • gateway.py: boot with a small ext4 data_drive via new _ensure_ca_volume() (creates on first use, reuses after).
  • infra_vm.py: the gateway guest init mounts /dev/vdb at mitmproxy's confdir (/home/mitmproxy/.mitmproxy) before the data plane starts, so mitmproxy reuses the on-disk CA rather than regenerating it.

Tests

  • New TestGatewayCaVolume (create/reuse) mirroring the orchestrator's registry-volume tests.
  • Updated test_boots_the_gateway_vm_and_seeds_the_token to assert the CA volume rides as data_drive.
  • Updated test_gateway_init_starts_only_the_data_plane to assert the /dev/vdb mount lands before bot_bottle.gateway.bootstrap.
  • 78 unit tests green across gateway/infra_vm/orchestrator/gateway_init.

Operational caveat

The first gateway restart after this lands still mints one final CA (into the new, initially-empty volume); it persists from then on. To skip that one-time break, seed the volume with the current gateway's CA material before restarting.

Closes #510

🤖 Generated with Claude Code

## Problem Rebuilding/restarting the Firecracker gateway minted a fresh mitmproxy CA, breaking every already-running bottle with `SSL certificate verification failed` on egress. The #450 CA-persistence fix existed only for the docker backend (`host_gateway_ca_dir()` bind-mount); the firecracker gateway had no equivalent, so its CA lived only in the ephemeral per-boot rootfs. ## Fix Give the gateway VM a persistent CA volume, mirroring the orchestrator's registry volume: - `gateway.py`: boot with a small ext4 `data_drive` via new `_ensure_ca_volume()` (creates on first use, reuses after). - `infra_vm.py`: the gateway guest init mounts `/dev/vdb` at mitmproxy's confdir (`/home/mitmproxy/.mitmproxy`) **before** the data plane starts, so mitmproxy reuses the on-disk CA rather than regenerating it. ## Tests - New `TestGatewayCaVolume` (create/reuse) mirroring the orchestrator's registry-volume tests. - Updated `test_boots_the_gateway_vm_and_seeds_the_token` to assert the CA volume rides as `data_drive`. - Updated `test_gateway_init_starts_only_the_data_plane` to assert the `/dev/vdb` mount lands before `bot_bottle.gateway.bootstrap`. - 78 unit tests green across gateway/infra_vm/orchestrator/gateway_init. ## Operational caveat The first gateway restart after this lands still mints one final CA (into the new, initially-empty volume); it persists from then on. To skip that one-time break, seed the volume with the current gateway's CA material before restarting. Closes #510 🤖 Generated with [Claude Code](https://claude.com/claude-code)
didericis-claude added 1 commit 2026-07-26 17:15:37 -04:00
fix(firecracker): persist the gateway mitmproxy CA across rebuilds
prd-number-check / require-numbered-prds (pull_request) Successful in 6s
Details
tracker-policy-pr / check-pr (pull_request) Successful in 7s
Details
lint / lint (push) Successful in 55s
Details
test / unit (pull_request) Successful in 47s
Details
test / integration-docker (pull_request) Failing after 2m24s
Details
test / coverage (pull_request) Has been skipped
Details
0ddaa95c14 
The #450 CA-persistence fix was only implemented for the docker backend
(the host_gateway_ca_dir bind-mount). The firecracker gateway booted with
no persistent volume, so its mitmproxy CA lived only in the ephemeral
per-boot rootfs — every rebuild/restart minted a fresh CA that every
already-running bottle distrusted, failing egress TLS with "SSL
certificate verification failed".

Give the gateway VM a persistent CA volume, mirroring the orchestrator's
registry volume: boot with a small ext4 data_drive (_ensure_ca_volume)
and mount it at mitmproxy's confdir in the gateway guest init, before the
data plane starts, so mitmproxy reuses the on-disk CA instead of
regenerating it.

Closes #510

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
didericis-claude commented 2026-07-26 17:56:03 -04:00
Author
Collaborator
Copy Link
Copy Source

Superseded by the reprovision-on-gateway-bring-up strategy (PRD 0081, forthcoming PR). Rather than persist gateway-side state on a volume, the gateway reconciles all running bottles when it comes up — replacing each running agent's CA with the current gateway CA and re-provisioning each bottle's git-gate — giving one unified reprovision flow across CA / git-gate / egress tokens and free CA rotation on each restart. Closing in favor of that single PR.

Superseded by the reprovision-on-gateway-bring-up strategy (PRD 0081, forthcoming PR). Rather than persist gateway-side state on a volume, the gateway reconciles all running bottles when it comes up — replacing each running agent's CA with the current gateway CA and re-provisioning each bottle's git-gate — giving one unified reprovision flow across CA / git-gate / egress tokens and free CA rotation on each restart. Closing in favor of that single PR.
didericis-claude closed this pull request 2026-07-26 17:56:07 -04:00
Some optional checks failed
prd-number-check / require-numbered-prds (pull_request) Successful in 6s
Details
tracker-policy-pr / check-pr (pull_request) Successful in 7s
Required
Details
lint / lint (push) Successful in 55s
Details
test / unit (pull_request) Successful in 47s
Details
test / integration-docker (pull_request) Failing after 2m24s
Details
test / coverage (pull_request) Has been skipped
Details

Pull request closed

Please reopen this pull request to perform a merge.
Sign in to join this conversation.
Reviewers
No Reviewers
Labels
Clear labels
Compat/Breaking
Breaking change that won't be backward compatible
 Kind/Bug
Something is not working
 Kind/Documentation
Documentation changes
 Kind/Enhancement
Improve existing functionality
 Kind/Feature
New functionality
 Kind/Security
This is security issue
 Kind/Testing
Issue or pull request related to testing
Priority
Critical
1
The priority is critical
Priority
High
2
The priority is high
Priority
Low
4
The priority is low
Priority
Medium
3
The priority is medium
Status
Abandoned
3
Somebody has started to work on this but abandoned work
Status
Blocked
1
Something is blocking this issue or pull request
Status
Need More Info
2
Feedback is required to reproduce issue or to continue work
Status/Needs Triage
Awaiting initial classification
No Label
Milestone
No items
No Milestone
Assignees
Clear assignees
didericis-claude (didericis (claude)) didericis-codex (didericis (codex)) didericis-gemma (gemma) didericis
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: didericis/bot-bottle#511
Delete Branch "fix/firecracker-gateway-ca-persist-510"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?