fix(gateway): persist mitmproxy CA on the host, not a named volume (#450) #454

2026-07-21T14:17:20-04:00

didericis-claude commented

2026-07-21 14:17:20 -04:00

What & why

The shared gateway self-generates a mitmproxy CA that every bottle installs to trust its TLS interception. That CA was persisted on a Docker named volume, which survives docker rm but is silently wiped by docker volume prune / docker system prune --volumes during routine host maintenance. When that happens, the gateway mints a fresh CA on the next restart and every already-running bottle fails the TLS handshake — even after it re-resolves and reconnects to the moved gateway. This is the fourth re-attachment blocker described in #450, distinct from #443/#445.

Change

Persist the CA on the host filesystem, not a named volume. New host_gateway_ca_dir() → bot_bottle_root()/gateway-ca, bind-mounted into the infra/gateway container at mitmproxy's confdir. This mirrors how the shared DB and control-plane token already live under the app-data root. Docker never prunes a path under the root, and it stays directly inspectable + rotatable from the host.
Adopt-existing / generate-on-first-run comes for free: mitmproxy reuses an on-disk CA and generates one only when absent, so the bind-mount alone gives stable-across-restart behaviour.
Explicit rotation for deliberate rollover: rotate_gateway_ca() clears the persisted CA so the next start remints it, and python -m bot_bottle.orchestrator.rotate_ca wires that together with dropping the running gateway container (whose mitmproxy still holds the old CA in memory). Rotation stays an operator action — it doesn't auto-re-provision running bottles, which re-attach to install the new anchor.

Maps 1:1 to the issue's proposed solution (persist to host FS → bind-mount → adopt-on-startup → explicit rotation).

Scope

Fixes the Docker infra/gateway path — the "infra container" in the report (OrchestratorService, DockerGateway). The macOS (container-only volume) and Firecracker (VM-attached ext4 file) backends persist the CA by other means and are unaffected.

Tests

New rotate_gateway_ca unit tests: removes only mitmproxy-ca* material (leaves the non-CA trust bundle), defaults to the host CA dir, idempotent when empty.
New test_orchestrator_rotate_ca: the one-shot CLI clears the CA and force-removes both gateway container names; succeeds with no persisted CA.
Updated gateway + lifecycle tests to assert a host bind-mount (source under the app-data root, ending in /gateway-ca) rather than a named volume.
Full unit suite green (1991 tests).

Docs: PRD 0070 gains a "Gateway CA: host-resident, like the DB" subsection capturing the why + rotation.

Closes #450.

🤖 Generated with Claude Code

## What & why The shared gateway self-generates a mitmproxy CA that **every bottle installs** to trust its TLS interception. That CA was persisted on a Docker **named volume**, which survives `docker rm` but is **silently wiped by `docker volume prune` / `docker system prune --volumes`** during routine host maintenance. When that happens, the gateway mints a *fresh* CA on the next restart and every already-running bottle fails the TLS handshake — even after it re-resolves and reconnects to the moved gateway. This is the fourth re-attachment blocker described in #450, distinct from #443/#445. ## Change - **Persist the CA on the host filesystem, not a named volume.** New `host_gateway_ca_dir()` → `bot_bottle_root()/gateway-ca`, bind-mounted into the infra/gateway container at mitmproxy's confdir. This mirrors how the shared DB and control-plane token already live under the app-data root. Docker never prunes a path under the root, and it stays directly inspectable + rotatable from the host. - **Adopt-existing / generate-on-first-run** comes for free: mitmproxy reuses an on-disk CA and generates one only when absent, so the bind-mount alone gives stable-across-restart behaviour. - **Explicit rotation** for deliberate rollover: `rotate_gateway_ca()` clears the persisted CA so the next start remints it, and `python -m bot_bottle.orchestrator.rotate_ca` wires that together with dropping the running gateway container (whose mitmproxy still holds the old CA in memory). Rotation stays an operator action — it doesn't auto-re-provision running bottles, which re-attach to install the new anchor. Maps 1:1 to the issue's proposed solution (persist to host FS → bind-mount → adopt-on-startup → explicit rotation). ## Scope Fixes the **Docker** infra/gateway path — the "infra container" in the report (`OrchestratorService`, `DockerGateway`). The macOS (`container`-only volume) and Firecracker (VM-attached ext4 file) backends persist the CA by other means and are unaffected. ## Tests - New `rotate_gateway_ca` unit tests: removes only `mitmproxy-ca*` material (leaves the non-CA trust bundle), defaults to the host CA dir, idempotent when empty. - New `test_orchestrator_rotate_ca`: the one-shot CLI clears the CA and force-removes both gateway container names; succeeds with no persisted CA. - Updated gateway + lifecycle tests to assert a **host bind-mount** (source under the app-data root, ending in `/gateway-ca`) rather than a named volume. - Full unit suite green (1991 tests). Docs: PRD 0070 gains a "Gateway CA: host-resident, like the DB" subsection capturing the why + rotation. Closes #450. 🤖 Generated with [Claude Code](https://claude.com/claude-code)

didericis-claude added 1 commit 2026-07-21 14:17:21 -04:00

fix(gateway): persist mitmproxy CA on the host, not a named volume (#450 )

lint / lint (push) Successful in 49s

Details

tracker-policy-pr / check-pr (pull_request) Successful in 7s

Details

test / integration-docker (pull_request) Successful in 12s

Details

test / unit (pull_request) Successful in 36s

Details

test / integration-firecracker (pull_request) Successful in 3m27s

Details

test / coverage (pull_request) Successful in 16s

Details

test / publish-infra (pull_request) Has been skipped

Details

f52ac0ebbf

The shared gateway self-generates a mitmproxy CA that every bottle installs
to trust its TLS interception. It was persisted on a Docker named volume,
which survives `docker rm` but is silently wiped by `docker volume prune` /
`docker system prune --volumes` during routine host maintenance. When that
happens the gateway mints a fresh CA on restart, and every already-running
bottle fails the TLS handshake even after it re-resolves and reconnects to
the moved gateway — a re-attachment blocker distinct from #443/#445.

Move CA persistence to a host bind-mount under the app-data root
(`bot_bottle_root()/gateway-ca`, via `host_gateway_ca_dir()`), mirroring how
the shared DB and control-plane token already live on the host. Docker never
prunes a path under the root, and it stays inspectable + rotatable from the
host. mitmproxy already adopts an existing CA and generates one only on first
run, so the bind-mount gives adopt-existing/generate-on-first-run for free.

Add an explicit rollover path: `rotate_gateway_ca()` clears the persisted CA
so the next start remints it, and `python -m bot_bottle.orchestrator.rotate_ca`
wires that together with dropping the running gateway container (whose
mitmproxy still holds the old CA in memory). Rotation stays an operator action
— it doesn't auto-re-provision running bottles, which re-attach to pick up the
new anchor.

Scope: the Docker infra/gateway path (the "infra container" in the report).
The macOS (`container`-only volume) and Firecracker (VM-attached ext4) backends
persist the CA differently and are unaffected.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

didericis added 1 commit 2026-07-21 14:51:14 -04:00

fix(macos): persist gateway CA on host

tracker-policy-pr / check-pr (pull_request) Successful in 10s

Details

test / integration-docker (pull_request) Successful in 34s

Details

test / unit (pull_request) Successful in 40s

Details

lint / lint (push) Successful in 48s

Details

test / integration-firecracker (pull_request) Successful in 3m28s

Details

test / coverage (pull_request) Successful in 16s

Details

test / publish-infra (pull_request) Has been skipped

Details

60c9f8a971

didericis approved these changes 2026-07-21 15:01:24 -04:00

didericis merged commit 2cd44cf79a into main

2026-07-21 15:01:40 -04:00

didericis deleted branch fix/issue-450-gateway-ca-persistence

2026-07-21 15:01:42 -04:00

didericis-codex commented

2026-07-21 15:02:59 -04:00

macOS is now covered by this PR as of 60c9f8a.

The Apple "container-only volume" is the named ext4 volume mounted at /var/lib/bot-bottle; it exists specifically to keep SQLite access within one guest kernel. It did not cover mitmproxy’s CA directory at /home/mitmproxy/.mitmproxy, so recreating the infra container silently generated a new CA.

The macOS backend now bind-mounts the host-resident gateway-ca directory into mitmproxy’s confdir, matching the Docker fix. The DB remains on its container-only volume, while the CA survives both infra-container recreation and container volume prune. Regression coverage and PRD rationale are included.

macOS is now covered by this PR as of `60c9f8a`. The Apple "container-only volume" is the named ext4 volume mounted at `/var/lib/bot-bottle`; it exists specifically to keep SQLite access within one guest kernel. It did not cover mitmproxy’s CA directory at `/home/mitmproxy/.mitmproxy`, so recreating the infra container silently generated a new CA. The macOS backend now bind-mounts the host-resident `gateway-ca` directory into mitmproxy’s confdir, matching the Docker fix. The DB remains on its container-only volume, while the CA survives both infra-container recreation and `container volume prune`. Regression coverage and PRD rationale are included.

Sign in to join this conversation.

3 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: didericis/bot-bottle#454