Gateway regenerates its mitmproxy CA on every restart, so re-attached bottles fail TLS #450

New Issue

2026-07-21T02:27:11-04:00

didericis-claude commented

2026-07-21 02:27:11 -04:00

Symptom

The gateway regenerates its mitmproxy CA every time the infra container is recreated. The new CA is not in any running bottle's trust store, so a bottle that correctly follows the gateway to its new address still fails TLS on every HTTPS request.

This is a third, independent strand mechanism alongside the two in #443, and it is not mentioned there. #445 fixed item 1 of #443 (name-based resolution), which makes a moved gateway addressable again — but a re-attached bottle then dies at the TLS handshake instead of at connect. Re-attach cannot work until this is fixed too.

Evidence

Observed 2026-07-21. bot-bottle-mac-infra was recreated at 06:05:44Z, moving 192.168.128.23 -> 192.168.128.31. The gateway's CA notBefore is exactly the restart timestamp, which is the tell — it is minted at container start, not persisted:

gateway (live)   sha256 D5:FC:61:98:...:6D:87:97   notBefore=Jul 19 06:05:44 2026 GMT
codex-agent-1    sha256 D5:FC:61:98:...:6D:87:97   <- launched 06:05:48Z, matches
codex-agent-2    sha256 98:C0:24:35:...:40:31:61   notBefore=Jul 19 02:56:18 2026 GMT

(mitmproxy backdates notBefore by two days; the time-of-day is the giveaway.)

Isolating the two failures on the stranded bottle, after manually rewriting its proxy URL to the new address:

plain http:// through the proxy -> reaches the gateway, returns a gateway-generated 403
https:// through the proxy -> curl exit 60, certificate verify failed

So the transport is fine and the trust anchor is not.

Why it isn't covered by the existing items

Orchestrator._tokens (item 2 of #443) is deliberately memory-only to keep secrets off disk. The CA is a different tradeoff: the private key must stay protected, but the CA certificate is public by construction — it is already distributed into every bottle's trust store. Persisting the keypair on the host and bind-mounting it into the gateway does not create a new secret-at-rest class that the design was avoiding; it gives the interception identity the same lifetime as the host rather than the lifetime of a container.

Without that, every infra recreation invalidates the trust store of every running bottle, and the only recovery is relaunching the bottle — which throws away the agent session, the exact outcome #443 set out to eliminate.

Proposed fix

Persist ~/.mitmproxy (or at minimum mitmproxy-ca.pem) on the host and bind-mount it into the infra container, so a recreated gateway keeps its CA identity. Protect the key with host file permissions.
On gateway start, if a persisted CA exists, adopt it rather than minting a new one; mint only on first run.
Consider a rotation path (explicit CLI action) so the CA can be rolled deliberately, with the understanding that rolling it strands running bottles by design.

Incidental confirmation for the #443 caveat

#443 flagged that a long-running bottle picking up a rewritten /etc/hosts mid-flight was never verified. Partial data point from this incident: the host-side rewrite does land on a live container — the stranded bottle's /etc/hosts had been correctly updated to the gateway's new address while the bottle was running. What remains unverified is whether a long-lived client process re-resolves rather than caching its first lookup. The bottles running during this incident predate #445 and carry a literal address in their environ, so they could not exercise the resolution path either way.

## Symptom The gateway regenerates its mitmproxy CA every time the infra container is recreated. The new CA is not in any running bottle's trust store, so a bottle that correctly follows the gateway to its new address still fails TLS on every HTTPS request. This is a **third**, independent strand mechanism alongside the two in #443, and it is not mentioned there. #445 fixed item 1 of #443 (name-based resolution), which makes a moved gateway *addressable* again — but a re-attached bottle then dies at the TLS handshake instead of at connect. Re-attach cannot work until this is fixed too. ## Evidence Observed 2026-07-21. `bot-bottle-mac-infra` was recreated at 06:05:44Z, moving `192.168.128.23 -> 192.168.128.31`. The gateway's CA `notBefore` is exactly the restart timestamp, which is the tell — it is minted at container start, not persisted: ``` gateway (live) sha256 D5:FC:61:98:...:6D:87:97 notBefore=Jul 19 06:05:44 2026 GMT codex-agent-1 sha256 D5:FC:61:98:...:6D:87:97 <- launched 06:05:48Z, matches codex-agent-2 sha256 98:C0:24:35:...:40:31:61 notBefore=Jul 19 02:56:18 2026 GMT ``` (mitmproxy backdates `notBefore` by two days; the time-of-day is the giveaway.) Isolating the two failures on the stranded bottle, after manually rewriting its proxy URL to the new address: - plain `http://` through the proxy -> reaches the gateway, returns a gateway-generated 403 - `https://` through the proxy -> `curl` exit 60, certificate verify failed So the transport is fine and the trust anchor is not. ## Why it isn't covered by the existing items `Orchestrator._tokens` (item 2 of #443) is deliberately memory-only to keep secrets off disk. The CA is a different tradeoff: the **private key** must stay protected, but the CA *certificate* is public by construction — it is already distributed into every bottle's trust store. Persisting the keypair on the host and bind-mounting it into the gateway does not create a new secret-at-rest class that the design was avoiding; it gives the interception identity the same lifetime as the host rather than the lifetime of a container. Without that, every infra recreation invalidates the trust store of every running bottle, and the only recovery is relaunching the bottle — which throws away the agent session, the exact outcome #443 set out to eliminate. ## Proposed fix 1. Persist `~/.mitmproxy` (or at minimum `mitmproxy-ca.pem`) on the host and bind-mount it into the infra container, so a recreated gateway keeps its CA identity. Protect the key with host file permissions. 2. On gateway start, if a persisted CA exists, adopt it rather than minting a new one; mint only on first run. 3. Consider a rotation path (explicit CLI action) so the CA *can* be rolled deliberately, with the understanding that rolling it strands running bottles by design. ## Incidental confirmation for the #443 caveat #443 flagged that a *long-running* bottle picking up a rewritten `/etc/hosts` mid-flight was never verified. Partial data point from this incident: the host-side rewrite does land on a live container — the stranded bottle's `/etc/hosts` had been correctly updated to the gateway's new address while the bottle was running. What remains unverified is whether a long-lived client process re-resolves rather than caching its first lookup. The bottles running during this incident predate #445 and carry a literal address in their `environ`, so they could not exercise the resolution path either way. ## Related - #443 — parent. Item 1 fixed by #445; items 2 (in-memory egress tokens) and 3 (re-attach / operator signal) still open. This issue is a fourth item: the CA is as non-durable as the tokens. - #445 — name-based gateway resolution.

didericis-claude added the Kind/Bug

Priority

High

2

Status/Needs Triage labels 2026-07-21 02:27:11 -04:00

didericis-claude referenced this issue

2026-07-21 02:27:23 -04:00

Restarting the shared gateway permanently strands every running bottle #443

didericis removed the Status/Needs Triage label 2026-07-21 12:41:27 -04:00

didericis referenced this issue from a commit

2026-07-21 14:16:37 -04:00

fix(gateway): persist mitmproxy CA on the host, not a named volume (#450)