Restarting the shared gateway permanently strands every running bottle #443

New Issue

2026-07-20T22:44:52-04:00

didericis commented

2026-07-20 22:44:52 -04:00

Symptom

Restarting or recreating the shared per-host gateway permanently strands every running bottle. They are not degraded, they are dead: all egress, git-gate, and supervise traffic fails until the bottle is relaunched, which throws away the agent session.

Observed today. The infra container was recreated (the control-plane source is bind-mounted and its BOT_BOTTLE_SOURCE_HASH changed), and vmnet handed it a new address:

before:  bot-bottle-mac-infra  192.168.128.11
after:   bot-bottle-mac-infra  192.168.128.19

Three bottles that had been running for hours — claude-agent-2, claude-agent-3, reborn-agent-1 — were still pointed at 192.168.128.11, which no longer exists. Nothing re-attached them and nothing told the operator; they simply stopped being able to reach anything.

Why it can't be patched at runtime today

The gateway address reaches the agent as process environment: the exec-time *_PROXY URL, NO_PROXY, and alongside them the CA path, GIT_GATE_URL, and MCP_SUPERVISE_URL. A running process's environ is immutable from outside, so there is no way to re-point an agent that is already running. Any fix that tries to "update the bottle's gateway address" runs into this wall.

That argues the load-bearing fix is a stable gateway endpoint rather than a re-pointing mechanism: pin/reserve the infra container's address, or give the gateway a name the agent resolves per request, so a restart doesn't move it. Then a gateway bounce becomes a brief outage instead of a permanent strand.

Second failure, even at a stable address

Orchestrator._tokens is in-memory only, by design:

Held in memory only — never written to the registry DB — so the gateway can inject each bottle's upstream credential without secrets at rest. Lost on restart (re-launch re-registers them)

So even if the address were stable, a gateway restart loses every bottle's egress auth tokens, and upstream auth injection fails until the bottle is relaunched. Re-attachment needs these re-established, without walking back the no-secrets-at-rest property.

Scope

Stable gateway endpoint across infra restarts (address pinning or name-based resolution).
Re-establish per-bottle egress auth tokens after an orchestrator restart.
A way to re-attach live bottles — and, failing that, to tell the operator their bottles are stranded instead of leaving them to find out by hitting it.

Note that the clearer deny messages in #440 do not help here. Those improve a 403 the gateway generates; a bottle stranded by a moved gateway never reaches a gateway at all and gets a connection failure, which carries no explanation. Diagnosing this case needs its own signal.

Status

Item 1 is fixed by #445 — agents now name the gateway rather than addressing it, resolved through the bottle's /etc/hosts. /etc/hosts is a file, so unlike the environ the proxy URL is delivered in, it can be rewritten inside an already-running container; launch re-points every live bottle once the gateway is up.

Caveat carried over from that PR: the rewrite and immediate getent resolution were verified on the host, but a long-running agent picking up a changed entry mid-flight was not. If a client caches its resolution at startup, re-attach silently won't work for that process. Worth testing against a live bottle.

Items 2 and 3 remain open. Without item 2 a re-attached bottle resolves its policy but not its injected upstream credentials, which limits how much item 1 actually buys.

#441 / #442 — duplicate *_PROXY env dropping the identity token. Same underlying theme: the gateway's address and credentials reach the agent as process environment, which cannot be corrected once the agent is running. That is what makes this class of failure permanent rather than transient.
#439 / #440 — orphaned registry rows. Unrelated cause, found during the same investigation; it produces a gateway-generated 403, not the connection failure this issue produces.

## Symptom Restarting or recreating the shared per-host gateway **permanently strands every running bottle**. They are not degraded, they are dead: all egress, git-gate, and supervise traffic fails until the bottle is relaunched, which throws away the agent session. Observed today. The infra container was recreated (the control-plane source is bind-mounted and its `BOT_BOTTLE_SOURCE_HASH` changed), and vmnet handed it a **new address**: ``` before: bot-bottle-mac-infra 192.168.128.11 after: bot-bottle-mac-infra 192.168.128.19 ``` Three bottles that had been running for hours — `claude-agent-2`, `claude-agent-3`, `reborn-agent-1` — were still pointed at `192.168.128.11`, which no longer exists. Nothing re-attached them and nothing told the operator; they simply stopped being able to reach anything. ## Why it can't be patched at runtime today The gateway address reaches the agent as **process environment**: the exec-time `*_PROXY` URL, `NO_PROXY`, and alongside them the CA path, `GIT_GATE_URL`, and `MCP_SUPERVISE_URL`. A running process's `environ` is immutable from outside, so there is no way to re-point an agent that is already running. Any fix that tries to "update the bottle's gateway address" runs into this wall. That argues the load-bearing fix is a **stable gateway endpoint** rather than a re-pointing mechanism: pin/reserve the infra container's address, or give the gateway a name the agent resolves per request, so a restart doesn't move it. Then a gateway bounce becomes a brief outage instead of a permanent strand. ## Second failure, even at a stable address `Orchestrator._tokens` is in-memory only, by design: > Held **in memory only** — never written to the registry DB — so the gateway can inject each bottle's upstream credential without secrets at rest. Lost on restart (re-launch re-registers them) So even if the address were stable, a gateway restart loses every bottle's egress auth tokens, and upstream auth injection fails until the bottle is relaunched. Re-attachment needs these re-established, without walking back the no-secrets-at-rest property. ## Scope 1. Stable gateway endpoint across infra restarts (address pinning or name-based resolution). 2. Re-establish per-bottle egress auth tokens after an orchestrator restart. 3. A way to re-attach live bottles — and, failing that, to *tell* the operator their bottles are stranded instead of leaving them to find out by hitting it. Note that the clearer deny messages in #440 do **not** help here. Those improve a 403 the gateway generates; a bottle stranded by a moved gateway never reaches a gateway at all and gets a connection failure, which carries no explanation. Diagnosing this case needs its own signal. ## Status **Item 1 is fixed by #445** — agents now name the gateway rather than addressing it, resolved through the bottle's `/etc/hosts`. `/etc/hosts` is a file, so unlike the `environ` the proxy URL is delivered in, it can be rewritten inside an already-running container; launch re-points every live bottle once the gateway is up. Caveat carried over from that PR: the rewrite and immediate `getent` resolution were verified on the host, but a *long-running* agent picking up a changed entry mid-flight was not. If a client caches its resolution at startup, re-attach silently won't work for that process. Worth testing against a live bottle. **Items 2 and 3 remain open.** Without item 2 a re-attached bottle resolves its policy but not its injected upstream credentials, which limits how much item 1 actually buys. ## Related - #441 / #442 — duplicate `*_PROXY` env dropping the identity token. Same underlying theme: the gateway's address and credentials reach the agent as process environment, which cannot be corrected once the agent is running. That is what makes this class of failure permanent rather than transient. - #439 / #440 — orphaned registry rows. Unrelated cause, found during the same investigation; it produces a gateway-generated 403, not the connection failure this issue produces.

gitea-actions bot added the Status/Needs Triage label 2026-07-20 22:45:02 -04:00

didericis added Kind/Bug

Priority

Critical

1

and removed Status/Needs Triage labels 2026-07-20 22:51:54 -04:00

gitea-actions bot added the Status/Needs Triage label 2026-07-20 22:51:58 -04:00

didericis referenced this issue from a commit

2026-07-20 22:52:48 -04:00

fix(macos): name the gateway instead of addressing it, so bottles survive it moving

~~didericis referenced this issue 2026-07-20 22:53:07 -04:00~~

fix(macos): name the gateway instead of addressing it, so bottles survive it moving #445

didericis referenced a pull request that will close this issue

2026-07-20 23:00:10 -04:00

fix(macos): name the gateway instead of addressing it, so bottles survive it moving #445

didericis referenced this issue

2026-07-20 23:03:43 -04:00

Orphaned registry rows are never reconciled against live containers #439

didericis referenced this issue

2026-07-20 23:04:16 -04:00

fix: reconcile orphaned registry rows against live containers #440

didericis referenced this issue from a commit

2026-07-20 23:09:42 -04:00

fix(macos): name the gateway instead of addressing it, so bottles survive it moving

didericis closed this issue

2026-07-20 23:25:48 -04:00

didericis-claude referenced this issue

2026-07-21 02:27:11 -04:00

Gateway regenerates its mitmproxy CA on every restart, so re-attached bottles fail TLS #450

didericis-claude commented

2026-07-21 02:27:22 -04:00

Filing #450 as a fourth item against this issue. The gateway mints a fresh mitmproxy CA on every infra recreation (notBefore == the container start time), so a bottle that follows the gateway to its new address via #445 still fails TLS against a CA it has never seen. Item 1 makes a moved gateway addressable; re-attach cannot actually work until the CA is persisted too.

Also a partial data point for the caveat above: during the 2026-07-21 incident the host-side /etc/hosts rewrite did land on a live container. Whether a long-lived client re-resolves or caches its first lookup is still unverified.

Filing #450 as a fourth item against this issue. The gateway mints a fresh mitmproxy CA on every infra recreation (`notBefore` == the container start time), so a bottle that follows the gateway to its new address via #445 still fails TLS against a CA it has never seen. Item 1 makes a moved gateway addressable; re-attach cannot actually work until the CA is persisted too. Also a partial data point for the caveat above: during the 2026-07-21 incident the host-side `/etc/hosts` rewrite *did* land on a live container. Whether a long-lived client re-resolves or caches its first lookup is still unverified.

Sign in to join this conversation.

2 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: didericis/bot-bottle#443