fix(macos): name the gateway instead of addressing it, so bottles survive it moving #445

Merged

didericis merged 2 commits from fix/gateway-stable-hostname into main 2026-07-20 23:25:47 -04:00

Conversation 0 Commits 2 Files Changed 7

+330 -17

didericis commented 2026-07-20 22:53:07 -04:00

Owner

Copy Link

Copy Source

closes #443.

Problem

The gateway's address is DHCP-assigned and moves whenever the infra container is recreated. Every agent-facing URL embedded it, and the proxy URL reaches the agent as process environment at container exec time — which cannot be rewritten from outside once the agent is running. So a moved gateway stranded every running bottle permanently, recoverable only by relaunching and discarding the agent session.

Seen live: infra moved 192.168.128.11 -> .19 and three bottles that had been up for hours were left pointing at an address that no longer existed.

Approach

Give the agent a stable name rather than an address. GATEWAY_HOSTNAME replaces the IP in the egress proxy URL, NO_PROXY, git-http, and supervise URLs, and resolves through the bottle's own /etc/hosts.

The key property: environ is immutable for a running process, but /etc/hosts is a file — it can be rewritten inside a container that is already running. That is what makes re-attachment possible at all.

• Launch writes the entry before anything execs (every agent URL names the gateway, so it must resolve for the first connection).
• Launch also re-points every running bottle once the gateway is up, so a bottle stranded by an earlier restart re-attaches in place instead of needing a relaunch.
• The rewrite is idempotent (replaces its own marked line, preserves localhost and the container's own entry) and keeps the file's inode, so a pre-created /etc/hosts isn't replaced by a root-owned copy.
• Best-effort per bottle: one container that refuses the write doesn't stop the others or fail the launch.

Security note

The write goes through a new exec_container_as_root. The agent runs as node and cannot rewrite /etc/hosts, so it cannot repoint its own gateway name; the host can. That asymmetry is deliberate and there's a test pinning it.

Constraints found

Apple Container 1.0 has no container-name DNS on a user network (the only nameserver an agent sees is vmnet's, which doesn't know container names) and container run has no --add-host — both verified on the live host — so the entry is written by exec after start rather than declared at run.

Not covered

The other half of #443: Orchestrator._tokens is in-memory only, so per-bottle egress auth tokens are still lost across an orchestrator restart. A re-attached bottle resolves its policy but not its injected upstream credentials. Left deliberately, since fixing it touches the no-secrets-at-rest property.

Tests

10 new (test_macos_gateway_hosts.py) plus updates to the wiring tests that asserted address-based URLs. 1891 unit tests pass, pyright clean, pylint 9.82.

Merge state

#442 has landed; this branch has been rebased onto main and is now a single independent commit. No ordering constraint remains.

enumerate.CONTAINER_NAME_PREFIX is exported both here and in #440 — identical change, so whichever lands second may need a trivial conflict resolution.

closes #443. ## Problem The gateway's address is DHCP-assigned and moves whenever the infra container is recreated. Every agent-facing URL embedded it, and the proxy URL reaches the agent as **process environment** at `container exec` time — which cannot be rewritten from outside once the agent is running. So a moved gateway stranded every running bottle permanently, recoverable only by relaunching and discarding the agent session. Seen live: infra moved `192.168.128.11 -> .19` and three bottles that had been up for hours were left pointing at an address that no longer existed. ## Approach Give the agent a stable **name** rather than an address. `GATEWAY_HOSTNAME` replaces the IP in the egress proxy URL, `NO_PROXY`, git-http, and supervise URLs, and resolves through the bottle's own `/etc/hosts`. The key property: `environ` is immutable for a running process, but `/etc/hosts` is a **file** — it can be rewritten inside a container that is already running. That is what makes re-attachment possible at all. - Launch writes the entry **before anything execs** (every agent URL names the gateway, so it must resolve for the first connection). - Launch also re-points every running bottle once the gateway is up, so a bottle stranded by an earlier restart re-attaches in place instead of needing a relaunch. - The rewrite is idempotent (replaces its own marked line, preserves `localhost` and the container's own entry) and keeps the file's inode, so a pre-created `/etc/hosts` isn't replaced by a root-owned copy. - Best-effort per bottle: one container that refuses the write doesn't stop the others or fail the launch. ## Security note The write goes through a new `exec_container_as_root`. The agent runs as `node` and **cannot** rewrite `/etc/hosts`, so it cannot repoint its own gateway name; the host can. That asymmetry is deliberate and there's a test pinning it. ## Constraints found Apple Container 1.0 has no container-name DNS on a user network (the only nameserver an agent sees is vmnet's, which doesn't know container names) and `container run` has no `--add-host` — both verified on the live host — so the entry is written by exec after start rather than declared at run. ## Not covered The other half of #443: `Orchestrator._tokens` is in-memory only, so per-bottle egress auth tokens are still lost across an orchestrator restart. A re-attached bottle resolves its policy but not its injected upstream credentials. Left deliberately, since fixing it touches the no-secrets-at-rest property. ## Tests 10 new (`test_macos_gateway_hosts.py`) plus updates to the wiring tests that asserted address-based URLs. 1891 unit tests pass, pyright clean, pylint 9.82. ## Merge state #442 has landed; this branch has been rebased onto `main` and is now a single independent commit. No ordering constraint remains. `enumerate.CONTAINER_NAME_PREFIX` is exported both here and in #440 — identical change, so whichever lands second may need a trivial conflict resolution.

didericis changed target branch from fix/duplicate-proxy-env-drops-identity-token to main 2026-07-20 22:55:12 -04:00

didericis referenced this pull request 2026-07-20 23:04:16 -04:00

fix: reconcile orphaned registry rows against live containers #440

didericis referenced this pull request 2026-07-20 23:07:39 -04:00

Restarting the shared gateway permanently strands every running bottle #443

didericis added 1 commit 2026-07-20 23:09:42 -04:00

fix(macos): name the gateway instead of addressing it, so bottles survive it moving ... 2f8539c2c7

The shared gateway's address is DHCP-assigned and changes whenever the infra
container is recreated — a source-hash bump, an image upgrade, a crash. Every
agent-facing URL embedded that address, and the proxy URL reaches the agent
as process environment at `container exec` time. A running process's environ
cannot be rewritten from outside, so a moved gateway stranded every running
bottle permanently: not degraded, unreachable, until relaunched and its agent
session thrown away.

Give the agent a stable name instead. `GATEWAY_HOSTNAME` replaces the address
in the egress proxy URL, NO_PROXY, git-http, and supervise URLs, and resolves
through the bottle's own /etc/hosts. Unlike environ that is a file, so it can
be rewritten inside a container that is already running — which is the whole
point: a gateway that returns at a new address is picked up by live bottles.

Launch writes the entry before anything execs (every agent URL names the
gateway, so it must resolve for the first connection), and re-points every
running bottle once the gateway is up, so one stranded by an earlier restart
re-attaches instead of needing a relaunch.

The write needs root and the agent runs as `node`: the host can repoint a
bottle's gateway name, the agent cannot repoint its own. Keep that asymmetry.

Apple Container 1.0 has no container-name DNS on a user network and
`container run` has no --add-host, so the entry is written by exec after
start rather than declared at run.

Does not address the other half of #443: per-bottle egress auth tokens are
held in memory by the orchestrator and are still lost across a restart, so a
re-attached bottle resolves its policy but not its injected credentials.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

didericis force-pushed fix/gateway-stable-hostname from ae46779fe5 to 2f8539c2c7 2026-07-20 23:09:42 -04:00 Compare

didericis added 1 commit 2026-07-20 23:12:23 -04:00

test(macos): cover root container exec helper 5e01c28016

didericis merged commit 5e01c28016 into main 2026-07-20 23:25:47 -04:00

didericis deleted branch fix/gateway-stable-hostname 2026-07-20 23:25:49 -04:00

didericis-claude referenced this pull request 2026-07-21 02:27:11 -04:00

Gateway regenerates its mitmproxy CA on every restart, so re-attached bottles fail TLS #450

didericis-claude referenced this pull request 2026-07-21 02:27:23 -04:00

Restarting the shared gateway permanently strands every running bottle #443

Sign in to join this conversation.

Reviewers

No Reviewers

Labels

Clear labels

Compat/Breaking

Breaking change that won't be backward compatible

Kind/Bug

Something is not working

Kind/Documentation

Documentation changes

Kind/Enhancement

Improve existing functionality

Kind/Feature

New functionality

Kind/Security

This is security issue

Kind/Testing

Issue or pull request related to testing

Priority

Critical

1

The priority is critical

Priority

High

2

The priority is high

Priority

Low

4

The priority is low

Priority

Medium

3

The priority is medium

Reviewed

Confirmed

1

Issue has been confirmed

Reviewed

Duplicate

2

This issue or pull request already exists

Reviewed

Invalid

3

Invalid issue

Reviewed

Won't Fix

3

This issue won't be fixed

Status

Abandoned

3

Somebody has started to work on this but abandoned work

Status

Blocked

1

Something is blocking this issue or pull request

Status

Need More Info

2

Feedback is required to reproduce issue or to continue work

Status/Needs Triage

Awaiting initial classification

No Label

Milestone

No items

No Milestone

Assignees

Clear assignees

didericis-claude (didericis (claude)) didericis-codex (didericis (codex)) didericis-gemma (gemma) didericis

No Assignees

1 Participants

Notifications

Subscribe

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: didericis/bot-bottle#445