Codex bottles get zero egress: duplicate *_PROXY env drops the identity token #441

New Issue

2026-07-20T22:41:06-04:00

didericis commented

2026-07-20 22:41:06 -04:00

Symptom

Every Codex bottle is denied all egress. Claude bottles on the same host, same gateway, same manifests, work fine. Codex fails on its model calls, its wss://chatgpt.com/backend-api/codex/responses reconnects, and its MCP servers:

MCP client for `codex_apps` failed to start: HTTP 403: egress: ... not attributed ...
Unexpected status 403 Forbidden, url: wss://chatgpt.com/backend-api/codex/responses

The registry row is correct — right slug, right source IP, seconds old. That is what made this look like a registration bug for so long.

Root cause

container exec --env does not override the run-time environment on Apple Container — it appends. launch._agent_env_entries baked a token-less *_PROXY into container run, and _identity_proxy_env applied the token-bearing value at exec, on the assumption that exec wins.

The agent's environ therefore holds two entries, token-less first:

HTTPS_PROXY=http://<gw>:9099            <- baked, no token   (first)
HTTPS_PROXY=http://<CREDS>@<gw>:9099    <- exec, with token  (second)

Verified directly in /proc/<pid>/environ for both a Claude and a Codex agent — both have the duplicate. The difference is only in how each runtime resolves it:

Node (Claude Code) reads the last entry -> gets the token -> works.
Rust std::env::var (Codex) reads the first -> no token -> /resolve fail-closes on the (source_ip, identity_token) pair -> every host denied.

Three comments asserted the false invariant (launch.py:269, bottle.py:71-73), which is why it survived review: it was true in practice for the only agent anyone tested.

Fix

PR #442 drops the run-time proxy vars entirely, making the exec-time env the single source. A token-less proxy URL has no legitimate consumer — the init process is sleep and everything that egresses arrives via exec. A process that egresses before the exec-time env still fails closed, since the agent network is host-only.

## Symptom Every Codex bottle is denied all egress. Claude bottles on the same host, same gateway, same manifests, work fine. Codex fails on its model calls, its `wss://chatgpt.com/backend-api/codex/responses` reconnects, and its MCP servers: ``` MCP client for `codex_apps` failed to start: HTTP 403: egress: ... not attributed ... Unexpected status 403 Forbidden, url: wss://chatgpt.com/backend-api/codex/responses ``` The registry row is **correct** — right slug, right source IP, seconds old. That is what made this look like a registration bug for so long. ## Root cause `container exec --env` does **not** override the run-time environment on Apple Container — it *appends*. `launch._agent_env_entries` baked a token-less `*_PROXY` into `container run`, and `_identity_proxy_env` applied the token-bearing value at exec, on the assumption that exec wins. The agent's `environ` therefore holds two entries, token-less first: ``` HTTPS_PROXY=http://<gw>:9099 <- baked, no token (first) HTTPS_PROXY=http://<CREDS>@<gw>:9099 <- exec, with token (second) ``` Verified directly in `/proc/<pid>/environ` for both a Claude and a Codex agent — **both** have the duplicate. The difference is only in how each runtime resolves it: - **Node** (Claude Code) reads the **last** entry -> gets the token -> works. - **Rust `std::env::var`** (Codex) reads the **first** -> no token -> `/resolve` fail-closes on the (source_ip, identity_token) pair -> every host denied. Three comments asserted the false invariant (`launch.py:269`, `bottle.py:71-73`), which is why it survived review: it was true in practice for the only agent anyone tested. ## Fix PR #442 drops the run-time proxy vars entirely, making the exec-time env the single source. A token-less proxy URL has no legitimate consumer — the init process is `sleep` and everything that egresses arrives via exec. A process that egresses before the exec-time env still fails closed, since the agent network is host-only. ## Related - #439 / #440 — orphaned registry rows, a genuinely separate cause of the same 403. - The misleading deny message that hid both is fixed in #440.

didericis referenced a pull request that will close this issue

2026-07-20 22:41:07 -04:00

fix(macos): supply proxy env only at exec, so Codex keeps its identity token #442

gitea-actions bot added the Status/Needs Triage label 2026-07-20 22:41:13 -04:00

didericis referenced this issue

2026-07-20 22:44:53 -04:00

Restarting the shared gateway permanently strands every running bottle #443

didericis added Kind/Bug

Priority

Critical

1

and removed Status/Needs Triage labels 2026-07-20 22:52:38 -04:00