Orphaned registry rows are never reconciled against live containers #439

New Issue

2026-07-20T22:30:20-04:00

didericis commented

2026-07-20 22:30:20 -04:00

Corrected. This issue originally opened with a Codex bottle being denied all egress and presented orphaned rows as the cause. That was wrong — those 403s were #441 (duplicate *_PROXY env dropping the identity token); the Codex bottle's registry row was correct every time. The defect described below is real and was found along the way, but it is not what produced that symptom. Rewritten to say only what was actually verified.

The defect

Registry rows outlive their containers, and nothing ever cleans them up.

A row only leaves the registry two ways: an explicit teardown_bottle (the launcher's cleanup callback), or the same-IP supersede sweep in register(). Neither runs if the launching CLI dies hard — SIGKILL, a closed terminal, a host sleep. There is no reconciliation of registry rows against live containers anywhere in the codebase, so a row left behind stays active forever.

Verified on the host: bot-bottle-claude-agent-1 was registered active at 192.168.128.6 with no container running (not even stopped) and no /git namespace. A pure orphan.

Why it matters

RegistryStore.by_source_ip fail-closes when len(rows) != 1 — zero rows and ambiguous rows both resolve to "no policy", and a bottle with no policy is denied every host. Source IPs are recycled by the backend's DHCP, so an orphan sitting on an address is a live collision hazard for whatever bottle lands there next.

Being precise about status: the accumulation is observed and the fail-closed-on-ambiguity behaviour is plain in the code, but I never caught an orphan actually colliding with a new bottle. This is a reachable hazard, not a witnessed failure. It is worth fixing on its own terms — an active row for a container that no longer exists is wrong regardless of whether it has bitten yet — but it should not be credited with any outage so far.

Fix

PR #440:

RegistryStore.reap_absent / Orchestrator.reconcile + POST /reconcile, called from the macOS launch path before registering, so each launch self-heals.
Restores the invariant: at most one active row per live address, and none for a dead one. Rows younger than a grace window are exempt, so reconciliation can't race an in-flight launch.
Separately, the block message now names the real fault (unattributed / orchestrator unreachable / unparseable policy) instead of blaming the allowlist. That message is why this issue and #441 were conflated for as long as they were: a bottle with no policy and a bottle whose policy lacks the host produced identical text.

Note on deployment

The control-plane source is bind-mounted with a BOT_BOTTLE_SOURCE_HASH staleness check, so the first launch after this lands recreates the shared infra container — which strands any running bottle, see #443.

Resolved from the original writeup

The "first 403 exactly 30.0 minutes in" mystery is explained: the Codex session never made a successful call at all. Its first network activity was a background list_models refresh half an hour in, and that was denied like everything else. The cause was #441 throughout.

> **Corrected.** This issue originally opened with a Codex bottle being denied all egress and presented orphaned rows as the cause. That was wrong — those 403s were #441 (duplicate `*_PROXY` env dropping the identity token); the Codex bottle's registry row was correct every time. The defect described below is real and was found along the way, but it is **not** what produced that symptom. Rewritten to say only what was actually verified. ## The defect Registry rows outlive their containers, and nothing ever cleans them up. A row only leaves the registry two ways: an explicit `teardown_bottle` (the launcher's cleanup callback), or the same-IP supersede sweep in `register()`. Neither runs if the launching CLI dies hard — SIGKILL, a closed terminal, a host sleep. There is **no reconciliation of registry rows against live containers** anywhere in the codebase, so a row left behind stays `active` forever. **Verified on the host:** `bot-bottle-claude-agent-1` was registered `active` at `192.168.128.6` with no container running (not even stopped) and no `/git` namespace. A pure orphan. ## Why it matters `RegistryStore.by_source_ip` fail-closes when `len(rows) != 1` — zero rows *and* ambiguous rows both resolve to "no policy", and a bottle with no policy is denied every host. Source IPs are recycled by the backend's DHCP, so an orphan sitting on an address is a live collision hazard for whatever bottle lands there next. **Being precise about status:** the accumulation is observed and the fail-closed-on-ambiguity behaviour is plain in the code, but I never caught an orphan actually colliding with a new bottle. This is a reachable hazard, not a witnessed failure. It is worth fixing on its own terms — an `active` row for a container that no longer exists is wrong regardless of whether it has bitten yet — but it should not be credited with any outage so far. ## Fix PR #440: - `RegistryStore.reap_absent` / `Orchestrator.reconcile` + `POST /reconcile`, called from the macOS launch path before registering, so each launch self-heals. - Restores the invariant: at most one active row per live address, and none for a dead one. Rows younger than a grace window are exempt, so reconciliation can't race an in-flight launch. - Separately, the block message now names the real fault (unattributed / orchestrator unreachable / unparseable policy) instead of blaming the allowlist. That message is why this issue and #441 were conflated for as long as they were: a bottle with no policy and a bottle whose policy lacks the host produced *identical* text. ## Note on deployment The control-plane source is bind-mounted with a `BOT_BOTTLE_SOURCE_HASH` staleness check, so the first launch after this lands recreates the shared infra container — which strands any running bottle, see #443. ## Resolved from the original writeup The "first 403 exactly 30.0 minutes in" mystery is explained: the Codex session never made a successful call at all. Its first *network* activity was a background `list_models` refresh half an hour in, and that was denied like everything else. The cause was #441 throughout. ## Related - #441 / #442 — duplicate `*_PROXY` env drops the identity token. The actual cause of the Codex outage. - #443 — restarting the shared gateway strands running bottles.

gitea-actions bot added the Status/Needs Triage label 2026-07-20 22:30:26 -04:00

didericis referenced a pull request that will close this issue

2026-07-20 22:30:39 -04:00

fix: reconcile orphaned registry rows against live containers #440

didericis referenced this issue

2026-07-20 22:41:06 -04:00

Codex bottles get zero egress: duplicate *_PROXY env drops the identity token #441

didericis referenced this issue

2026-07-20 22:44:53 -04:00

Restarting the shared gateway permanently strands every running bottle #443

didericis changed title from ~~Orphaned registry rows brick the next bottle at a recycled source IP~~ to Orphaned registry rows are never reconciled against live containers

2026-07-20 23:03:43 -04:00

didericis added Kind/Enhancement

Priority

Medium

3

and removed Status/Needs Triage labels 2026-07-20 23:05:50 -04:00

didericis closed this issue

2026-07-21 00:05:04 -04:00

Sign in to join this conversation.