fix(orchestrator): supersede a prior active bottle at the same source IP
test / unit (pull_request) Successful in 1m4s
test / integration (pull_request) Failing after 30s
test / coverage (pull_request) Failing after 1m17s

register() used INSERT OR REPLACE keyed on bottle_id, so re-registering an
existing source_ip with a new bottle_id left TWO active rows for that IP.
by_source_ip fail-closes on ambiguity (>1 active -> None), so the reused IP
was then bricked to a 403 on egress resolve — even for hosts its own policy
allowed. Happy path (fresh IPs, teardown on stop) never hit it, but IP reuse,
crash recovery, or a persisted registry DB across an orchestrator restart
(now that ensure_running always recreates the container) all could.

register() now deletes any other active row at the same source_ip before
insert; the fail-closed ambiguity guard stays as defense in depth. Adds a
unit test for the supersede, and a docker integration test
(test_multitenant_isolation) that drives two bottles through one shared
gateway and asserts each gets only its own injected token and its own
allowlist — the core PRD 0070 multi-tenancy invariant, previously only
verified by hand.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WBMWTEtQdJ4W5UrWuLHCck
This commit is contained in:
2026-07-14 02:35:13 -04:00
parent 13a8bdd7ca
commit ea75fab3b4
3 changed files with 204 additions and 4 deletions
+14 -1
View File
@@ -149,7 +149,15 @@ class RegistryStore(DbStore):
policy: str = "",
) -> BottleRecord:
"""Register (or replace) a bottle. Mints a bottle_id / identity token
when not supplied. Live — this is the control-plane reload path."""
when not supplied. Live — this is the control-plane reload path.
Supersedes any *other* active bottle at the same source IP first:
`by_source_ip` fail-closes on ambiguity (>1 active row → None), so a
leftover row at a reused IP — from an untorn-down bottle, crash
recovery, or a persisted DB — would otherwise *brick* the new bottle's
attribution. The new registration is authoritative for its IP; the
stale rows go. INSERT OR REPLACE handles a same-`bottle_id` reload in
place (its own row is exempt from the sweep)."""
rec = BottleRecord(
bottle_id=bottle_id or secrets.token_hex(8),
source_ip=source_ip,
@@ -160,6 +168,11 @@ class RegistryStore(DbStore):
policy=policy,
)
with self._connect() as conn:
conn.execute(
"DELETE FROM orchestrator_bottles "
"WHERE source_ip = ? AND state = 'active' AND bottle_id != ?",
(rec.source_ip, rec.bottle_id),
)
conn.execute(
"INSERT OR REPLACE INTO orchestrator_bottles "
"(bottle_id, source_ip, identity_token, state, created_at, metadata, policy) "