fix(orchestrator): recreate the orchestrator container on ensure_running

The orchestrator runs the repo's code bind-mounted, but the Python process
loads it at startup and never reloads — and ensure_running reused a
healthy-but-stale container. So after a code change (e.g. the token fix), the
running orchestrator kept executing OLD control-plane code that dropped the
new /bottles 'tokens' field, and egress auth injection stayed broken no matter
how many times the gateway image was rebuilt.

ensure_running now always recreates the orchestrator container (cheap; the
registry DB persists and the current launch re-registers its in-memory
state). Combined with the gateway's image-staleness recreate, a fresh 'start'
now runs current code end to end.

Validated live: register an authed route + in-memory token -> a client at the
bottle IP curling through the gateway proxy receives the injected
'Authorization: Bearer <token>' header.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WBMWTEtQdJ4W5UrWuLHCck
This commit is contained in:
2026-07-14 02:16:06 -04:00
parent 3318bdce28
commit 13a8bdd7ca
2 changed files with 19 additions and 10 deletions
+9 -5
View File
@@ -127,13 +127,17 @@ class OrchestratorService:
gateway are left untouched. Raises `OrchestratorStartError` on
timeout."""
gateway = self._gateway()
gateway.ensure_built() # build the bundle image if missing (both use it)
if self.is_healthy():
gateway.ensure_running() # make sure the gateway is up too
return self.url
gateway.ensure_built() # rebuild the bundle image on a source change
gateway.ensure_running() # creates the shared network + (re)starts gateway
# Always (re)create the orchestrator container. It runs the repo's code
# bind-mounted, but the Python process loaded that code at startup and
# won't reload — so reusing a healthy-but-stale container would keep
# running OLD control-plane code (e.g. dropping the tokens field). Cheap
# (~seconds); the registry DB persists and the current launch
# re-registers its own in-memory state. (The dedicated orchestrator
# image follow-up replaces this with image-staleness detection.)
log.info("starting orchestrator container", context={"name": ORCHESTRATOR_NAME})
gateway.ensure_running() # creates the shared network + gateway
self._run_orchestrator_container()
deadline = time.monotonic() + startup_timeout