Only recreate the orchestrator container on a real source change #382

2026-07-14T04:07:48-04:00

didericis-claude commented

2026-07-14 04:07:48 -04:00

Closes #381.

Summary

OrchestratorService.ensure_running() (910267b8a8/bot_bottle/orchestrator/lifecycle.py) unconditionally recreated the bot-bottle-orchestrator container on every bottle launch. That was added deliberately so bind-mounted code changes take effect (the control-plane process never reloads its source) — but the orchestrator's per-bottle egress auth tokens live only in that process's memory (Orchestrator._tokens in bot_bottle/orchestrator/service.py), never persisted to disk by design. So starting a second bottle while a first was already running recreated the container out from under it, silently wiping the first bottle's already-injected token. Its next authed egress request then 403s with env var EGRESS_TOKEN_0 is unset, and it needs a full restart to recover — reproduced live in the session that filed #381.

The fix mirrors the gateway's existing image-staleness check (DockerGateway._running_image_is_current): the orchestrator container is now labeled with a content hash of its bind-mounted bot_bottle source, and ensure_running only recreates it when that hash no longer matches the running container's label — i.e. on an actual code change, not on every launch. A bottle launch that isn't accompanied by a code change now leaves a healthy orchestrator, and every other active bottle's in-memory tokens, alone.

Testing

python3 -m unittest discover -s tests/unit -p "test_orchestrator*.py" — 113 tests pass, including two new/rewritten cases in test_orchestrator_lifecycle.py covering the noop-when-unchanged and recreate-when-changed paths.
npx pyright bot_bottle/orchestrator/lifecycle.py — 0 errors.

Closes #381. ## Summary `OrchestratorService.ensure_running()` (https://gitea.dideric.is/didericis/bot-bottle/src/commit/910267b8a8cbdf433ffaa344a6528c7c1e8d3050/bot_bottle/orchestrator/lifecycle.py) unconditionally recreated the `bot-bottle-orchestrator` container on **every** bottle launch. That was added deliberately so bind-mounted code changes take effect (the control-plane process never reloads its source) — but the orchestrator's per-bottle egress auth tokens live only in that process's memory (`Orchestrator._tokens` in `bot_bottle/orchestrator/service.py`), never persisted to disk by design. So starting a *second* bottle while a *first* was already running recreated the container out from under it, silently wiping the first bottle's already-injected token. Its next authed egress request then 403s with `env var EGRESS_TOKEN_0 is unset`, and it needs a full restart to recover — reproduced live in the session that filed #381. The fix mirrors the gateway's existing image-staleness check (`DockerGateway._running_image_is_current`): the orchestrator container is now labeled with a content hash of its bind-mounted `bot_bottle` source, and `ensure_running` only recreates it when that hash no longer matches the running container's label — i.e. on an actual code change, not on every launch. A bottle launch that isn't accompanied by a code change now leaves a healthy orchestrator, and every other active bottle's in-memory tokens, alone. ## Testing - `python3 -m unittest discover -s tests/unit -p "test_orchestrator*.py"` — 113 tests pass, including two new/rewritten cases in `test_orchestrator_lifecycle.py` covering the noop-when-unchanged and recreate-when-changed paths. - `npx pyright bot_bottle/orchestrator/lifecycle.py` — 0 errors.

didericis-claude added 1 commit 2026-07-14 04:07:49 -04:00

fix(orchestrator): only recreate the orchestrator on a real source change

test / unit (pull_request) Successful in 1m7s

Details

test / integration (pull_request) Successful in 20s

Details

test / coverage (pull_request) Successful in 1m19s

Details

lint / lint (push) Successful in 2m7s

Details

test / unit (push) Successful in 1m11s

Details

test / integration (push) Successful in 28s

Details

test / coverage (push) Successful in 1m18s

Details

Update Quality Badges / update-badges (push) Successful in 1m9s

Details

910267b8a8

ensure_running() unconditionally recreated the bot-bottle-orchestrator
container on every bottle launch, added so bind-mounted code changes take
effect (the process never reloads). But the orchestrator's per-bottle
egress auth tokens live only in its process memory (Orchestrator._tokens),
by design never persisted to disk. So starting a second bottle recreated
the container and silently dropped the FIRST bottle's already-injected
token -- its next authed egress call 403s with "env var EGRESS_TOKEN_0 is
unset", and it needs a full restart to recover.

Now the orchestrator container is labeled with a content hash of its
bind-mounted bot_bottle source, mirroring the gateway's existing
image-staleness check. ensure_running only recreates on a hash mismatch
(a real code change), so a bottle launch that isn't accompanied by a code
change leaves a healthy orchestrator -- and every other active bottle's
in-memory tokens -- alone.

Fixes #381.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

didericis commented

2026-07-14 04:21:45 -04:00

@didericis-claude

This is a bad solution... we need a way to reprovision the secrets when an orchestrator gets rebuilt. Really should have that separate vault... but suppose this is ok for now.

@didericis-claude This is a bad solution... we need a way to reprovision the secrets when an orchestrator gets rebuilt. Really should have that separate vault... but suppose this is ok for now.

didericis-claude commented

2026-07-14 04:23:14 -04:00

Agreed — this PR is intentionally scoped to the specific bug (unnecessary recreation on every launch), not the broader problem of tokens surviving a real rebuild. A vault or out-of-process token store is the right fix for that, and worth a follow-up issue so it does not get lost.

didericis merged commit 910267b8a8 into main

2026-07-14 04:31:28 -04:00

didericis deleted branch fix-orchestrator-recreate-drops-tokens

2026-07-14 04:31:28 -04:00

Sign in to join this conversation.

2 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: didericis/bot-bottle#382