Starting a bottle recreates the orchestrator, dropping already-running bottles' egress tokens (403s) #381

New Issue

2026-07-14T04:04:59-04:00

didericis-claude commented

2026-07-14 04:04:59 -04:00

Repro

./cli.py start bottle A. Confirm it can make an authed egress call (e.g. its LLM provider) successfully.
While A is still running, ./cli.py start bottle B (a second, parallel agent).
Bottle A's next authed egress request 403s: egress: route for '<host>' declared auth but env var 'EGRESS_TOKEN_0' is unset.

A now needs a full restart to work again — it lost its injected credential permanently, even though nothing about A changed.

Diagnosis

OrchestratorService.ensure_running() (bot_bottle/orchestrator/lifecycle.py) unconditionally recreates the fixed-name bot-bottle-orchestrator container on every call — added in a prior fix ("recreate the orchestrator container on ensure_running") so bind-mounted code changes take effect, since the orchestrator's Python process never reloads its bind-mounted source.

launch_consolidated() calls service.ensure_running() on every single bottle launch. So starting bottle B tears down and restarts the orchestrator container that was already serving bottle A.

The orchestrator's per-bottle egress auth tokens live only in an in-process dict (Orchestrator._tokens in bot_bottle/orchestrator/service.py), keyed by bottle_id, deliberately never persisted to disk (secrets shouldn't hit disk, per the token-injection design). Recreating the container starts a fresh Python process with an empty _tokens dict — wiping bottle A's token even though only bottle B is launching. The registry DB (bottle metadata, policy, identity tokens) does persist across the restart, so A still resolves and looks "active" — it just has no injected credential anymore, so its next authed route 403s.

Root cause: ensure_running conflates "is the orchestrator container running" with "is it safe to recreate" — it recreates even when nothing about the source changed, unconditionally sacrificing every other active bottle's in-memory state.

Fix

Only recreate the orchestrator container when its bind-mounted source has actually changed (content hash of the bot_bottle package, stored as a container label and compared on each ensure_running, mirroring the existing image-staleness check already used for the gateway container). A bottle launch that isn't accompanied by a code change now leaves a healthy orchestrator — and its other bottles' in-memory tokens — alone.

## Repro 1. `./cli.py start` bottle A. Confirm it can make an authed egress call (e.g. its LLM provider) successfully. 2. While A is still running, `./cli.py start` bottle B (a second, parallel agent). 3. Bottle A's next authed egress request 403s: `egress: route for '<host>' declared auth but env var 'EGRESS_TOKEN_0' is unset`. A now needs a full restart to work again — it lost its injected credential permanently, even though nothing about A changed. ## Diagnosis `OrchestratorService.ensure_running()` (`bot_bottle/orchestrator/lifecycle.py`) unconditionally recreates the fixed-name `bot-bottle-orchestrator` container on **every** call — added in a prior fix ("recreate the orchestrator container on ensure_running") so bind-mounted code changes take effect, since the orchestrator's Python process never reloads its bind-mounted source. `launch_consolidated()` calls `service.ensure_running()` on every single bottle launch. So starting bottle B tears down and restarts the orchestrator container that was already serving bottle A. The orchestrator's per-bottle egress auth tokens live only in an in-process dict (`Orchestrator._tokens` in `bot_bottle/orchestrator/service.py`), keyed by `bottle_id`, deliberately never persisted to disk (secrets shouldn't hit disk, per the token-injection design). Recreating the container starts a fresh Python process with an empty `_tokens` dict — wiping bottle A's token even though only bottle B is launching. The registry DB (bottle metadata, policy, identity tokens) *does* persist across the restart, so A still resolves and looks "active" — it just has no injected credential anymore, so its next authed route 403s. Root cause: `ensure_running` conflates "is the orchestrator container running" with "is it safe to recreate" — it recreates even when nothing about the source changed, unconditionally sacrificing every other active bottle's in-memory state. ## Fix Only recreate the orchestrator container when its bind-mounted source has actually changed (content hash of the `bot_bottle` package, stored as a container label and compared on each `ensure_running`, mirroring the existing image-staleness check already used for the gateway container). A bottle launch that isn't accompanied by a code change now leaves a healthy orchestrator — and its other bottles' in-memory tokens — alone.

didericis referenced this issue from a commit

2026-07-14 04:07:30 -04:00

fix(orchestrator): only recreate the orchestrator on a real source change

didericis-claude referenced a pull request that will close this issue

2026-07-14 04:07:48 -04:00

Only recreate the orchestrator container on a real source change #382

didericis closed this issue

2026-07-14 04:31:28 -04:00

didericis referenced this issue from a commit

2026-07-14 16:24:24 -04:00

refactor(orchestrator): split conflated sidecar image into orchestrator + gateway images

didericis-claude referenced this issue