ensure_running() unconditionally recreated the bot-bottle-orchestrator
container on every bottle launch, added so bind-mounted code changes take
effect (the process never reloads). But the orchestrator's per-bottle
egress auth tokens live only in its process memory (Orchestrator._tokens),
by design never persisted to disk. So starting a second bottle recreated
the container and silently dropped the FIRST bottle's already-injected
token -- its next authed egress call 403s with "env var EGRESS_TOKEN_0 is
unset", and it needs a full restart to recover.
Now the orchestrator container is labeled with a content hash of its
bind-mounted bot_bottle source, mirroring the gateway's existing
image-staleness check. ensure_running only recreates on a hash mismatch
(a real code change), so a bottle launch that isn't accompanied by a code
change leaves a healthy orchestrator -- and every other active bottle's
in-memory tokens -- alone.
Fixes#381.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
The orchestrator runs the repo's code bind-mounted, but the Python process
loads it at startup and never reloads — and ensure_running reused a
healthy-but-stale container. So after a code change (e.g. the token fix), the
running orchestrator kept executing OLD control-plane code that dropped the
new /bottles 'tokens' field, and egress auth injection stayed broken no matter
how many times the gateway image was rebuilt.
ensure_running now always recreates the orchestrator container (cheap; the
registry DB persists and the current launch re-registers its in-memory
state). Combined with the gateway's image-staleness recreate, a fresh 'start'
now runs current code end to end.
Validated live: register an authed route + in-memory token -> a client at the
bottle IP curling through the gateway proxy receives the injected
'Authorization: Bearer <token>' header.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WBMWTEtQdJ4W5UrWuLHCck
Real-on-host testing surfaced two issues the unit-mocked slices couldn't:
1. The host (NixOS) firewall DROPS container->host traffic, so a host-process
orchestrator is unreachable from the gateway container. Fix: run the
orchestrator AS a container on the shared gateway network (PRD 0070's
"virtualize the orchestrator") — the gateway reaches it by container name
over docker DNS (container<->container, no firewall), and the host CLI
reaches it via a published loopback port.
2. The control plane crashed the connection on a dispatch error (e.g. a
broker failure) instead of returning 500.
Changes:
- lifecycle: OrchestratorProcess (host process) -> OrchestratorService
(containers). Runs the control plane in the bundle image with the repo
bind-mounted (orchestrator is stdlib-only), register-only stub broker so it
needs NO docker socket (the backend launches agents; the host manages both
containers). Registry DB persists via a host-root mount. ensure_running is
an idempotent singleton over both containers.
- gateway: BOT_BOTTLE_ORCHESTRATOR_URL is now the orchestrator's *by-name*
URL on the shared network (dropped the host.docker.internal hack).
- control_plane: _serve wraps dispatch — a failure returns 500, never crashes
the connection.
- OrchestratorProcess default broker -> stub (register-only) for docker.
Validated live end-to-end: both containers up, gateway->orchestrator by name
OK, register -> resolve-by-source-IP returns the bottle's policy from inside
the gateway.
pyright 0 errors; pylint 9.83/10; unit suite green (1760 tests; the 13
test_sidecar_init /bin/sleep errors are pre-existing NixOS-local noise).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WBMWTEtQdJ4W5UrWuLHCck
First sub-slice of docker launch integration: the foundation the CLI needs
to ensure exactly one orchestrator control plane + shared gateway is up
before registering/launching bottles. Does NOT touch the real launch path
yet — it's the lifecycle primitive the cut-over slices build on.
- OrchestratorProcess.ensure_running(): idempotent singleton — returns the
control-plane URL if a healthy one already answers /health, else spawns
`python -m bot_bottle.orchestrator --broker docker --gateway` detached
(start_new_session, output tee'd to <root>/orchestrator.log) and polls
/health until healthy or timeout (OrchestratorStartError).
- The control-plane port is the singleton key: a second orchestrator can't
bind it, so a stray double-start fails fast rather than forking a rival.
- Host-process (not container) per the PRD's dev-harness sequencing — it
already has the host user's docker access to broker launches; the
data-plane gateway it manages is the container.
pyright 0 errors; pylint 9.83/10; unit suite green (1714 tests; the 13
test_sidecar_init /bin/sleep errors are pre-existing NixOS-local noise).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WBMWTEtQdJ4W5UrWuLHCck
didericis