- `_build_images()` now builds Dockerfile.gateway → Dockerfile.orchestrator
→ Dockerfile.infra in order; Dockerfile.infra starts FROM bot-bottle-gateway
so the base must exist on clean hosts.
- Publish mapping corrected from `self.port:self.port` to `self.port:DEFAULT_PORT`
(8099) — gateway_init hardcodes the orchestrator on port 8099 inside the
container, so the host-side published port must map to that fixed internal port.
- BOT_BOTTLE_ORCHESTRATOR_URL inside the container now always points to
127.0.0.1:8099, not self.port, since gateway daemons reach the orchestrator
over loopback at the fixed internal port.
- Update test_ensure_running_builds_both_images → _all_images for the new
three-step build sequence; add test_publish_maps_host_port_to_fixed_internal_port
to lock in the port-mapping fix.
Addresses the P1 findings from the didericis-codex review on PR #432.
Add three new tests:
- noop when healthy but docker inspect fails (returns True → don't churn)
- build failure raises GatewayError
- _ensure_network creates the network when it doesn't exist
Also update the integration test to use new OrchestratorService API
(infra_name/image instead of orchestrator_name/gateway_name/gateway_image).
Brings diff-coverage from 86% to 90.3% against origin/main.
- Remove unused INFRA_IMAGE import from test_orchestrator_lifecycle
- Update integration test to use new single-container OrchestratorService
API (infra_name/image replaces orchestrator_name/gateway_name/gateway_image)
- Move type: ignore to the lambda line in gateway_init SIGHUP handler
- Break two long lines in test_orchestrator_lifecycle
Collapses the two-container Docker model (gateway + orchestrator) into one
bot-bottle-infra container, matching the macOS and Firecracker backends.
- Dockerfile.infra: now a shared gateway+orchestrator base (COPY bot_bottle
from orchestrator build, no CMD override)
- Dockerfile.infra.fc: new Firecracker-specific layer (buildah/crun/netavark)
- gateway_init: adds orchestrator daemon with _OPT_IN_DAEMONS gating so it
only starts when BOT_BOTTLE_GATEWAY_DAEMONS explicitly includes it
- orchestrator/lifecycle: OrchestratorService manages one infra container;
builds orchestrator (intermediate) then infra; live source bind-mounted at
/bot-bottle-src with PYTHONPATH so the subprocess uses the checkout
- backend/consolidated_util: extracts provision_bottle + teardown_consolidated
shared across all three backends; removes duplication in docker/fc/macos
consolidated_launch modules
- firecracker/infra_vm: builds four images (orchestrator→gateway→infra→infra.fc)
- All unit tests updated and passing (1878 tests)
- PRD status: Draft → Active
The control plane had no caller authentication — dispatch() routed every
request straight through, and the only guards anywhere were the
(source_ip, identity_token) checks inside /resolve and /attribute, which
authenticate the *bottle* a request is about, not the *caller*. On the
consolidated backends the agent shares a network with the control-plane port
(macOS: the infra container runs 8099 on the agent's --internal network;
docker: the orchestrator container sits on the shared gateway network), so a
prompt-injected agent could GET /bottles (enumerate siblings + policies),
PUT /bottles/<id>/policy (rewrite its own egress allowlist to allow-all),
POST /resolve (read the injected upstream credentials it's never meant to
see), and POST /supervise/respond (self-approve its own proposals). Every
README guarantee collapsed to the agent's discretion.
Fix: a per-host control-plane secret required on every route but GET /health,
compared with hmac.compare_digest. It is held only by the trusted callers and
never handed to an agent:
- minted + persisted 0600 at <root>/control-plane-token (paths.host_control_plane_token);
- injected as $BOT_BOTTLE_CONTROL_PLANE_TOKEN into the orchestrator + gateway
containers via bare `--env NAME` (value inherited from the launch process,
so it never lands on argv or in `container/docker inspect`);
- presented by the gateway's PolicyResolver (reads the env) on /resolve, and by
the host CLI's OrchestratorClient (reads the host file) on every call.
The agent container is never given the env var or the host file, so from a
bottle every /bottles*, /resolve, /attribute, and /supervise/* call now
returns 401 — closing the enumeration, allowlist-rewrite, credential-lift, and
self-approval. The existing (source_ip, identity_token) checks stay as
defense-in-depth.
Enforced when configured: macOS + docker inject the secret (→ enforced). With
no secret set the server runs open and warns loudly at startup — a
fail-visible fallback for the unit suite and for Firecracker, whose
port-scoped nft already blocks agents from 8099 (wiring the secret into its
infra-VM init is a clean fast-follow, left out here to avoid churning the
prebuilt-artifact hash).
Verified end-to-end on real Apple Container: infra comes up healthy, the host
CLI (with the secret) lists bottles while an unauthenticated GET /bottles gets
401, all five issue-#400 attacks from inside the agent get 401, and egress
policy still works (200 allowed / 403 denied) — proving the gateway
authenticates to /resolve with the secret. 1829 unit tests pass, pyright
clean, pylint 9.91.
Refs #400.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Re-enables the macos-container backend on the shared per-host orchestrator +
gateway, replacing the per-bottle companion container removed in #385. This is
the last backend in PRD 0070's roadmap.
Apple Container 1.0.0 forced three departures from the docker shape, each
verified against the live CLI (findings recorded in the networking spike):
- No `--ip`. The address is DHCP-assigned and knowable only once the container
runs, so the order inverts: gateway up -> run agent -> read its address ->
register. The identity token is minted by registration and therefore cannot
be in the agent's run-time env; it rides the proxy URL applied at
`container exec` time (bare `--env` names keep it off argv).
- No container DNS. The gateway can only be handed the control plane's IP, so
the orchestrator starts first and the gateway is pointed at its address.
- No `network connect`. Networks are fixed at run time, so the shared host-only
network is created up front; per-bottle networks would restart the gateway
on every launch and defeat the consolidation.
The agent runs with `--cap-drop CAP_NET_RAW`: Apple grants NET_RAW by default,
which would let an agent forge a neighbour's source address on the shared
segment. NET_ADMIN is already absent, so this closes the source-address half of
PRD 0070's attribution invariant.
Verified end-to-end on real Apple Container 1.0.0: both images build, the
control plane comes up healthy, the gateway reaches it by IP, and a registered
agent gets 200 for a host in its routes and 403 for one outside them. Bring-up
is idempotent — a second launch does not churn the singletons.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
One image — `bot-bottle-sidecars:latest`, built from `Dockerfile.sidecars` —
served two unrelated roles: the egress/git-gate/supervise *data plane* and
the orchestrator *control plane* (which ran the same image with the entrypoint
overridden to `python3 -m bot_bottle.orchestrator`). The control plane is
stdlib-only, so it needed none of the mitmproxy/git/gitleaks payload it was
riding on — while being the most secret-dense process on the host (PRD 0070's
"secret concentration").
Split into two purpose-built images:
- `Dockerfile.gateway` -> `bot-bottle-gateway:latest` — the data plane
(renamed from Dockerfile.sidecars; identical contents).
- `Dockerfile.orchestrator` -> `bot-bottle-orchestrator:latest` — a lean
`python:3.12-slim` runtime; the bind-mounted `bot_bottle` package supplies
the code (so the #381 source-hash recreate semantics are unchanged).
`OrchestratorService` now takes distinct `image` (control plane, default
`ORCHESTRATOR_IMAGE`) and `gateway_image` (data plane, default `GATEWAY_IMAGE`)
instead of feeding one `self.image` to both, and builds the lean image
(build-if-missing) before starting the container. The per-bottle bundle
constants in `backend/docker/sidecar_bundle.py` now alias the gateway
constants so a bundle and the shared gateway can never drift onto different
images. The `bot-bottle-sidecars` *image* name and `Dockerfile.sidecars` are
gone; the per-bottle *container* name prefix (`bot-bottle-sidecars-<slug>`) is
intentionally left for a separate change.
Verified end-to-end: both images build; the lean image runs the control plane;
`ensure_running` brings up the orchestrator on `bot-bottle-orchestrator:latest`
and the gateway on `bot-bottle-gateway:latest` (distinct images) and reports
healthy.
Closes#384.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WBMWTEtQdJ4W5UrWuLHCck
ensure_running() unconditionally recreated the bot-bottle-orchestrator
container on every bottle launch, added so bind-mounted code changes take
effect (the process never reloads). But the orchestrator's per-bottle
egress auth tokens live only in its process memory (Orchestrator._tokens),
by design never persisted to disk. So starting a second bottle recreated
the container and silently dropped the FIRST bottle's already-injected
token -- its next authed egress call 403s with "env var EGRESS_TOKEN_0 is
unset", and it needs a full restart to recover.
Now the orchestrator container is labeled with a content hash of its
bind-mounted bot_bottle source, mirroring the gateway's existing
image-staleness check. ensure_running only recreates on a hash mismatch
(a real code change), so a bottle launch that isn't accompanied by a code
change leaves a healthy orchestrator -- and every other active bottle's
in-memory tokens -- alone.
Fixes#381.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
The orchestrator runs the repo's code bind-mounted, but the Python process
loads it at startup and never reloads — and ensure_running reused a
healthy-but-stale container. So after a code change (e.g. the token fix), the
running orchestrator kept executing OLD control-plane code that dropped the
new /bottles 'tokens' field, and egress auth injection stayed broken no matter
how many times the gateway image was rebuilt.
ensure_running now always recreates the orchestrator container (cheap; the
registry DB persists and the current launch re-registers its in-memory
state). Combined with the gateway's image-staleness recreate, a fresh 'start'
now runs current code end to end.
Validated live: register an authed route + in-memory token -> a client at the
bottle IP curling through the gateway proxy receives the injected
'Authorization: Bearer <token>' header.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WBMWTEtQdJ4W5UrWuLHCck
Real-on-host testing surfaced two issues the unit-mocked slices couldn't:
1. The host (NixOS) firewall DROPS container->host traffic, so a host-process
orchestrator is unreachable from the gateway container. Fix: run the
orchestrator AS a container on the shared gateway network (PRD 0070's
"virtualize the orchestrator") — the gateway reaches it by container name
over docker DNS (container<->container, no firewall), and the host CLI
reaches it via a published loopback port.
2. The control plane crashed the connection on a dispatch error (e.g. a
broker failure) instead of returning 500.
Changes:
- lifecycle: OrchestratorProcess (host process) -> OrchestratorService
(containers). Runs the control plane in the bundle image with the repo
bind-mounted (orchestrator is stdlib-only), register-only stub broker so it
needs NO docker socket (the backend launches agents; the host manages both
containers). Registry DB persists via a host-root mount. ensure_running is
an idempotent singleton over both containers.
- gateway: BOT_BOTTLE_ORCHESTRATOR_URL is now the orchestrator's *by-name*
URL on the shared network (dropped the host.docker.internal hack).
- control_plane: _serve wraps dispatch — a failure returns 500, never crashes
the connection.
- OrchestratorProcess default broker -> stub (register-only) for docker.
Validated live end-to-end: both containers up, gateway->orchestrator by name
OK, register -> resolve-by-source-IP returns the bottle's policy from inside
the gateway.
pyright 0 errors; pylint 9.83/10; unit suite green (1760 tests; the 13
test_sidecar_init /bin/sleep errors are pre-existing NixOS-local noise).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WBMWTEtQdJ4W5UrWuLHCck
First sub-slice of docker launch integration: the foundation the CLI needs
to ensure exactly one orchestrator control plane + shared gateway is up
before registering/launching bottles. Does NOT touch the real launch path
yet — it's the lifecycle primitive the cut-over slices build on.
- OrchestratorProcess.ensure_running(): idempotent singleton — returns the
control-plane URL if a healthy one already answers /health, else spawns
`python -m bot_bottle.orchestrator --broker docker --gateway` detached
(start_new_session, output tee'd to <root>/orchestrator.log) and polls
/health until healthy or timeout (OrchestratorStartError).
- The control-plane port is the singleton key: a second orchestrator can't
bind it, so a stray double-start fails fast rather than forking a rival.
- Host-process (not container) per the PRD's dev-harness sequencing — it
already has the host user's docker access to broker launches; the
data-plane gateway it manages is the container.
pyright 0 errors; pylint 9.83/10; unit suite green (1714 tests; the 13
test_sidecar_init /bin/sleep errors are pre-existing NixOS-local noise).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WBMWTEtQdJ4W5UrWuLHCck
didericis-claude
didericis