feat(orchestrator): per-host consolidated orchestrator + multi-tenant gateway (PRD 0070) #380
No Assignees
Notifications
Due Date
No due date set.
Dependencies
No dependencies set.
Reference: didericis/bot-bottle#380
Reference in New Issue
Block a user
Delete Branch "orchestrator-cutover"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
The full PRD 0070 stack, collapsed into one PR (38 commits, slices 1–13d). Replaces the per-bottle sidecar bundle with a single per-host orchestrator + one shared multi-tenant gateway, and cuts the docker backend over to it.
test_sandbox_escapeis green end-to-end on real docker.What it does
/resolve, HTTP control plane, register-only stub broker (no docker socket).Slice map (commit history preserved)
DockerBottleBackend.launchover, dropping per-bottle networks/CA/bundle.Validated on real docker
test_sandbox_escapepasses all 5 attacks through the shared gateway (hostname/IP not-in-allowlist, HTTP exfil, DNS exfil, git-gate secret-push). Real-on-host testing found + fixed bugs unit mocks couldn't (firewall/containerization, IP-collision with the orchestrator container, missing gateway hooks, stale image). pyright 0, pylint 9.83, unit suite green.Not in this PR (tracked)
bot_bottle/orchestrator/).sidecars→gatewayimage rename, iroh transport, pull-model launcher → #322.bot-bottle-sidecarsimage must be rebuilt when its flat sources change (ensure_builtonly builds-if-missing) — a content-hash/--rebuildpath is a follow-up.Collapses the stacked review PRs #352, #356–#365, #367–#379.
🤖 Generated with Claude Code
273f51223dtodcf81ee2ceFirst implementation slice of PRD 0070, the backend-neutral consolidation core as a plain-process dev-harness (no VM packaging yet): * orchestrator/registry.py — SQLite (WAL) runtime-state store on the existing DbStore/TableMigrations base. Live bottle registry keyed by source IP + per-bottle identity token, with fail-closed attribution: a request resolves to a bottle only when its source IP AND identity token both match exactly one active record (unknown/ambiguous IP, empty token, or token mismatch all deny). Tokens are 256-bit urandom. * orchestrator/control_plane.py — the HTTP control plane (the universal transport chosen in 0070): register / deregister / list / attribute / health. Routing is a pure dispatch() so it is socket-free testable; Handler/ControlPlaneServer/make_server are a thin stdlib adapter. register/deregister are the live-reload path; listing redacts tokens. * orchestrator/__main__.py — `python -m bot_bottle.orchestrator` harness. Launch/teardown, the launch broker, and the egress/git/supervise data plane come in later slices. 24 unit tests (attribution matrix, persistence, dispatch, one real-socket round-trip). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WBMWTEtQdJ4W5UrWuLHCckSecond slice of PRD 0070, still a backend-neutral dev-harness: * orchestrator/broker.py — the launch-broker contract. A LaunchRequest is structured (static ids/flags only — bottle id, pool slot, a content-addressed image_ref; never a path/argv) and signed as a compact HS256 JWT so the broker verifies PROVENANCE before acting: a compromised co-located component can't forge a launch without the shared secret. verify_request is fail-closed (bad sig / malformed / off-schema -> raise). Stdlib only (no runtime deps). Ships a StubBroker that records verified requests for the harness/tests. * orchestrator/service.py — the Orchestrator: owns the registry and brokers the lifecycle. launch_bottle mints the bottle + sends a signed launch, rolling the registry entry back if the launch fails (no orphans); teardown_bottle brokers teardown then deregisters; attribute delegates. * control_plane.py — POST /bottles now launches, DELETE tears down (both go through the Orchestrator + broker). dispatch/server take an Orchestrator. * __main__.py wires an ephemeral secret + StubBroker for the harness. Tests: broker sign/verify round-trip, tamper/wrong-secret/malformed/off-schema rejection, StubBroker fail-closed; Orchestrator launch->registry->attribute, teardown, rollback-on-broker-failure; control-plane updated for launch/teardown. Full suite green (only the pre-existing /bin/sleep errors); harness does launch -> attribute -> teardown over HTTP. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WBMWTEtQdJ4W5UrWuLHCckThe first concrete LaunchBroker, proving the orchestrator -> backend seam on the cheapest backend (the sidecar bundle is already containers): * orchestrator/docker_broker.py — DockerBroker runs a container on a verified launch (`docker run --detach --name <bottle> --label ... <image_ref>`) and removes it on teardown (`docker rm --force`, idempotent on an already-absent container). The argv is built only from the request's static ids/flags, so nothing free-form reaches docker; provenance/schema verification is inherited from LaunchBroker.submit. * __main__.py gains `--broker {stub,docker}` so the harness can drive real containers. Slice 3 launches a single container from image_ref (the seam); the full agent + sidecar bundle is a later slice. Tests: unit (docker mocked) — argv from static fields, launch/teardown call the right commands, missing-image and docker-failure raise, teardown idempotent on missing, forged token never touches docker; integration (gated on a reachable daemon) — launch creates a real container, teardown removes it. Full suite green (only pre-existing /bin/sleep errors); integration verified locally against real docker. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WBMWTEtQdJ4W5UrWuLHCckThe core consolidation win: one persistent sidecar per host, shared by every bottle, instead of a sidecar bundle per bottle. Safe to share because the attribution invariant (source IP + identity token) lets the sidecar map each request to the right bottle. * orchestrator/sidecar.py — a backend-neutral `Sidecar` lifecycle contract (mirrors LaunchBroker) + a `DockerSidecar` impl. The defining behaviour is idempotent singleton: `ensure_running` starts the instance if absent and is a no-op if it's already up, so N launches never spawn N sidecars; `stop` is idempotent. * orchestrator/dockerutil.py — a shared `run_docker` helper; DockerBroker now uses it too (DRY with slice 3). * service.py — the Orchestrator holds an optional `Sidecar`, exposes `ensure_sidecar()` + `sidecar_status()`. * control_plane.py — `GET /sidecar` reports it; __main__ gains `--sidecar-image` and ensures the single sidecar on startup. Tests: unit (docker mocked) — is_running, ensure idempotent (no-op when up, starts when absent), failure raises, stop idempotent; Orchestrator sidecar wiring/status; control-plane /sidecar; integration (gated) — ensure is a real idempotent singleton (one container after two ensures), stop removes. Full suite green (only pre-existing /bin/sleep errors); integration verified locally against real docker. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WBMWTEtQdJ4W5UrWuLHCckAnswers "where do we build the consolidated sidecar": nowhere, until now. * sidecar.py — `Sidecar.ensure_built()` (default no-op) + `DockerSidecar` now defaults its image to the real bundle (`bot-bottle-sidecars`) and `ensure_built()` builds it from `Dockerfile.sidecars` when `docker image inspect` shows it's missing (no-op when present or when no dockerfile is configured, e.g. a pre-pulled image). `image_exists()` added. * service.py — `ensure_sidecar()` now builds then runs. * __main__.py — `--sidecar` runs the consolidated bundle (build-if-missing). Scope note: this builds + launches the bundle *container*; making the running instance functional across bottles needs the per-bottle, source-IP-keyed multi-tenant config + registration/reload, and routing agent bottles to it — the next slices (added to PRD 0070's roadmap). Tests: unit (docker mocked) — image_exists, ensure_built builds when missing / no-op when present / no-op without a dockerfile / raises on build failure; ensure_sidecar builds-then-runs; integration (gated, no heavy build) — image_exists reflects real docker state. Full suite green (only pre-existing /bin/sleep errors). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WBMWTEtQdJ4W5UrWuLHCckThe orchestrator side of the multi-tenant consolidated sidecar: hold each bottle's sidecar policy and serve it by verified source IP, with live reload. One shared sidecar can now get per-bottle config keyed on who's calling. * registry.py — a `policy` column (migration v3, opaque JSON the sidecar interprets) on BottleRecord; `register(..., policy=)` stores it, `set_policy(bottle_id, policy)` updates it live, and `attribute` returns it (the source-IP-keyed resolution). * service.py — `launch_bottle(..., policy=)` and `set_policy`. * control_plane.py — `POST /bottles` accepts `policy`; `PUT /bottles/<id>/policy` live-reloads it; `POST /resolve` returns {bottle_id, policy} for a verified (source_ip, token) — the per-request call the multi-tenant sidecar makes; `/attribute` stays identity-only. Scope note: this is the control-plane / state half. The data-plane half — the egress mitmproxy addon (and git-gate) selecting allowlist / DLP / token-injection per client IP by calling `/resolve` — is the next slice (route agent bottles through the shared sidecar). The orchestrator stays policy-agnostic: it stores and serves the blob verbatim. Tests: registry policy store/update/persist; Orchestrator launch-with-policy + live set_policy; control-plane resolve returns policy (403 on bad token), PUT policy updates / 404 / 400. Verified live over HTTP (launch -> resolve -> PUT reload -> resolve reflects). Full suite green. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WBMWTEtQdJ4W5UrWuLHCckThe egress addon now selects each request's Config by the calling bottle's source IP, so one shared sidecar serves every bottle. Opt-in and fail-closed; single-tenant behaviour is unchanged. Orchestrator side (source-IP-primary attribution, per the PRD invariant): * registry: `by_source_ip` (the single active bottle at a source IP — network-layer attribution); `attribute` now composes it + the token. * service: `resolve(source_ip, token="")` — with a token, strict attribution; without, source IP alone. * control_plane: `POST /resolve`'s identity_token is now OPTIONAL (absent → source-IP-only); split cleanly from the token-required `/attribute`. * policy_resolver: `resolve` token now optional. Egress side: * egress_addon_core: `resolve_client_config(resolver, client_ip, token)` — fetches + parses the client's Config, **fail-closed**: unattributed, a resolver error, or an unparseable policy all yield deny-all (no routes). Host-testable; `PolicyResolverLike` Protocol keeps it import-free. * egress_addon: consolidated mode when `BOT_BOTTLE_ORCHESTRATOR_URL` is set → `_active_config(flow)` resolves per client IP (reads + strips the `x-bot-bottle-identity` header); `request()` uses it. Unset → the static routes file, exactly as before. `PolicyResolver` added to the bundle. Security note: source-IP-only resolution is safe where the IP is unspoofable (Firecracker /31 + nft) AND the control plane is reachable only by the trusted sidecar; the identity token, when the agent injects it, strengthens it on weaker backends. Scope note: the egress data plane is now multi-tenant. Remaining to be fully live: the network topology routing every bottle's proxy to the one shared sidecar, git-gate multitenancy, and agent-side identity-token injection. Tests: registry by_source_ip; orchestrator resolve (with/without token); control-plane /resolve token-optional; resolver token-optional; resolve_client_config fail-closed matrix. All 182 egress tests still pass (single-tenant unchanged). Full suite green. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WBMWTEtQdJ4W5UrWuLHCckComposes the orchestrator primitives into the register/teardown sequence that replaces the per-bottle bundle: launch_consolidated(egress_plan, git_gate_plan): ensure orchestrator + gateway up -> read the gateway network's subnet + the gateway's own address -> allocate the bottle a pinned source IP (skip the gateway + live bottles) -> register (egress policy blob + slug metadata) -> provision its git-gate repos/creds into the gateway. Returns a LaunchContext (bottle_id, identity_token, source_ip, network, gateway_ip, orchestrator_url) — everything the agent container needs to attach. Rolls the registration back if provisioning fails, so no orphan. teardown_consolidated(bottle_id): deregister + deprovision (idempotent). The agent `docker run` itself stays the backend's job (provider provisioning); this owns the orchestrator-facing wiring so the sequence is unit-testable — collaborators mocked, next_free_ip + registration_inputs run for real (IP allocation + policy blob asserted end to end). pyright 0 errors; pylint 9.83/10; unit suite green (1755 tests; the 13 test_sidecar_init /bin/sleep errors are pre-existing NixOS-local noise). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WBMWTEtQdJ4W5UrWuLHCckThe per-bottle model rendered a compose project with the agent + a sidecar bundle on two per-bottle networks. Consolidated has no sidecars — one shared gateway serves everyone — so this renders JUST the agent, attached to the external shared gateway network at the orchestrator-allocated pinned IP, and pointed at the gateway's address for egress. - consolidated_agent_compose(plan, *, gateway_ip, source_ip, network): agent service only; networks {<gateway>: {ipv4_address: source_ip}} on the external gateway network; HTTPS_PROXY -> http://<gateway_ip>:9099; NO_PROXY includes the gateway address so git-http + supervise (also on the gateway) bypass the proxy and are reached directly. No depends_on/sidecars. - Pure (takes the launch-time LaunchContext values), so it's unit-testable without docker. pyright 0 errors; pylint 9.83/10; unit suite green (the 13 test_sidecar_init /bin/sleep errors are pre-existing NixOS-local noise). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WBMWTEtQdJ4W5UrWuLHCckRewire DockerBottleBackend.launch to the consolidated model, replacing the per-bottle sidecar bundle. VALIDATED END-TO-END on real docker: test_sandbox_escape passes all 5 attacks (egress DLP + git-gate gitleaks) through the shared gateway. launch now: - mints git-gate dynamic keys (if any), then launch_consolidated() to register the bottle + provision its git-gate state into the gateway and get the agent's attach context (pinned source IP, gateway address); - installs the SHARED gateway CA (gateway.ca_cert_pem) into the agent; - renders the agent-only consolidated compose on the gateway network at the pinned IP, proxied through the gateway; - points the agent's git-gate insteadOf (http://<gw>:9420) and supervise MCP (http://<gw>:9100) at the gateway (DockerBottlePlan.agent_git_gate_url / agent_supervise_url); - teardown = compose down + teardown_consolidated (dereg + deprovision). Dropped the per-bottle networks, per-bottle egress CA, and bundle service. Fixes surfaced by the real e2e (couldn't be caught by unit mocks): - provision_git_gate installs the bottle-agnostic pre-receive/access hooks into the gateway (were cp'd per-bundle before); - source-IP allocation reads the *actual* container IPs on the network (gateway + orchestrator + agents), not just gateway + registry — the orchestrator container sits on the network and was colliding. Unit tests for the old bundle launch rewritten against the new collaborators. NOTE for reviewers: the gateway/bundle image (bot-bottle-sidecars) must be rebuilt when its flat sources change — `ensure_built` only builds-if-missing, so a stale image silently runs the OLD single-tenant daemons. A content-hash / --rebuild path is a follow-up. pyright 0 errors; pylint 9.83/10; unit suite green (1760 tests; the 13 test_sidecar_init /bin/sleep errors are pre-existing NixOS-local noise); test_sandbox_escape green on real docker. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WBMWTEtQdJ4W5UrWuLHCck2583ef56fctoea75fab3b4The test brings up the orchestrator container, which bind-mounts the repo path into a container on the host daemon. Under the Gitea act_runner the job runs in a container sharing the host docker socket, so that path (/workspace/...) doesn't exist on the host daemon and the mount fails ('mkdir /workspace: read-only file system'). Same host-bind-mount constraint that already skips test_sandbox_escape and the other bottle-bringup tests, so gate it the same way (GITEA_ACTIONS). Runs for real on a normal docker host. This unblocks the integration + coverage CI jobs, which failed only because this test errored (coverage.sh runs under set -e; no percentage gate tripped). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WBMWTEtQdJ4W5UrWuLHCck