`_supervise_token_block` and `_await_token_response` called the synchronous
`PolicyResolver` directly from mitmproxy's async request handling. Those RPCs
funnel through `PolicyResolver._post_json`, which blocks on
`urllib.request.urlopen`. Because the poll loop runs for the entire
operator-approval window, a slow or unreachable orchestrator would repeatedly
freeze the proxy event loop — stalling every other bottle's traffic — despite
`_await_token_response`'s contract of not blocking the loop (#471 review,
review #443).
Dispatch both the propose and poll RPCs via `asyncio.to_thread` so the
blocking urllib call runs in a worker thread and the event loop keeps serving
other flows. Fail-closed semantics are unchanged: `PolicyResolveError` still
propagates through the await and is caught exactly as before.
Regression test records the thread each RPC executes on and asserts it is not
the loop thread; it fails against the pre-fix inline calls and passes with the
dispatch. Full egress-addon + policy-resolver + supervise-server suites green
(155 tests).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The previous fix interpolated the real per-bottle identity token into the
`warn()` message shown when `claude mcp add supervise` fails, so a
registration failure would leak the credential into host terminal output
and any collected launch logs (#476 review, PR #471 comment).
Render a `<bottle-identity-token>` placeholder instead. The recovery hint
still shows the required `--header x-bot-bottle-identity: …` shape, and the
operator substitutes the value from inside the bottle (it rides in the
agent's HTTPS_PROXY credentials) — so the token never reaches the host log.
The token truthiness still gates whether the header is shown at all.
Test updated to assert the header name and placeholder are present and the
raw token is absent.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The manual registration command shown in the fallback warning when
`claude mcp add supervise` fails was missing the `--header
x-bot-bottle-identity: <token>` option. Since (source_ip,
identity_token) attribution is now mandatory on the supervise server,
the missing header caused every subsequent supervise request to fail
closed as unattributed — defeating supervision for the entire bottle.
Reuse the already-constructed `token` variable to build the same
`manual_header` string the automatic path uses; include it in the
printed command. Tests verify both the happy path and the fallback
include (or correctly omit) the header.
Addresses P2 finding in #471 (comment 7, didericis-codex 2026-07-25).
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Remove symbols with no remaining consumer:
- `gateway_name` property on Docker/MacosInfraService — nothing uses it; the
launch flow reads `gateway().name` off the Gateway service. (+ its docker test.)
- firecracker `infra_vm` EGRESS_PORT / SUPERVISE_PORT / GIT_HTTP_PORT — dead
duplicates; launch.py uses supervisor.types / docker.egress / a local const.
- macOS `INFRA_DB_VOLUME` back-compat alias — unused since the DB-volume test
moved to `ORCHESTRATOR_DB_VOLUME`.
- docker `infra.py`: the unused `ORCHESTRATOR_SOURCE_HASH_LABEL` import and the
`__all__` re-exports of the orchestrator constants (consumers import them from
`docker.orchestrator`, not `docker.infra`).
- macOS `infra.py`: the unused `GatewayError` import + re-export.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Add a backend-neutral `InfraService` ABC (backend/infra_service.py) so all three
backends present the same composer contract: `orchestrator()` / `gateway()`
accessors, `ensure_running(*, startup_timeout) -> str` (the host control-plane
URL), `stop()`, plus concrete `url()` / `is_healthy()` that delegate to the
orchestrator. `DockerInfraService` and `MacosInfraService` now subclass it.
Give firecracker the missing class: `FirecrackerInfraService`
(backend/firecracker/infra.py) wraps the `infra_vm` substrate as the pair
coordinator (adopt-or-boot-both under the singleton flock). `infra_vm` keeps only
the plane-agnostic substrate; its `ensure_running` / `_adopt` / `InfraEndpoint`
move to the class, and the coordinator helpers it now calls cross-module go
public (`singleton_lock` / `expected_version` / `adoptable` /
`record_booted_version`).
Unify the return type on the way: `ensure_running` returns the URL string
everywhere (was `str` for docker, two different `InfraEndpoint`s for
macOS/firecracker), and those `InfraEndpoint`s are deleted — the services are the
source of truth (callers read `gateway().address()` / `orchestrator().url()`).
docker's `url` property becomes the inherited `url()`. backend.py /
consolidated_launch / image_builder + tests updated.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Sweep for vestiges of the old combined-plane model and the pre-split shared
rootfs. Two are load-bearing, the rest are stale docs/comments:
- Bug: macOS `enumerate_active` only excluded the gateway container from the
agent list, so after the split the orchestrator container
(`bot-bottle-mac-orchestrator`, also `bot-bottle-`-prefixed) was enumerated as
a phantom agent. Exclude both infra containers; test covers it.
- Dead code: the gateway `bootstrap.py` still carried an `orchestrator` daemon
spec + `_OPT_IN_DAEMONS` + a signing-key/JWT env branch, all for the old
combined container where the gateway process could also run the control plane.
No backend ever requests it now — removed; the key-stripping stays as
defense-in-depth.
Stale-comment reframes: "the/single infra container" -> the orchestrator +
gateway pair (or the specific plane); "shared rootfs / bb_role init / one
published rootfs" -> the per-plane rootfs + `role_init`; the deleted
Dockerfile.infra references in Dockerfile.orchestrator/.gateway; and the macOS
"one infra container ... same address" docstring + its now-false
share-one-address test (the planes are distinct containers with distinct
addresses).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Each infra VM now boots its own rootfs instead of a shared combined image, so
the exposed gateway VM no longer carries buildah + the control-plane code it
never runs (a fatter, less-isolated exposed surface — the opposite of the plane
split's intent). The combined image bought only "one artifact"; each VM already
kept a full copy of the shared rootfs, so nothing was saved at boot.
* orchestrator rootfs — control plane + buildah (Dockerfile.orchestrator.fc,
FROM orchestrator); the slim gateway rootfs boots bot-bottle-gateway:latest
directly. Dockerfile.infra.fc (the combined image) is deleted.
* `_infra_init` splits into `_orchestrator_init` / `_gateway_init` (shared
preamble via `_init_head`); each per-plane rootfs bakes only its role init,
so the `bb_role` cmdline branch is gone.
* infra_artifact is role-parametrized: a per-role package
(bot-bottle-firecracker-<role>), version hash, URL, cache dir, and candidate
subdir. `ensure_built` pulls both; `_expected_version` combines both markers.
* publish_infra builds the images once, then builds + publishes an
orchestrator and a gateway artifact under DIR/<role>/.
coverage.sh's candidate-dir plumbing is layout-agnostic (publish_infra --output
now fills DIR/<role>/, which ensure_artifact_gz reads). Full suite green.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The orchestrator/gateway container split retired the docker backend's combined
`bot-bottle-infra` container, leaving `Dockerfile.infra` used only as the base
layer for the firecracker rootfs image — and its header still (falsely) claimed
"used directly by the Docker backend."
Fold its one `COPY --from orchestrator` into `Dockerfile.infra.fc` (now
`FROM bot-bottle-gateway` directly + the copy + buildah), delete Dockerfile.infra,
and drop the now-dead build step + artifact-version entry. `build_infra_images_with_docker`
builds three images instead of four; the firecracker rootfs is unchanged in
content (still gateway + orchestrator package + buildah, one artifact).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Pull the control plane's host-side logic out of `infra_vm` into
`FirecrackerOrchestrator` (backend/firecracker/orchestrator.py): booting the
orchestrator microVM on its link with the persistent registry volume (/dev/vdb),
seeding the host-canonical signing key over SSH, waiting for /health, and the
buildah SSH target. `url()` == `gateway_url()` (the gateway resolves the
orchestrator by guest IP via bb_orch). This completes the trio — docker, macOS,
firecracker — behind both the Gateway and Orchestrator ABCs.
`infra_vm` stays the pair coordinator + shared substrate: `ensure_running` now
mints nothing itself — it constructs both services (lazy import to break the
cycle), calls `orchestrator.ensure_running()` first, then
`gateway.connect_to_orchestrator(orch.gateway_url(), orch.mint_gateway_token())`.
The plane-agnostic boot primitives graduate to public API (`_boot_vm`/
`_push_secret` -> `boot_vm`/`push_secret`) now that they're consumed only across
modules; `InfraVm` loses its `orchestrator_url` (moved to the service) and
`InfraEndpoint.orchestrator` is now the `FirecrackerOrchestrator` service.
Container-lifecycle tests split into test_firecracker_orchestrator; the infra_vm
tests keep the substrate + pair-coordinator coverage.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Extract the Apple-Container control-plane lifecycle out of `MacosInfraService`
into `MacosOrchestrator` (backend/macos_container/orchestrator.py): the
container run on the host-only control network, the container-only DB volume,
the source-hash recreate gate, health polling against the resolved
control-network address, and `probe_orchestrator_url`. `url()` == `gateway_url()`
here (Apple has no container DNS, so one resolved address serves the CLI and the
gateway).
`MacosInfraService` now composes `orchestrator()` + `gateway()`: ensure the
networks, build both images (each service self-builds via `ensure_built` — added
to `MacosGateway` too), bring the orchestrator up first, then connect the gateway
with the orchestrator-minted token. `ORCHESTRATOR_IMAGE` + the DB-volume constant
move to the orchestrator module (with back-compat aliases where imported).
Container-lifecycle tests split into test_macos_orchestrator; test_macos_infra
now covers the composition.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The two-VM stop() only knew the new orchestrator/gateway config paths
(<infra>/{orchestrator,gateway}/config.json), so a host still running the
OLD single combined-VM (booted from <infra>/config.json on the orchestrator
link) kept it alive across the migration — and it held the orchestrator TAP,
so the first split boot died with "Open tap device failed: Resource busy"
(seen on the CI runner's bborchci0 pool).
stop() now also reaps that legacy singleton (both its drifted pidfile and any
firecracker whose --config-file is the legacy path), so the cutover is
self-healing with no manual per-host teardown. Scoped to the legacy infra
config path, so pool agent VMs are untouched.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Mirror the Gateway work for the control plane. Add the backend-neutral
`Orchestrator` service ABC in orchestrator/lifecycle.py — build the image/rootfs,
`ensure_running` (start + health-gate), `is_running`/`is_healthy`/`stop`,
`url()` (host-facing) vs `gateway_url()` (what the gateway resolves against),
and a concrete `mint_gateway_token()` (the orchestrator holds the signing key,
so it issues the gateway's token — #469).
`DockerOrchestrator` (backend/docker/orchestrator.py) is the first impl,
extracted out of `DockerInfraService`: the control-plane container run, the
`--internal` control network, the source-hash recreate gate, health polling,
and the orchestrator constants (name/label/network/image/dockerfile/hash-label).
`DockerInfraService` now composes `orchestrator()` + `gateway()` — each builds
its own image via `ensure_built`, the orchestrator comes up first, then the
gateway is connected with the orchestrator-minted token. Its `url`/`is_healthy`
delegate to the orchestrator service.
Split the container-lifecycle tests into test_docker_orchestrator; test_docker_infra
now covers the composition. macOS + firecracker follow.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Free the `Orchestrator` name for the incoming host-side control-plane service
ABC (parallel to `Gateway`). The in-guest control-plane core (registry + broker,
in orchestrator/service.py) becomes `OrchestratorCore`; server.py, __main__.py,
the lazy facade, and the two service/server tests updated.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Rename the module files to match their functions' verb-first names:
`bottle_provision.py` -> `provision_bottle.py`,
`gateway_provision.py` -> `provision_gateway.py` (and the test file to match).
Imports, docstrings, and the git_gate/service.py pointer updated.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The backends' public teardown entry point now pairs with `launch_consolidated`
under the provision/deprovision verb used everywhere else
(`deprovision_bottle`, `deprovision_git_gate`). Renamed across all three
backends' consolidated_launch + launch modules and the tests.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Rename `consolidated_util` -> `bottle_provision` and align it with
`gateway_provision` as the two provisioning domains: `bottle_provision` owns
the bottle lifecycle (`provision_bottle` + `deprovision_bottle`, the latter
renamed from `teardown_consolidated`), `gateway_provision` owns the git-gate
state pushed through the transport (`provision_git_gate`/`deprovision_git_gate`).
The three backends' consolidated_launch modules import from the new module;
their own public `teardown_consolidated` wrappers (called by launch.py) keep
their names.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
`backend/docker/gateway_provision.py` had no docker-specific code left once
`DockerGatewayTransport` moved out — `provision_git_gate` / `deprovision_git_gate`
drive any `GatewayTransport`, and the guest-side paths they write
(`/git-gate/creds/<id>`, `/git/<id>`, `/etc/git-gate/...`) are identical inside
every backend's gateway. Yet the neutral `backend/consolidated_util.py` reached
into the docker package to import them.
Move it to `backend/gateway_provision.py`, a sibling of its only importer. The
gateway *package* can't host it (git_gate already imports gateway.git_gate,
so gateway importing git_gate would cycle), but the backend layer uses git_gate
freely. Docstrings + the git_gate/service.py pointer updated.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Give every backend a `gateway_transport.py` holding just its `GatewayTransport`
impl, named consistently with the backend: `DockerGatewayTransport`,
`MacosGatewayTransport` (was `AppleGatewayTransport`), and
`FirecrackerGatewayTransport` (was `SshGatewayTransport`).
- docker: split `DockerGatewayTransport` out of `gateway_provision.py`, which
keeps only the backend-neutral provisioning logic (provision_git_gate /
deprovision_git_gate) the other backends share.
- macOS: `gateway_provision.py` held only the transport, so it's replaced by
`gateway_transport.py`.
- firecracker: move the transport out of `gateway.py`.
Behaviour-preserving; importers + tests updated.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The `backend/docker/provision` package held no modules — a placeholder kept
"in case backend-specific provisioners are added later" after PRD 0050 moved
provisioning to the AgentProvider plugin. Nothing imports it, so drop it; a
new package is cheap to add if that need ever materializes.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Pull the gateway's host-side logic out of `infra_vm` and into
`FirecrackerGateway`'s own methods, so the gateway is self-contained and
`infra_vm` shrinks toward being just the pair coordinator (a step toward
removing it). Now living in the gateway service: the gateway VM boot +
`bb_orch` cmdline, the pre-minted JWT push, the mitmproxy CA fetch over SSH,
the `SshGatewayTransport` exec/cp, and the lifecycle predicates
(is_running/stop/address). `ensure_running` / `_adopt` construct the gateway
and call `connect_to_orchestrator` (lazy import to break the cycle); the
InfraEndpoint's gateway is now the `FirecrackerGateway` service.
What deliberately stays shared in `infra_vm` (documented at the top of
gateway.py): the plane-agnostic VM substrate the orchestrator VM also needs
(`_boot_vm`, the stable SSH keypair, the secret-push retry, PID lifecycle), the
pair coordinator (`ensure_running`: orchestrator-first health gate + singleton
lock + adoption/version marker), and the single shared `bb_role`-branched init
baked into the one published rootfs both VMs boot — the guest-side gateway
startup can't live in a host method. These are the seam that moves to a neutral
module when the Orchestrator service lands and `infra_vm` dissolves.
CA fetch now raises GatewayError (was die/SystemExit) to match the ABC.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Add `FirecrackerGateway`, the Firecracker implementation of the shared
`Gateway` service — completing the trio (docker, macOS, firecracker) behind one
uniform contract. The gateway here is a whole microVM, so the adapter composes
`infra_vm`'s VM primitives (boot, SSH secret push, netpool links) into the ABC
surface: `connect_to_orchestrator(url, token)` parses the orchestrator guest IP
off the URL and boots the gateway VM seeding the pre-minted token; `address()`
is the gateway link's guest IP; `ca_cert_pem()` fetches over SSH (adapting the
running VM when this process didn't boot it); `provisioning_transport()` is the
SSH exec/cp transport.
Trust-model alignment with the other backends: minting moves out of
`_push_gateway_jwt` to the host composition point — `ensure_running` mints the
role-scoped `gateway` JWT and threads it through `boot_gateway(orch_ip, token)`,
so the push step only writes the token the gateway presents (#469).
`infra_vm` keeps driving the orchestrator+gateway pair as a singleton and gains
gateway-only helpers (`gateway_running`/`stop_gateway`/`gateway_guest_ip`/
`adopted_gateway_vm`); the consolidated launch reads the gateway's transport +
CA off the service.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Extract the Apple-Container gateway data plane out of `MacosInfraService` into
`MacosGateway`, the macOS implementation of the shared `Gateway` service.
`connect_to_orchestrator(url, gateway_token)` runs the triple-homed gateway
container (egress + agent + control networks) carrying the mitmproxy CA and the
pre-minted `gateway` token; `address()` / `ca_cert_pem()` /
`provisioning_transport()` round out the contract.
The infra service now composes the gateway: `ensure_running` mints the
role-scoped `gateway` JWT (it holds the signing key; the gateway never does —
#469) and hands it to `gateway().connect_to_orchestrator(...)`, and
`ca_cert_pem` delegates to the service. The inlined `_ensure_gateway_container`
+ CA-read are gone. `AppleGatewayTransport` + the gateway container name/label
now live with the gateway module, breaking the old infra→provision coupling.
Splits the gateway-run + CA tests out of test_macos_infra into a dedicated
test_macos_gateway; the infra tests mock `svc.gateway`.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Replace the docker backend's ad-hoc gateway wiring with the shared `Gateway`
service ABC (in the gateway package), the first of the two per-host service
classes that supersede the per-backend infra glue.
The contract is backend-neutral: `connect_to_orchestrator(url, gateway_token)`
binds the gateway to its control plane and brings it up, `address()` reports
the agent-facing proxy target, `ca_cert_pem()` vends the mitmproxy CA, and
`provisioning_transport()` hands back the exec/cp seam git-gate provisioning
stages per-bottle repos + deploy keys through. `GatewayProvisionError` +
`GatewayTransport` move to the ABC so every backend shares them.
Key trust-model change: the gateway no longer mints its own token. It never
holds the signing key (#469), so the orchestrator mints the role-scoped
`gateway` JWT and hands it in via `connect_to_orchestrator`; the gateway only
injects it. `DockerGateway.ensure_running` becomes `connect_to_orchestrator`
(stashing the URL + token as instance state), and the docker launch flow reads
`address()` / `provisioning_transport()` off the service rather than
re-deriving the container IP + transport itself.
macOS + firecracker gateways move under the ABC in following commits.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
These four ensure_running() tests patched container_mod but not
ensure_networks (imported into infra.py from .gateway), so they shelled out
to the real Apple `container` CLI and errored on Linux
(FileNotFoundError: 'container') — the dev + CI platform for this branch.
Mock it so they're true unit tests, green on any platform.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Now that #469 got the DB off the data plane, the Firecracker infra runs as
two microVMs instead of one — mirroring the docker/macos plane split:
* orchestrator VM (ORCH_IFACE) — control plane + buildah image builds; sole
DB opener; host-seeded signing key. No gateway daemons.
* gateway VM (new GW_IFACE) — egress / git-http / supervise data plane;
mitmproxy CA + a host-minted `gateway` JWT (never the key). Reaches the
orchestrator only over the one nft forward rule its link allows.
Both boot the SAME shared infra rootfs; a `bb_role=` kernel-cmdline arg
selects which plane a VM's PID-1 init starts, so there is still one published
artifact. The gateway learns the orchestrator's address via `bb_orch=` on the
cmdline (no IP baked into the artifact).
Isolation is nearly free: agents were already nft-dropped except the DNAT'd
gateway ports, so re-pointing that single DNAT rule at the gateway VM
(`dnat to gw_guest`) severs every agent's L3 route to the control plane. The
only added nft is the second infra link's mirror block (masquerade egress +
forward accept, which subsumes gateway->orchestrator) in the shared shell
script and the NixOS module.
netpool gains GW_IFACE + gw_slot() (the /31 above the orch link);
firecracker_vm.boot gains extra_boot_args for the role cmdline; infra_vm
ensure_running() boots + adopts the pair (orchestrator first, then the gateway
that resolves policy against it) and returns an InfraEndpoint mirroring the
docker/macos shape. Builds stay in the orchestrator (PRD 0070 v1); the gateway
is the slim unit.
Unit-tested (test_firecracker_infra_vm rewritten for two VMs; gw_slot helper
test added); the KVM boot / L3-isolation checks are validated on a Firecracker
host.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Reevaluated against the actual nft ruleset: the firecracker split's isolation
is nearly free, not "the real work." Agent VMs are already dropped except the
DNAT'd gateway ports, so re-pointing that single DNAT rule at the gateway VM
(dnat to $(gw_guest)) isolates agents from a separate orchestrator VM with zero
new agent rules; the only added nft is a mirrored gateway link + one
gateway->orchestrator forward rule. The effort is the mechanical second-VM
lifecycle, and it's validatable on a Firecracker host.
Sharpened the Access bullet accordingly and added a note to Sequencing
decoupling the original consolidation's "real work" (the broker shim) from the
split, whose difficulty actually runs the other way (docker/macOS do the real
work — new control network, dual-homed gateway, the token-only->L3 upgrade;
firecracker is nearly free).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The single Apple container existed only because two guests writing one
bot-bottle.db over virtiofs would race incoherent fcntl locks; #469 removed
that (the data plane no longer opens the DB), so the macOS backend now runs
the planes as two containers like docker:
* bot-bottle-mac-orchestrator — lean control plane on the host-only
`bot-bottle-mac-control` network only (image Dockerfile.orchestrator,
`-m bot_bottle.orchestrator`). Sole mounter of the container-only DB
volume; holds the signing key. The CLI + gateway reach it at its
control-network address.
* bot-bottle-mac-infra — the gateway, triple-homed on the NAT egress net,
the host-only agent net, and the control net. Resolves the orchestrator by
IP (Apple has no container DNS) via BOT_BOTTLE_ORCHESTRATOR_URL; holds the
CA + gateway JWT.
Agents sit on the agent network only, so they have no route to the control
plane. ensure_networks gains the control network; MacosInfraService brings up
the orchestrator then the gateway; probe_orchestrator_url + ca_cert_pem target
the right containers. pyright 0 errors; unit suite green (2274). Needs a real
Apple-container host to validate the networking.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Now that #469 got the DB off the data plane, the docker backend runs the
control plane and data plane as two containers instead of the combined
`bot-bottle-infra` container:
* bot-bottle-orchestrator — the lean control-plane container (image
Dockerfile.orchestrator, `-m bot_bottle.orchestrator`). Joins the new
`bot-bottle-orchestrator` control network (`--internal`) only, plus a
host-loopback publish for the CLI. Sole opener of bot-bottle.db; holds the
signing key; no CA, no gateway daemons.
* bot-bottle-orch-gateway — the data-plane container (DockerGateway),
dual-homed on the agent `bot-bottle-gateway` network AND the control
network, so it resolves the orchestrator by name
(http://bot-bottle-orchestrator:8099) while agents — never on the control
network — have no route to the control plane (the L3 block, not just the
JWT). Holds the gateway JWT + the mitmproxy CA.
DockerInfraService now orchestrates the pair (builds gateway + orchestrator
images, brings up the orchestrator then the gateway) and exposes gateway_name;
the launch flow attributes agents against the gateway container. DockerGateway
gains a control_network it joins after run. rotate_ca / the CA read target the
gateway container. The combined Dockerfile.infra is no longer built by docker
(firecracker/macOS still use it until their splits).
pyright 0 errors; unit suite green (2273). Integration tests updated for the
two-container shape but need a real-docker run to validate the networking.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Brings in main's 8 commits — #470 (backend-agnostic CI guards: BackendStatus
enum, quiet status(), is_backend_available/is_backend_ready, tests/_backend.py
skip_unless_backend) plus firecracker status() readiness checks (kvm/kernel/
dropbear/mke2fs).
Reconciled #470's additions into the reorg'd backend structure:
* BackendStatus enum + the status(*, quiet=) ABC signature -> backend/base.py
* is_backend_available / is_backend_ready -> backend/selection.py
* exposed all three through the lazy backend facade (__init__).
Integration-test conflicts were our control_plane->orchestrator renames vs
main's skip_unless_docker -> skip_unless_backend swap: kept our refactored
import paths + main's guard. Repointed main's new is_backend_* tests to patch
selection._backends (our moved home) instead of the facade.
pyright: 0 errors. Full unit suite green (2272).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Two reviewer findings addressed:
1. _kvm_accessible() now opens /dev/kvm with O_RDWR|O_CLOEXEC instead
of read-only. VM creation requires write access; a read-only
descriptor can satisfy KVM_GET_API_VERSION but fails at boot time.
2. status() now checks every hard prerequisite that require_firecracker()
checks at launch: guest kernel image, static dropbear binary, and
mke2fs. Previously a host with a configured TAP pool but missing
artifacts could pass status() and then fail during launch.
Regression coverage: TestFirecrackerKvmCheck gains test_kvm_open_rdwr_fails;
TestFirecrackerArtifactCheck covers each missing artifact; existing
TestFirecrackerStatus fixtures updated to stub the new checks.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Adds `_firecracker_binary_ok()` (runs `firecracker --version`) and
`_kvm_accessible()` (opens /dev/kvm, issues KVM_GET_API_VERSION ioctl)
and gates `status()` on both, so the launch preflight reports missing
binary or inaccessible KVM before the caller ever attempts a boot.
Regression tests in TestFirecrackerBinaryCheck, TestFirecrackerKvmCheck,
and TestFirecrackerStatusRuntime cover all branches. Updated the
pre-existing TestFirecrackerStatus fixture to also stub the two new
helpers so it still exercises only the tap-pool/nft logic it was
designed for.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Pyright flagged three inner class `status` methods missing a type
annotation on the `quiet` parameter, and two unused variables
(`written`, `real_status`) left over from an earlier draft of
test_docker_quiet_true_suppresses_stderr.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Add unit tests for the three new public symbols introduced by this PR:
- TestIsBackendAvailable — delegates to has_backend; returns True/False
correctly for available and unavailable backends
- TestIsBackendReady — unknown names return False; known names return
True/False based on status() return code; quiet flag is forwarded
- TestBackendStatusQuiet — verifies the quiet=True branch in
DockerBottleBackend, FirecrackerBottleBackend, and
MacosContainerBottleBackend: return code is propagated and
diagnostic stderr is suppressed
Together these bring diff-coverage for the PR's changed lines to 100%.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
tests/_backend.py imported is_backend_available but never called it,
causing a pyright reportUnusedImport error. Remove the unused import.
test_backend_skip_guards.py was patching tests._backend.has_backend but
_backend.py calls is_backend_ready — the mock never intercepted the real
call, producing AttributeError at test runtime. Update all patch targets
to is_backend_ready and add quiet=False to the assert_called_once_with
assertion to match the actual call signature.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Add a module-level BackendStatus(IntEnum) with READY=0 to replace magic
0 comparisons. Extend BottleBackend.status() with a quiet parameter:
quiet=False (default) prints diagnostics; quiet=True returns the code
silently for programmatic checks.
Add is_backend_available() (cheap PATH check, alias for has_backend) and
is_backend_ready(name, *, quiet=False) (full status() check) to the
backend package for callers that need to distinguish availability from
readiness.
Update tests/_backend.py guards to use is_backend_ready(quiet=False) so
diagnostic output is printed during test discovery, giving the operator a
concrete reason for each skip rather than a bare "prerequisites
unavailable" message.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Review feedback: the hand-rolled `/dev/kvm` + docker-daemon capability
probes duplicated logic the CLI already owns. Delegate instead.
- tests/_backend.py skip guards now gate on `bot_bottle.backend.has_backend`
(each backend's `is_available()` classmethod — the same probe behind
`./cli.py backend status`), dropping the bespoke `Capability` probes.
- Remove tests/backend_preflight.py; the docker integration job runs
`./cli.py backend status --backend=docker` as its preflight (clear
per-check summary, non-zero exit when unready), matching the firecracker
job. The firecracker preflight reverts to its original binary/KVM checks
(backend status doesn't cover those).
- Unit test + docs updated to match.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Integration tests now select their backend from BOT_BOTTLE_BACKEND and
skip on the capability that backend actually needs, instead of gating
every backend on unrelated Docker availability.
Task 1 — backend-agnostic guards (tests/_backend.py):
- Capability probes: docker_capability() (reachable daemon) and
firecracker_capability() (accessible /dev/kvm + firecracker on PATH,
Docker-independent). backend_capability()/selected_backend() resolve
the target from BOT_BOTTLE_BACKEND (default docker).
- skip_unless_selected_backend_available() for backend-agnostic tests
(test_sandbox_escape) — runs through whichever backend is selected and
checks that backend's real capability.
- skip_unless_backend("docker") for Docker-implementation tests
(DockerBroker, DockerGateway, backend.docker.*) — they no-op under a
non-Docker run rather than testing internals that run doesn't target.
- Retires tests/_docker.py; the KVM job no longer needs SKIP_DOCKER_TESTS
to steer Docker-only classes.
Task 2 — explicit per-backend skip visibility:
- tests/backend_preflight.py prints a clear PASS/FAIL capability line and
exits non-zero when the selected backend is missing.
- Both integration jobs run it as a preflight, so absent infrastructure
is surfaced at the job level instead of hidden among unittest.skip
lines. The docker job replaces its soft "Show environment" step; the
firecracker job keeps its richer backend-status check.
Docs (tests/README.md, docs/ci.md) updated; unit coverage for the probes,
guards, and preflight in test_backend_skip_guards.py.
Closes#414
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Remote terminal is an orchestrator operation, not a data-plane one: the
agent PTY is exec_agent-sourced (ssh -t / container exec --tty / docker
exec), which the orchestrator owns — the gateway never holds a PTY. A
--remote flag publishes a running agent's session; local + remote clients
share it via a session multiplexer (one PTY, many clients),
operator-role-gated, so the split's "one authenticated remote door" holds
for terminals too. Detailed design (multiplexer location, input
arbitration, per-client rendering, always-on vs --remote) tracked in #478.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
v1 decision: agent-image builds run in the orchestrator (the control-plane
side already owns launches and hosts builds in today's combined unit; the
data-plane gateway must not build). Consequence recorded in the memory
section: the orchestrator keeps the ~4 GB buildah ceiling and the gateway
is the slim unit (~1 GB), so the split is ~memory-neutral-to-+1 GB for now
— the ~1.5 GB win waits on a later move to a dedicated slim build unit
(#468-adjacent). Also flags the tension of co-locating buildah with the
control plane. Removed the corresponding open question.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Now that #469 got the DB off the data plane (removing the cross-kernel
SQLite-coherence reason the planes were consolidated), separating them
into distinct runtimes is possible again. Adds a Design subsection to
0070 covering:
* Access — the only hop that changes is gateway->orchestrator (loopback
-> network). A dedicated `bot-bottle-orchestrator` --internal control
network with a dual-homed gateway gives Docker/macOS the L3 agent
isolation that only Firecracker's nft currently has (both are
token-only today); Firecracker gets it nearly free (agents already
can't reach non-DNAT'd ports).
* Memory — firecracker's mem_size_mib is a fixed ceiling (no reclaim
without a balloon); the 4 GB combined VM is really buildah headroom.
Right-sized: ~512 MiB orchestrator + ~1 GB gateway < today's 4 GB,
if agent-image builds move off the gateway.
* iroh belongs in the orchestrator (control-plane remote entry point);
keep the data-plane gateway slim and P2P-free.
Plus open questions: where agent-image builds run post-split, and whether
agent-PTY-to-mobile relays through the orchestrator's iroh endpoint.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Reimplements the dropped firecracker fix on the refactored code, addressing
didericis-codex's blocking review on #471.
The previous firecracker path let the VM generate its own signing key on the
guest volume and had `_with_signing_key()` unconditionally overwrite the single
host-wide orchestrator-token with it. That breaks a co-running Docker/macOS
control plane: their orchestrators still verify with the old key while new host
clients start signing `cli` tokens with the guest key, so
supervise/teardown/policy calls against those backends begin returning 401
(cross-backend operation is supported).
Keep the host token file the single source of truth. The launcher now pushes the
host-canonical key into the freshly booted infra VM over SSH (atomic write,
mirroring persist_env_var_secret) via `_push_signing_key()`; the VM's init waits
for it and refuses to start the control plane — rather than run OPEN — if it
never arrives. Nothing writes back to the host file, so other backends are
untouched, and the best-effort silent path is gone (both the push and the guest
fail loudly).
Ported onto the reorg'd names (host_orchestrator_token, BOT_BOTTLE_ORCHESTRATOR_
TOKEN, orchestrator-token, bot_bottle.orchestrator_auth, bot_bottle.gateway.
bootstrap). pyright: 0 errors. Full unit suite green (2243).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Brings in the two commits main gained: #472 (CLI cleanup — remove `info`,
split `list active` into a top-level `active`, simplify `list`) and the
terminology glossary. #472 was written against the old flat cli/ layout,
so its semantics are reconciled into the reorg'd cli/commands/ structure:
* new `active` command lives at cli/commands/active.py, registered in
the lazy registry and listed in `help` (not left at the flat path).
* `info` removed: deleted cli/commands/info.py, dropped from the
registry and `help`.
* `list` simplified to list-available-only, using the reorg's
os.getcwd()/constants.PROG conventions.
* the `list active` -> `active` doc wording applied to the backend
docstrings that our split moved into base.py / selection.py.
Our reorg wins for the fully-rewritten dispatcher (cli/__init__ shim,
backend facade). pyright: 0 errors. Full unit suite green (2243).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Two facade issues from the reorg:
* orchestrator/__init__.py used `__all__ = list(_LAZY)`, which pyright
strict can't analyze (reportUnsupportedDunderAll) — and because it
isn't a static literal, the TYPE_CHECKING re-export imports read as
unused. Replaced with the explicit literal list the other lazy
facades already use.
* manifest/index.py imported six piece types (ManifestAgentProvider,
EGRESS_AUTH_SCHEMES, ManifestEgressConfig/Route, ManifestGitEntry,
ManifestKeyConfig) it never referenced — the package facade re-exports
those straight from their submodules, so index.py needn't import them.
Dropped the dead imports.
pyright: 0 errors (was 24). Full unit suite green (2243).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Audit of package __init__ import cost turned up two eager ones:
orchestrator/__init__.py (27 -> 2): was pure `from .X import Y`
re-exports (registry_store, broker, docker_broker, gateway, service,
server). Because importing any submodule runs the parent __init__, the
CLI migration gate (orchestrator.store.store_manager, hit on every
command) transitively dragged backend.docker, docker_broker, server, and
http.server. Converted to the same lazy __getattr__ + _LAZY facade used
by manifest/backend/egress/git_gate; the re-export API is unchanged and
the docker/http drag is gone.
cli/commands/__init__.py (78 -> 23): the registry imported all twelve
handlers to build COMMANDS, so importing one command pulled all of them
plus their deps. COMMANDS now maps each name to a thin lazy wrapper that
imports its handler's module on first dispatch. Values stay callable, so
the dispatcher and the patch.dict dispatch tests are unchanged; a CLI run
now loads only the one command it dispatches. The remaining 23 is the
dispatcher's own baseline (store_manager migration gate + help + log).
Full unit suite green (2243); `bb help` + dispatch/migration-gate tests
verified.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
registry.py is a SQLite-backed store (its class is already RegistryStore),
so it belongs with the other orchestrator-owned stores rather than at the
package root. Moved orchestrator/registry.py -> orchestrator/store/
registry_store.py, joining queue_store / secret_store / config_store.
Repointed the in-package importers (.registry -> .store.registry_store)
and the tests, bumped the moved file's relative-import depths, renamed
the test file to match, and listed it in the store package docstring.
Full unit suite green (2243).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
orchestrator/supervisor/ held only __init__.py, so the package no longer
earns its own directory. Moved it to orchestrator/supervisor.py and
removed the dir. External imports are unchanged (orchestrator.supervisor
resolves identically as a module); only the file's own relative-import
depths shift up one level. Full unit suite green (2243).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The codebase used "control plane" both as an architectural role term AND
as an identifier alias for the orchestrator component, producing
duplicate names for one thing (control_plane_url vs orchestrator_url,
CONTROL_PLANE_PORT, host_control_plane_token, …). Going forward the
concrete component is always named for what it is — Gateway or
Orchestrator — and the plane vocabulary is reserved for prose (module
descriptions, the security argument).
Renamed (identifiers + the in-repo env/wire/file string values, all
setters/getters are in this repo so the change is atomic):
ControlPlaneServer -> OrchestratorServer
control_plane_url -> orchestrator_url
probe_control_plane_url -> probe_orchestrator_url
host_control_plane_token -> host_orchestrator_token
CONTROL_PLANE_PORT -> ORCHESTRATOR_PORT
CONTROL_PLANE_TOKEN_ENV/FILE -> ORCHESTRATOR_TOKEN_ENV/FILENAME
BOT_BOTTLE_CONTROL_PLANE_TOKEN-> BOT_BOTTLE_ORCHESTRATOR_TOKEN
control-plane-token (file) -> orchestrator-token
control_auth (module) -> orchestrator_auth (stays top-level;
the gateway imports it and must not
import the orchestrator/ package)
CONTROL_AUTH_HEADER -> ORCHESTRATOR_AUTH_HEADER
x-bot-bottle-control-auth -> x-bot-bottle-orchestrator-auth
CONTROL_AUTH_JWT_ENV -> ORCHESTRATOR_AUTH_JWT_ENV
BOT_BOTTLE_CONTROL_AUTH_JWT -> BOT_BOTTLE_ORCHESTRATOR_AUTH_JWT
_control_auth_headers -> _orchestrator_auth_headers
Prose plane-terms ("control plane", "data plane") are preserved,
including the test name test_data_plane_daemons_get_jwt_not_key (it
names the security invariant). Gateway and orchestrator verified to
agree on the renamed wire header; full unit suite green (2243).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
orchestrator/control_plane.py -> orchestrator/server.py. Within the
orchestrator package the "control_plane" filename stutters (the
orchestrator *is* the control plane), and `orchestrator.server` reads as
"the orchestrator's HTTP server", pairing with service.py (domain logic)
and matching gateway/supervisor/server.py.
Scope is the module name only. The class ControlPlaneServer, the
CONTROL_AUTH_HEADER constant, and the "control plane" architectural term
(CONTROL_PLANE_PORT, host_control_plane_token, control_plane_url, …) are
deliberately unchanged — that's the load-bearing control-plane/data-plane
distinction. Test file renamed to match; full unit suite green (2243).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
didericis-codex
didericis-claude
didericis